2024-03-04T15:42:28.722 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T15:42:28.722 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235 --verbose --name tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi --owner scheduled_tonay@teuthology --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235/orig.config.yaml 2024-03-04T15:42:28.778 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T15:42:28.897 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235 branch: wip-asm0deuz-testing-2024-03-04-1211 description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: tonay@ibm.com first_in_suite: false job_id: '7581235' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.1 ' name: tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-asm0deuz-testing-2024-03-04-1211 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-asm0deuz-testing-2024-03-04-1211 sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 owner: scheduled_tonay@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 3905 sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 sleep_before_teardown: 0 subset: 1/14 suite: orch:cephadm suite_branch: wip-asm0deuz-testing-2024-03-04-1211 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPQOb5Abv1OZIbkRjvIYIsiGaWcSXRg72uD1itWszrWmuDYmlbWwlZZ1NkclDfTumZcyG//l2WDWYSNbadXBoo= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEzoCGrb5maYMP2IHy53/n2pgqD7FOQ/cAlr36Z5wmwvFEyl0DSC1ka+4yUgvjuq0E89DjowqdWUO/gWhKO0WsA= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.1 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.1 roleless: true - print: '**** done end installing v18.2.1 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/tonay/src/git.ceph.com_ceph-c_wip-asm0deuz-testing-2024-03-04-1211/qa/suites/orch/cephadm/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_15:07:38 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: tonay verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-04T15:42:28.897 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa; will attempt to use it 2024-03-04T15:42:28.898 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks 2024-03-04T15:42:28.898 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T15:42:28.899 INFO:teuthology.task.internal:Checking packages... 2024-03-04T15:42:28.923 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9ca0b5de35ed230e84387abdd8e619ed8e4f3282' 2024-03-04T15:42:28.923 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T15:42:28.924 INFO:teuthology.packaging:ref: None 2024-03-04T15:42:28.924 INFO:teuthology.packaging:tag: None 2024-03-04T15:42:28.924 INFO:teuthology.packaging:branch: wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T15:42:28.924 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:42:28.924 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T15:42:29.074 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1826.g9ca0b5de 2024-03-04T15:42:29.076 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T15:42:29.083 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T15:42:29.083 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T15:42:29.098 INFO:teuthology.task.internal:Saving configuration 2024-03-04T15:42:29.116 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T15:42:29.122 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:42:29.154 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:37:10.749215', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPQOb5Abv1OZIbkRjvIYIsiGaWcSXRg72uD1itWszrWmuDYmlbWwlZZ1NkclDfTumZcyG//l2WDWYSNbadXBoo='} 2024-03-04T15:42:29.186 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:37:10.747864', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEzoCGrb5maYMP2IHy53/n2pgqD7FOQ/cAlr36Z5wmwvFEyl0DSC1ka+4yUgvjuq0E89DjowqdWUO/gWhKO0WsA='} 2024-03-04T15:42:29.186 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T15:42:29.191 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-04T15:42:29.191 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-04T15:42:29.191 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T15:42:29.280 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fccccfa00d0>, signals=[15]) 2024-03-04T15:42:29.281 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T15:42:29.289 INFO:teuthology.task.internal:Opening connections... 2024-03-04T15:42:29.290 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T15:42:29.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:42:29.413 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-03-04T15:42:29.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:42:29.514 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T15:42:29.520 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-04T15:42:29.539 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-04T15:42:29.539 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:42:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:42:29.597 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:42:29.597 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:42:29.597 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:42:29.597 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-04T15:42:29.634 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-03-04T15:42:29.652 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-03-04T15:42:29.653 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-03-04T15:42:29.709 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-03-04T15:42:29.710 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-03-04T15:42:29.710 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-03-04T15:42:29.710 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:42:29.710 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-03-04T15:42:29.710 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:42:29.710 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:42:29.711 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-03-04T15:42:29.711 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:42:29.711 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:42:29.711 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:42:29.711 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:42:29.711 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:42:29.712 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:42:29.712 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-03-04T15:42:29.742 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T15:42:29.751 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T15:42:29.805 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T15:42:29.806 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:42:29.809 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:42:29.825 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T15:42:29.832 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T15:42:29.832 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:42:29.865 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:42:29.966 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T15:42:30.039 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T15:42:30.053 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-04T15:42:30.053 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-04T15:42:30.053 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:42:30.054 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:42:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:42:30.070 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-04T15:42:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:42:30.071 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-03-04T15:42:30.126 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:42:30.127 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-361.el9.x86_64 2024-03-04T15:42:30.127 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2024-03-04T15:42:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:42:30.128 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-361.el9.x86_64 2024-03-04T15:42:30.128 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-03-04T15:42:32.755 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 7.0 MB/s | 6.7 MB 00:00 2024-03-04T15:42:32.800 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 6.7 MB/s | 6.7 MB 00:01 2024-03-04T15:42:34.430 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-04T15:42:34.484 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-04T15:42:35.306 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-04T15:42:35.327 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-04T15:42:45.921 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 19 MB 00:09 2024-03-04T15:42:50.041 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-04T15:42:51.597 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-03-04T15:42:53.307 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2024-03-04T15:42:57.548 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 929 kB/s | 19 MB 00:21 2024-03-04T15:42:58.315 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 9.3 kB/s | 1.7 kB 00:00 2024-03-04T15:43:00.580 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:43:00.580 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:43:00.673 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:43:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:43:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:43:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-03-04T15:43:00.686 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:43:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 100 M 2024-03-04T15:43:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 123 M 2024-03-04T15:43:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:43:01.250 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 21 MB/s | 6.0 MB 00:00 2024-03-04T15:43:01.508 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 38 MB/s | 21 MB 00:00 2024-03-04T15:43:01.875 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-04T15:43:01.925 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 41 MB/s | 39 MB 00:00 2024-03-04T15:43:02.236 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 34 MB/s | 33 MB 00:00 2024-03-04T15:43:02.237 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:43:02.237 INFO:teuthology.orchestra.run.smithi116.stdout:Total 64 MB/s | 100 MB 00:01 2024-03-04T15:43:03.115 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:43:03.489 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:43:03.489 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:43:03.583 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 15 kB 00:00 2024-03-04T15:43:04.829 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:43:04.830 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:43:05.217 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-03-04T15:43:07.552 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:43:08.188 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:43:08.317 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:43:10.134 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-03-04T15:43:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:43:11.270 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:43:12.399 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:43:12.399 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:43:12.507 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:43:12.518 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:43:12.518 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-04T15:43:12.518 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:43:12.518 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:43:12.518 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:43:12.518 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-04T15:43:12.518 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:43:12.518 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:43:12.519 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:43:12.519 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:43:12.519 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:43:12.519 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:43:12.519 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-03-04T15:43:12.519 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:43:12.519 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 100 M 2024-03-04T15:43:12.520 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 123 M 2024-03-04T15:43:12.520 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:43:12.877 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 19 MB/s | 6.0 MB 00:00 2024-03-04T15:43:13.169 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 35 MB/s | 21 MB 00:00 2024-03-04T15:43:13.619 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 37 MB/s | 39 MB 00:01 2024-03-04T15:43:13.945 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 31 MB/s | 33 MB 00:01 2024-03-04T15:43:13.947 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:43:13.948 INFO:teuthology.orchestra.run.smithi152.stdout:Total 70 MB/s | 100 MB 00:01 2024-03-04T15:43:14.839 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:43:15.235 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:43:15.235 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:43:16.033 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:43:16.439 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:43:16.602 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:43:16.602 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:43:19.546 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:43:20.203 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:43:20.332 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:43:21.237 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:43:23.124 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:43:23.304 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:43:28.246 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:43:28.364 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:43:33.193 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:43:50.431 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:43:50.432 INFO:teuthology.orchestra.run.smithi116.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:43:50.432 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:43:50.517 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:43:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:43:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:43:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:43:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:43:51.444 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:43:51.444 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:43:51.444 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:43:51.444 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:43:51.444 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:43:51.444 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:43:51.445 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:43:51.445 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:43:51.445 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:43:51.908 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:43:52.059 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:43:52.059 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:43:52.059 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:43:52.060 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:43:52.060 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:43:52.060 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-04T15:43:52.060 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:43:52.060 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:43:52.181 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:43:52.182 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:43:52.182 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-03-04T15:43:52.216 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-03-04T15:43:52.216 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-03-04T15:43:52.216 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-03-04T15:43:52.216 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-03-04T15:43:52.216 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-03-04T15:43:52.216 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:43:52.216 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-03-04T15:43:52.216 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:43:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-03-04T15:43:52.218 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:43:52.218 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:43:52.218 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:43:52.219 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:43:52.219 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-03-04T15:43:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:43:54.194 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-03-04T15:43:54.196 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-04T15:43:54.211 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.G5qav0o6ji 2024-03-04T15:43:54.212 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.G5qav0o6ji 2024-03-04T15:43:54.281 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.G5qav0o6ji 2024-03-04T15:43:54.430 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.G5qav0o6ji is 6KB 2024-03-04T15:43:54.478 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.G5qav0o6ji 2024-03-04T15:43:54.492 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:43:54.560 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:43:54.561 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:43:54.561 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:43:54.561 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:43:54.562 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:43:54.718 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-03-04T15:44:02.462 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:44:02.462 INFO:teuthology.orchestra.run.smithi152.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:44:02.462 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:44:02.548 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:44:03.024 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:44:03.024 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:44:03.024 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:44:03.024 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:44:03.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:44:03.458 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:44:03.458 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:44:03.458 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:44:03.458 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:44:03.458 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:44:03.459 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:44:03.459 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:44:03.459 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:44:03.981 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:44:04.135 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:44:04.135 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:44:04.135 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:44:04.135 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:44:04.135 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:44:04.135 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-04T15:44:04.135 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:44:04.135 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:44:04.257 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:44:04.257 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:44:04.257 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2024-03-04T15:44:04.294 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2024-03-04T15:44:04.294 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2024-03-04T15:44:04.294 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.06 2024-03-04T15:44:04.294 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 68.el9 2024-03-04T15:44:04.294 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2024-03-04T15:44:04.294 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:44:04.294 INFO:teuthology.orchestra.run.smithi152.stdout:Group : Unspecified 2024-03-04T15:44:04.294 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 8264254 2024-03-04T15:44:04.294 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2024-03-04T15:44:04.294 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : builder@centos.org 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2024-03-04T15:44:04.295 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:44:04.296 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:44:04.296 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:44:04.297 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:44:04.297 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:44:04.926 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2024-03-04T15:44:06.426 INFO:teuthology.orchestra.run.smithi152.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:44:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:done 2024-03-04T15:44:06.466 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-03-04T15:44:06.480 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.8cEV66xjyp 2024-03-04T15:44:06.481 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8cEV66xjyp 2024-03-04T15:44:06.550 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.8cEV66xjyp 2024-03-04T15:44:06.702 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.8cEV66xjyp is 6KB 2024-03-04T15:44:06.752 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.8cEV66xjyp 2024-03-04T15:44:06.766 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:44:06.831 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:44:06.831 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:44:06.831 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:44:06.831 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:44:06.832 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:44:06.980 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2024-03-04T15:44:24.736 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-04T15:44:24.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T15:44:24.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:44:36.992 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-04T15:44:36.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-04T15:44:36.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:45:11.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-03-04T15:45:20.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-04T15:45:20.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:45:24.750 DEBUG:teuthology.orchestra.remote:timed out 2024-03-04T15:45:33.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T15:45:33.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:45:34.157 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-04T15:45:34.596 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T15:45:34.597 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:45:34.597 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-04T15:45:34.610 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:45:34.611 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:45:34.611 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:45:34.611 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:45:35.611 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-04T15:45:35.612 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:45:35.682 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-03-04T15:45:35.729 DEBUG:teuthology.parallel:result is None 2024-03-04T15:46:20.503 DEBUG:teuthology.orchestra.remote:timed out 2024-03-04T15:46:32.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-04T15:46:32.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:46:32.973 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-04T15:46:33.386 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-04T15:46:33.386 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:46:33.386 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-03-04T15:46:33.401 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:46:33.402 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:46:33.402 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:46:33.402 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:46:34.403 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-04T15:46:34.403 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:46:34.490 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-03-04T15:46:34.531 DEBUG:teuthology.parallel:result is None 2024-03-04T15:46:34.532 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T15:46:34.537 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T15:46:34.537 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:46:34.540 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:46:34.557 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T15:46:34.648 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T15:46:34.655 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T15:46:34.655 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:46:34.658 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:46:34.689 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T15:46:34.697 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T15:46:34.697 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:46:34.723 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:46:34.762 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:46:34.776 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:46:34.782 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:46:34.796 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:46:34.798 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T15:46:34.805 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T15:46:34.805 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:46:34.807 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:46:34.847 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T15:46:34.855 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T15:46:34.856 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:46:34.877 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:46:34.903 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:46:34.984 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:46:35.038 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:46:35.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:46:35.122 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:46:35.122 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:46:35.190 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:46:35.250 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:46:35.283 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:46:35.374 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:46:35.409 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:46:35.409 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:46:35.477 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-04T15:46:35.479 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-03-04T15:46:35.522 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:46:35.567 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:46:36.013 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T15:46:36.024 INFO:teuthology.task.internal:Starting timer... 2024-03-04T15:46:36.024 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T15:46:36.039 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T15:46:36.047 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-04T15:46:36.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-04T15:46:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-04T15:46:36.106 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-03-04T15:46:36.148 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-03-04T15:46:36.149 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T15:46:36.150 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-04T15:46:36.181 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-04T15:46:36.181 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-03-04T15:46:36.211 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-03-04T15:46:36.212 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-03-04T15:46:36.212 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:46:36.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:46:36.249 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:46:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:46:36.279 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T15:46:36.279 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:46:36.324 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:46:36.356 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T15:46:36.366 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T15:46:36.383 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T15:46:36.384 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T15:50:50.197 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-03-04T15:50:50.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T15:50:50.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:50:50.326 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-04T15:50:50.408 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T15:50:50.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-04T15:50:50.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:50:50.539 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-04T15:50:50.619 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-04T15:50:50.619 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T15:50:50.630 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T15:50:50.630 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:50:50.631 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:50:50.634 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:50:50.634 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:50:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:50:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:50:50.690 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:50:50.706 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:50:50.710 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-04T15:50:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-04T15:50:50.736 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-03-04T15:50:50.740 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:50:50.749 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-03-04T15:50:50.757 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:50:50.766 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:50:50.782 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:50:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:50:50.819 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:50:50.819 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-04T15:50:50.819 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:50:50.819 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:50:50.819 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:50:50.819 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:50:50.832 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:50:50.836 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:50:50.836 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-04T15:50:50.836 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:50:50.836 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:50:50.836 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:50:50.836 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:50:50.836 INFO:teuthology.run_tasks:Running task install... 2024-03-04T15:50:50.846 DEBUG:teuthology.task.install:project ceph 2024-03-04T15:50:50.846 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'}} 2024-03-04T15:50:50.846 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.1', 'flavor': 'default', 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'} 2024-03-04T15:50:50.846 INFO:teuthology.task.install:Using flavor: default 2024-03-04T15:50:50.852 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-04T15:50:50.852 INFO:teuthology.task.install:extra packages: [] 2024-03-04T15:50:50.853 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-03-04T15:50:50.853 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-04T15:50:50.853 INFO:teuthology.packaging:ref: None 2024-03-04T15:50:50.853 INFO:teuthology.packaging:tag: v18.2.1 2024-03-04T15:50:50.853 INFO:teuthology.packaging:branch: None 2024-03-04T15:50:50.853 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:50:51.266 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v18.2.1^{} -> None 2024-03-04T15:50:51.266 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-03-04T15:51:00.189 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.1^{} -> 7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-03-04T15:51:00.190 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-03-04T15:51:00.193 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-03-04T15:51:00.193 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-04T15:51:00.193 INFO:teuthology.packaging:ref: None 2024-03-04T15:51:00.193 INFO:teuthology.packaging:tag: v18.2.1 2024-03-04T15:51:00.193 INFO:teuthology.packaging:branch: None 2024-03-04T15:51:00.194 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:51:00.194 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-03-04T15:51:00.194 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-03-04T15:51:00.344 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-03-04T15:51:00.345 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-03-04T15:51:00.367 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-03-04T15:51:00.367 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-03-04T15:51:00.416 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T15:51:00.417 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:51:00.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T15:51:00.426 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T15:51:00.426 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:51:00.426 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T15:51:00.450 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-04T15:51:00.451 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-04T15:51:00.451 INFO:teuthology.packaging:ref: None 2024-03-04T15:51:00.451 INFO:teuthology.packaging:tag: v18.2.1 2024-03-04T15:51:00.451 INFO:teuthology.packaging:branch: None 2024-03-04T15:51:00.451 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:51:00.451 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T15:51:00.459 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-04T15:51:00.459 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-04T15:51:00.459 INFO:teuthology.packaging:ref: None 2024-03-04T15:51:00.459 INFO:teuthology.packaging:tag: v18.2.1 2024-03-04T15:51:00.460 INFO:teuthology.packaging:branch: None 2024-03-04T15:51:00.460 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:51:00.460 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T15:51:00.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T15:51:00.530 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T15:51:00.606 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T15:51:00.614 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T15:51:00.673 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-03-04T15:51:00.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-03-04T15:51:00.682 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-03-04T15:51:00.683 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-03-04T15:51:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:74 files removed 2024-03-04T15:51:00.931 INFO:teuthology.orchestra.run.smithi152.stdout:74 files removed 2024-03-04T15:51:00.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw 2024-03-04T15:51:00.958 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw 2024-03-04T15:51:01.412 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 567 kB/s | 76 kB 00:00 2024-03-04T15:51:01.412 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 515 kB/s | 76 kB 00:00 2024-03-04T15:51:01.579 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 78 kB/s | 9.4 kB 00:00 2024-03-04T15:51:01.596 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 70 kB/s | 9.4 kB 00:00 2024-03-04T15:51:01.746 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 17 kB/s | 2.2 kB 00:00 2024-03-04T15:51:01.763 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 17 kB/s | 2.2 kB 00:00 2024-03-04T15:51:02.324 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-04T15:51:02.374 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-04T15:51:04.133 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-03-04T15:51:04.174 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-04T15:51:05.034 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-04T15:51:05.059 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-04T15:51:11.799 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 3.4 MB/s | 19 MB 00:05 2024-03-04T15:51:13.741 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 19 MB 00:07 2024-03-04T15:51:15.885 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-04T15:51:17.551 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-03-04T15:51:17.944 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-04T15:51:19.977 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 15 kB 00:00 2024-03-04T15:51:21.096 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 6.0 MB/s | 20 MB 00:03 2024-03-04T15:51:21.738 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-03-04T15:51:26.052 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-04T15:51:26.803 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-04T15:51:28.654 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:51:28.667 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:51:28.667 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-04T15:51:28.667 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:51:28.667 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:51:28.667 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-03-04T15:51:28.667 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-03-04T15:51:28.667 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-03-04T15:51:28.667 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T15:51:28.668 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-03-04T15:51:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T15:51:28.670 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T15:51:28.670 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:51:28.670 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:51:28.670 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:51:28.670 INFO:teuthology.orchestra.run.smithi116.stdout:Install 28 Packages 2024-03-04T15:51:28.670 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 2 Packages 2024-03-04T15:51:28.670 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:51:28.670 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 53 M 2024-03-04T15:51:28.671 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:51:28.920 INFO:teuthology.orchestra.run.smithi116.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 30 MB/s | 5.2 MB 00:00 2024-03-04T15:51:28.945 INFO:teuthology.orchestra.run.smithi116.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 966 kB/s | 24 kB 00:00 2024-03-04T15:51:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-03-04T15:51:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 11 MB/s | 474 kB 00:00 2024-03-04T15:51:29.137 INFO:teuthology.orchestra.run.smithi116.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 20 MB/s | 7.7 MB 00:00 2024-03-04T15:51:29.154 INFO:teuthology.orchestra.run.smithi116.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 2.7 MB/s | 45 kB 00:00 2024-03-04T15:51:29.213 INFO:teuthology.orchestra.run.smithi116.stdout:(7/30): librgw2-18.2.1-0.el9.x86_64.rpm 23 MB/s | 4.5 MB 00:00 2024-03-04T15:51:29.238 INFO:teuthology.orchestra.run.smithi116.stdout:(8/30): python3-ceph-common-18.2.1-0.el9.x86_64 1.5 MB/s | 124 kB 00:00 2024-03-04T15:51:29.272 INFO:teuthology.orchestra.run.smithi116.stdout:(9/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 2.7 MB/s | 161 kB 00:00 2024-03-04T15:51:29.297 INFO:teuthology.orchestra.run.smithi116.stdout:(10/30): python3-rados-18.2.1-0.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-04T15:51:29.322 INFO:teuthology.orchestra.run.smithi116.stdout:(11/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 3.9 MB/s | 99 kB 00:00 2024-03-04T15:51:29.347 INFO:teuthology.orchestra.run.smithi116.stdout:(12/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 3.8 MB/s | 297 kB 00:00 2024-03-04T15:51:29.388 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:51:29.391 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:51:29.391 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-04T15:51:29.391 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:51:29.391 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:51:29.391 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-03-04T15:51:29.391 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-03-04T15:51:29.391 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-03-04T15:51:29.391 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T15:51:29.392 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T15:51:29.393 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T15:51:29.394 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:51:29.394 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:51:29.394 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:51:29.394 INFO:teuthology.orchestra.run.smithi152.stdout:Install 28 Packages 2024-03-04T15:51:29.394 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2024-03-04T15:51:29.394 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:51:29.394 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 53 M 2024-03-04T15:51:29.394 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:51:29.606 INFO:teuthology.orchestra.run.smithi116.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 21 MB/s | 18 MB 00:00 2024-03-04T15:51:29.640 INFO:teuthology.orchestra.run.smithi116.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 333 kB/s | 106 kB 00:00 2024-03-04T15:51:29.654 INFO:teuthology.orchestra.run.smithi152.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 29 MB/s | 5.2 MB 00:00 2024-03-04T15:51:29.665 INFO:teuthology.orchestra.run.smithi116.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 757 kB/s | 44 kB 00:00 2024-03-04T15:51:29.679 INFO:teuthology.orchestra.run.smithi152.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 967 kB/s | 24 kB 00:00 2024-03-04T15:51:29.690 INFO:teuthology.orchestra.run.smithi116.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 464 kB/s | 159 kB 00:00 2024-03-04T15:51:29.721 INFO:teuthology.orchestra.run.smithi152.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 15 MB/s | 658 kB 00:00 2024-03-04T15:51:29.754 INFO:teuthology.orchestra.run.smithi152.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-03-04T15:51:29.863 INFO:teuthology.orchestra.run.smithi152.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 20 MB/s | 7.7 MB 00:00 2024-03-04T15:51:29.874 INFO:teuthology.orchestra.run.smithi116.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 661 kB 00:00 2024-03-04T15:51:29.880 INFO:teuthology.orchestra.run.smithi152.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 2.7 MB/s | 45 kB 00:00 2024-03-04T15:51:29.905 INFO:teuthology.orchestra.run.smithi152.stdout:(7/30): python3-ceph-common-18.2.1-0.el9.x86_64 4.9 MB/s | 124 kB 00:00 2024-03-04T15:51:29.916 INFO:teuthology.orchestra.run.smithi116.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 282 kB 00:00 2024-03-04T15:51:29.931 INFO:teuthology.orchestra.run.smithi152.stdout:(8/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 6.3 MB/s | 161 kB 00:00 2024-03-04T15:51:29.933 INFO:teuthology.orchestra.run.smithi116.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 170 kB/s | 41 kB 00:00 2024-03-04T15:51:29.956 INFO:teuthology.orchestra.run.smithi152.stdout:(9/30): python3-rados-18.2.1-0.el9.x86_64.rpm 12 MB/s | 321 kB 00:00 2024-03-04T15:51:29.981 INFO:teuthology.orchestra.run.smithi152.stdout:(10/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 12 MB/s | 297 kB 00:00 2024-03-04T15:51:30.007 INFO:teuthology.orchestra.run.smithi152.stdout:(11/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 3.9 MB/s | 99 kB 00:00 2024-03-04T15:51:30.074 INFO:teuthology.orchestra.run.smithi152.stdout:(12/30): librgw2-18.2.1-0.el9.x86_64.rpm 14 MB/s | 4.5 MB 00:00 2024-03-04T15:51:30.158 INFO:teuthology.orchestra.run.smithi116.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 391 kB/s | 111 kB 00:00 2024-03-04T15:51:30.234 INFO:teuthology.orchestra.run.smithi116.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 974 kB/s | 309 kB 00:00 2024-03-04T15:51:30.251 INFO:teuthology.orchestra.run.smithi116.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 282 kB/s | 26 kB 00:00 2024-03-04T15:51:30.309 INFO:teuthology.orchestra.run.smithi116.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 645 kB/s | 49 kB 00:00 2024-03-04T15:51:30.315 INFO:teuthology.orchestra.run.smithi152.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 22 MB/s | 18 MB 00:00 2024-03-04T15:51:30.334 INFO:teuthology.orchestra.run.smithi116.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 806 kB/s | 67 kB 00:00 2024-03-04T15:51:30.357 INFO:teuthology.orchestra.run.smithi152.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 302 kB/s | 106 kB 00:00 2024-03-04T15:51:30.391 INFO:teuthology.orchestra.run.smithi152.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 600 kB/s | 44 kB 00:00 2024-03-04T15:51:30.416 INFO:teuthology.orchestra.run.smithi152.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 499 kB/s | 159 kB 00:00 2024-03-04T15:51:30.560 INFO:teuthology.orchestra.run.smithi116.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 852 kB/s | 192 kB 00:00 2024-03-04T15:51:30.625 INFO:teuthology.orchestra.run.smithi152.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 661 kB 00:00 2024-03-04T15:51:30.684 INFO:teuthology.orchestra.run.smithi152.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 967 kB/s | 282 kB 00:00 2024-03-04T15:51:30.709 INFO:teuthology.orchestra.run.smithi152.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 141 kB/s | 41 kB 00:00 2024-03-04T15:51:30.827 INFO:teuthology.orchestra.run.smithi116.stdout:(26/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2024-03-04T15:51:30.869 INFO:teuthology.orchestra.run.smithi116.stdout:(27/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 838 kB 00:00 2024-03-04T15:51:30.952 INFO:teuthology.orchestra.run.smithi116.stdout:(28/30): librbd1-18.2.1-0.el9.x86_64.rpm 36 MB/s | 3.0 MB 00:00 2024-03-04T15:51:31.011 INFO:teuthology.orchestra.run.smithi116.stdout:(29/30): librados2-18.2.1-0.el9.x86_64.rpm 18 MB/s | 3.3 MB 00:00 2024-03-04T15:51:31.059 INFO:teuthology.orchestra.run.smithi152.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 255 kB/s | 111 kB 00:00 2024-03-04T15:51:31.128 INFO:teuthology.orchestra.run.smithi116.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 2.8 MB/s | 1.6 MB 00:00 2024-03-04T15:51:31.132 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:51:31.132 INFO:teuthology.orchestra.run.smithi116.stdout:Total 21 MB/s | 53 MB 00:02 2024-03-04T15:51:31.133 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:51:31.167 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:51:31.167 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:51:31.176 INFO:teuthology.orchestra.run.smithi152.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-03-04T15:51:31.210 INFO:teuthology.orchestra.run.smithi152.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 587 kB/s | 309 kB 00:00 2024-03-04T15:51:31.294 INFO:teuthology.orchestra.run.smithi152.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-03-04T15:51:31.319 INFO:teuthology.orchestra.run.smithi152.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-03-04T15:51:31.461 INFO:teuthology.orchestra.run.smithi152.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-04T15:51:31.603 INFO:teuthology.orchestra.run.smithi152.stdout:(26/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2024-03-04T15:51:31.611 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:51:31.611 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:51:31.670 INFO:teuthology.orchestra.run.smithi152.stdout:(27/30): librados2-18.2.1-0.el9.x86_64.rpm 49 MB/s | 3.3 MB 00:00 2024-03-04T15:51:31.703 INFO:teuthology.orchestra.run.smithi152.stdout:(28/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-03-04T15:51:31.779 INFO:teuthology.orchestra.run.smithi152.stdout:(29/30): librbd1-18.2.1-0.el9.x86_64.rpm 28 MB/s | 3.0 MB 00:00 2024-03-04T15:51:31.829 INFO:teuthology.orchestra.run.smithi152.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 4.3 MB/s | 1.6 MB 00:00 2024-03-04T15:51:31.833 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:51:31.833 INFO:teuthology.orchestra.run.smithi152.stdout:Total 22 MB/s | 53 MB 00:02 2024-03-04T15:51:31.834 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:51:31.869 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:51:31.869 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:51:32.316 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:51:32.316 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:51:32.987 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:51:33.070 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-04T15:51:33.099 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-04T15:51:33.157 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-04T15:51:33.474 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-04T15:51:33.523 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-03-04T15:51:33.599 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-03-04T15:51:33.679 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-03-04T15:51:33.693 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:51:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-03-04T15:51:33.779 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-03-04T15:51:33.780 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-04T15:51:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-04T15:51:33.841 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-04T15:51:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-03-04T15:51:33.925 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-03-04T15:51:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-03-04T15:51:34.010 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-03-04T15:51:34.017 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-04T15:51:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-04T15:51:34.151 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-04T15:51:34.192 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-04T15:51:34.224 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-04T15:51:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-04T15:51:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-04T15:51:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-04T15:51:34.380 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-04T15:51:34.408 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-03-04T15:51:34.498 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-03-04T15:51:34.584 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-03-04T15:51:34.609 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-03-04T15:51:34.626 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-04T15:51:34.658 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-03-04T15:51:34.680 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-03-04T15:51:34.709 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-04T15:51:34.746 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-03-04T15:51:34.753 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-03-04T15:51:34.783 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-03-04T15:51:34.823 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-04T15:51:34.843 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-03-04T15:51:34.870 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-03-04T15:51:34.936 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-03-04T15:51:35.043 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-04T15:51:35.077 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-04T15:51:35.118 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-04T15:51:35.142 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-04T15:51:35.170 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-04T15:51:35.206 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-04T15:51:35.223 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-04T15:51:35.238 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-04T15:51:35.261 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-03-04T15:51:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-03-04T15:51:35.408 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-03-04T15:51:35.438 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-03-04T15:51:35.537 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-04T15:51:35.569 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-03-04T15:51:35.659 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-03-04T15:51:35.690 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-03-04T15:51:35.732 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-04T15:51:36.133 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-04T15:51:36.163 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-03-04T15:51:36.226 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-03-04T15:51:36.292 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-03-04T15:51:36.324 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-03-04T15:51:37.487 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-03-04T15:51:37.518 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-03-04T15:51:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-03-04T15:51:38.096 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-03-04T15:51:38.178 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-03-04T15:51:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T15:51:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T15:51:38.179 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:51:38.214 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-03-04T15:51:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-03-04T15:51:38.471 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-03-04T15:51:39.011 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-03-04T15:51:39.049 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-03-04T15:51:39.148 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-03-04T15:51:39.148 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T15:51:39.148 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T15:51:39.149 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:51:39.192 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-03-04T15:51:47.798 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-03-04T15:51:47.798 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-03-04T15:51:47.798 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-03-04T15:51:47.798 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-03-04T15:51:47.798 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-03-04T15:51:47.798 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-03-04T15:51:47.799 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-03-04T15:51:47.799 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-03-04T15:51:47.799 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:51:48.569 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-03-04T15:51:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-03-04T15:51:48.594 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:51:48.594 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T15:51:48.594 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T15:51:48.594 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T15:51:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:51:48.595 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-04T15:51:48.680 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-04T15:51:48.681 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-04T15:51:48.930 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-03-04T15:51:48.930 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-03-04T15:51:48.930 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-03-04T15:51:48.931 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-03-04T15:51:48.931 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-03-04T15:51:48.931 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-03-04T15:51:48.931 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-03-04T15:51:48.931 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-03-04T15:51:48.931 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:51:49.665 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-03-04T15:51:49.689 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-03-04T15:51:49.689 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:51:49.690 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T15:51:49.690 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T15:51:49.690 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T15:51:49.690 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:51:49.691 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-04T15:51:49.761 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-04T15:51:49.761 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-03-04T15:51:49.762 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-03-04T15:51:49.762 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-03-04T15:51:49.762 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-03-04T15:51:49.762 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-03-04T15:51:49.762 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-03-04T15:51:49.762 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-03-04T15:51:49.762 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-03-04T15:51:49.762 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-03-04T15:51:49.762 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-03-04T15:51:49.763 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-03-04T15:51:49.763 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-03-04T15:51:49.763 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-03-04T15:51:49.763 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-04T15:51:49.763 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-04T15:51:49.763 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-04T15:51:49.763 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-04T15:51:49.763 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-04T15:51:49.764 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-03-04T15:51:49.765 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-04T15:51:49.765 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-03-04T15:51:49.775 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-04T15:51:49.776 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-04T15:51:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-04T15:51:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-03-04T15:51:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-03-04T15:51:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-03-04T15:51:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-03-04T15:51:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-03-04T15:51:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-03-04T15:51:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-03-04T15:51:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-03-04T15:51:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-03-04T15:51:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-03-04T15:51:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-03-04T15:51:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-03-04T15:51:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-03-04T15:51:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-04T15:51:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-04T15:51:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-04T15:51:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-04T15:51:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-04T15:51:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-04T15:51:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-04T15:51:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-04T15:51:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-04T15:51:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-04T15:51:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-04T15:51:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-04T15:51:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-04T15:51:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-04T15:51:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-04T15:51:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-03-04T15:51:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-04T15:51:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-03-04T15:51:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-04T15:51:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:51:50.955 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-03-04T15:51:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.955 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:51:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T15:51:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-04T15:51:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T15:51:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T15:51:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T15:51:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T15:51:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T15:51:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T15:51:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:51:50.958 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:51:51.334 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-test 2024-03-04T15:51:51.889 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:25 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:51:51.947 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-04T15:51:51.947 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:51:51.947 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-03-04T15:51:51.947 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.947 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.948 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-03-04T15:51:51.949 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T15:51:51.950 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T15:51:51.950 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:51:51.950 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:51:52.062 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:51:52.063 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:51:52.063 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-04T15:51:52.063 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:51:52.063 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:51:52.063 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-03-04T15:51:52.063 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-04T15:51:52.064 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T15:51:52.064 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-03-04T15:51:52.064 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T15:51:52.064 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T15:51:52.064 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T15:51:52.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:51:52.064 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:51:52.064 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:51:52.064 INFO:teuthology.orchestra.run.smithi116.stdout:Install 6 Packages 2024-03-04T15:51:52.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:51:52.065 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 41 M 2024-03-04T15:51:52.065 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 167 M 2024-03-04T15:51:52.065 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:51:52.193 INFO:teuthology.orchestra.run.smithi116.stdout:(1/6): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.3 MB/s | 165 kB 00:00 2024-03-04T15:51:52.311 INFO:teuthology.orchestra.run.smithi116.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 300 kB 00:00 2024-03-04T15:51:52.316 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-test 2024-03-04T15:51:52.327 INFO:teuthology.orchestra.run.smithi116.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 469 kB/s | 63 kB 00:00 2024-03-04T15:51:52.378 INFO:teuthology.orchestra.run.smithi116.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 186 kB 00:00 2024-03-04T15:51:52.403 INFO:teuthology.orchestra.run.smithi116.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-04T15:51:52.762 INFO:teuthology.orchestra.run.smithi116.stdout:(6/6): ceph-test-18.2.1-0.el9.x86_64.rpm 57 MB/s | 40 MB 00:00 2024-03-04T15:51:52.765 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:51:52.765 INFO:teuthology.orchestra.run.smithi116.stdout:Total 58 MB/s | 41 MB 00:00 2024-03-04T15:51:52.765 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:51:52.784 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:51:52.784 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:51:52.899 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:26 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:51:53.070 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:51:53.070 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:51:53.070 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-04T15:51:53.070 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-03-04T15:51:53.071 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:51:53.072 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 41 M 2024-03-04T15:51:53.072 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 167 M 2024-03-04T15:51:53.072 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:51:53.137 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:51:53.137 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:51:53.193 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.3 MB/s | 165 kB 00:00 2024-03-04T15:51:53.344 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 421 kB/s | 63 kB 00:00 2024-03-04T15:51:53.377 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 986 kB/s | 300 kB 00:00 2024-03-04T15:51:53.402 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 3.1 MB/s | 186 kB 00:00 2024-03-04T15:51:53.427 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-04T15:51:53.663 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:51:53.735 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-03-04T15:51:53.777 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-03-04T15:51:53.786 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): ceph-test-18.2.1-0.el9.x86_64.rpm 56 MB/s | 40 MB 00:00 2024-03-04T15:51:53.790 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:51:53.790 INFO:teuthology.orchestra.run.smithi152.stdout:Total 56 MB/s | 41 MB 00:00 2024-03-04T15:51:53.790 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:51:53.811 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:51:53.811 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:51:53.820 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-03-04T15:51:53.862 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-04T15:51:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-03-04T15:51:54.166 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:51:54.167 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:51:54.700 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:51:54.760 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-03-04T15:51:54.794 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-03-04T15:51:54.838 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-03-04T15:51:54.878 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-04T15:51:54.905 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-03-04T15:51:57.616 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-03-04T15:51:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-03-04T15:51:58.675 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-03-04T15:51:58.755 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-03-04T15:51:59.868 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-03-04T15:51:59.869 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/6 2024-03-04T15:51:59.869 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/6 2024-03-04T15:51:59.869 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-03-04T15:51:59.869 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-03-04T15:51:59.869 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-03-04T15:52:00.245 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-03-04T15:52:00.245 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:00.246 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:00.246 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:18.2.1-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-03-04T15:52:00.246 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:18.2.1-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T15:52:00.246 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T15:52:00.246 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:00.246 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:00.552 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph 2024-03-04T15:52:00.899 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-03-04T15:52:00.899 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/6 2024-03-04T15:52:00.899 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/6 2024-03-04T15:52:00.899 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-03-04T15:52:00.899 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-03-04T15:52:00.900 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-03-04T15:52:01.112 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:35 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:01.282 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:01.284 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:01.284 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-04T15:52:01.284 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:01.284 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:01.285 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-03-04T15:52:01.285 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-04T15:52:01.285 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-03-04T15:52:01.285 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-03-04T15:52:01.285 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-03-04T15:52:01.285 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-03-04T15:52:01.285 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-03-04T15:52:01.285 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T15:52:01.285 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T15:52:01.285 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T15:52:01.286 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T15:52:01.286 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T15:52:01.286 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T15:52:01.286 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T15:52:01.286 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T15:52:01.286 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T15:52:01.286 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T15:52:01.286 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T15:52:01.286 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T15:52:01.286 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T15:52:01.287 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T15:52:01.287 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T15:52:01.287 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T15:52:01.287 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T15:52:01.287 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T15:52:01.287 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T15:52:01.287 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T15:52:01.287 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T15:52:01.287 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T15:52:01.287 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T15:52:01.288 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T15:52:01.288 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T15:52:01.288 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T15:52:01.288 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T15:52:01.288 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:01.288 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:01.288 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:01.288 INFO:teuthology.orchestra.run.smithi116.stdout:Install 33 Packages 2024-03-04T15:52:01.288 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:01.289 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 30 M 2024-03-04T15:52:01.289 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 110 M 2024-03-04T15:52:01.289 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:01.310 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-03-04T15:52:01.311 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:01.311 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:01.311 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:18.2.1-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-03-04T15:52:01.311 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:18.2.1-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T15:52:01.311 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T15:52:01.311 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:01.311 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:01.483 INFO:teuthology.orchestra.run.smithi116.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 59 kB/s | 6.4 kB 00:00 2024-03-04T15:52:01.541 INFO:teuthology.orchestra.run.smithi116.stdout:(2/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 13 MB/s | 2.1 MB 00:00 2024-03-04T15:52:01.583 INFO:teuthology.orchestra.run.smithi116.stdout:(3/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 6.9 MB/s | 1.4 MB 00:00 2024-03-04T15:52:01.608 INFO:teuthology.orchestra.run.smithi116.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.5 MB/s | 242 kB 00:00 2024-03-04T15:52:01.631 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph 2024-03-04T15:52:01.709 INFO:teuthology.orchestra.run.smithi116.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-04T15:52:01.942 INFO:teuthology.orchestra.run.smithi116.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-03-04T15:52:01.967 INFO:teuthology.orchestra.run.smithi116.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 402 kB/s | 144 kB 00:00 2024-03-04T15:52:01.986 INFO:teuthology.orchestra.run.smithi116.stdout:(8/33): python3-markupsafe-1.1.1-12.el9.x86_64. 115 kB/s | 32 kB 00:00 2024-03-04T15:52:02.001 INFO:teuthology.orchestra.run.smithi116.stdout:(9/33): python3-pytz-2021.1-5.el9.noarch.rpm 812 kB/s | 47 kB 00:00 2024-03-04T15:52:02.018 INFO:teuthology.orchestra.run.smithi116.stdout:(10/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-04T15:52:02.046 INFO:teuthology.orchestra.run.smithi116.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-04T15:52:02.105 INFO:teuthology.orchestra.run.smithi116.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-04T15:52:02.127 INFO:teuthology.orchestra.run.smithi116.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 947 kB/s | 103 kB 00:00 2024-03-04T15:52:02.157 INFO:teuthology.orchestra.run.smithi116.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 1.1 MB/s | 124 kB 00:00 2024-03-04T15:52:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 113 kB 00:00 2024-03-04T15:52:02.199 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:36 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:02.204 INFO:teuthology.orchestra.run.smithi116.stdout:(16/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-04T15:52:02.295 INFO:teuthology.orchestra.run.smithi116.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-03-04T15:52:02.321 INFO:teuthology.orchestra.run.smithi116.stdout:(18/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 260 kB/s | 43 kB 00:00 2024-03-04T15:52:02.346 INFO:teuthology.orchestra.run.smithi116.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-03-04T15:52:02.367 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:02.370 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:02.370 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-04T15:52:02.370 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:02.370 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:02.370 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-03-04T15:52:02.370 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-04T15:52:02.370 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-03-04T15:52:02.370 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T15:52:02.371 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T15:52:02.372 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T15:52:02.373 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T15:52:02.373 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T15:52:02.373 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:02.373 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:02.373 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:02.373 INFO:teuthology.orchestra.run.smithi152.stdout:Install 33 Packages 2024-03-04T15:52:02.373 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:02.373 INFO:teuthology.orchestra.run.smithi116.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-03-04T15:52:02.374 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 30 M 2024-03-04T15:52:02.374 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 110 M 2024-03-04T15:52:02.374 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:02.396 INFO:teuthology.orchestra.run.smithi116.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-03-04T15:52:02.422 INFO:teuthology.orchestra.run.smithi116.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-03-04T15:52:02.447 INFO:teuthology.orchestra.run.smithi116.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-03-04T15:52:02.472 INFO:teuthology.orchestra.run.smithi116.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-03-04T15:52:02.498 INFO:teuthology.orchestra.run.smithi116.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-03-04T15:52:02.523 INFO:teuthology.orchestra.run.smithi116.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-04T15:52:02.548 INFO:teuthology.orchestra.run.smithi116.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-04T15:52:02.554 INFO:teuthology.orchestra.run.smithi152.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 67 kB/s | 6.4 kB 00:00 2024-03-04T15:52:02.573 INFO:teuthology.orchestra.run.smithi116.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-03-04T15:52:02.598 INFO:teuthology.orchestra.run.smithi116.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-03-04T15:52:02.604 INFO:teuthology.orchestra.run.smithi152.stdout:(2/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 9.9 MB/s | 1.4 MB 00:00 2024-03-04T15:52:02.624 INFO:teuthology.orchestra.run.smithi116.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 892 kB/s | 90 kB 00:00 2024-03-04T15:52:02.649 INFO:teuthology.orchestra.run.smithi116.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-04T15:52:02.655 INFO:teuthology.orchestra.run.smithi152.stdout:(3/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2024-03-04T15:52:02.679 INFO:teuthology.orchestra.run.smithi152.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.5 MB/s | 242 kB 00:00 2024-03-04T15:52:02.682 INFO:teuthology.orchestra.run.smithi116.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-03-04T15:52:02.725 INFO:teuthology.orchestra.run.smithi116.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-04T15:52:02.728 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:02.728 INFO:teuthology.orchestra.run.smithi116.stdout:Total 21 MB/s | 30 MB 00:01 2024-03-04T15:52:02.729 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:02.763 INFO:teuthology.orchestra.run.smithi152.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-03-04T15:52:02.767 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:02.767 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:03.039 INFO:teuthology.orchestra.run.smithi152.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 41 MB/s | 18 MB 00:00 2024-03-04T15:52:03.063 INFO:teuthology.orchestra.run.smithi152.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 376 kB/s | 144 kB 00:00 2024-03-04T15:52:03.089 INFO:teuthology.orchestra.run.smithi152.stdout:(8/33): python3-markupsafe-1.1.1-12.el9.x86_64. 105 kB/s | 32 kB 00:00 2024-03-04T15:52:03.114 INFO:teuthology.orchestra.run.smithi152.stdout:(9/33): python3-pytz-2021.1-5.el9.noarch.rpm 632 kB/s | 47 kB 00:00 2024-03-04T15:52:03.129 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:03.129 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:03.139 INFO:teuthology.orchestra.run.smithi152.stdout:(10/33): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2024-03-04T15:52:03.164 INFO:teuthology.orchestra.run.smithi152.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-04T15:52:03.223 INFO:teuthology.orchestra.run.smithi152.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-04T15:52:03.240 INFO:teuthology.orchestra.run.smithi152.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-04T15:52:03.265 INFO:teuthology.orchestra.run.smithi152.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 1.2 MB/s | 124 kB 00:00 2024-03-04T15:52:03.291 INFO:teuthology.orchestra.run.smithi152.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 1.7 MB/s | 113 kB 00:00 2024-03-04T15:52:03.316 INFO:teuthology.orchestra.run.smithi152.stdout:(16/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-04T15:52:03.616 INFO:teuthology.orchestra.run.smithi152.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-03-04T15:52:03.666 INFO:teuthology.orchestra.run.smithi152.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 458 kB/s | 172 kB 00:00 2024-03-04T15:52:03.700 INFO:teuthology.orchestra.run.smithi152.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-04T15:52:03.742 INFO:teuthology.orchestra.run.smithi152.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2024-03-04T15:52:03.767 INFO:teuthology.orchestra.run.smithi152.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 795 kB/s | 358 kB 00:00 2024-03-04T15:52:03.792 INFO:teuthology.orchestra.run.smithi152.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 251 kB/s | 23 kB 00:00 2024-03-04T15:52:03.818 INFO:teuthology.orchestra.run.smithi152.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-04T15:52:03.851 INFO:teuthology.orchestra.run.smithi152.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 234 kB/s | 20 kB 00:00 2024-03-04T15:52:03.902 INFO:teuthology.orchestra.run.smithi152.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 426 kB/s | 46 kB 00:00 2024-03-04T15:52:03.927 INFO:teuthology.orchestra.run.smithi152.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 724 kB/s | 79 kB 00:00 2024-03-04T15:52:03.952 INFO:teuthology.orchestra.run.smithi152.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-03-04T15:52:03.985 INFO:teuthology.orchestra.run.smithi152.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-04T15:52:04.019 INFO:teuthology.orchestra.run.smithi152.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 977 kB/s | 90 kB 00:00 2024-03-04T15:52:04.044 INFO:teuthology.orchestra.run.smithi152.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-03-04T15:52:04.120 INFO:teuthology.orchestra.run.smithi152.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-04T15:52:04.170 INFO:teuthology.orchestra.run.smithi152.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-04T15:52:04.195 INFO:teuthology.orchestra.run.smithi152.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-04T15:52:04.202 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:04.202 INFO:teuthology.orchestra.run.smithi152.stdout:Total 16 MB/s | 30 MB 00:01 2024-03-04T15:52:04.202 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:04.239 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:04.240 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:04.366 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:04.418 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-04T15:52:04.459 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-04T15:52:04.500 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-04T15:52:04.591 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-04T15:52:04.604 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:04.604 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:04.661 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-04T15:52:04.703 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-04T15:52:04.774 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-04T15:52:04.828 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-04T15:52:04.875 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-04T15:52:04.932 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-04T15:52:04.994 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-04T15:52:05.185 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-04T15:52:05.237 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-04T15:52:05.284 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-04T15:52:05.327 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-04T15:52:05.357 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-04T15:52:05.433 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-04T15:52:05.468 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-04T15:52:05.501 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-04T15:52:05.542 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-04T15:52:05.577 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-04T15:52:05.611 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-04T15:52:05.700 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-04T15:52:05.742 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-04T15:52:05.822 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-04T15:52:05.911 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:05.959 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-04T15:52:05.981 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-04T15:52:06.013 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-04T15:52:06.049 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-04T15:52:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-04T15:52:06.146 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-04T15:52:06.232 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-04T15:52:06.238 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-03-04T15:52:06.264 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-03-04T15:52:06.274 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-04T15:52:06.288 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-03-04T15:52:06.288 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:06.288 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T15:52:06.288 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T15:52:06.288 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T15:52:06.288 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:06.346 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-04T15:52:06.408 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-04T15:52:06.454 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-04T15:52:06.503 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-04T15:52:06.565 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-04T15:52:06.758 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-04T15:52:06.816 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-04T15:52:06.855 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-04T15:52:06.889 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-04T15:52:06.919 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-04T15:52:06.996 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-04T15:52:07.030 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-04T15:52:07.063 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-04T15:52:07.104 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-04T15:52:07.138 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-04T15:52:07.172 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-04T15:52:07.262 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-04T15:52:07.295 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-04T15:52:07.367 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-04T15:52:07.514 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-04T15:52:07.636 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-04T15:52:07.791 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-03-04T15:52:07.816 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-03-04T15:52:07.840 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-03-04T15:52:07.841 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:07.841 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T15:52:07.841 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T15:52:07.841 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T15:52:07.841 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:07.857 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-03-04T15:52:07.885 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-03-04T15:52:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T15:52:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T15:52:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T15:52:07.886 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:08.349 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-03-04T15:52:08.372 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-03-04T15:52:08.372 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:08.372 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T15:52:08.372 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T15:52:08.373 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T15:52:08.373 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:08.575 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-03-04T15:52:08.598 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-03-04T15:52:08.598 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:08.598 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T15:52:08.598 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T15:52:08.598 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T15:52:08.598 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-03-04T15:52:09.376 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-03-04T15:52:09.408 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-03-04T15:52:09.408 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:09.408 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T15:52:09.408 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T15:52:09.408 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T15:52:09.409 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:09.868 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-03-04T15:52:09.892 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-03-04T15:52:09.892 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:09.892 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T15:52:09.892 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T15:52:09.892 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T15:52:09.892 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:10.102 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-03-04T15:52:10.125 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-03-04T15:52:10.125 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:10.125 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T15:52:10.126 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T15:52:10.126 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T15:52:10.126 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:10.193 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-03-04T15:52:10.538 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-03-04T15:52:10.538 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-03-04T15:52:10.538 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-03-04T15:52:10.538 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-03-04T15:52:10.538 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-03-04T15:52:10.538 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-03-04T15:52:10.538 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-03-04T15:52:10.538 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-04T15:52:10.538 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-04T15:52:10.539 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-04T15:52:10.540 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-04T15:52:10.540 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-04T15:52:10.540 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-04T15:52:10.540 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-04T15:52:10.540 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-04T15:52:10.540 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-04T15:52:10.540 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-04T15:52:10.540 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-04T15:52:10.540 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-04T15:52:10.541 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-04T15:52:10.541 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-04T15:52:10.541 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-04T15:52:10.541 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-04T15:52:11.468 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-04T15:52:11.468 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:11.468 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:11.468 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:11.468 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:11.468 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:11.468 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-03-04T15:52:11.469 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:11.469 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:11.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T15:52:11.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T15:52:11.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T15:52:11.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T15:52:11.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T15:52:11.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T15:52:11.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T15:52:11.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T15:52:11.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T15:52:11.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T15:52:11.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T15:52:11.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T15:52:11.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T15:52:11.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T15:52:11.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T15:52:11.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T15:52:11.471 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:11.471 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:11.700 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-base 2024-03-04T15:52:12.050 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-03-04T15:52:12.050 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-03-04T15:52:12.050 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-03-04T15:52:12.051 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-04T15:52:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-04T15:52:12.053 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-04T15:52:12.053 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-04T15:52:12.053 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-04T15:52:12.263 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:46 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:12.374 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:12.427 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:12.428 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-04T15:52:12.428 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:12.488 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install cephadm 2024-03-04T15:52:13.051 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:47 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:13.052 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-04T15:52:13.052 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:13.052 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:13.052 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:13.052 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:13.052 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:13.052 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-03-04T15:52:13.052 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:13.053 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:13.053 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T15:52:13.053 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T15:52:13.053 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T15:52:13.053 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T15:52:13.053 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T15:52:13.053 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T15:52:13.053 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T15:52:13.054 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T15:52:13.055 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T15:52:13.055 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T15:52:13.055 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T15:52:13.055 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T15:52:13.055 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T15:52:13.055 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T15:52:13.055 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:13.055 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:13.218 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:13.218 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 221 k 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 213 k 2024-03-04T15:52:13.219 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:13.289 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm-18.2.1-0.el9.noarch.rpm 3.1 MB/s | 221 kB 00:00 2024-03-04T15:52:13.290 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:13.290 INFO:teuthology.orchestra.run.smithi116.stdout:Total 3.0 MB/s | 221 kB 00:00 2024-03-04T15:52:13.290 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:13.295 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:13.295 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:13.367 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:13.438 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-base 2024-03-04T15:52:13.759 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-03-04T15:52:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-03-04T15:52:14.009 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:48 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:14.122 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:14.174 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:14.175 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-04T15:52:14.175 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:14.232 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-03-04T15:52:14.235 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install cephadm 2024-03-04T15:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-03-04T15:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-03-04T15:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:14.797 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:48 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:14.847 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-immutable-object-cache 2024-03-04T15:52:14.966 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 221 k 2024-03-04T15:52:14.967 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 213 k 2024-03-04T15:52:14.968 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:15.033 INFO:teuthology.orchestra.run.smithi152.stdout:cephadm-18.2.1-0.el9.noarch.rpm 3.3 MB/s | 221 kB 00:00 2024-03-04T15:52:15.033 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:15.034 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.2 MB/s | 221 kB 00:00 2024-03-04T15:52:15.034 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:15.039 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:15.039 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:15.042 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:15.042 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:15.116 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:49 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:15.484 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-03-04T15:52:15.571 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-03-04T15:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-03-04T15:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-03-04T15:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-04T15:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 142 k 2024-03-04T15:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 395 k 2024-03-04T15:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 2.3 MB/s | 142 kB 00:00 2024-03-04T15:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout:Total 2.2 MB/s | 142 kB 00:00 2024-03-04T15:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T15:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:15.968 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-03-04T15:52:16.298 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-03-04T15:52:16.299 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:16.299 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:16.299 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-03-04T15:52:16.299 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:16.299 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:16.544 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-immutable-object-cache 2024-03-04T15:52:16.981 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:16.982 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:16.982 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:16.982 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:16.982 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:16.982 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:17.120 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:51 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:17.223 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr 2024-03-04T15:52:17.287 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:17.288 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:17.288 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-03-04T15:52:17.288 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:17.289 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:17.289 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-03-04T15:52:17.289 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:17.289 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:17.289 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:17.289 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-04T15:52:17.289 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:17.289 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 142 k 2024-03-04T15:52:17.289 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 395 k 2024-03-04T15:52:17.289 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:17.343 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 2.6 MB/s | 142 kB 00:00 2024-03-04T15:52:17.344 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:17.344 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.5 MB/s | 142 kB 00:00 2024-03-04T15:52:17.344 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:17.351 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:17.351 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:17.404 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:17.404 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:17.540 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:17.603 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:17.627 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:17.627 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:17.627 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T15:52:17.627 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:17.798 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:51 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:17.915 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:17.968 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:17.969 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-04T15:52:17.969 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:18.029 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-dashboard 2024-03-04T15:52:18.592 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:18.592 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:18.592 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:18.592 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:18.592 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:18.592 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:18.593 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:52 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:18.757 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:18.758 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:18.758 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-04T15:52:18.758 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:18.759 INFO:teuthology.orchestra.run.smithi116.stdout:Install 7 Packages 2024-03-04T15:52:18.760 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:18.760 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.1 M 2024-03-04T15:52:18.760 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 8.5 M 2024-03-04T15:52:18.760 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:18.841 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr 2024-03-04T15:52:19.231 INFO:teuthology.orchestra.run.smithi116.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 213 kB/s | 23 kB 00:00 2024-03-04T15:52:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 117 kB/s | 15 kB 00:00 2024-03-04T15:52:19.299 INFO:teuthology.orchestra.run.smithi116.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 10 MB/s | 1.8 MB 00:00 2024-03-04T15:52:19.349 INFO:teuthology.orchestra.run.smithi116.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 77 kB/s | 9.0 kB 00:00 2024-03-04T15:52:19.391 INFO:teuthology.orchestra.run.smithi116.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 334 kB/s | 31 kB 00:00 2024-03-04T15:52:19.408 INFO:teuthology.orchestra.run.smithi116.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 257 kB/s | 41 kB 00:00 2024-03-04T15:52:19.417 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:53 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-03-04T15:52:19.493 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:19.493 INFO:teuthology.orchestra.run.smithi116.stdout:Total 2.8 MB/s | 2.1 MB 00:00 2024-03-04T15:52:19.493 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:19.501 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:19.501 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:19.531 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:19.563 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:19.564 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:19.584 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:19.585 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-04T15:52:19.585 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:19.653 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-dashboard 2024-03-04T15:52:19.864 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:19.917 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-04T15:52:19.951 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-04T15:52:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-04T15:52:20.038 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-04T15:52:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-03-04T15:52:20.221 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:54 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:20.300 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-03-04T15:52:20.337 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-03-04T15:52:20.390 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:20.391 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:20.391 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-04T15:52:20.391 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:20.391 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout:Install 7 Packages 2024-03-04T15:52:20.392 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:20.393 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.1 M 2024-03-04T15:52:20.393 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 8.5 M 2024-03-04T15:52:20.393 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:20.713 INFO:teuthology.orchestra.run.smithi152.stdout:(1/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 121 kB/s | 15 kB 00:00 2024-03-04T15:52:20.738 INFO:teuthology.orchestra.run.smithi152.stdout:(2/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 159 kB/s | 23 kB 00:00 2024-03-04T15:52:20.780 INFO:teuthology.orchestra.run.smithi152.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 9.4 MB/s | 1.8 MB 00:00 2024-03-04T15:52:20.873 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-03-04T15:52:20.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-03-04T15:52:20.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-03-04T15:52:20.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-03-04T15:52:20.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-04T15:52:20.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-04T15:52:20.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-04T15:52:20.955 INFO:teuthology.orchestra.run.smithi152.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 37 kB/s | 9.0 kB 00:00 2024-03-04T15:52:20.989 INFO:teuthology.orchestra.run.smithi152.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 163 kB/s | 41 kB 00:00 2024-03-04T15:52:21.014 INFO:teuthology.orchestra.run.smithi152.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 131 kB/s | 31 kB 00:00 2024-03-04T15:52:21.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-04T15:52:21.190 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:21.191 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:21.191 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-03-04T15:52:21.191 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-03-04T15:52:21.191 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-03-04T15:52:21.191 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-04T15:52:21.191 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-04T15:52:21.191 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T15:52:21.191 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T15:52:21.191 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:21.192 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:21.206 INFO:teuthology.orchestra.run.smithi152.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 753 kB/s | 188 kB 00:00 2024-03-04T15:52:21.209 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:21.210 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.5 MB/s | 2.1 MB 00:00 2024-03-04T15:52:21.210 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:21.218 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:21.218 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:21.287 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:21.287 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:21.384 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-04T15:52:21.596 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:21.648 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-04T15:52:21.682 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-04T15:52:21.729 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-04T15:52:21.769 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-04T15:52:21.798 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-03-04T15:52:21.939 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:55 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:22.032 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-03-04T15:52:22.059 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-03-04T15:52:22.103 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T15:52:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T15:52:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T15:52:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T15:52:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T15:52:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T15:52:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:22.106 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:22.106 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:22.106 INFO:teuthology.orchestra.run.smithi116.stdout:Install 12 Packages 2024-03-04T15:52:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:22.107 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 37 M 2024-03-04T15:52:22.107 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 209 M 2024-03-04T15:52:22.107 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:22.348 INFO:teuthology.orchestra.run.smithi116.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 38 MB/s | 7.4 MB 00:00 2024-03-04T15:52:22.365 INFO:teuthology.orchestra.run.smithi116.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 148 kB/s | 32 kB 00:00 2024-03-04T15:52:22.398 INFO:teuthology.orchestra.run.smithi116.stdout:(3/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 352 kB/s | 17 kB 00:00 2024-03-04T15:52:22.432 INFO:teuthology.orchestra.run.smithi116.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 509 kB/s | 34 kB 00:00 2024-03-04T15:52:22.663 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-03-04T15:52:22.663 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-03-04T15:52:22.663 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-03-04T15:52:22.664 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-03-04T15:52:22.664 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-04T15:52:22.664 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-04T15:52:22.664 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-04T15:52:22.682 INFO:teuthology.orchestra.run.smithi116.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.6 MB/s | 3.0 MB 00:00 2024-03-04T15:52:22.849 INFO:teuthology.orchestra.run.smithi116.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-03-04T15:52:22.916 INFO:teuthology.orchestra.run.smithi116.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.9 MB/s | 4.8 MB 00:00 2024-03-04T15:52:22.941 INFO:teuthology.orchestra.run.smithi116.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 843 kB/s | 217 kB 00:00 2024-03-04T15:52:22.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-04T15:52:22.998 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:22.998 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:22.998 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-03-04T15:52:22.998 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-03-04T15:52:22.999 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-03-04T15:52:22.999 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-04T15:52:22.999 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-04T15:52:23.000 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T15:52:23.000 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T15:52:23.000 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:23.000 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:23.050 INFO:teuthology.orchestra.run.smithi116.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 194 kB 00:00 2024-03-04T15:52:23.117 INFO:teuthology.orchestra.run.smithi116.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 802 kB 00:00 2024-03-04T15:52:23.198 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-04T15:52:23.392 INFO:teuthology.orchestra.run.smithi116.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-04T15:52:23.756 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:57 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:23.784 INFO:teuthology.orchestra.run.smithi116.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 335 kB/s | 245 kB 00:00 2024-03-04T15:52:23.786 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:23.787 INFO:teuthology.orchestra.run.smithi116.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-04T15:52:23.803 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:23.830 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:23.830 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:23.922 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:23.923 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:23.923 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-04T15:52:23.923 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T15:52:23.924 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T15:52:23.925 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:23.925 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:23.925 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:23.925 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-03-04T15:52:23.925 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:23.925 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 37 M 2024-03-04T15:52:23.925 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 209 M 2024-03-04T15:52:23.925 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:24.270 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 37 MB/s | 7.4 MB 00:00 2024-03-04T15:52:24.287 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:24.287 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:24.295 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 140 kB/s | 32 kB 00:00 2024-03-04T15:52:24.321 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 345 kB/s | 17 kB 00:00 2024-03-04T15:52:24.363 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 510 kB/s | 34 kB 00:00 2024-03-04T15:52:24.696 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-03-04T15:52:24.771 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-03-04T15:52:24.796 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-04T15:52:24.979 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:25.039 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 802 kB 00:00 2024-03-04T15:52:25.039 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-04T15:52:25.071 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-04T15:52:25.164 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-04T15:52:25.208 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-04T15:52:25.385 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T15:52:25.406 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-03-04T15:52:25.415 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T15:52:25.431 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 498 kB/s | 194 kB 00:00 2024-03-04T15:52:25.500 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-04T15:52:25.731 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 1.8 MB/s | 3.0 MB 00:01 2024-03-04T15:52:26.023 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 398 kB/s | 245 kB 00:00 2024-03-04T15:52:26.025 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:26.026 INFO:teuthology.orchestra.run.smithi152.stdout:Total 18 MB/s | 37 MB 00:02 2024-03-04T15:52:26.047 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:26.073 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:26.074 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:26.193 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-04T15:52:26.310 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-04T15:52:26.507 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:26.507 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:27.207 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:27.271 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-04T15:52:27.310 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-04T15:52:27.403 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-04T15:52:27.447 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-04T15:52:27.558 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-04T15:52:27.612 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T15:52:27.646 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T15:52:27.732 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-04T15:52:28.375 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-04T15:52:28.490 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-04T15:52:28.567 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-04T15:52:28.590 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-03-04T15:52:29.760 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-04T15:52:30.789 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-04T15:52:30.819 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-03-04T15:52:31.288 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-03-04T15:52:31.288 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-03-04T15:52:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-04T15:52:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-04T15:52:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-04T15:52:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T15:52:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T15:52:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-04T15:52:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-04T15:52:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-04T15:52:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-04T15:52:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-04T15:52:31.772 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-04T15:52:31.772 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T15:52:31.773 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:31.774 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:32.162 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-rook 2024-03-04T15:52:32.730 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:06 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:32.901 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:32.901 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:32.901 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-04T15:52:32.901 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:32.901 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:32.901 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-03-04T15:52:32.901 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-04T15:52:32.901 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T15:52:32.901 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T15:52:32.901 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout:Install 13 Packages 2024-03-04T15:52:32.902 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:32.903 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.0 M 2024-03-04T15:52:32.903 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 26 M 2024-03-04T15:52:32.903 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:33.164 INFO:teuthology.orchestra.run.smithi116.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 491 kB/s | 50 kB 00:00 2024-03-04T15:52:33.181 INFO:teuthology.orchestra.run.smithi116.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 205 kB/s | 24 kB 00:00 2024-03-04T15:52:33.198 INFO:teuthology.orchestra.run.smithi116.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 133 kB/s | 18 kB 00:00 2024-03-04T15:52:33.273 INFO:teuthology.orchestra.run.smithi116.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 174 kB 00:00 2024-03-04T15:52:33.307 INFO:teuthology.orchestra.run.smithi116.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-03-04T15:52:33.332 INFO:teuthology.orchestra.run.smithi116.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 211 kB 00:00 2024-03-04T15:52:33.349 INFO:teuthology.orchestra.run.smithi116.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2024-03-04T15:52:33.458 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-03-04T15:52:33.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-03-04T15:52:33.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-04T15:52:33.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-04T15:52:33.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-04T15:52:33.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T15:52:33.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T15:52:33.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-04T15:52:33.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-04T15:52:33.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-04T15:52:33.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-04T15:52:33.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-04T15:52:33.466 INFO:teuthology.orchestra.run.smithi116.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-04T15:52:33.483 INFO:teuthology.orchestra.run.smithi116.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 184 kB/s | 32 kB 00:00 2024-03-04T15:52:33.599 INFO:teuthology.orchestra.run.smithi116.stdout:(10/13): python3-google-auth-2.28.0-1.el9.noarc 813 kB/s | 203 kB 00:00 2024-03-04T15:52:33.617 INFO:teuthology.orchestra.run.smithi116.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 442 kB/s | 59 kB 00:00 2024-03-04T15:52:33.659 INFO:teuthology.orchestra.run.smithi116.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-04T15:52:33.793 INFO:teuthology.orchestra.run.smithi116.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-04T15:52:33.794 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:33.794 INFO:teuthology.orchestra.run.smithi116.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-03-04T15:52:33.795 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:33.807 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:33.807 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:33.901 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-04T15:52:33.901 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:33.901 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:33.901 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-03-04T15:52:33.901 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T15:52:33.901 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T15:52:33.901 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T15:52:33.901 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T15:52:33.901 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T15:52:33.902 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T15:52:33.902 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T15:52:33.902 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T15:52:33.902 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T15:52:33.902 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T15:52:33.902 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T15:52:33.902 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:33.902 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:33.904 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:33.905 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:34.288 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-rook 2024-03-04T15:52:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:34.598 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-04T15:52:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-04T15:52:34.707 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-04T15:52:34.753 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-04T15:52:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-04T15:52:34.864 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:08 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-04T15:52:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-04T15:52:35.031 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:35.033 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:35.033 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-04T15:52:35.033 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:35.033 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:35.033 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-03-04T15:52:35.033 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-04T15:52:35.033 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T15:52:35.033 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T15:52:35.033 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T15:52:35.033 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2024-03-04T15:52:35.034 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:35.035 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.0 M 2024-03-04T15:52:35.035 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 26 M 2024-03-04T15:52:35.035 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:35.045 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-04T15:52:35.193 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 471 kB/s | 50 kB 00:00 2024-03-04T15:52:35.209 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 198 kB/s | 24 kB 00:00 2024-03-04T15:52:35.235 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 122 kB/s | 18 kB 00:00 2024-03-04T15:52:35.276 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-04T15:52:35.302 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-04T15:52:35.327 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.2 MB/s | 211 kB 00:00 2024-03-04T15:52:35.352 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 570 kB/s | 43 kB 00:00 2024-03-04T15:52:35.469 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-04T15:52:35.494 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-04T15:52:35.561 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): python3-google-auth-2.28.0-1.el9.noarc 971 kB/s | 203 kB 00:00 2024-03-04T15:52:35.587 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-03-04T15:52:35.620 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-04T15:52:35.774 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-04T15:52:35.787 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.2 MB/s | 1.0 MB 00:00 2024-03-04T15:52:35.789 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:35.789 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-04T15:52:35.789 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:35.801 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:35.802 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-04T15:52:35.868 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-04T15:52:35.901 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:35.901 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:35.916 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-04T15:52:35.938 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-03-04T15:52:36.452 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:36.624 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-04T15:52:36.679 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-04T15:52:36.734 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-04T15:52:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-03-04T15:52:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-03-04T15:52:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-04T15:52:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-04T15:52:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-04T15:52:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-04T15:52:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-04T15:52:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-04T15:52:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-04T15:52:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-04T15:52:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-04T15:52:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-04T15:52:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-04T15:52:36.784 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-04T15:52:36.822 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-04T15:52:36.916 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-04T15:52:37.023 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-04T15:52:37.073 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-04T15:52:37.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-04T15:52:37.137 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:37.137 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:37.137 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-03-04T15:52:37.137 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:37.138 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:37.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-cephadm 2024-03-04T15:52:37.790 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-04T15:52:37.844 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-04T15:52:37.878 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-04T15:52:37.925 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-04T15:52:37.948 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-03-04T15:52:38.019 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:12 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T15:52:38.194 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:38.195 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:38.195 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:38.195 INFO:teuthology.orchestra.run.smithi116.stdout:Install 6 Packages 2024-03-04T15:52:38.195 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:38.195 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 6.8 M 2024-03-04T15:52:38.195 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 33 M 2024-03-04T15:52:38.195 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:38.470 INFO:teuthology.orchestra.run.smithi116.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.1 MB/s | 132 kB 00:00 2024-03-04T15:52:38.578 INFO:teuthology.orchestra.run.smithi116.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.0 MB/s | 227 kB 00:00 2024-03-04T15:52:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-03-04T15:52:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-03-04T15:52:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-04T15:52:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-04T15:52:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-04T15:52:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-04T15:52:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-04T15:52:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-04T15:52:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-04T15:52:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-04T15:52:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-04T15:52:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-04T15:52:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-04T15:52:38.887 INFO:teuthology.orchestra.run.smithi116.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 187 kB/s | 58 kB 00:00 2024-03-04T15:52:38.946 INFO:teuthology.orchestra.run.smithi116.stdout:(4/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-04T15:52:38.996 INFO:teuthology.orchestra.run.smithi116.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 471 kB/s | 51 kB 00:00 2024-03-04T15:52:39.234 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T15:52:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T15:52:39.236 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T15:52:39.236 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T15:52:39.236 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:39.236 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:39.405 INFO:teuthology.orchestra.run.smithi116.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.5 MB/s | 5.8 MB 00:01 2024-03-04T15:52:39.409 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:39.409 INFO:teuthology.orchestra.run.smithi116.stdout:Total 5.6 MB/s | 6.8 MB 00:01 2024-03-04T15:52:39.409 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:39.418 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:39.418 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:39.491 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-cephadm 2024-03-04T15:52:39.506 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:39.506 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:39.807 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:39.987 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-04T15:52:40.039 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-04T15:52:40.064 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:14 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:40.232 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:40.233 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:40.233 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-04T15:52:40.233 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:40.233 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:40.233 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-03-04T15:52:40.233 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-04T15:52:40.233 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T15:52:40.233 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T15:52:40.233 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T15:52:40.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T15:52:40.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T15:52:40.234 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:40.234 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:40.234 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:40.234 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-03-04T15:52:40.234 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:40.235 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 6.8 M 2024-03-04T15:52:40.235 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 33 M 2024-03-04T15:52:40.235 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:40.429 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.1 MB/s | 132 kB 00:00 2024-03-04T15:52:40.529 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.0 MB/s | 227 kB 00:00 2024-03-04T15:52:40.565 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-04T15:52:40.658 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-04T15:52:40.671 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 409 kB/s | 58 kB 00:00 2024-03-04T15:52:40.722 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-04T15:52:40.727 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-04T15:52:40.746 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 680 kB/s | 51 kB 00:00 2024-03-04T15:52:40.751 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-03-04T15:52:41.022 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.1 MB/s | 5.8 MB 00:00 2024-03-04T15:52:41.023 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:41.024 INFO:teuthology.orchestra.run.smithi152.stdout:Total 8.6 MB/s | 6.8 MB 00:00 2024-03-04T15:52:41.024 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:41.033 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:41.033 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:41.123 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:41.124 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:41.424 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:41.575 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-03-04T15:52:41.576 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-03-04T15:52:41.576 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-04T15:52:41.576 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-04T15:52:41.576 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-04T15:52:41.576 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-04T15:52:41.586 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-04T15:52:41.650 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-04T15:52:41.878 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-04T15:52:41.878 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:41.878 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:41.879 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-03-04T15:52:41.879 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T15:52:41.879 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T15:52:41.879 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T15:52:41.879 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T15:52:41.879 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T15:52:41.879 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:41.879 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:42.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-fuse 2024-03-04T15:52:42.167 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-04T15:52:42.252 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-04T15:52:42.314 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-04T15:52:42.337 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-03-04T15:52:42.699 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:16 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:42.865 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:42.866 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:42.866 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:42.866 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:42.866 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:42.866 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-03-04T15:52:42.866 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-04T15:52:42.866 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T15:52:42.866 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:42.866 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:42.866 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:42.867 INFO:teuthology.orchestra.run.smithi116.stdout:Install 2 Packages 2024-03-04T15:52:42.867 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:42.867 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 918 k 2024-03-04T15:52:42.867 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 2.7 M 2024-03-04T15:52:42.867 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:42.965 INFO:teuthology.orchestra.run.smithi116.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 8.3 MB/s | 839 kB 00:00 2024-03-04T15:52:43.040 INFO:teuthology.orchestra.run.smithi116.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 453 kB/s | 79 kB 00:00 2024-03-04T15:52:43.041 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:43.042 INFO:teuthology.orchestra.run.smithi116.stdout:Total 5.1 MB/s | 918 kB 00:00 2024-03-04T15:52:43.042 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:43.061 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:43.061 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:43.135 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:43.135 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-03-04T15:52:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-03-04T15:52:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-04T15:52:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-04T15:52:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-04T15:52:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-04T15:52:43.320 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:43.453 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-04T15:52:43.531 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-03-04T15:52:43.545 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-04T15:52:43.545 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:43.545 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:43.545 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-03-04T15:52:43.545 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T15:52:43.545 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T15:52:43.545 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T15:52:43.546 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T15:52:43.546 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T15:52:43.546 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:43.546 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:43.798 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-fuse 2024-03-04T15:52:44.286 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-03-04T15:52:44.286 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-03-04T15:52:44.365 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:18 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:44.533 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:44.533 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:44.533 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:44.533 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:44.533 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:44.533 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-03-04T15:52:44.533 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-04T15:52:44.533 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T15:52:44.534 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:44.534 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:44.534 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:44.534 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-03-04T15:52:44.534 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:44.534 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 918 k 2024-03-04T15:52:44.535 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 2.7 M 2024-03-04T15:52:44.535 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:44.623 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-04T15:52:44.623 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:44.623 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:44.623 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-04T15:52:44.623 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:44.624 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:44.650 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 7.1 MB/s | 839 kB 00:00 2024-03-04T15:52:44.733 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 396 kB/s | 79 kB 00:00 2024-03-04T15:52:44.735 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:44.735 INFO:teuthology.orchestra.run.smithi152.stdout:Total 4.5 MB/s | 918 kB 00:00 2024-03-04T15:52:44.735 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:44.755 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:44.755 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:44.830 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:44.830 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:44.868 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados-devel 2024-03-04T15:52:45.023 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:45.164 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-04T15:52:45.232 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-03-04T15:52:45.446 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:19 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:45.610 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 127 k 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 456 k 2024-03-04T15:52:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:45.678 INFO:teuthology.orchestra.run.smithi116.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.9 MB/s | 127 kB 00:00 2024-03-04T15:52:45.679 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:45.679 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.8 MB/s | 127 kB 00:00 2024-03-04T15:52:45.679 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:45.684 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:45.684 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:45.741 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:45.741 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:45.886 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:45.931 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-03-04T15:52:45.931 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-03-04T15:52:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:46.243 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-04T15:52:46.244 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:46.244 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:46.244 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-04T15:52:46.244 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:46.244 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:46.530 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados-devel 2024-03-04T15:52:46.744 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:46.744 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:46.744 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:46.744 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:46.744 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:46.744 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:46.981 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs2 2024-03-04T15:52:47.109 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:21 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:47.276 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:47.276 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:47.276 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:47.276 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:47.277 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:47.277 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-03-04T15:52:47.277 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:47.277 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:47.277 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:47.277 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-04T15:52:47.277 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:47.277 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 127 k 2024-03-04T15:52:47.277 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 456 k 2024-03-04T15:52:47.277 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:47.337 INFO:teuthology.orchestra.run.smithi152.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 2.1 MB/s | 127 kB 00:00 2024-03-04T15:52:47.338 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:47.338 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.0 MB/s | 127 kB 00:00 2024-03-04T15:52:47.338 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:47.343 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:47.343 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:47.400 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:47.400 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:47.541 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:47.544 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:21 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:47.644 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:47.660 INFO:teuthology.orchestra.run.smithi116.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:47.713 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:47.714 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-04T15:52:47.714 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:47.778 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs-devel 2024-03-04T15:52:48.083 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:48.350 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:22 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:48.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:48.377 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:48.377 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:48.377 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:48.377 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:48.377 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:48.518 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:48.519 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:48.519 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:48.519 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:48.519 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:48.519 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-03-04T15:52:48.519 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:48.519 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:48.519 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:48.519 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-04T15:52:48.519 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:48.520 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 31 k 2024-03-04T15:52:48.520 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 139 k 2024-03-04T15:52:48.520 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:48.600 INFO:teuthology.orchestra.run.smithi116.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 385 kB/s | 31 kB 00:00 2024-03-04T15:52:48.601 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:48.601 INFO:teuthology.orchestra.run.smithi116.stdout:Total 379 kB/s | 31 kB 00:00 2024-03-04T15:52:48.601 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:48.604 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:48.604 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:48.632 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:48.632 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:48.674 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs2 2024-03-04T15:52:48.725 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:48.857 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:49.262 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:23 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:49.379 INFO:teuthology.orchestra.run.smithi152.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:49.433 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:49.434 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-04T15:52:49.434 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:49.505 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs-devel 2024-03-04T15:52:49.576 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:49.577 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:49.577 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:49.577 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:49.577 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:49.577 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:49.757 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados2 2024-03-04T15:52:50.075 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:24 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:50.244 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:50.244 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 31 k 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 139 k 2024-03-04T15:52:50.245 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:50.306 INFO:teuthology.orchestra.run.smithi152.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 506 kB/s | 31 kB 00:00 2024-03-04T15:52:50.307 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:50.307 INFO:teuthology.orchestra.run.smithi152.stdout:Total 495 kB/s | 31 kB 00:00 2024-03-04T15:52:50.307 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:50.311 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:50.311 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:50.320 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:24 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:50.345 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:50.345 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:50.433 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:50.486 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:50.487 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-04T15:52:50.487 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:50.546 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:50.555 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librbd1 2024-03-04T15:52:50.952 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:51.111 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:25 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:51.223 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:51.235 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:51.235 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:51.235 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:51.236 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:51.236 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:51.236 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-04T15:52:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:51.348 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rados 2024-03-04T15:52:51.399 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados2 2024-03-04T15:52:51.906 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:25 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:51.964 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:25 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:52.020 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:52.072 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:52.073 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-04T15:52:52.073 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:52.077 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:52.130 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:52.131 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-04T15:52:52.131 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:52.136 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rgw 2024-03-04T15:52:52.196 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librbd1 2024-03-04T15:52:52.694 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:26 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:52.755 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:26 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:52.807 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:52.860 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-04T15:52:52.861 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:52.868 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:52.921 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:52.922 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-04T15:52:52.922 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:52.927 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-cephfs 2024-03-04T15:52:52.992 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rados 2024-03-04T15:52:53.492 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:27 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:53.555 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:27 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:53.605 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:53.656 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:53.657 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-04T15:52:53.657 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:53.669 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:53.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rbd 2024-03-04T15:52:53.724 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:53.725 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-04T15:52:53.725 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:53.791 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rgw 2024-03-04T15:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:28 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:54.351 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:28 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:54.445 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:54.446 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-04T15:52:54.446 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:54.464 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:54.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-fuse 2024-03-04T15:52:54.516 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:54.517 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-04T15:52:54.517 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:54.579 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-cephfs 2024-03-04T15:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:29 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:55.146 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:29 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-03-04T15:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-04T15:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 86 k 2024-03-04T15:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 226 k 2024-03-04T15:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:55.260 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:55.303 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.3 MB/s | 86 kB 00:00 2024-03-04T15:52:55.304 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:55.304 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.2 MB/s | 86 kB 00:00 2024-03-04T15:52:55.305 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:55.311 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:55.311 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:55.313 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:55.314 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-04T15:52:55.314 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:55.366 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:55.366 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:55.382 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rbd 2024-03-04T15:52:55.501 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:55.601 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:55.958 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:29 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:56.048 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:56.072 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-03-04T15:52:56.125 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:56.126 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-04T15:52:56.126 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:56.190 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-fuse 2024-03-04T15:52:56.393 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:56.393 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:56.393 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:56.393 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:56.393 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:56.393 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:56.639 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-mirror 2024-03-04T15:52:56.758 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:30 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:56.925 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:56.926 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:56.926 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:56.926 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:56.926 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:56.926 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-03-04T15:52:56.926 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:56.926 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:56.926 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:56.926 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-04T15:52:56.927 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:56.927 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 86 k 2024-03-04T15:52:56.927 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 226 k 2024-03-04T15:52:56.927 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:56.993 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.3 MB/s | 86 kB 00:00 2024-03-04T15:52:56.993 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:56.994 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.3 MB/s | 86 kB 00:00 2024-03-04T15:52:56.994 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:57.001 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:57.001 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:57.057 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:57.058 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:57.195 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:57.207 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:31 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:57.317 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:57.379 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:52:57.380 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:57.380 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:57.380 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:57.381 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:52:57.381 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-03-04T15:52:57.381 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:57.381 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:52:57.381 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:52:57.381 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-04T15:52:57.381 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:57.381 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 3.0 M 2024-03-04T15:52:57.381 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 12 M 2024-03-04T15:52:57.381 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:52:57.473 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 32 MB/s | 3.0 MB 00:00 2024-03-04T15:52:57.474 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:57.474 INFO:teuthology.orchestra.run.smithi116.stdout:Total 32 MB/s | 3.0 MB 00:00 2024-03-04T15:52:57.474 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:52:57.483 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:52:57.483 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:52:57.555 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:52:57.555 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:52:57.735 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:57.973 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:52:58.033 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:58.033 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:58.033 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:58.033 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:52:58.033 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:58.033 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:58.033 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:52:58.056 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:58.056 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:58.056 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T15:52:58.056 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T15:52:58.057 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T15:52:58.057 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:58.263 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2024-03-04T15:52:58.839 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:32 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:52:59.011 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:52:59.011 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:59.011 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-04T15:52:59.011 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:59.012 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:52:59.012 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-03-04T15:52:59.012 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:59.012 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:52:59.012 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:52:59.012 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-04T15:52:59.012 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:52:59.012 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.0 M 2024-03-04T15:52:59.012 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 12 M 2024-03-04T15:52:59.012 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:52:59.104 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 33 MB/s | 3.0 MB 00:00 2024-03-04T15:52:59.104 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:52:59.104 INFO:teuthology.orchestra.run.smithi152.stdout:Total 32 MB/s | 3.0 MB 00:00 2024-03-04T15:52:59.105 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:52:59.114 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:52:59.114 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:52:59.186 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:52:59.187 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:52:59.215 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:59.215 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:59.216 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:52:59.216 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-03-04T15:52:59.216 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:52:59.216 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:52:59.447 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-nbd 2024-03-04T15:52:59.607 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:52:59.654 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:59.678 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:52:59.679 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:52:59.679 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T15:52:59.679 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T15:52:59.679 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T15:52:59.679 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:53:00.021 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:34 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:53:00.189 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T15:53:00.190 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:53:00.190 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-04T15:53:00.190 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:53:00.190 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T15:53:00.190 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-03-04T15:53:00.190 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:53:00.190 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T15:53:00.190 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T15:53:00.191 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-04T15:53:00.191 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:53:00.191 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 171 k 2024-03-04T15:53:00.191 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 494 k 2024-03-04T15:53:00.191 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T15:53:00.251 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.7 MB/s | 171 kB 00:00 2024-03-04T15:53:00.252 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:53:00.252 INFO:teuthology.orchestra.run.smithi116.stdout:Total 2.7 MB/s | 171 kB 00:00 2024-03-04T15:53:00.252 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T15:53:00.260 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T15:53:00.260 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T15:53:00.315 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T15:53:00.316 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T15:53:00.470 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T15:53:00.584 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:53:00.828 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:53:00.828 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:53:00.828 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:53:00.829 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-03-04T15:53:00.829 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:53:00.829 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:53:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:53:01.084 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-nbd 2024-03-04T15:53:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:53:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:53:01.381 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T15:53:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-03-04T15:53:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:53:01.381 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T15:53:01.614 DEBUG:teuthology.parallel:result is None 2024-03-04T15:53:01.661 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:35 ago on Mon 04 Mar 2024 03:51:26 PM UTC. 2024-03-04T15:53:01.832 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:53:01.833 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 171 k 2024-03-04T15:53:01.834 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 494 k 2024-03-04T15:53:01.834 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T15:53:01.898 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.6 MB/s | 171 kB 00:00 2024-03-04T15:53:01.899 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:53:01.899 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.6 MB/s | 171 kB 00:00 2024-03-04T15:53:01.899 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T15:53:01.906 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T15:53:01.906 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T15:53:01.963 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T15:53:01.964 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T15:53:02.109 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T15:53:02.221 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:53:02.666 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:53:02.983 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-03-04T15:53:02.984 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:53:02.984 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T15:53:02.984 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-03-04T15:53:02.984 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:53:02.984 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T15:53:03.264 DEBUG:teuthology.parallel:result is None 2024-03-04T15:53:03.264 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-04T15:53:03.264 INFO:teuthology.packaging:ref: None 2024-03-04T15:53:03.265 INFO:teuthology.packaging:tag: v18.2.1 2024-03-04T15:53:03.265 INFO:teuthology.packaging:branch: None 2024-03-04T15:53:03.265 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:53:03.265 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-03-04T15:53:03.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-03-04T15:53:03.422 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T15:53:03.446 INFO:teuthology.orchestra.run.smithi116.stdout:18.2.1-0.el9 2024-03-04T15:53:03.447 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-03-04T15:53:03.447 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-03-04T15:53:03.449 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-04T15:53:03.449 INFO:teuthology.packaging:ref: None 2024-03-04T15:53:03.449 INFO:teuthology.packaging:tag: v18.2.1 2024-03-04T15:53:03.449 INFO:teuthology.packaging:branch: None 2024-03-04T15:53:03.449 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:53:03.449 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-03-04T15:53:03.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-03-04T15:53:03.600 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T15:53:03.621 INFO:teuthology.orchestra.run.smithi152.stdout:18.2.1-0.el9 2024-03-04T15:53:03.621 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-03-04T15:53:03.622 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-03-04T15:53:03.623 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-04T15:53:03.624 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:53:03.624 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T15:53:03.656 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:53:03.656 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T15:53:03.689 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-04T15:53:03.690 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:53:03.690 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T15:53:03.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T15:53:03.793 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:53:03.793 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T15:53:03.820 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T15:53:03.887 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-04T15:53:03.888 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:53:03.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T15:53:03.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T15:53:03.985 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:53:03.985 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T15:53:04.011 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T15:53:04.077 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-04T15:53:04.078 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:53:04.078 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T15:53:04.105 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T15:53:04.173 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:53:04.173 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T15:53:04.200 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T15:53:04.267 INFO:teuthology.run_tasks:Running task print... 2024-03-04T15:53:04.277 INFO:teuthology.task.print:**** done install task... 2024-03-04T15:53:04.278 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-04T15:53:04.390 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.1', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'} 2024-03-04T15:53:04.390 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.1 2024-03-04T15:53:04.391 INFO:tasks.cephadm:Cluster fsid is 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:53:04.391 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-04T15:53:04.391 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-04T15:53:04.391 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi116': '172.21.15.116', 'mon.smithi152': '172.21.15.152'} 2024-03-04T15:53:04.391 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-04T15:53:04.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-03-04T15:53:04.420 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname $(hostname -s) 2024-03-04T15:53:04.448 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-03-04T15:53:04.448 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:53:04.601 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77294/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1826-g9ca0b5de', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1826.g9ca0b5de'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-04 12:37:35.725599', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-asm0deuz-testing-2024-03-04-1211', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-04T15:53:04.935 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref reef, sha1 5cc7a7679b7ba0f0befea649a39bb50c530f1e24 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-03-04T15:53:04.937 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/reef/5cc7a7679b7ba0f0befea649a39bb50c530f1e24/centos/9/x86_64/flavors/default/cephadm 2024-03-04T15:53:04.937 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/5cc7a7679b7ba0f0befea649a39bb50c530f1e24/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:53:05.477 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 214740 Mar 4 15:53 /home/ubuntu/cephtest/cephadm 2024-03-04T15:53:05.477 DEBUG:teuthology.orchestra.run.smithi152:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/5cc7a7679b7ba0f0befea649a39bb50c530f1e24/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:53:05.878 INFO:teuthology.orchestra.run.smithi152.stdout:-rw-r--r--. 1 ubuntu ubuntu 214740 Mar 4 15:53 /home/ubuntu/cephtest/cephadm 2024-03-04T15:53:05.879 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:53:05.897 DEBUG:teuthology.orchestra.run.smithi152:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:53:05.927 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.1 on all hosts... 2024-03-04T15:53:05.928 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-03-04T15:53:05.940 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-03-04T15:53:06.833 INFO:teuthology.orchestra.run.smithi152.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-03-04T15:53:06.837 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-03-04T15:53:36.253 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-04T15:53:36.253 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-03-04T15:53:36.254 INFO:teuthology.orchestra.run.smithi152.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-03-04T15:53:36.254 INFO:teuthology.orchestra.run.smithi152.stdout: "repo_digests": [ 2024-03-04T15:53:36.254 INFO:teuthology.orchestra.run.smithi152.stdout: "quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf", 2024-03-04T15:53:36.254 INFO:teuthology.orchestra.run.smithi152.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-03-04T15:53:36.254 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-03-04T15:53:36.254 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-04T15:53:38.798 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T15:53:38.798 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-03-04T15:53:38.798 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-03-04T15:53:38.798 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-03-04T15:53:38.798 INFO:teuthology.orchestra.run.smithi116.stdout: "quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf", 2024-03-04T15:53:38.798 INFO:teuthology.orchestra.run.smithi116.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-03-04T15:53:38.798 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T15:53:38.798 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T15:53:38.816 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-03-04T15:53:38.871 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph 2024-03-04T15:53:38.906 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-03-04T15:53:38.932 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /etc/ceph 2024-03-04T15:53:38.976 INFO:tasks.cephadm:Writing seed config... 2024-03-04T15:53:38.977 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-04T15:53:38.977 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-04T15:53:38.977 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-04T15:53:38.977 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-04T15:53:38.977 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-04T15:53:38.977 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-04T15:53:38.977 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-04T15:53:38.977 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-04T15:53:38.978 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-04T15:53:38.979 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-04T15:53:38.980 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-04T15:53:38.980 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:53:38.980 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-04T15:53:38.995 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4960653c-da3f-11ee-95c6-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-04T15:53:38.995 DEBUG:teuthology.orchestra.run.smithi116:mon.smithi116> sudo journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi116.service 2024-03-04T15:53:39.037 INFO:tasks.cephadm:Bootstrapping... 2024-03-04T15:53:39.037 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 -v bootstrap --fsid 4960653c-da3f-11ee-95c6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.116 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:53:39.193 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:53:39.193 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.1', '-v', 'bootstrap', '--fsid', '4960653c-da3f-11ee-95c6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.116', '--skip-admin-label'] 2024-03-04T15:53:39.218 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:53:39.219 INFO:teuthology.orchestra.run.smithi116.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-04T15:53:39.219 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying podman|docker is present... 2024-03-04T15:53:39.242 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:53:39.242 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying lvm2 is present... 2024-03-04T15:53:39.242 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying time synchronization is in place... 2024-03-04T15:53:39.250 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T15:53:39.250 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T15:53:39.255 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T15:53:39.256 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T15:53:39.261 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-04T15:53:39.267 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-04T15:53:39.268 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-04T15:53:39.268 INFO:teuthology.orchestra.run.smithi116.stdout:Repeating the final host check... 2024-03-04T15:53:39.290 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:53:39.290 INFO:teuthology.orchestra.run.smithi116.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-04T15:53:39.290 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl is present 2024-03-04T15:53:39.290 INFO:teuthology.orchestra.run.smithi116.stdout:lvcreate is present 2024-03-04T15:53:39.299 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T15:53:39.299 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T15:53:39.302 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T15:53:39.302 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T15:53:39.308 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-04T15:53:39.314 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-04T15:53:39.314 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-04T15:53:39.314 INFO:teuthology.orchestra.run.smithi116.stdout:Host looks OK 2024-03-04T15:53:39.314 INFO:teuthology.orchestra.run.smithi116.stdout:Cluster fsid: 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:53:39.314 INFO:teuthology.orchestra.run.smithi116.stdout:Acquiring lock 140299592034528 on /run/cephadm/4960653c-da3f-11ee-95c6-87774f69a715.lock 2024-03-04T15:53:39.315 INFO:teuthology.orchestra.run.smithi116.stdout:Lock 140299592034528 acquired on /run/cephadm/4960653c-da3f-11ee-95c6-87774f69a715.lock 2024-03-04T15:53:39.315 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 3300 ... 2024-03-04T15:53:39.315 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 6789 ... 2024-03-04T15:53:39.315 INFO:teuthology.orchestra.run.smithi116.stdout:Base mon IP(s) is [172.21.15.116:3300, 172.21.15.116:6789], mon addrv is [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-04T15:53:39.318 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.116 metric 100 2024-03-04T15:53:39.319 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.116 metric 100 2024-03-04T15:53:39.321 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-04T15:53:39.321 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-04T15:53:39.321 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-03-04T15:53:39.323 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-04T15:53:39.323 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-04T15:53:39.323 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T15:53:39.323 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-04T15:53:39.324 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-04T15:53:39.324 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T15:53:39.324 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-04T15:53:39.324 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-04T15:53:39.324 INFO:teuthology.orchestra.run.smithi116.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-04T15:53:39.325 INFO:teuthology.orchestra.run.smithi116.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-04T15:53:39.325 INFO:teuthology.orchestra.run.smithi116.stdout:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-03-04T15:53:40.451 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-03-04T15:53:40.451 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.1... 2024-03-04T15:53:40.451 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Getting image source signatures 2024-03-04T15:53:40.451 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:a733d3c618b71f19c168ebecd1953429dce2c1631835ca182e9551c36dce5989 2024-03-04T15:53:40.451 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-03-04T15:53:40.451 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying config sha256:5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-03-04T15:53:40.451 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-04T15:53:41.364 INFO:teuthology.orchestra.run.smithi116.stdout:ceph: stdout ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-03-04T15:53:41.364 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-03-04T15:53:41.364 INFO:teuthology.orchestra.run.smithi116.stdout:Extracting ceph user uid/gid from container image... 2024-03-04T15:53:42.319 INFO:teuthology.orchestra.run.smithi116.stdout:stat: stdout 167 167 2024-03-04T15:53:42.319 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial keys... 2024-03-04T15:53:43.342 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQCG7uVlLytPJhAAofyklfjPI1HvIaNzqGsrZg== 2024-03-04T15:53:44.230 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQCH7uVlJNuYJhAAC5l5Lt8NuSqt04232vmzrw== 2024-03-04T15:53:45.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQCI7uVlf34GIxAABU+ExpYz9KypQuS2utsGQQ== 2024-03-04T15:53:45.113 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial monmap... 2024-03-04T15:53:46.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T15:53:46.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-03-04T15:53:46.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:53:46.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T15:53:46.078 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool for smithi116 [v2:172.21.15.116:3300,v1:172.21.15.116:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T15:53:46.078 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = pacific 2024-03-04T15:53:46.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: set fsid to 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:53:46.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T15:53:46.078 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:53:46.078 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mon... 2024-03-04T15:53:47.074 INFO:teuthology.orchestra.run.smithi116.stdout:create mon.smithi116 on 2024-03-04T15:53:47.275 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-04T15:53:47.412 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-04T15:53:47.589 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4960653c-da3f-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-4960653c-da3f-11ee-95c6-87774f69a715.target. 2024-03-04T15:53:47.590 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-4960653c-da3f-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-4960653c-da3f-11ee-95c6-87774f69a715.target. 2024-03-04T15:53:47.801 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi116 2024-03-04T15:53:47.801 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi116.service: Unit ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi116.service not loaded. 2024-03-04T15:53:47.948 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4960653c-da3f-11ee-95c6-87774f69a715.target.wants/ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi116.service → /etc/systemd/system/ceph-4960653c-da3f-11ee-95c6-87774f69a715@.service. 2024-03-04T15:53:48.470 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:48 smithi116 podman[28721]: 2024-03-04 15:53:48.403052314 +0000 UTC m=+0.337707094 container start e28bbe08b2f1f3ab0e6cc7e04b8a3ccd7bb9b8ca34306928a4e75ea333cda798 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-03-04T15:53:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:53:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-04T15:53:48.517 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:53:48.517 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T15:53:48.517 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-04T15:53:48.518 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T15:53:48.518 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon to start... 2024-03-04T15:53:48.518 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon... 2024-03-04T15:53:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:48 smithi116 ceph-mon[28741]: mkfs 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:53:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:48 smithi116 ceph-mon[28741]: mon.smithi116 is new leader, mons smithi116 in quorum (ranks 0) 2024-03-04T15:53:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:48 smithi116 bash[28721]: e28bbe08b2f1f3ab0e6cc7e04b8a3ccd7bb9b8ca34306928a4e75ea333cda798 2024-03-04T15:53:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:48 smithi116 systemd[1]: Started Ceph mon.smithi116 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:53:49.633 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:49 smithi116 ceph-mon[28741]: mon.smithi116 is new leader, mons smithi116 in quorum (ranks 0) 2024-03-04T15:53:49.633 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:49 smithi116 ceph-mon[28741]: monmap e1: 1 mons at {smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:53:49.633 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:49 smithi116 ceph-mon[28741]: fsmap 2024-03-04T15:53:49.633 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:49 smithi116 ceph-mon[28741]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T15:53:49.633 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:49 smithi116 ceph-mon[28741]: mgrmap e1: no daemons active 2024-03-04T15:53:49.633 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:49 smithi116 ceph-mon[28741]: from='client.? 172.21.15.116:0/69614670' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-04T15:53:49.641 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout cluster: 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout id: 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout services: 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi116 (age 0.601958s) 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout data: 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-04T15:53:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pgs: 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.986+0000 7f7fb36e9700 1 Processor -- start 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.986+0000 7f7fb36e9700 1 -- start start 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.986+0000 7f7fb36e9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac107e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.986+0000 7f7fb36e9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac108430 con 0x7f7fac107a40 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.987+0000 7f7fb1485700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac107e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.987+0000 7f7fb1485700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac107e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53348/0 (socket says 172.21.15.116:53348) 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.987+0000 7f7fb1485700 1 -- 172.21.15.116:0/1552051959 learned_addr learned my addr 172.21.15.116:0/1552051959 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.987+0000 7f7fb1485700 1 -- 172.21.15.116:0/1552051959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac108570 con 0x7f7fac107a40 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.987+0000 7f7fb1485700 1 --2- 172.21.15.116:0/1552051959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac107e60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0098a0 tx=0x7f7f9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f61635be8178b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.987+0000 7f7fa3fff700 1 -- 172.21.15.116:0/1552051959 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f9c004180 con 0x7f7fac107a40 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.987+0000 7f7fa3fff700 1 -- 172.21.15.116:0/1552051959 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c0042e0 con 0x7f7fac107a40 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.987+0000 7f7fa3fff700 1 -- 172.21.15.116:0/1552051959 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f9c0036a0 con 0x7f7fac107a40 2024-03-04T15:53:49.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.988+0000 7f7fb36e9700 1 -- 172.21.15.116:0/1552051959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 msgr2=0x7f7fac107e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.988+0000 7f7fb36e9700 1 --2- 172.21.15.116:0/1552051959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac107e60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0098a0 tx=0x7f7f9c009bb0 comp rx=0 tx=0).stop 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.988+0000 7f7fb36e9700 1 -- 172.21.15.116:0/1552051959 shutdown_connections 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.988+0000 7f7fb36e9700 1 --2- 172.21.15.116:0/1552051959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac107e60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.988+0000 7f7fb36e9700 1 -- 172.21.15.116:0/1552051959 >> 172.21.15.116:0/1552051959 conn(0x7f7fac100040 msgr2=0x7f7fac1024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.988+0000 7f7fb36e9700 1 -- 172.21.15.116:0/1552051959 shutdown_connections 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.988+0000 7f7fb36e9700 1 -- 172.21.15.116:0/1552051959 wait complete. 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.989+0000 7f7fb36e9700 1 Processor -- start 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.989+0000 7f7fb36e9700 1 -- start start 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.989+0000 7f7fb36e9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.989+0000 7f7fb36e9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac199e30 con 0x7f7fac107a40 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.989+0000 7f7fb1485700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.989+0000 7f7fb1485700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac1998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53360/0 (socket says 172.21.15.116:53360) 2024-03-04T15:53:49.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.989+0000 7f7fb1485700 1 -- 172.21.15.116:0/69614670 learned_addr learned my addr 172.21.15.116:0/69614670 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:49.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.989+0000 7f7fb1485700 1 -- 172.21.15.116:0/69614670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c009550 con 0x7f7fac107a40 2024-03-04T15:53:49.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.989+0000 7f7fb1485700 1 --2- 172.21.15.116:0/69614670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac1998f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c004000 tx=0x7f7f9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:49.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.990+0000 7f7fa27fc700 1 -- 172.21.15.116:0/69614670 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f9c003ce0 con 0x7f7fac107a40 2024-03-04T15:53:49.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.990+0000 7f7fa27fc700 1 -- 172.21.15.116:0/69614670 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c020070 con 0x7f7fac107a40 2024-03-04T15:53:49.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.990+0000 7f7fb36e9700 1 -- 172.21.15.116:0/69614670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac19a030 con 0x7f7fac107a40 2024-03-04T15:53:49.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.990+0000 7f7fa27fc700 1 -- 172.21.15.116:0/69614670 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f9c011630 con 0x7f7fac107a40 2024-03-04T15:53:49.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.990+0000 7f7fb36e9700 1 -- 172.21.15.116:0/69614670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac19a4d0 con 0x7f7fac107a40 2024-03-04T15:53:49.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.990+0000 7f7fa27fc700 1 -- 172.21.15.116:0/69614670 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7f9c022030 con 0x7f7fac107a40 2024-03-04T15:53:49.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.990+0000 7f7fa27fc700 1 -- 172.21.15.116:0/69614670 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7f9c025d00 con 0x7f7fac107a40 2024-03-04T15:53:49.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.991+0000 7f7fb36e9700 1 -- 172.21.15.116:0/69614670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac193140 con 0x7f7fac107a40 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:48.995+0000 7f7fa27fc700 1 -- 172.21.15.116:0/69614670 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f7f9c025480 con 0x7f7fac107a40 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:49.065+0000 7f7fb36e9700 1 -- 172.21.15.116:0/69614670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f7fac02cc70 con 0x7f7fac107a40 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:49.065+0000 7f7fa27fc700 1 -- 172.21.15.116:0/69614670 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f7f9c011a90 con 0x7f7fac107a40 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:49.066+0000 7f7fb36e9700 1 -- 172.21.15.116:0/69614670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 msgr2=0x7f7fac1998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:49.066+0000 7f7fb36e9700 1 --2- 172.21.15.116:0/69614670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac1998f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c004000 tx=0x7f7f9c004750 comp rx=0 tx=0).stop 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:49.066+0000 7f7fb36e9700 1 -- 172.21.15.116:0/69614670 shutdown_connections 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:49.066+0000 7f7fb36e9700 1 --2- 172.21.15.116:0/69614670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fac107a40 0x7f7fac1998f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:49.066+0000 7f7fb36e9700 1 -- 172.21.15.116:0/69614670 >> 172.21.15.116:0/69614670 conn(0x7f7fac100040 msgr2=0x7f7fac18dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:49.066+0000 7f7fb36e9700 1 -- 172.21.15.116:0/69614670 shutdown_connections 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:49.066+0000 7f7fb36e9700 1 -- 172.21.15.116:0/69614670 wait complete. 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:mon is available 2024-03-04T15:53:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:Assimilating anything we can from ceph.conf... 2024-03-04T15:53:50.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:53:50.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T15:53:50.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:53:50.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-04T15:53:50.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.039+0000 7fae43a22700 1 Processor -- start 2024-03-04T15:53:50.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.039+0000 7fae43a22700 1 -- start start 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.039+0000 7fae43a22700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.039+0000 7fae43a22700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c106e10 con 0x7fae3c106420 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.039+0000 7fae417be700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.039+0000 7fae417be700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53362/0 (socket says 172.21.15.116:53362) 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.039+0000 7fae417be700 1 -- 172.21.15.116:0/2324349952 learned_addr learned my addr 172.21.15.116:0/2324349952 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.040+0000 7fae417be700 1 -- 172.21.15.116:0/2324349952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c107620 con 0x7fae3c106420 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.040+0000 7fae417be700 1 --2- 172.21.15.116:0/2324349952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c106840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fae380098a0 tx=0x7fae38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c52b9d1e513b6b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.040+0000 7fae33fff700 1 -- 172.21.15.116:0/2324349952 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae38004180 con 0x7fae3c106420 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.040+0000 7fae33fff700 1 -- 172.21.15.116:0/2324349952 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae380042e0 con 0x7fae3c106420 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.040+0000 7fae33fff700 1 -- 172.21.15.116:0/2324349952 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae380036a0 con 0x7fae3c106420 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.040+0000 7fae43a22700 1 -- 172.21.15.116:0/2324349952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 msgr2=0x7fae3c106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.040+0000 7fae43a22700 1 --2- 172.21.15.116:0/2324349952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c106840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fae380098a0 tx=0x7fae38009bb0 comp rx=0 tx=0).stop 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.040+0000 7fae43a22700 1 -- 172.21.15.116:0/2324349952 shutdown_connections 2024-03-04T15:53:50.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.040+0000 7fae43a22700 1 --2- 172.21.15.116:0/2324349952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c106840 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.040+0000 7fae43a22700 1 -- 172.21.15.116:0/2324349952 >> 172.21.15.116:0/2324349952 conn(0x7fae3c101a00 msgr2=0x7fae3c103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.041+0000 7fae43a22700 1 -- 172.21.15.116:0/2324349952 shutdown_connections 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.041+0000 7fae43a22700 1 -- 172.21.15.116:0/2324349952 wait complete. 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.041+0000 7fae43a22700 1 Processor -- start 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.041+0000 7fae43a22700 1 -- start start 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.041+0000 7fae43a22700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.041+0000 7fae43a22700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c199dc0 con 0x7fae3c106420 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.041+0000 7fae417be700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.041+0000 7fae417be700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53378/0 (socket says 172.21.15.116:53378) 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.041+0000 7fae417be700 1 -- 172.21.15.116:0/184109712 learned_addr learned my addr 172.21.15.116:0/184109712 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.042+0000 7fae417be700 1 -- 172.21.15.116:0/184109712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38009550 con 0x7fae3c106420 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.042+0000 7fae417be700 1 --2- 172.21.15.116:0/184109712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c199880 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fae38004000 tx=0x7fae38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:50.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.042+0000 7fae327fc700 1 -- 172.21.15.116:0/184109712 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae38003ce0 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.042+0000 7fae327fc700 1 -- 172.21.15.116:0/184109712 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae38020070 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.042+0000 7fae43a22700 1 -- 172.21.15.116:0/184109712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c199fc0 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.042+0000 7fae327fc700 1 -- 172.21.15.116:0/184109712 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae38011630 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.042+0000 7fae43a22700 1 -- 172.21.15.116:0/184109712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c19a460 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.042+0000 7fae327fc700 1 -- 172.21.15.116:0/184109712 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fae38022030 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.042+0000 7fae327fc700 1 -- 172.21.15.116:0/184109712 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fae38025d00 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.043+0000 7fae43a22700 1 -- 172.21.15.116:0/184109712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c193160 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.045+0000 7fae327fc700 1 -- 172.21.15.116:0/184109712 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fae38025480 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.111+0000 7fae43a22700 1 -- 172.21.15.116:0/184109712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fae3c02cf20 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.114+0000 7fae327fc700 1 -- 172.21.15.116:0/184109712 <== mon.0 v2:172.21.15.116:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fae38011c20 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.114+0000 7fae327fc700 1 -- 172.21.15.116:0/184109712 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7fae38038590 con 0x7fae3c106420 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.115+0000 7fae43a22700 1 -- 172.21.15.116:0/184109712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 msgr2=0x7fae3c199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.115+0000 7fae43a22700 1 --2- 172.21.15.116:0/184109712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c199880 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fae38004000 tx=0x7fae38004750 comp rx=0 tx=0).stop 2024-03-04T15:53:50.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.115+0000 7fae43a22700 1 -- 172.21.15.116:0/184109712 shutdown_connections 2024-03-04T15:53:50.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.115+0000 7fae43a22700 1 --2- 172.21.15.116:0/184109712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae3c106420 0x7fae3c199880 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:50.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.115+0000 7fae43a22700 1 -- 172.21.15.116:0/184109712 >> 172.21.15.116:0/184109712 conn(0x7fae3c101a00 msgr2=0x7fae3c18dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:50.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.115+0000 7fae43a22700 1 -- 172.21.15.116:0/184109712 shutdown_connections 2024-03-04T15:53:50.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:50.116+0000 7fae43a22700 1 -- 172.21.15.116:0/184109712 wait complete. 2024-03-04T15:53:50.678 INFO:teuthology.orchestra.run.smithi116.stdout:Generating new minimal ceph.conf... 2024-03-04T15:53:50.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:50 smithi116 ceph-mon[28741]: from='client.? 172.21.15.116:0/184109712' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T15:53:50.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:50 smithi116 ceph-mon[28741]: from='client.? 172.21.15.116:0/184109712' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-04T15:53:51.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:51 smithi116 ceph-mon[28741]: from='client.? 172.21.15.116:0/784946158' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:53:51.735 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.060+0000 7f8160f1a700 1 Processor -- start 2024-03-04T15:53:51.735 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.060+0000 7f8160f1a700 1 -- start start 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.061+0000 7f8160f1a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c104b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.061+0000 7f8160f1a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c1050f0 con 0x7f815c104700 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.061+0000 7f815a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c104b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.061+0000 7f815a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c104b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53392/0 (socket says 172.21.15.116:53392) 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.061+0000 7f815a59c700 1 -- 172.21.15.116:0/580775257 learned_addr learned my addr 172.21.15.116:0/580775257 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.061+0000 7f815a59c700 1 -- 172.21.15.116:0/580775257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815c105230 con 0x7f815c104700 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.061+0000 7f815a59c700 1 --2- 172.21.15.116:0/580775257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c104b20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f81500098a0 tx=0x7f8150009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443e98ae423a78e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.061+0000 7f815959a700 1 -- 172.21.15.116:0/580775257 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8150004180 con 0x7f815c104700 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.062+0000 7f815959a700 1 -- 172.21.15.116:0/580775257 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f81500042e0 con 0x7f815c104700 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.062+0000 7f815959a700 1 -- 172.21.15.116:0/580775257 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8150011a50 con 0x7f815c104700 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.062+0000 7f8160f1a700 1 -- 172.21.15.116:0/580775257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 msgr2=0x7f815c104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.062+0000 7f8160f1a700 1 --2- 172.21.15.116:0/580775257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c104b20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f81500098a0 tx=0x7f8150009bb0 comp rx=0 tx=0).stop 2024-03-04T15:53:51.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.062+0000 7f8160f1a700 1 -- 172.21.15.116:0/580775257 shutdown_connections 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.062+0000 7f8160f1a700 1 --2- 172.21.15.116:0/580775257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c104b20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.062+0000 7f8160f1a700 1 -- 172.21.15.116:0/580775257 >> 172.21.15.116:0/580775257 conn(0x7f815c100040 msgr2=0x7f815c1024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.062+0000 7f8160f1a700 1 -- 172.21.15.116:0/580775257 shutdown_connections 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.062+0000 7f8160f1a700 1 -- 172.21.15.116:0/580775257 wait complete. 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.063+0000 7f8160f1a700 1 Processor -- start 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.063+0000 7f8160f1a700 1 -- start start 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.063+0000 7f8160f1a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c1070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.063+0000 7f8160f1a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c107620 con 0x7f815c104700 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.063+0000 7f815a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c1070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.063+0000 7f815a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c1070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53406/0 (socket says 172.21.15.116:53406) 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.063+0000 7f815a59c700 1 -- 172.21.15.116:0/784946158 learned_addr learned my addr 172.21.15.116:0/784946158 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.063+0000 7f815a59c700 1 -- 172.21.15.116:0/784946158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8150009550 con 0x7f815c104700 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.064+0000 7f815a59c700 1 --2- 172.21.15.116:0/784946158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c1070e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8150009520 tx=0x7f8150011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.064+0000 7f814b7fe700 1 -- 172.21.15.116:0/784946158 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8150003a80 con 0x7f815c104700 2024-03-04T15:53:51.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.064+0000 7f814b7fe700 1 -- 172.21.15.116:0/784946158 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8150003be0 con 0x7f815c104700 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.064+0000 7f8160f1a700 1 -- 172.21.15.116:0/784946158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815c105730 con 0x7f815c104700 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.064+0000 7f814b7fe700 1 -- 172.21.15.116:0/784946158 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81500197b0 con 0x7f815c104700 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.064+0000 7f8160f1a700 1 -- 172.21.15.116:0/784946158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815c105bd0 con 0x7f815c104700 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.064+0000 7f814b7fe700 1 -- 172.21.15.116:0/784946158 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f815002d440 con 0x7f815c104700 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.064+0000 7f814b7fe700 1 -- 172.21.15.116:0/784946158 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f81500247c0 con 0x7f815c104700 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.065+0000 7f8160f1a700 1 -- 172.21.15.116:0/784946158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815c0623c0 con 0x7f815c104700 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.067+0000 7f814b7fe700 1 -- 172.21.15.116:0/784946158 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f815001ac30 con 0x7f815c104700 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.133+0000 7f8160f1a700 1 -- 172.21.15.116:0/784946158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f815c106590 con 0x7f815c104700 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.133+0000 7f814b7fe700 1 -- 172.21.15.116:0/784946158 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f8150024980 con 0x7f815c104700 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.134+0000 7f8160f1a700 1 -- 172.21.15.116:0/784946158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 msgr2=0x7f815c1070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.134+0000 7f8160f1a700 1 --2- 172.21.15.116:0/784946158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c1070e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8150009520 tx=0x7f8150011ab0 comp rx=0 tx=0).stop 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.134+0000 7f8160f1a700 1 -- 172.21.15.116:0/784946158 shutdown_connections 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.134+0000 7f8160f1a700 1 --2- 172.21.15.116:0/784946158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815c104700 0x7f815c1070e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.134+0000 7f8160f1a700 1 -- 172.21.15.116:0/784946158 >> 172.21.15.116:0/784946158 conn(0x7f815c100040 msgr2=0x7f815c101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:51.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.134+0000 7f8160f1a700 1 -- 172.21.15.116:0/784946158 shutdown_connections 2024-03-04T15:53:51.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:51.134+0000 7f8160f1a700 1 -- 172.21.15.116:0/784946158 wait complete. 2024-03-04T15:53:51.739 INFO:teuthology.orchestra.run.smithi116.stdout:Restarting the monitor... 2024-03-04T15:53:51.991 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:51 smithi116 systemd[1]: Stopping Ceph mon.smithi116 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T15:53:52.315 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:51 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116[28736]: 2024-03-04T15:53:51.988+0000 7f9a4d910700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi116 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:53:52.316 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:51 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116[28736]: 2024-03-04T15:53:51.988+0000 7f9a4d910700 -1 mon.smithi116@0(leader) e1 *** Got Signal Terminated *** 2024-03-04T15:53:52.316 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:52 smithi116 podman[28997]: 2024-03-04 15:53:52.111856033 +0000 UTC m=+0.219143200 container died e28bbe08b2f1f3ab0e6cc7e04b8a3ccd7bb9b8ca34306928a4e75ea333cda798 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) 2024-03-04T15:53:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:52 smithi116 podman[28997]: 2024-03-04 15:53:52.413923524 +0000 UTC m=+0.521210742 container cleanup e28bbe08b2f1f3ab0e6cc7e04b8a3ccd7bb9b8ca34306928a4e75ea333cda798 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-04T15:53:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:52 smithi116 bash[28997]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116 2024-03-04T15:53:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:52 smithi116 podman[29010]: 2024-03-04 15:53:52.556402526 +0000 UTC m=+0.442714175 container remove e28bbe08b2f1f3ab0e6cc7e04b8a3ccd7bb9b8ca34306928a4e75ea333cda798 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-04T15:53:53.055 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:52 smithi116 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi116.service: Deactivated successfully. 2024-03-04T15:53:53.055 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:52 smithi116 systemd[1]: Stopped Ceph mon.smithi116 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:53:53.055 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:52 smithi116 systemd[1]: Starting Ceph mon.smithi116 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T15:53:53.055 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:52 smithi116 podman[29102]: 2024-03-04 15:53:52.965646065 +0000 UTC m=+0.113995808 container create 60e0bf118c4de686f5559d34e2b95dda8785a9e2b515c8e2b2dfd708a3c11169 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, io.buildah.version=1.29.1) 2024-03-04T15:53:53.056 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:52 smithi116 podman[29102]: 2024-03-04 15:53:52.868734626 +0000 UTC m=+0.017084375 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph:v18.2.1 2024-03-04T15:53:53.248 INFO:teuthology.orchestra.run.smithi116.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-04T15:53:53.362 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 podman[29102]: 2024-03-04 15:53:53.159439558 +0000 UTC m=+0.307789307 container init 60e0bf118c4de686f5559d34e2b95dda8785a9e2b515c8e2b2dfd708a3c11169 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-04T15:53:53.362 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 podman[29102]: 2024-03-04 15:53:53.162211163 +0000 UTC m=+0.310560907 container start 60e0bf118c4de686f5559d34e2b95dda8785a9e2b515c8e2b2dfd708a3c11169 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-03-04T15:53:53.362 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:53:53.362 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-03-04T15:53:53.362 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: pidfile_write: ignore empty --pid-file 2024-03-04T15:53:53.362 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: load: jerasure load: lrc 2024-03-04T15:53:53.362 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: RocksDB version: 7.9.2 2024-03-04T15:53:53.362 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Git sha 0 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Compile date 2023-12-11 22:07:34 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: DB SUMMARY 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: DB Session ID: 4SB0HW5LO86H9FGTVVUY 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi116/store.db dir, Total Num: 1, files: 000008.sst 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi116/store.db: 000009.log size: 97496 ; 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:53:53.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T15:53:53.364 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.env: 0x5604ebe33720 2024-03-04T15:53:53.364 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T15:53:53.364 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.info_log: 0x5604eedbd2e0 2024-03-04T15:53:53.364 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:53:53.364 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.statistics: (nil) 2024-03-04T15:53:53.364 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.use_fsync: 0 2024-03-04T15:53:53.365 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:53:53.365 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.db_log_dir: 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.wal_dir: 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:53:53.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:53:53.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:53:53.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:53:53.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:53:53.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:53:53.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:53:53.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.write_buffer_manager: 0x5604ee02e140 2024-03-04T15:53:53.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:53:53.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:53:53.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.unordered_write: 0 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.row_cache: None 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.wal_filter: None 2024-03-04T15:53:53.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.wal_compression: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:53:53.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_open_files: -1 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:53:53.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Compression algorithms supported: 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: kZSTD supported: 0 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi116/store.db/MANIFEST-000010 2024-03-04T15:53:53.371 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.merge_operator: 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_filter: None 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5604eedbd3e0) 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_type: 0 2024-03-04T15:53:53.372 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: data_block_index_type: 0 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_shortening: 1 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: checksum: 4 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: no_block_cache: 0 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache: 0x5604ee0cf1f0 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_options: 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: capacity : 536870912 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: num_shard_bits : 4 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: strict_capacity_limit : 0 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_compressed: (nil) 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: persistent_cache: (nil) 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_size: 4096 2024-03-04T15:53:53.373 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_size_deviation: 10 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_restart_interval: 16 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_block_restart_interval: 1 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: metadata_block_size: 4096 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: partition_filters: 0 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: use_delta_encoding: 1 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: filter_policy: bloomfilter 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: whole_key_filtering: 1 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: verify_compression: 0 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: format_version: 5 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: enable_index_compression: 1 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_align: 0 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: max_auto_readahead_size: 262144 2024-03-04T15:53:53.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: prepopulate_block_cache: 0 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: initial_auto_readahead_size: 8192 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compression: NoCompression 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.num_levels: 7 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:53:53.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:53:53.376 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:53:53.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:53:53.378 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:53:53.379 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.ttl: 2592000 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T15:53:53.380 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi116/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 46a12290-7bb5-4234-a8dd-39d260c12269 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567633191027, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567633191992, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92846, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 90863, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83578, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709567633, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "46a12290-7bb5-4234-a8dd-39d260c12269", "db_session_id": "4SB0HW5LO86H9FGTVVUY", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567633192075, "job": 1, "event": "recovery_finished"} 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi116/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5604ee16c000 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: DB pointer 0x5604ee158000 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T15:53:53.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** DB Stats ** 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: L0 2/0 92.54 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:53:53.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Sum 2/0 92.54 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:53:53.383 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative compaction: 0.00 GB write, 30.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval compaction: 0.00 GB write, 30.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Block cache BinnedLRUCache@0x5604ee0cf1f0#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: starting mon.smithi116 rank 0 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi116 fsid 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: mon.smithi116@-1(???) e1 preinit fsid 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: mon.smithi116@-1(???).mds e1 new map 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: mon.smithi116@-1(???).mds e1 print_map 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: e1 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:53:53.384 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: legacy client fscid: -1 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: No filesystems configured 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: mon.smithi116@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: mon.smithi116@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: mon.smithi116@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: mon.smithi116@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: mon.smithi116@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta expand map: {default=false} 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta from 'false' to 'false' 2024-03-04T15:53:53.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta expanded map: {default=false} 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta expand map: {default=info} 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta from 'info' to 'info' 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta expanded map: {default=info} 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta expand map: {default=daemon} 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta from 'daemon' to 'daemon' 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta expanded map: {default=daemon} 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta expand map: {default=debug} 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta from 'debug' to 'debug' 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: expand_channel_meta expanded map: {default=debug} 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 bash[29102]: 60e0bf118c4de686f5559d34e2b95dda8785a9e2b515c8e2b2dfd708a3c11169 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 systemd[1]: Started Ceph mon.smithi116 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: mon.smithi116 is new leader, mons smithi116 in quorum (ranks 0) 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: monmap e1: 1 mons at {smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: fsmap 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T15:53:53.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:53 smithi116 ceph-mon[29120]: mgrmap e1: no daemons active 2024-03-04T15:53:54.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.721+0000 7f3bc5c50700 1 Processor -- start 2024-03-04T15:53:54.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.721+0000 7f3bc5c50700 1 -- start start 2024-03-04T15:53:54.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.721+0000 7f3bc5c50700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc0104b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:54.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.721+0000 7f3bc5c50700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc0105110 con 0x7f3bc0104720 2024-03-04T15:53:54.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.722+0000 7f3bbf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc0104b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:54.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.722+0000 7f3bbf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc0104b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53422/0 (socket says 172.21.15.116:53422) 2024-03-04T15:53:54.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.722+0000 7f3bbf7fe700 1 -- 172.21.15.116:0/2717410692 learned_addr learned my addr 172.21.15.116:0/2717410692 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:54.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.722+0000 7f3bbf7fe700 1 -- 172.21.15.116:0/2717410692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc0105250 con 0x7f3bc0104720 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.722+0000 7f3bbf7fe700 1 --2- 172.21.15.116:0/2717410692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc0104b40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009e80 tx=0x7f3bb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35bf50326f57c087 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.722+0000 7f3bbe7fc700 1 -- 172.21.15.116:0/2717410692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bb000e070 con 0x7f3bc0104720 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.722+0000 7f3bbe7fc700 1 -- 172.21.15.116:0/2717410692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3bb0003e90 con 0x7f3bc0104720 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.722+0000 7f3bbe7fc700 1 -- 172.21.15.116:0/2717410692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bb0004380 con 0x7f3bc0104720 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.723+0000 7f3bc5c50700 1 -- 172.21.15.116:0/2717410692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 msgr2=0x7f3bc0104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.723+0000 7f3bc5c50700 1 --2- 172.21.15.116:0/2717410692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc0104b40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009e80 tx=0x7f3bb000b330 comp rx=0 tx=0).stop 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.723+0000 7f3bc5c50700 1 -- 172.21.15.116:0/2717410692 shutdown_connections 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.723+0000 7f3bc5c50700 1 --2- 172.21.15.116:0/2717410692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc0104b40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.723+0000 7f3bc5c50700 1 -- 172.21.15.116:0/2717410692 >> 172.21.15.116:0/2717410692 conn(0x7f3bc0100040 msgr2=0x7f3bc01024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.723+0000 7f3bc5c50700 1 -- 172.21.15.116:0/2717410692 shutdown_connections 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.723+0000 7f3bc5c50700 1 -- 172.21.15.116:0/2717410692 wait complete. 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.724+0000 7f3bc5c50700 1 Processor -- start 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.724+0000 7f3bc5c50700 1 -- start start 2024-03-04T15:53:54.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.724+0000 7f3bc5c50700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.724+0000 7f3bc5c50700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc019a620 con 0x7f3bc0104720 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.724+0000 7f3bbf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.724+0000 7f3bbf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc019a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53430/0 (socket says 172.21.15.116:53430) 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.724+0000 7f3bbf7fe700 1 -- 172.21.15.116:0/4051042559 learned_addr learned my addr 172.21.15.116:0/4051042559 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.724+0000 7f3bbf7fe700 1 -- 172.21.15.116:0/4051042559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb0009b30 con 0x7f3bc0104720 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.724+0000 7f3bbf7fe700 1 --2- 172.21.15.116:0/4051042559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc019a0e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000b7b0 tx=0x7f3bb0004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.725+0000 7f3bbcff9700 1 -- 172.21.15.116:0/4051042559 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bb000e040 con 0x7f3bc0104720 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.725+0000 7f3bbcff9700 1 -- 172.21.15.116:0/4051042559 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3bb0003c30 con 0x7f3bc0104720 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.725+0000 7f3bc5c50700 1 -- 172.21.15.116:0/4051042559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc019a820 con 0x7f3bc0104720 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.725+0000 7f3bbcff9700 1 -- 172.21.15.116:0/4051042559 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bb001b740 con 0x7f3bc0104720 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.725+0000 7f3bc5c50700 1 -- 172.21.15.116:0/4051042559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc019ad20 con 0x7f3bc0104720 2024-03-04T15:53:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.725+0000 7f3bbcff9700 1 -- 172.21.15.116:0/4051042559 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3bb0019040 con 0x7f3bc0104720 2024-03-04T15:53:54.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.725+0000 7f3bbcff9700 1 -- 172.21.15.116:0/4051042559 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3bb001a6d0 con 0x7f3bc0104720 2024-03-04T15:53:54.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.726+0000 7f3bc5c50700 1 -- 172.21.15.116:0/4051042559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc00623c0 con 0x7f3bc0104720 2024-03-04T15:53:54.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.728+0000 7f3bbcff9700 1 -- 172.21.15.116:0/4051042559 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f3bb0029060 con 0x7f3bc0104720 2024-03-04T15:53:54.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.793+0000 7f3bc5c50700 1 -- 172.21.15.116:0/4051042559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f3bc019daa0 con 0x7f3bc0104720 2024-03-04T15:53:54.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.795+0000 7f3bbcff9700 1 -- 172.21.15.116:0/4051042559 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3bb001f070 con 0x7f3bc0104720 2024-03-04T15:53:54.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.796+0000 7f3bc5c50700 1 -- 172.21.15.116:0/4051042559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 msgr2=0x7f3bc019a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:54.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.796+0000 7f3bc5c50700 1 --2- 172.21.15.116:0/4051042559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc019a0e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000b7b0 tx=0x7f3bb0004920 comp rx=0 tx=0).stop 2024-03-04T15:53:54.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.796+0000 7f3bc5c50700 1 -- 172.21.15.116:0/4051042559 shutdown_connections 2024-03-04T15:53:54.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.796+0000 7f3bc5c50700 1 --2- 172.21.15.116:0/4051042559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0104720 0x7f3bc019a0e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:54.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.796+0000 7f3bc5c50700 1 -- 172.21.15.116:0/4051042559 >> 172.21.15.116:0/4051042559 conn(0x7f3bc0100040 msgr2=0x7f3bc0100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:54.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.796+0000 7f3bc5c50700 1 -- 172.21.15.116:0/4051042559 shutdown_connections 2024-03-04T15:53:54.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:53.796+0000 7f3bc5c50700 1 -- 172.21.15.116:0/4051042559 wait complete. 2024-03-04T15:53:54.353 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-04T15:53:54.367 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:53:54.367 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mgr... 2024-03-04T15:53:54.367 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-04T15:53:54.368 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-04T15:53:54.368 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:8443 ... 2024-03-04T15:53:54.618 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4960653c-da3f-11ee-95c6-87774f69a715@mgr.smithi116.jwdhop 2024-03-04T15:53:54.618 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4960653c-da3f-11ee-95c6-87774f69a715@mgr.smithi116.jwdhop.service: Unit ceph-4960653c-da3f-11ee-95c6-87774f69a715@mgr.smithi116.jwdhop.service not loaded. 2024-03-04T15:53:54.772 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4960653c-da3f-11ee-95c6-87774f69a715.target.wants/ceph-4960653c-da3f-11ee-95c6-87774f69a715@mgr.smithi116.jwdhop.service → /etc/systemd/system/ceph-4960653c-da3f-11ee-95c6-87774f69a715@.service. 2024-03-04T15:53:54.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:54 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/4051042559' entity='client.admin' 2024-03-04T15:53:55.285 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:53:55.285 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-04T15:53:55.292 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:53:55.292 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T15:53:55.292 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-04T15:53:55.292 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T15:53:55.300 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:53:55.300 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-04T15:53:55.307 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:53:55.307 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T15:53:55.307 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-04T15:53:55.307 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-03-04T15:53:55.307 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr to start... 2024-03-04T15:53:55.307 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr... 2024-03-04T15:53:55.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:55 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1947382913' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:53:56.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "4960653c-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:53:56.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "smithi116" 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:53:56.409 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:53:56.410 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:53:56.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:53:48.465875+0000", 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.712+0000 7f3f116e9700 1 Processor -- start 2024-03-04T15:53:56.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.712+0000 7f3f116e9700 1 -- start start 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.712+0000 7f3f116e9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.712+0000 7f3f116e9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c106e10 con 0x7f3f0c106420 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.713+0000 7f3f0affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.713+0000 7f3f0affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49256/0 (socket says 172.21.15.116:49256) 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.713+0000 7f3f0affd700 1 -- 172.21.15.116:0/3106436140 learned_addr learned my addr 172.21.15.116:0/3106436140 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.713+0000 7f3f0affd700 1 -- 172.21.15.116:0/3106436140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c107620 con 0x7f3f0c106420 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.713+0000 7f3f0affd700 1 --2- 172.21.15.116:0/3106436140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c106840 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3f000098a0 tx=0x7f3f00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec23fe182d717f4b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.713+0000 7f3f09ffb700 1 -- 172.21.15.116:0/3106436140 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f00004180 con 0x7f3f0c106420 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.713+0000 7f3f09ffb700 1 -- 172.21.15.116:0/3106436140 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3f000042e0 con 0x7f3f0c106420 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.713+0000 7f3f09ffb700 1 -- 172.21.15.116:0/3106436140 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f00011a50 con 0x7f3f0c106420 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.714+0000 7f3f116e9700 1 -- 172.21.15.116:0/3106436140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 msgr2=0x7f3f0c106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.714+0000 7f3f116e9700 1 --2- 172.21.15.116:0/3106436140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c106840 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3f000098a0 tx=0x7f3f00009bb0 comp rx=0 tx=0).stop 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.714+0000 7f3f116e9700 1 -- 172.21.15.116:0/3106436140 shutdown_connections 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.714+0000 7f3f116e9700 1 --2- 172.21.15.116:0/3106436140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c106840 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.714+0000 7f3f116e9700 1 -- 172.21.15.116:0/3106436140 >> 172.21.15.116:0/3106436140 conn(0x7f3f0c101a00 msgr2=0x7f3f0c103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.714+0000 7f3f116e9700 1 -- 172.21.15.116:0/3106436140 shutdown_connections 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.714+0000 7f3f116e9700 1 -- 172.21.15.116:0/3106436140 wait complete. 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.715+0000 7f3f116e9700 1 Processor -- start 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.715+0000 7f3f116e9700 1 -- start start 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.715+0000 7f3f116e9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c19a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.715+0000 7f3f116e9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c19a680 con 0x7f3f0c106420 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.715+0000 7f3f0affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c19a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.715+0000 7f3f0affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c19a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49258/0 (socket says 172.21.15.116:49258) 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.715+0000 7f3f0affd700 1 -- 172.21.15.116:0/1947382913 learned_addr learned my addr 172.21.15.116:0/1947382913 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:56.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.715+0000 7f3f0affd700 1 -- 172.21.15.116:0/1947382913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f00009550 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.716+0000 7f3f0affd700 1 --2- 172.21.15.116:0/1947382913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c19a140 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3f00003de0 tx=0x7f3f00011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.716+0000 7f3ef3fff700 1 -- 172.21.15.116:0/1947382913 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f00003930 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.716+0000 7f3ef3fff700 1 -- 172.21.15.116:0/1947382913 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3f00003a90 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.716+0000 7f3f116e9700 1 -- 172.21.15.116:0/1947382913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c19a880 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.716+0000 7f3ef3fff700 1 -- 172.21.15.116:0/1947382913 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f000196c0 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.716+0000 7f3f116e9700 1 -- 172.21.15.116:0/1947382913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c19ad20 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.716+0000 7f3ef3fff700 1 -- 172.21.15.116:0/1947382913 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3f00019ca0 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.716+0000 7f3ef3fff700 1 -- 172.21.15.116:0/1947382913 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3f0002c3a0 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.717+0000 7f3f116e9700 1 -- 172.21.15.116:0/1947382913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c0623c0 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.719+0000 7f3ef3fff700 1 -- 172.21.15.116:0/1947382913 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f3f0001abc0 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.790+0000 7f3f116e9700 1 -- 172.21.15.116:0/1947382913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3f0c10a850 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.791+0000 7f3ef3fff700 1 -- 172.21.15.116:0/1947382913 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f3f0001a430 con 0x7f3f0c106420 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.792+0000 7f3f116e9700 1 -- 172.21.15.116:0/1947382913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 msgr2=0x7f3f0c19a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:56.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.792+0000 7f3f116e9700 1 --2- 172.21.15.116:0/1947382913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c19a140 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3f00003de0 tx=0x7f3f00011ab0 comp rx=0 tx=0).stop 2024-03-04T15:53:56.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.792+0000 7f3f116e9700 1 -- 172.21.15.116:0/1947382913 shutdown_connections 2024-03-04T15:53:56.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.792+0000 7f3f116e9700 1 --2- 172.21.15.116:0/1947382913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f0c106420 0x7f3f0c19a140 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:56.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.792+0000 7f3f116e9700 1 -- 172.21.15.116:0/1947382913 >> 172.21.15.116:0/1947382913 conn(0x7f3f0c101a00 msgr2=0x7f3f0c1030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:56.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.792+0000 7f3f116e9700 1 -- 172.21.15.116:0/1947382913 shutdown_connections 2024-03-04T15:53:56.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:55.792+0000 7f3f116e9700 1 -- 172.21.15.116:0/1947382913 wait complete. 2024-03-04T15:53:56.416 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (1/15)... 2024-03-04T15:53:59.434 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:53:59 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3351089566' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "4960653c-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:53:59.472 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "smithi116" 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:53:59.473 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:53:59.474 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:53:59.475 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:53:48.465875+0000", 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.818+0000 7fa365a10700 1 Processor -- start 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.818+0000 7fa365a10700 1 -- start start 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.818+0000 7fa365a10700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa360106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.818+0000 7fa365a10700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa360106e10 con 0x7fa360106420 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.818+0000 7fa35effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa360106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:59.476 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.818+0000 7fa35effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa360106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49264/0 (socket says 172.21.15.116:49264) 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.818+0000 7fa35effd700 1 -- 172.21.15.116:0/1332078982 learned_addr learned my addr 172.21.15.116:0/1332078982 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.818+0000 7fa35effd700 1 -- 172.21.15.116:0/1332078982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa360107620 con 0x7fa360106420 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.819+0000 7fa35effd700 1 --2- 172.21.15.116:0/1332078982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa360106840 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa3500098a0 tx=0x7fa350009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f7bc445ef64125 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.819+0000 7fa35dffb700 1 -- 172.21.15.116:0/1332078982 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa350004180 con 0x7fa360106420 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.819+0000 7fa35dffb700 1 -- 172.21.15.116:0/1332078982 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa3500042e0 con 0x7fa360106420 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.819+0000 7fa35dffb700 1 -- 172.21.15.116:0/1332078982 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa350011a50 con 0x7fa360106420 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.819+0000 7fa365a10700 1 -- 172.21.15.116:0/1332078982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 msgr2=0x7fa360106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.819+0000 7fa365a10700 1 --2- 172.21.15.116:0/1332078982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa360106840 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa3500098a0 tx=0x7fa350009bb0 comp rx=0 tx=0).stop 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.820+0000 7fa365a10700 1 -- 172.21.15.116:0/1332078982 shutdown_connections 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.820+0000 7fa365a10700 1 --2- 172.21.15.116:0/1332078982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa360106840 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.820+0000 7fa365a10700 1 -- 172.21.15.116:0/1332078982 >> 172.21.15.116:0/1332078982 conn(0x7fa360101a00 msgr2=0x7fa360103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.820+0000 7fa365a10700 1 -- 172.21.15.116:0/1332078982 shutdown_connections 2024-03-04T15:53:59.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.820+0000 7fa365a10700 1 -- 172.21.15.116:0/1332078982 wait complete. 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.820+0000 7fa365a10700 1 Processor -- start 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.820+0000 7fa365a10700 1 -- start start 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa365a10700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa36019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa365a10700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36019a5f0 con 0x7fa360106420 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa35effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa36019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa35effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa36019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49276/0 (socket says 172.21.15.116:49276) 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa35effd700 1 -- 172.21.15.116:0/3351089566 learned_addr learned my addr 172.21.15.116:0/3351089566 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa35effd700 1 -- 172.21.15.116:0/3351089566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa350009550 con 0x7fa360106420 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa35effd700 1 --2- 172.21.15.116:0/3351089566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa36019a0b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa350006af0 tx=0x7fa350011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa364a0e700 1 -- 172.21.15.116:0/3351089566 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa350003b90 con 0x7fa360106420 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa365a10700 1 -- 172.21.15.116:0/3351089566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36019a7f0 con 0x7fa360106420 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa364a0e700 1 -- 172.21.15.116:0/3351089566 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa35001a8a0 con 0x7fa360106420 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa365a10700 1 -- 172.21.15.116:0/3351089566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36019ac90 con 0x7fa360106420 2024-03-04T15:53:59.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.821+0000 7fa364a0e700 1 -- 172.21.15.116:0/3351089566 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa350019880 con 0x7fa360106420 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.822+0000 7fa364a0e700 1 -- 172.21.15.116:0/3351089566 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa35002d440 con 0x7fa360106420 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.822+0000 7fa364a0e700 1 -- 172.21.15.116:0/3351089566 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa350024870 con 0x7fa360106420 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.822+0000 7fa365a10700 1 -- 172.21.15.116:0/3351089566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c005320 con 0x7fa360106420 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.826+0000 7fa364a0e700 1 -- 172.21.15.116:0/3351089566 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa35001aa10 con 0x7fa360106420 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.898+0000 7fa365a10700 1 -- 172.21.15.116:0/3351089566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa34c005190 con 0x7fa360106420 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.898+0000 7fa364a0e700 1 -- 172.21.15.116:0/3351089566 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fa35001abb0 con 0x7fa360106420 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.899+0000 7fa365a10700 1 -- 172.21.15.116:0/3351089566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 msgr2=0x7fa36019a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.899+0000 7fa365a10700 1 --2- 172.21.15.116:0/3351089566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa36019a0b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa350006af0 tx=0x7fa350011a90 comp rx=0 tx=0).stop 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.899+0000 7fa365a10700 1 -- 172.21.15.116:0/3351089566 shutdown_connections 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.899+0000 7fa365a10700 1 --2- 172.21.15.116:0/3351089566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa360106420 0x7fa36019a0b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.899+0000 7fa365a10700 1 -- 172.21.15.116:0/3351089566 >> 172.21.15.116:0/3351089566 conn(0x7fa360101a00 msgr2=0x7fa3601027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.900+0000 7fa365a10700 1 -- 172.21.15.116:0/3351089566 shutdown_connections 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:53:58.900+0000 7fa365a10700 1 -- 172.21.15.116:0/3351089566 wait complete. 2024-03-04T15:53:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (2/15)... 2024-03-04T15:54:02.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: Activating manager daemon smithi116.jwdhop 2024-03-04T15:54:02.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: mgrmap e2: smithi116.jwdhop(active, starting, since 0.00321208s) 2024-03-04T15:54:02.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: from='mgr.14100 172.21.15.116:0/3997135866' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:54:02.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: from='mgr.14100 172.21.15.116:0/3997135866' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:54:02.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: from='mgr.14100 172.21.15.116:0/3997135866' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:54:02.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: from='mgr.14100 172.21.15.116:0/3997135866' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T15:54:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: from='mgr.14100 172.21.15.116:0/3997135866' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi116.jwdhop", "id": "smithi116.jwdhop"}]: dispatch 2024-03-04T15:54:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: Manager daemon smithi116.jwdhop is now available 2024-03-04T15:54:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: from='mgr.14100 172.21.15.116:0/3997135866' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:54:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: from='mgr.14100 172.21.15.116:0/3997135866' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/trash_purge_schedule"}]: dispatch 2024-03-04T15:54:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: from='mgr.14100 172.21.15.116:0/3997135866' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: from='mgr.14100 172.21.15.116:0/3997135866' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:01 smithi116 ceph-mon[29120]: from='mgr.14100 172.21.15.116:0/3997135866' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:02.502 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:54:02.502 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:54:02.502 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "4960653c-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:54:02.502 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:54:02.502 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:54:02.502 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:54:02.502 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:54:02.502 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:02.502 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:54:02.502 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "smithi116" 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:54:02.503 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:54:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:54:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:53:48.465875+0000", 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.851+0000 7fbc0088a700 1 Processor -- start 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.851+0000 7fbc0088a700 1 -- start start 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.851+0000 7fbc0088a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc105770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.851+0000 7fbc0088a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbfc105d40 con 0x7fbbfc105350 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.852+0000 7fbbfa59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc105770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.852+0000 7fbbfa59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc105770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49296/0 (socket says 172.21.15.116:49296) 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.852+0000 7fbbfa59c700 1 -- 172.21.15.116:0/3381930144 learned_addr learned my addr 172.21.15.116:0/3381930144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.852+0000 7fbbfa59c700 1 -- 172.21.15.116:0/3381930144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbfc106550 con 0x7fbbfc105350 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.852+0000 7fbbfa59c700 1 --2- 172.21.15.116:0/3381930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc105770 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0098a0 tx=0x7fbbec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143d80c3d2abdb12 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.852+0000 7fbbf959a700 1 -- 172.21.15.116:0/3381930144 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbec004180 con 0x7fbbfc105350 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.852+0000 7fbbf959a700 1 -- 172.21.15.116:0/3381930144 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbbec0042e0 con 0x7fbbfc105350 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.852+0000 7fbbf959a700 1 -- 172.21.15.116:0/3381930144 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbec011a50 con 0x7fbbfc105350 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.853+0000 7fbc0088a700 1 -- 172.21.15.116:0/3381930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 msgr2=0x7fbbfc105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.853+0000 7fbc0088a700 1 --2- 172.21.15.116:0/3381930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc105770 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0098a0 tx=0x7fbbec009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.853+0000 7fbc0088a700 1 -- 172.21.15.116:0/3381930144 shutdown_connections 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.853+0000 7fbc0088a700 1 --2- 172.21.15.116:0/3381930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc105770 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.853+0000 7fbc0088a700 1 -- 172.21.15.116:0/3381930144 >> 172.21.15.116:0/3381930144 conn(0x7fbbfc1008d0 msgr2=0x7fbbfc102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.853+0000 7fbc0088a700 1 -- 172.21.15.116:0/3381930144 shutdown_connections 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.853+0000 7fbc0088a700 1 -- 172.21.15.116:0/3381930144 wait complete. 2024-03-04T15:54:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.854+0000 7fbc0088a700 1 Processor -- start 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.854+0000 7fbc0088a700 1 -- start start 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.854+0000 7fbc0088a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc190520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.854+0000 7fbc0088a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbfc190a60 con 0x7fbbfc105350 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.854+0000 7fbbfa59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc190520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.854+0000 7fbbfa59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc190520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49298/0 (socket says 172.21.15.116:49298) 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.854+0000 7fbbfa59c700 1 -- 172.21.15.116:0/1877513107 learned_addr learned my addr 172.21.15.116:0/1877513107 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.854+0000 7fbbfa59c700 1 -- 172.21.15.116:0/1877513107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec009550 con 0x7fbbfc105350 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.855+0000 7fbbfa59c700 1 --2- 172.21.15.116:0/1877513107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc190520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbbec003de0 tx=0x7fbbec011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.855+0000 7fbbeb7fe700 1 -- 172.21.15.116:0/1877513107 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbec003930 con 0x7fbbfc105350 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.855+0000 7fbc0088a700 1 -- 172.21.15.116:0/1877513107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbfc18eb70 con 0x7fbbfc105350 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.855+0000 7fbbeb7fe700 1 -- 172.21.15.116:0/1877513107 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbbec003a90 con 0x7fbbfc105350 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.855+0000 7fbbeb7fe700 1 -- 172.21.15.116:0/1877513107 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbec0196c0 con 0x7fbbfc105350 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.855+0000 7fbc0088a700 1 -- 172.21.15.116:0/1877513107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbfc18f010 con 0x7fbbfc105350 2024-03-04T15:54:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.855+0000 7fbbeb7fe700 1 -- 172.21.15.116:0/1877513107 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 2) v1 ==== 45039+0+0 (secure 0 0 0) 0x7fbbec019980 con 0x7fbbfc105350 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.856+0000 7fbbeb7fe700 1 -- 172.21.15.116:0/1877513107 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbbec04cc00 con 0x7fbbfc105350 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.856+0000 7fbc0088a700 1 -- 172.21.15.116:0/1877513107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbfc0623c0 con 0x7fbbfc105350 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.858+0000 7fbbeb7fe700 1 -- 172.21.15.116:0/1877513107 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fbbec019c30 con 0x7fbbfc105350 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.931+0000 7fbc0088a700 1 -- 172.21.15.116:0/1877513107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbbfc109780 con 0x7fbbfc105350 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.931+0000 7fbbeb7fe700 1 -- 172.21.15.116:0/1877513107 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fbbec010380 con 0x7fbbfc105350 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.932+0000 7fbc0088a700 1 -- 172.21.15.116:0/1877513107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 msgr2=0x7fbbfc190520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.932+0000 7fbc0088a700 1 --2- 172.21.15.116:0/1877513107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc190520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbbec003de0 tx=0x7fbbec011ab0 comp rx=0 tx=0).stop 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.932+0000 7fbc0088a700 1 -- 172.21.15.116:0/1877513107 shutdown_connections 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.932+0000 7fbc0088a700 1 --2- 172.21.15.116:0/1877513107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbfc105350 0x7fbbfc190520 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.932+0000 7fbc0088a700 1 -- 172.21.15.116:0/1877513107 >> 172.21.15.116:0/1877513107 conn(0x7fbbfc1008d0 msgr2=0x7fbbfc0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.932+0000 7fbc0088a700 1 -- 172.21.15.116:0/1877513107 shutdown_connections 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:01.932+0000 7fbc0088a700 1 -- 172.21.15.116:0/1877513107 wait complete. 2024-03-04T15:54:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (3/15)... 2024-03-04T15:54:03.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:02 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1877513107' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:54:03.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:02 smithi116 ceph-mon[29120]: mgrmap e3: smithi116.jwdhop(active, since 1.00703s) 2024-03-04T15:54:05.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:05 smithi116 ceph-mon[29120]: mgrmap e4: smithi116.jwdhop(active, since 2s) 2024-03-04T15:54:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:09 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/717357350' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:54:12.568 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "4960653c-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:54:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "smithi116" 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:54:12.570 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:54:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:12.572 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:53:48.465875+0000", 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.204+0000 7f93a9909700 1 Processor -- start 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.204+0000 7f93a9909700 1 -- start start 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.204+0000 7f93a9909700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a41092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.204+0000 7f93a9909700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a4079e80 con 0x7f93a4106f00 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.204+0000 7f93a2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a41092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.204+0000 7f93a2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a41092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36584/0 (socket says 172.21.15.116:36584) 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.204+0000 7f93a2ffd700 1 -- 172.21.15.116:0/3052202187 learned_addr learned my addr 172.21.15.116:0/3052202187 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.204+0000 7f93a2ffd700 1 -- 172.21.15.116:0/3052202187 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a4109830 con 0x7f93a4106f00 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a2ffd700 1 --2- 172.21.15.116:0/3052202187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a41092f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f93940098a0 tx=0x7f9394009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34467cd0ae3f7eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a1ffb700 1 -- 172.21.15.116:0/3052202187 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9394004180 con 0x7f93a4106f00 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a1ffb700 1 -- 172.21.15.116:0/3052202187 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f93940042e0 con 0x7f93a4106f00 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a1ffb700 1 -- 172.21.15.116:0/3052202187 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9394011a50 con 0x7f93a4106f00 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a9909700 1 -- 172.21.15.116:0/3052202187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 msgr2=0x7f93a41092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a9909700 1 --2- 172.21.15.116:0/3052202187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a41092f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f93940098a0 tx=0x7f9394009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a9909700 1 -- 172.21.15.116:0/3052202187 shutdown_connections 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a9909700 1 --2- 172.21.15.116:0/3052202187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a41092f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a9909700 1 -- 172.21.15.116:0/3052202187 >> 172.21.15.116:0/3052202187 conn(0x7f93a41008d0 msgr2=0x7f93a4102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a9909700 1 -- 172.21.15.116:0/3052202187 shutdown_connections 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.205+0000 7f93a9909700 1 -- 172.21.15.116:0/3052202187 wait complete. 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.206+0000 7f93a9909700 1 Processor -- start 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.206+0000 7f93a9909700 1 -- start start 2024-03-04T15:54:12.574 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.206+0000 7f93a9909700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.206+0000 7f93a9909700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a419a610 con 0x7f93a4106f00 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.207+0000 7f93a2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.207+0000 7f93a2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36598/0 (socket says 172.21.15.116:36598) 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.207+0000 7f93a2ffd700 1 -- 172.21.15.116:0/717357350 learned_addr learned my addr 172.21.15.116:0/717357350 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.207+0000 7f93a2ffd700 1 -- 172.21.15.116:0/717357350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394009550 con 0x7f93a4106f00 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.207+0000 7f93a2ffd700 1 --2- 172.21.15.116:0/717357350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a419a0d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9394003de0 tx=0x7f9394011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.207+0000 7f93a8907700 1 -- 172.21.15.116:0/717357350 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9394003930 con 0x7f93a4106f00 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.207+0000 7f93a8907700 1 -- 172.21.15.116:0/717357350 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9394003a90 con 0x7f93a4106f00 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.207+0000 7f93a9909700 1 -- 172.21.15.116:0/717357350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a419a810 con 0x7f93a4106f00 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.207+0000 7f93a8907700 1 -- 172.21.15.116:0/717357350 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9394019730 con 0x7f93a4106f00 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.207+0000 7f93a9909700 1 -- 172.21.15.116:0/717357350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a419acb0 con 0x7f93a4106f00 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.208+0000 7f93a8907700 1 -- 172.21.15.116:0/717357350 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f9394019960 con 0x7f93a4106f00 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.208+0000 7f93a9909700 1 -- 172.21.15.116:0/717357350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a410a580 con 0x7f93a4106f00 2024-03-04T15:54:12.575 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.208+0000 7f93a8907700 1 --2- 172.21.15.116:0/717357350 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f938c038320 0x7f938c03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.209+0000 7f93a8907700 1 -- 172.21.15.116:0/717357350 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f939404bf90 con 0x7f93a4106f00 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.209+0000 7f93a27fc700 1 --2- 172.21.15.116:0/717357350 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f938c038320 0x7f938c03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.209+0000 7f93a27fc700 1 --2- 172.21.15.116:0/717357350 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f938c038320 0x7f938c03a7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9398006fd0 tx=0x7f9398006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.214+0000 7f93a8907700 1 -- 172.21.15.116:0/717357350 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9394019c10 con 0x7f93a4106f00 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.489+0000 7f93a9909700 1 -- 172.21.15.116:0/717357350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f93a410a7a0 con 0x7f93a4106f00 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.489+0000 7f93a8907700 1 -- 172.21.15.116:0/717357350 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f939404bdb0 con 0x7f93a4106f00 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.492+0000 7f93a9909700 1 -- 172.21.15.116:0/717357350 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f938c038320 msgr2=0x7f938c03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.492+0000 7f93a9909700 1 --2- 172.21.15.116:0/717357350 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f938c038320 0x7f938c03a7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9398006fd0 tx=0x7f9398006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.492+0000 7f93a9909700 1 -- 172.21.15.116:0/717357350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 msgr2=0x7f93a419a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.492+0000 7f93a9909700 1 --2- 172.21.15.116:0/717357350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a419a0d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9394003de0 tx=0x7f9394011ab0 comp rx=0 tx=0).stop 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.492+0000 7f93a9909700 1 -- 172.21.15.116:0/717357350 shutdown_connections 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.492+0000 7f93a9909700 1 --2- 172.21.15.116:0/717357350 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f938c038320 0x7f938c03a7e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.492+0000 7f93a9909700 1 --2- 172.21.15.116:0/717357350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93a4106f00 0x7f93a419a0d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:12.576 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.492+0000 7f93a9909700 1 -- 172.21.15.116:0/717357350 >> 172.21.15.116:0/717357350 conn(0x7f93a41008d0 msgr2=0x7f93a4101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:12.577 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.492+0000 7f93a9909700 1 -- 172.21.15.116:0/717357350 shutdown_connections 2024-03-04T15:54:12.577 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:08.493+0000 7f93a9909700 1 -- 172.21.15.116:0/717357350 wait complete. 2024-03-04T15:54:12.577 INFO:teuthology.orchestra.run.smithi116.stdout:mgr is available 2024-03-04T15:54:13.434 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:13 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3381344168' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T15:54:13.719 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:54:13.719 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T15:54:13.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.958+0000 7f16b4a4b700 1 Processor -- start 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.958+0000 7f16b4a4b700 1 -- start start 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.958+0000 7f16b4a4b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0105750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.958+0000 7f16b4a4b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b0105d20 con 0x7f16b0105330 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.958+0000 7f16ae59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.958+0000 7f16ae59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36608/0 (socket says 172.21.15.116:36608) 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.958+0000 7f16ae59c700 1 -- 172.21.15.116:0/3124396466 learned_addr learned my addr 172.21.15.116:0/3124396466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.959+0000 7f16ae59c700 1 -- 172.21.15.116:0/3124396466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b0106530 con 0x7f16b0105330 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.959+0000 7f16ae59c700 1 --2- 172.21.15.116:0/3124396466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0105750 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f16a00098a0 tx=0x7f16a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f3d0dda65fc99f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.959+0000 7f16ad59a700 1 -- 172.21.15.116:0/3124396466 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a0004180 con 0x7f16b0105330 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.959+0000 7f16ad59a700 1 -- 172.21.15.116:0/3124396466 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f16a00042e0 con 0x7f16b0105330 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.959+0000 7f16ad59a700 1 -- 172.21.15.116:0/3124396466 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a0011a50 con 0x7f16b0105330 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.960+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3124396466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 msgr2=0x7f16b0105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.960+0000 7f16b4a4b700 1 --2- 172.21.15.116:0/3124396466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0105750 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f16a00098a0 tx=0x7f16a0009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.960+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3124396466 shutdown_connections 2024-03-04T15:54:13.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.960+0000 7f16b4a4b700 1 --2- 172.21.15.116:0/3124396466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0105750 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.960+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3124396466 >> 172.21.15.116:0/3124396466 conn(0x7f16b01008b0 msgr2=0x7f16b0102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.960+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3124396466 shutdown_connections 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.960+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3124396466 wait complete. 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.960+0000 7f16b4a4b700 1 Processor -- start 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f16b4a4b700 1 -- start start 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f16b4a4b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0079b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f16b4a4b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b019ce80 con 0x7f16b0105330 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f16ae59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0079b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f16ae59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0079b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36616/0 (socket says 172.21.15.116:36616) 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f16ae59c700 1 -- 172.21.15.116:0/3381344168 learned_addr learned my addr 172.21.15.116:0/3381344168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f16ae59c700 1 -- 172.21.15.116:0/3381344168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a0009550 con 0x7f16b0105330 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f16ae59c700 1 --2- 172.21.15.116:0/3381344168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0079b40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f16a0003de0 tx=0x7f16a0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f169f7fe700 1 -- 172.21.15.116:0/3381344168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a0003930 con 0x7f16b0105330 2024-03-04T15:54:13.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f169f7fe700 1 -- 172.21.15.116:0/3381344168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f16a0003a90 con 0x7f16b0105330 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3381344168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b019cfc0 con 0x7f16b0105330 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f169f7fe700 1 -- 172.21.15.116:0/3381344168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a00196c0 con 0x7f16b0105330 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.961+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3381344168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b019d400 con 0x7f16b0105330 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.962+0000 7f169f7fe700 1 -- 172.21.15.116:0/3381344168 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f16a001a430 con 0x7f16b0105330 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.962+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3381344168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b00623c0 con 0x7f16b0105330 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.963+0000 7f169f7fe700 1 --2- 172.21.15.116:0/3381344168 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f16980382d0 0x7f169803a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.963+0000 7f169f7fe700 1 -- 172.21.15.116:0/3381344168 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f16a004bfb0 con 0x7f16b0105330 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.963+0000 7f16add9b700 1 --2- 172.21.15.116:0/3381344168 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f16980382d0 0x7f169803a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.963+0000 7f16add9b700 1 --2- 172.21.15.116:0/3381344168 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f16980382d0 0x7f169803a790 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f16a4006fd0 tx=0x7f16a4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:12.967+0000 7f169f7fe700 1 -- 172.21.15.116:0/3381344168 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f16a0010560 con 0x7f16b0105330 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.151+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3381344168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f16b019dcb0 con 0x7f16b0105330 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.151+0000 7f169f7fe700 1 -- 172.21.15.116:0/3381344168 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f16a00281c0 con 0x7f16b0105330 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.153+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3381344168 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f16980382d0 msgr2=0x7f169803a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.153+0000 7f16b4a4b700 1 --2- 172.21.15.116:0/3381344168 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f16980382d0 0x7f169803a790 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f16a4006fd0 tx=0x7f16a4006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.153+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3381344168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 msgr2=0x7f16b0079b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:13.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.153+0000 7f16b4a4b700 1 --2- 172.21.15.116:0/3381344168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0079b40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f16a0003de0 tx=0x7f16a0011ab0 comp rx=0 tx=0).stop 2024-03-04T15:54:13.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.154+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3381344168 shutdown_connections 2024-03-04T15:54:13.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.154+0000 7f16b4a4b700 1 --2- 172.21.15.116:0/3381344168 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7f16980382d0 0x7f169803a790 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:13.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.154+0000 7f16b4a4b700 1 --2- 172.21.15.116:0/3381344168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16b0105330 0x7f16b0079b40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:13.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.154+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3381344168 >> 172.21.15.116:0/3381344168 conn(0x7f16b01008b0 msgr2=0x7f16b0101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:13.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.154+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3381344168 shutdown_connections 2024-03-04T15:54:13.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:13.154+0000 7f16b4a4b700 1 -- 172.21.15.116:0/3381344168 wait complete. 2024-03-04T15:54:13.724 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling cephadm module... 2024-03-04T15:54:14.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:14 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2076687407' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.159+0000 7fae9715e700 1 Processor -- start 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.159+0000 7fae9715e700 1 -- start start 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.159+0000 7fae9715e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9007a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.159+0000 7fae9715e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9007a8e0 con 0x7fae9007beb0 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.159+0000 7fae94efa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9007a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.159+0000 7fae94efa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9007a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36624/0 (socket says 172.21.15.116:36624) 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.159+0000 7fae94efa700 1 -- 172.21.15.116:0/101657804 learned_addr learned my addr 172.21.15.116:0/101657804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.159+0000 7fae94efa700 1 -- 172.21.15.116:0/101657804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9007aa20 con 0x7fae9007beb0 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae94efa700 1 --2- 172.21.15.116:0/101657804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9007a310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fae840098a0 tx=0x7fae84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351ae27e9bacedd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae8f7fe700 1 -- 172.21.15.116:0/101657804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae84004180 con 0x7fae9007beb0 2024-03-04T15:54:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae8f7fe700 1 -- 172.21.15.116:0/101657804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fae840042e0 con 0x7fae9007beb0 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae8f7fe700 1 -- 172.21.15.116:0/101657804 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae84011a50 con 0x7fae9007beb0 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae9715e700 1 -- 172.21.15.116:0/101657804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 msgr2=0x7fae9007a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae9715e700 1 --2- 172.21.15.116:0/101657804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9007a310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fae840098a0 tx=0x7fae84009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae9715e700 1 -- 172.21.15.116:0/101657804 shutdown_connections 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae9715e700 1 --2- 172.21.15.116:0/101657804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9007a310 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae9715e700 1 -- 172.21.15.116:0/101657804 >> 172.21.15.116:0/101657804 conn(0x7fae90101a00 msgr2=0x7fae90103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae9715e700 1 -- 172.21.15.116:0/101657804 shutdown_connections 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.160+0000 7fae9715e700 1 -- 172.21.15.116:0/101657804 wait complete. 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.161+0000 7fae9715e700 1 Processor -- start 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.161+0000 7fae9715e700 1 -- start start 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.161+0000 7fae9715e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9019c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.161+0000 7fae9715e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9019c840 con 0x7fae9007beb0 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.161+0000 7fae94efa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9019c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.161+0000 7fae94efa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9019c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36634/0 (socket says 172.21.15.116:36634) 2024-03-04T15:54:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.161+0000 7fae94efa700 1 -- 172.21.15.116:0/2076687407 learned_addr learned my addr 172.21.15.116:0/2076687407 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.161+0000 7fae94efa700 1 -- 172.21.15.116:0/2076687407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae84009550 con 0x7fae9007beb0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.161+0000 7fae94efa700 1 --2- 172.21.15.116:0/2076687407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9019c300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fae84006af0 tx=0x7fae84011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.162+0000 7fae8dffb700 1 -- 172.21.15.116:0/2076687407 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae84003b90 con 0x7fae9007beb0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.162+0000 7fae8dffb700 1 -- 172.21.15.116:0/2076687407 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fae8401a8a0 con 0x7fae9007beb0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.162+0000 7fae8dffb700 1 -- 172.21.15.116:0/2076687407 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae84019880 con 0x7fae9007beb0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.162+0000 7fae9715e700 1 -- 172.21.15.116:0/2076687407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9019ca40 con 0x7fae9007beb0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.162+0000 7fae9715e700 1 -- 172.21.15.116:0/2076687407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9019cee0 con 0x7fae9007beb0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.163+0000 7fae8dffb700 1 -- 172.21.15.116:0/2076687407 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fae8401aa10 con 0x7fae9007beb0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.163+0000 7fae9715e700 1 -- 172.21.15.116:0/2076687407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9010c780 con 0x7fae9007beb0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.163+0000 7fae8dffb700 1 --2- 172.21.15.116:0/2076687407 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fae800382e0 0x7fae8003a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.163+0000 7fae8dffb700 1 -- 172.21.15.116:0/2076687407 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fae8404c620 con 0x7fae9007beb0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.163+0000 7fae8ffff700 1 --2- 172.21.15.116:0/2076687407 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fae800382e0 0x7fae8003a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.163+0000 7fae8ffff700 1 --2- 172.21.15.116:0/2076687407 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fae800382e0 0x7fae8003a7a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fae7c006fd0 tx=0x7fae7c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.168+0000 7fae8dffb700 1 -- 172.21.15.116:0/2076687407 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fae8401a430 con 0x7fae9007beb0 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.408+0000 7fae9715e700 1 -- 172.21.15.116:0/2076687407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fae9019fd20 con 0x7fae9007beb0 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.461+0000 7fae8dffb700 1 -- 172.21.15.116:0/2076687407 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fae8400fda0 con 0x7fae9007beb0 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.462+0000 7fae8dffb700 1 -- 172.21.15.116:0/2076687407 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fae84018350 con 0x7fae9007beb0 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.465+0000 7fae9715e700 1 -- 172.21.15.116:0/2076687407 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fae800382e0 msgr2=0x7fae8003a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.465+0000 7fae9715e700 1 --2- 172.21.15.116:0/2076687407 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fae800382e0 0x7fae8003a7a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fae7c006fd0 tx=0x7fae7c006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.465+0000 7fae9715e700 1 -- 172.21.15.116:0/2076687407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 msgr2=0x7fae9019c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.465+0000 7fae9715e700 1 --2- 172.21.15.116:0/2076687407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9019c300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fae84006af0 tx=0x7fae84011a90 comp rx=0 tx=0).stop 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.465+0000 7fae9715e700 1 -- 172.21.15.116:0/2076687407 shutdown_connections 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.465+0000 7fae9715e700 1 --2- 172.21.15.116:0/2076687407 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fae800382e0 0x7fae8003a7a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.465+0000 7fae9715e700 1 --2- 172.21.15.116:0/2076687407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae9007beb0 0x7fae9019c300 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.465+0000 7fae9715e700 1 -- 172.21.15.116:0/2076687407 >> 172.21.15.116:0/2076687407 conn(0x7fae90101a00 msgr2=0x7fae901030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.465+0000 7fae9715e700 1 -- 172.21.15.116:0/2076687407 shutdown_connections 2024-03-04T15:54:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:14.465+0000 7fae9715e700 1 -- 172.21.15.116:0/2076687407 wait complete. 2024-03-04T15:54:15.700 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:15 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2076687407' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-04T15:54:15.700 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:15 smithi116 ceph-mon[29120]: mgrmap e5: smithi116.jwdhop(active, since 12s) 2024-03-04T15:54:16.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:16 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/722103145' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T15:54:17.074 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "smithi116.jwdhop", 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.440+0000 7fa4af05a700 1 Processor -- start 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.440+0000 7fa4af05a700 1 -- start start 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.440+0000 7fa4af05a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a8105c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.440+0000 7fa4af05a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8106230 con 0x7fa4a8105840 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.440+0000 7fa4acdf6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a8105c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.440+0000 7fa4acdf6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a8105c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41810/0 (socket says 172.21.15.116:41810) 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.440+0000 7fa4acdf6700 1 -- 172.21.15.116:0/908521212 learned_addr learned my addr 172.21.15.116:0/908521212 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.441+0000 7fa4acdf6700 1 -- 172.21.15.116:0/908521212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a8106560 con 0x7fa4a8105840 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.441+0000 7fa4acdf6700 1 --2- 172.21.15.116:0/908521212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a8105c60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0098a0 tx=0x7fa49c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490946e15bf53333 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.441+0000 7fa4a77fe700 1 -- 172.21.15.116:0/908521212 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa49c004180 con 0x7fa4a8105840 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.441+0000 7fa4a77fe700 1 -- 172.21.15.116:0/908521212 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa49c0042e0 con 0x7fa4a8105840 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.441+0000 7fa4a77fe700 1 -- 172.21.15.116:0/908521212 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa49c011a50 con 0x7fa4a8105840 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.442+0000 7fa4af05a700 1 -- 172.21.15.116:0/908521212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 msgr2=0x7fa4a8105c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.442+0000 7fa4af05a700 1 --2- 172.21.15.116:0/908521212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a8105c60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0098a0 tx=0x7fa49c009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.442+0000 7fa4af05a700 1 -- 172.21.15.116:0/908521212 shutdown_connections 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.442+0000 7fa4af05a700 1 --2- 172.21.15.116:0/908521212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a8105c60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.442+0000 7fa4af05a700 1 -- 172.21.15.116:0/908521212 >> 172.21.15.116:0/908521212 conn(0x7fa4a8100040 msgr2=0x7fa4a81024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.442+0000 7fa4af05a700 1 -- 172.21.15.116:0/908521212 shutdown_connections 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.442+0000 7fa4af05a700 1 -- 172.21.15.116:0/908521212 wait complete. 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.442+0000 7fa4af05a700 1 Processor -- start 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.442+0000 7fa4af05a700 1 -- start start 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4af05a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4af05a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a819a670 con 0x7fa4a8105840 2024-03-04T15:54:17.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4acdf6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4acdf6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41820/0 (socket says 172.21.15.116:41820) 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4acdf6700 1 -- 172.21.15.116:0/722103145 learned_addr learned my addr 172.21.15.116:0/722103145 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4acdf6700 1 -- 172.21.15.116:0/722103145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c009550 con 0x7fa4a8105840 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4acdf6700 1 --2- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a819a130 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa49c003de0 tx=0x7fa49c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4a5ffb700 1 -- 172.21.15.116:0/722103145 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa49c003930 con 0x7fa4a8105840 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4af05a700 1 -- 172.21.15.116:0/722103145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a819a870 con 0x7fa4a8105840 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4a5ffb700 1 -- 172.21.15.116:0/722103145 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa49c003a90 con 0x7fa4a8105840 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4a5ffb700 1 -- 172.21.15.116:0/722103145 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa49c019730 con 0x7fa4a8105840 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.443+0000 7fa4af05a700 1 -- 172.21.15.116:0/722103145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a819ad10 con 0x7fa4a8105840 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.444+0000 7fa4a5ffb700 1 -- 172.21.15.116:0/722103145 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa49c0199b0 con 0x7fa4a8105840 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.444+0000 7fa4af05a700 1 -- 172.21.15.116:0/722103145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a80623c0 con 0x7fa4a8105840 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.445+0000 7fa4a5ffb700 1 --2- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fa498038320 0x7fa49803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.445+0000 7fa4a5ffb700 1 -- 172.21.15.116:0/722103145 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa49c04c100 con 0x7fa4a8105840 2024-03-04T15:54:17.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.445+0000 7fa4a7fff700 1 -- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fa498038320 msgr2=0x7fa49803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/961078374 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.445+0000 7fa4a7fff700 1 --2- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fa498038320 0x7fa49803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.449+0000 7fa4a5ffb700 1 -- 172.21.15.116:0/722103145 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa49c010dc0 con 0x7fa4a8105840 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.646+0000 7fa4a7fff700 1 -- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fa498038320 msgr2=0x7fa49803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/961078374 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.646+0000 7fa4a7fff700 1 --2- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fa498038320 0x7fa49803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.698+0000 7fa4af05a700 1 -- 172.21.15.116:0/722103145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa4a8109790 con 0x7fa4a8105840 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.698+0000 7fa4a5ffb700 1 -- 172.21.15.116:0/722103145 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa49c010380 con 0x7fa4a8105840 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.700+0000 7fa4af05a700 1 -- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fa498038320 msgr2=0x7fa49803a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.700+0000 7fa4af05a700 1 --2- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fa498038320 0x7fa49803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.700+0000 7fa4af05a700 1 -- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 msgr2=0x7fa4a819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.700+0000 7fa4af05a700 1 --2- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a819a130 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa49c003de0 tx=0x7fa49c011ab0 comp rx=0 tx=0).stop 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.700+0000 7fa4af05a700 1 -- 172.21.15.116:0/722103145 shutdown_connections 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.700+0000 7fa4af05a700 1 --2- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fa498038320 0x7fa49803a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.700+0000 7fa4af05a700 1 --2- 172.21.15.116:0/722103145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4a8105840 0x7fa4a819a130 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.700+0000 7fa4af05a700 1 -- 172.21.15.116:0/722103145 >> 172.21.15.116:0/722103145 conn(0x7fa4a8100040 msgr2=0x7fa4a8104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:17.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.701+0000 7fa4af05a700 1 -- 172.21.15.116:0/722103145 shutdown_connections 2024-03-04T15:54:17.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:15.701+0000 7fa4af05a700 1 -- 172.21.15.116:0/722103145 wait complete. 2024-03-04T15:54:17.079 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-04T15:54:17.079 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 5... 2024-03-04T15:54:21.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: Active manager daemon smithi116.jwdhop restarted 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: Activating manager daemon smithi116.jwdhop 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: osdmap e2: 0 total, 0 up, 0 in 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: mgrmap e6: smithi116.jwdhop(active, starting, since 0.00328039s) 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi116.jwdhop", "id": "smithi116.jwdhop"}]: dispatch 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: Manager daemon smithi116.jwdhop is now available 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:54:21.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:21 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:54:22.463 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:22 smithi116 ceph-mon[29120]: Found migration_current of "None". Setting to last migration. 2024-03-04T15:54:22.463 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:22 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:54:22.463 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:22 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/trash_purge_schedule"}]: dispatch 2024-03-04T15:54:22.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:22 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:22.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:22 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:22.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:22 smithi116 ceph-mon[29120]: mgrmap e7: smithi116.jwdhop(active, since 1.00557s) 2024-03-04T15:54:22.666 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:54:22.666 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-04T15:54:22.667 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T15:54:22.667 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:54:22.667 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.492+0000 7fd636618700 1 Processor -- start 2024-03-04T15:54:22.667 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.493+0000 7fd636618700 1 -- start start 2024-03-04T15:54:22.667 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.493+0000 7fd636618700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd630106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:22.667 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.493+0000 7fd636618700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd630106e20 con 0x7fd630106430 2024-03-04T15:54:22.667 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.493+0000 7fd62ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd630106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:22.667 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.493+0000 7fd62ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd630106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41828/0 (socket says 172.21.15.116:41828) 2024-03-04T15:54:22.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.493+0000 7fd62ffff700 1 -- 172.21.15.116:0/3364501989 learned_addr learned my addr 172.21.15.116:0/3364501989 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:22.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.493+0000 7fd62ffff700 1 -- 172.21.15.116:0/3364501989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd630107630 con 0x7fd630106430 2024-03-04T15:54:22.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.493+0000 7fd62ffff700 1 --2- 172.21.15.116:0/3364501989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd630106850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd6200098a0 tx=0x7fd620009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923a0455f502f9e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:22.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.493+0000 7fd62effd700 1 -- 172.21.15.116:0/3364501989 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd620004180 con 0x7fd630106430 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.494+0000 7fd62effd700 1 -- 172.21.15.116:0/3364501989 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd6200042e0 con 0x7fd630106430 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.494+0000 7fd62effd700 1 -- 172.21.15.116:0/3364501989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd620011a50 con 0x7fd630106430 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.494+0000 7fd636618700 1 -- 172.21.15.116:0/3364501989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 msgr2=0x7fd630106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.494+0000 7fd636618700 1 --2- 172.21.15.116:0/3364501989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd630106850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd6200098a0 tx=0x7fd620009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.494+0000 7fd636618700 1 -- 172.21.15.116:0/3364501989 shutdown_connections 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.494+0000 7fd636618700 1 --2- 172.21.15.116:0/3364501989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd630106850 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.494+0000 7fd636618700 1 -- 172.21.15.116:0/3364501989 >> 172.21.15.116:0/3364501989 conn(0x7fd6301019d0 msgr2=0x7fd630103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.494+0000 7fd636618700 1 -- 172.21.15.116:0/3364501989 shutdown_connections 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.494+0000 7fd636618700 1 -- 172.21.15.116:0/3364501989 wait complete. 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.495+0000 7fd636618700 1 Processor -- start 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.495+0000 7fd636618700 1 -- start start 2024-03-04T15:54:22.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.495+0000 7fd636618700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd63019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.495+0000 7fd636618700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63019a600 con 0x7fd630106430 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.496+0000 7fd62ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd63019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.496+0000 7fd62ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd63019a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41838/0 (socket says 172.21.15.116:41838) 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.496+0000 7fd62ffff700 1 -- 172.21.15.116:0/2865795421 learned_addr learned my addr 172.21.15.116:0/2865795421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.496+0000 7fd62ffff700 1 -- 172.21.15.116:0/2865795421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd620009550 con 0x7fd630106430 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.496+0000 7fd62ffff700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd63019a0c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd620006af0 tx=0x7fd620011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.496+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd620003b90 con 0x7fd630106430 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.496+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd62001a8a0 con 0x7fd630106430 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.496+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd620019880 con 0x7fd630106430 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.496+0000 7fd636618700 1 -- 172.21.15.116:0/2865795421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63019a800 con 0x7fd630106430 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.496+0000 7fd636618700 1 -- 172.21.15.116:0/2865795421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63019aca0 con 0x7fd630106430 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.497+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fd620019af0 con 0x7fd630106430 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.497+0000 7fd62d7fa700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 0x7fd61803a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:22.677 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.497+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 --> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd61803af00 con 0x7fd618038330 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.497+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd62004c690 con 0x7fd630106430 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.497+0000 7fd62f7fe700 1 -- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 msgr2=0x7fd61803a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/961078374 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.498+0000 7fd62f7fe700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 0x7fd61803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.699+0000 7fd62f7fe700 1 -- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 msgr2=0x7fd61803a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/961078374 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:17.699+0000 7fd62f7fe700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 0x7fd61803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:18.100+0000 7fd62f7fe700 1 -- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 msgr2=0x7fd61803a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/961078374 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:18.100+0000 7fd62f7fe700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 0x7fd61803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:18.901+0000 7fd62f7fe700 1 -- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 msgr2=0x7fd61803a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/961078374 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:18.901+0000 7fd62f7fe700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 0x7fd61803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:20.502+0000 7fd62f7fe700 1 -- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 msgr2=0x7fd61803a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/961078374 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:20.502+0000 7fd62f7fe700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 0x7fd61803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:21.168+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 6) v1 ==== 45050+0+0 (secure 0 0 0) 0x7fd620016d60 con 0x7fd630106430 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:21.168+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 msgr2=0x7fd61803a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T15:54:22.678 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:21.168+0000 7fd62d7fa700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 0x7fd61803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.169+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7fd62002acf0 con 0x7fd630106430 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.169+0000 7fd62d7fa700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd61803bc60 0x7fd61803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.169+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd61803af00 con 0x7fd61803bc60 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.170+0000 7fd62f7fe700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd61803bc60 0x7fd61803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.171+0000 7fd62f7fe700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd61803bc60 0x7fd61803e050 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd624003a10 tx=0x7fd6240092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.171+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fd61803af00 con 0x7fd61803bc60 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.177+0000 7fd636618700 1 -- 172.21.15.116:0/2865795421 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd63019b400 con 0x7fd61803bc60 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.177+0000 7fd62d7fa700 1 -- 172.21.15.116:0/2865795421 <== mgr.14120 v2:172.21.15.116:6800/1340152843 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd63019b400 con 0x7fd61803bc60 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 -- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd61803bc60 msgr2=0x7fd61803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd61803bc60 0x7fd61803e050 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd624003a10 tx=0x7fd6240092b0 comp rx=0 tx=0).stop 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 -- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 msgr2=0x7fd63019a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd63019a0c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd620006af0 tx=0x7fd620011a90 comp rx=0 tx=0).stop 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 -- 172.21.15.116:0/2865795421 shutdown_connections 2024-03-04T15:54:22.679 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd61803bc60 0x7fd61803e050 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:22.680 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd630106430 0x7fd63019a0c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:22.680 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 --2- 172.21.15.116:0/2865795421 >> [v2:172.21.15.116:6800/961078374,v1:172.21.15.116:6801/961078374] conn(0x7fd618038330 0x7fd61803a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:22.680 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 -- 172.21.15.116:0/2865795421 >> 172.21.15.116:0/2865795421 conn(0x7fd6301019d0 msgr2=0x7fd630102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:22.680 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 -- 172.21.15.116:0/2865795421 shutdown_connections 2024-03-04T15:54:22.680 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:22.178+0000 7fd636618700 1 -- 172.21.15.116:0/2865795421 wait complete. 2024-03-04T15:54:22.680 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 5 is available 2024-03-04T15:54:22.680 INFO:teuthology.orchestra.run.smithi116.stdout:Setting orchestrator backend to cephadm... 2024-03-04T15:54:23.333 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:23 smithi116 ceph-mon[29120]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T15:54:23.334 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:23 smithi116 ceph-mon[29120]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T15:54:23.952 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.096+0000 7fd9b49c5700 1 Processor -- start 2024-03-04T15:54:23.952 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.096+0000 7fd9b49c5700 1 -- start start 2024-03-04T15:54:23.952 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.097+0000 7fd9b49c5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b007a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:23.952 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.097+0000 7fd9b49c5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b007a900 con 0x7fd9b007bed0 2024-03-04T15:54:23.952 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.097+0000 7fd9ae59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b007a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.097+0000 7fd9ae59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b007a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41908/0 (socket says 172.21.15.116:41908) 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.097+0000 7fd9ae59c700 1 -- 172.21.15.116:0/743828065 learned_addr learned my addr 172.21.15.116:0/743828065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.097+0000 7fd9ae59c700 1 -- 172.21.15.116:0/743828065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b007aa40 con 0x7fd9b007bed0 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.097+0000 7fd9ae59c700 1 --2- 172.21.15.116:0/743828065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b007a330 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00098a0 tx=0x7fd9a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad746f84b8dc5a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.097+0000 7fd9ad59a700 1 -- 172.21.15.116:0/743828065 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9a0004180 con 0x7fd9b007bed0 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.097+0000 7fd9ad59a700 1 -- 172.21.15.116:0/743828065 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd9a00042e0 con 0x7fd9b007bed0 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.097+0000 7fd9ad59a700 1 -- 172.21.15.116:0/743828065 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9a0011a50 con 0x7fd9b007bed0 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.098+0000 7fd9b49c5700 1 -- 172.21.15.116:0/743828065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 msgr2=0x7fd9b007a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.098+0000 7fd9b49c5700 1 --2- 172.21.15.116:0/743828065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b007a330 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00098a0 tx=0x7fd9a0009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.098+0000 7fd9b49c5700 1 -- 172.21.15.116:0/743828065 shutdown_connections 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.098+0000 7fd9b49c5700 1 --2- 172.21.15.116:0/743828065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b007a330 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.098+0000 7fd9b49c5700 1 -- 172.21.15.116:0/743828065 >> 172.21.15.116:0/743828065 conn(0x7fd9b01019d0 msgr2=0x7fd9b0103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.098+0000 7fd9b49c5700 1 -- 172.21.15.116:0/743828065 shutdown_connections 2024-03-04T15:54:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.098+0000 7fd9b49c5700 1 -- 172.21.15.116:0/743828065 wait complete. 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.099+0000 7fd9b49c5700 1 Processor -- start 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.099+0000 7fd9b49c5700 1 -- start start 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.099+0000 7fd9b49c5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b019c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.099+0000 7fd9b49c5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b019c850 con 0x7fd9b007bed0 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.099+0000 7fd9ae59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b019c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.099+0000 7fd9ae59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b019c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41924/0 (socket says 172.21.15.116:41924) 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.099+0000 7fd9ae59c700 1 -- 172.21.15.116:0/431262992 learned_addr learned my addr 172.21.15.116:0/431262992 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.099+0000 7fd9ae59c700 1 -- 172.21.15.116:0/431262992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a0009550 con 0x7fd9b007bed0 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.100+0000 7fd9ae59c700 1 --2- 172.21.15.116:0/431262992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b019c310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0000c00 tx=0x7fd9a0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.100+0000 7fd99f7fe700 1 -- 172.21.15.116:0/431262992 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9a0003b90 con 0x7fd9b007bed0 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.100+0000 7fd99f7fe700 1 -- 172.21.15.116:0/431262992 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd9a00248b0 con 0x7fd9b007bed0 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.100+0000 7fd9b49c5700 1 -- 172.21.15.116:0/431262992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b019ca50 con 0x7fd9b007bed0 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.100+0000 7fd99f7fe700 1 -- 172.21.15.116:0/431262992 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9a001a8e0 con 0x7fd9b007bed0 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.100+0000 7fd9b49c5700 1 -- 172.21.15.116:0/431262992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b019cef0 con 0x7fd9b007bed0 2024-03-04T15:54:23.954 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.101+0000 7fd99f7fe700 1 -- 172.21.15.116:0/431262992 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7fd9a001ab50 con 0x7fd9b007bed0 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.101+0000 7fd99f7fe700 1 --2- 172.21.15.116:0/431262992 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd998038250 0x7fd99803a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.101+0000 7fd99f7fe700 1 -- 172.21.15.116:0/431262992 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd9a004c190 con 0x7fd9b007bed0 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.101+0000 7fd9add9b700 1 --2- 172.21.15.116:0/431262992 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd998038250 0x7fd99803a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.101+0000 7fd9b49c5700 1 -- 172.21.15.116:0/431262992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b00623c0 con 0x7fd9b007bed0 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.101+0000 7fd9add9b700 1 --2- 172.21.15.116:0/431262992 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd998038250 0x7fd99803a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4006fd0 tx=0x7fd9a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.106+0000 7fd99f7fe700 1 -- 172.21.15.116:0/431262992 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd9a00233a0 con 0x7fd9b007bed0 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.224+0000 7fd99f7fe700 1 -- 172.21.15.116:0/431262992 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fd9a002dc90 con 0x7fd9b007bed0 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.331+0000 7fd9b49c5700 1 -- 172.21.15.116:0/431262992 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9b0103ca0 con 0x7fd998038250 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.339+0000 7fd99f7fe700 1 -- 172.21.15.116:0/431262992 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd9b0103ca0 con 0x7fd998038250 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.341+0000 7fd9b49c5700 1 -- 172.21.15.116:0/431262992 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd998038250 msgr2=0x7fd99803a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.341+0000 7fd9b49c5700 1 --2- 172.21.15.116:0/431262992 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd998038250 0x7fd99803a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4006fd0 tx=0x7fd9a4006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.341+0000 7fd9b49c5700 1 -- 172.21.15.116:0/431262992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 msgr2=0x7fd9b019c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.341+0000 7fd9b49c5700 1 --2- 172.21.15.116:0/431262992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b019c310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0000c00 tx=0x7fd9a0011ab0 comp rx=0 tx=0).stop 2024-03-04T15:54:23.955 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.341+0000 7fd9b49c5700 1 -- 172.21.15.116:0/431262992 shutdown_connections 2024-03-04T15:54:23.956 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.341+0000 7fd9b49c5700 1 --2- 172.21.15.116:0/431262992 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd998038250 0x7fd99803a710 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:23.956 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.341+0000 7fd9b49c5700 1 --2- 172.21.15.116:0/431262992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b007bed0 0x7fd9b019c310 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:23.956 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.341+0000 7fd9b49c5700 1 -- 172.21.15.116:0/431262992 >> 172.21.15.116:0/431262992 conn(0x7fd9b01019d0 msgr2=0x7fd9b0103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:23.956 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.342+0000 7fd9b49c5700 1 -- 172.21.15.116:0/431262992 shutdown_connections 2024-03-04T15:54:23.956 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:23.342+0000 7fd9b49c5700 1 -- 172.21.15.116:0/431262992 wait complete. 2024-03-04T15:54:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:24 smithi116 ceph-mon[29120]: mgrmap e8: smithi116.jwdhop(active, since 2s) 2024-03-04T15:54:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:24 smithi116 ceph-mon[29120]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:24 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:24 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:54:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:24 smithi116 ceph-mon[29120]: [04/Mar/2024:15:54:23] ENGINE Bus STARTING 2024-03-04T15:54:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:24 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.369+0000 7fe0df1ec700 1 Processor -- start 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.369+0000 7fe0df1ec700 1 -- start start 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.370+0000 7fe0df1ec700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d81092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.370+0000 7fe0df1ec700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8079e80 con 0x7fe0d8106f00 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.370+0000 7fe0dcf88700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d81092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.370+0000 7fe0dcf88700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d81092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41932/0 (socket says 172.21.15.116:41932) 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.370+0000 7fe0dcf88700 1 -- 172.21.15.116:0/728351747 learned_addr learned my addr 172.21.15.116:0/728351747 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.370+0000 7fe0dcf88700 1 -- 172.21.15.116:0/728351747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d8109830 con 0x7fe0d8106f00 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.370+0000 7fe0dcf88700 1 --2- 172.21.15.116:0/728351747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d81092f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0098a0 tx=0x7fe0cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4a2918be787072 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.370+0000 7fe0d77fe700 1 -- 172.21.15.116:0/728351747 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0cc004180 con 0x7fe0d8106f00 2024-03-04T15:54:25.076 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.370+0000 7fe0d77fe700 1 -- 172.21.15.116:0/728351747 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe0cc0042e0 con 0x7fe0d8106f00 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.370+0000 7fe0d77fe700 1 -- 172.21.15.116:0/728351747 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0cc011a50 con 0x7fe0d8106f00 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.371+0000 7fe0df1ec700 1 -- 172.21.15.116:0/728351747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 msgr2=0x7fe0d81092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.371+0000 7fe0df1ec700 1 --2- 172.21.15.116:0/728351747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d81092f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0098a0 tx=0x7fe0cc009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.371+0000 7fe0df1ec700 1 -- 172.21.15.116:0/728351747 shutdown_connections 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.371+0000 7fe0df1ec700 1 --2- 172.21.15.116:0/728351747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d81092f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.371+0000 7fe0df1ec700 1 -- 172.21.15.116:0/728351747 >> 172.21.15.116:0/728351747 conn(0x7fe0d81008d0 msgr2=0x7fe0d8102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.371+0000 7fe0df1ec700 1 -- 172.21.15.116:0/728351747 shutdown_connections 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.371+0000 7fe0df1ec700 1 -- 172.21.15.116:0/728351747 wait complete. 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.371+0000 7fe0df1ec700 1 Processor -- start 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0df1ec700 1 -- start start 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0df1ec700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0df1ec700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d819a610 con 0x7fe0d8106f00 2024-03-04T15:54:25.077 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0dcf88700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0dcf88700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d819a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41934/0 (socket says 172.21.15.116:41934) 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0dcf88700 1 -- 172.21.15.116:0/1900041608 learned_addr learned my addr 172.21.15.116:0/1900041608 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0dcf88700 1 -- 172.21.15.116:0/1900041608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc009550 con 0x7fe0d8106f00 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0dcf88700 1 --2- 172.21.15.116:0/1900041608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d819a0d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc003de0 tx=0x7fe0cc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0d5ffb700 1 -- 172.21.15.116:0/1900041608 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0cc003930 con 0x7fe0d8106f00 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0df1ec700 1 -- 172.21.15.116:0/1900041608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d819a810 con 0x7fe0d8106f00 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0d5ffb700 1 -- 172.21.15.116:0/1900041608 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe0cc003a90 con 0x7fe0d8106f00 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.372+0000 7fe0d5ffb700 1 -- 172.21.15.116:0/1900041608 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0cc019730 con 0x7fe0d8106f00 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.373+0000 7fe0df1ec700 1 -- 172.21.15.116:0/1900041608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d819acb0 con 0x7fe0d8106f00 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.373+0000 7fe0d5ffb700 1 -- 172.21.15.116:0/1900041608 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fe0cc0199b0 con 0x7fe0d8106f00 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.373+0000 7fe0df1ec700 1 -- 172.21.15.116:0/1900041608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d810a580 con 0x7fe0d8106f00 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.373+0000 7fe0d5ffb700 1 --2- 172.21.15.116:0/1900041608 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe0c8038320 0x7fe0c803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.374+0000 7fe0d5ffb700 1 -- 172.21.15.116:0/1900041608 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe0cc04c580 con 0x7fe0d8106f00 2024-03-04T15:54:25.078 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.374+0000 7fe0d7fff700 1 --2- 172.21.15.116:0/1900041608 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe0c8038320 0x7fe0c803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.374+0000 7fe0d7fff700 1 --2- 172.21.15.116:0/1900041608 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe0c8038320 0x7fe0c803a7e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4006fd0 tx=0x7fe0c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.378+0000 7fe0d5ffb700 1 -- 172.21.15.116:0/1900041608 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe0cc010560 con 0x7fe0d8106f00 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.571+0000 7fe0df1ec700 1 -- 172.21.15.116:0/1900041608 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0d8101da0 con 0x7fe0c8038320 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.571+0000 7fe0d5ffb700 1 -- 172.21.15.116:0/1900041608 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fe0d8101da0 con 0x7fe0c8038320 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.573+0000 7fe0df1ec700 1 -- 172.21.15.116:0/1900041608 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe0c8038320 msgr2=0x7fe0c803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.573+0000 7fe0df1ec700 1 --2- 172.21.15.116:0/1900041608 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe0c8038320 0x7fe0c803a7e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4006fd0 tx=0x7fe0c4006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.573+0000 7fe0df1ec700 1 -- 172.21.15.116:0/1900041608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 msgr2=0x7fe0d819a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.573+0000 7fe0df1ec700 1 --2- 172.21.15.116:0/1900041608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d819a0d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc003de0 tx=0x7fe0cc011ab0 comp rx=0 tx=0).stop 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.573+0000 7fe0df1ec700 1 -- 172.21.15.116:0/1900041608 shutdown_connections 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.573+0000 7fe0df1ec700 1 --2- 172.21.15.116:0/1900041608 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe0c8038320 0x7fe0c803a7e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.573+0000 7fe0df1ec700 1 --2- 172.21.15.116:0/1900041608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0d8106f00 0x7fe0d819a0d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.573+0000 7fe0df1ec700 1 -- 172.21.15.116:0/1900041608 >> 172.21.15.116:0/1900041608 conn(0x7fe0d81008d0 msgr2=0x7fe0d8101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.573+0000 7fe0df1ec700 1 -- 172.21.15.116:0/1900041608 shutdown_connections 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:24.574+0000 7fe0df1ec700 1 -- 172.21.15.116:0/1900041608 wait complete. 2024-03-04T15:54:25.079 INFO:teuthology.orchestra.run.smithi116.stdout:Generating ssh key... 2024-03-04T15:54:25.420 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:25 smithi116 ceph-mon[29120]: [04/Mar/2024:15:54:23] ENGINE Serving on https://172.21.15.116:7150 2024-03-04T15:54:25.420 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:25 smithi116 ceph-mon[29120]: [04/Mar/2024:15:54:23] ENGINE Serving on http://172.21.15.116:8765 2024-03-04T15:54:25.420 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:25 smithi116 ceph-mon[29120]: [04/Mar/2024:15:54:23] ENGINE Bus STARTED 2024-03-04T15:54:26.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:26 smithi116 ceph-mon[29120]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:26.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:26 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:26.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:26 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:26.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.439+0000 7fa781a6f700 1 Processor -- start 2024-03-04T15:54:26.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.439+0000 7fa781a6f700 1 -- start start 2024-03-04T15:54:26.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.439+0000 7fa781a6f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c104b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:26.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.439+0000 7fa781a6f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c105120 con 0x7fa77c104730 2024-03-04T15:54:26.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.439+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c104b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:26.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.439+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c104b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42540/0 (socket says 172.21.15.116:42540) 2024-03-04T15:54:26.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.439+0000 7fa77b7fe700 1 -- 172.21.15.116:0/2241270096 learned_addr learned my addr 172.21.15.116:0/2241270096 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:26.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.439+0000 7fa77b7fe700 1 -- 172.21.15.116:0/2241270096 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c105260 con 0x7fa77c104730 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.439+0000 7fa77b7fe700 1 --2- 172.21.15.116:0/2241270096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c104b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0098a0 tx=0x7fa76c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51dceef238244285 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.440+0000 7fa77a7fc700 1 -- 172.21.15.116:0/2241270096 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa76c004180 con 0x7fa77c104730 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.440+0000 7fa77a7fc700 1 -- 172.21.15.116:0/2241270096 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa76c0042e0 con 0x7fa77c104730 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.440+0000 7fa77a7fc700 1 -- 172.21.15.116:0/2241270096 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa76c011a50 con 0x7fa77c104730 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.440+0000 7fa781a6f700 1 -- 172.21.15.116:0/2241270096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 msgr2=0x7fa77c104b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.440+0000 7fa781a6f700 1 --2- 172.21.15.116:0/2241270096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c104b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0098a0 tx=0x7fa76c009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.440+0000 7fa781a6f700 1 -- 172.21.15.116:0/2241270096 shutdown_connections 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.440+0000 7fa781a6f700 1 --2- 172.21.15.116:0/2241270096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c104b50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.440+0000 7fa781a6f700 1 -- 172.21.15.116:0/2241270096 >> 172.21.15.116:0/2241270096 conn(0x7fa77c100040 msgr2=0x7fa77c1024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.440+0000 7fa781a6f700 1 -- 172.21.15.116:0/2241270096 shutdown_connections 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.440+0000 7fa781a6f700 1 -- 172.21.15.116:0/2241270096 wait complete. 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.441+0000 7fa781a6f700 1 Processor -- start 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.441+0000 7fa781a6f700 1 -- start start 2024-03-04T15:54:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.441+0000 7fa781a6f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.441+0000 7fa781a6f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c19a690 con 0x7fa77c104730 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.441+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.441+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c19a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42552/0 (socket says 172.21.15.116:42552) 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.441+0000 7fa77b7fe700 1 -- 172.21.15.116:0/3704271422 learned_addr learned my addr 172.21.15.116:0/3704271422 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.442+0000 7fa77b7fe700 1 -- 172.21.15.116:0/3704271422 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c009550 con 0x7fa77c104730 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.442+0000 7fa77b7fe700 1 --2- 172.21.15.116:0/3704271422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c19a150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa76c003de0 tx=0x7fa76c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.442+0000 7fa778ff9700 1 -- 172.21.15.116:0/3704271422 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa76c003930 con 0x7fa77c104730 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.442+0000 7fa781a6f700 1 -- 172.21.15.116:0/3704271422 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c19a890 con 0x7fa77c104730 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.442+0000 7fa778ff9700 1 -- 172.21.15.116:0/3704271422 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa76c003a90 con 0x7fa77c104730 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.442+0000 7fa778ff9700 1 -- 172.21.15.116:0/3704271422 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa76c0196c0 con 0x7fa77c104730 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.442+0000 7fa781a6f700 1 -- 172.21.15.116:0/3704271422 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c19ad30 con 0x7fa77c104730 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.443+0000 7fa778ff9700 1 -- 172.21.15.116:0/3704271422 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa76c0199a0 con 0x7fa77c104730 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.443+0000 7fa778ff9700 1 --2- 172.21.15.116:0/3704271422 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa764038320 0x7fa76403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.443+0000 7fa781a6f700 1 -- 172.21.15.116:0/3704271422 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c0623c0 con 0x7fa77c104730 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.443+0000 7fa778ff9700 1 -- 172.21.15.116:0/3704271422 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa76c04c3f0 con 0x7fa77c104730 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.443+0000 7fa77affd700 1 --2- 172.21.15.116:0/3704271422 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa764038320 0x7fa76403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.443+0000 7fa77affd700 1 --2- 172.21.15.116:0/3704271422 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa764038320 0x7fa76403a7e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa770006fd0 tx=0x7fa770006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.447+0000 7fa778ff9700 1 -- 172.21.15.116:0/3704271422 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa76c019ca0 con 0x7fa77c104730 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:25.637+0000 7fa781a6f700 1 -- 172.21.15.116:0/3704271422 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa77c102370 con 0x7fa764038320 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.222+0000 7fa778ff9700 1 -- 172.21.15.116:0/3704271422 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa77c102370 con 0x7fa764038320 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.224+0000 7fa781a6f700 1 -- 172.21.15.116:0/3704271422 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa764038320 msgr2=0x7fa76403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.224+0000 7fa781a6f700 1 --2- 172.21.15.116:0/3704271422 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa764038320 0x7fa76403a7e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa770006fd0 tx=0x7fa770006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.224+0000 7fa781a6f700 1 -- 172.21.15.116:0/3704271422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 msgr2=0x7fa77c19a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.224+0000 7fa781a6f700 1 --2- 172.21.15.116:0/3704271422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c19a150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa76c003de0 tx=0x7fa76c011ab0 comp rx=0 tx=0).stop 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.224+0000 7fa781a6f700 1 -- 172.21.15.116:0/3704271422 shutdown_connections 2024-03-04T15:54:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.224+0000 7fa781a6f700 1 --2- 172.21.15.116:0/3704271422 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa764038320 0x7fa76403a7e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.224+0000 7fa781a6f700 1 --2- 172.21.15.116:0/3704271422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa77c104730 0x7fa77c19a150 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.224+0000 7fa781a6f700 1 -- 172.21.15.116:0/3704271422 >> 172.21.15.116:0/3704271422 conn(0x7fa77c100040 msgr2=0x7fa77c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.225+0000 7fa781a6f700 1 -- 172.21.15.116:0/3704271422 shutdown_connections 2024-03-04T15:54:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:26.225+0000 7fa781a6f700 1 -- 172.21.15.116:0/3704271422 wait complete. 2024-03-04T15:54:27.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:27 smithi116 ceph-mon[29120]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:27.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:27 smithi116 ceph-mon[29120]: Generating ssh key... 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/TEcKv95RWBJzAgjyMTHoBBcVhuqcve6m4cv3aesWwtjWrsVWrPk634jq5sl9THVBkVfPlRVm3ilmahHMRIW8gFURCzYRSLVQwNiDswu+vFjGHafIv4QOg1T564eg4fEJ5xA9AGEqcaJT4B4rtyncNR7d0H0sNKDP5d50wh8RiE+71VsUR0keD+hbjvBKjzzlCNw443tWQNB0HKUoKfke4neWfQUgIjBNfmMKOxKROwFr4wdQYfG2/D8SQ2HLIOB99/aH3Wof+Pod5JWARFY546tTzfBoBtgnX1Z5RgbvYYyi+WDfNg6X00qDojOSp6tJo2KR+PFP80mHu6AUApPvRRMqDUZl+EFtKJxZ9Rj04KYPfzBmN/AdeksMRiesaY7wG4iMMlBirNpa1xBDo7wL5O5x/kHOj/+puoity0mW5S/USyGnhh7CuZ+NPaXbbmGd8tUnoP062e4au5AtfFG++UwOWffZ3c9lEokBMDEhW59Xsi7jS439J4DTzTatKtk= ceph-4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.109+0000 7fd4f4373700 1 Processor -- start 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.110+0000 7fd4f4373700 1 -- start start 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.110+0000 7fd4f4373700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.110+0000 7fd4f4373700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec106e10 con 0x7fd4ec106420 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.110+0000 7fd4f210f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.110+0000 7fd4f210f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42554/0 (socket says 172.21.15.116:42554) 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.110+0000 7fd4f210f700 1 -- 172.21.15.116:0/3003654938 learned_addr learned my addr 172.21.15.116:0/3003654938 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.110+0000 7fd4f210f700 1 -- 172.21.15.116:0/3003654938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec107620 con 0x7fd4ec106420 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.110+0000 7fd4f210f700 1 --2- 172.21.15.116:0/3003654938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec106840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd4e80098a0 tx=0x7fd4e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76edb61904e73076 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.111+0000 7fd4f110d700 1 -- 172.21.15.116:0/3003654938 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e8004180 con 0x7fd4ec106420 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.111+0000 7fd4f110d700 1 -- 172.21.15.116:0/3003654938 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd4e80042e0 con 0x7fd4ec106420 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.111+0000 7fd4f110d700 1 -- 172.21.15.116:0/3003654938 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e8011a50 con 0x7fd4ec106420 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.111+0000 7fd4f4373700 1 -- 172.21.15.116:0/3003654938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 msgr2=0x7fd4ec106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.111+0000 7fd4f4373700 1 --2- 172.21.15.116:0/3003654938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec106840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd4e80098a0 tx=0x7fd4e8009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.111+0000 7fd4f4373700 1 -- 172.21.15.116:0/3003654938 shutdown_connections 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.111+0000 7fd4f4373700 1 --2- 172.21.15.116:0/3003654938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec106840 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.111+0000 7fd4f4373700 1 -- 172.21.15.116:0/3003654938 >> 172.21.15.116:0/3003654938 conn(0x7fd4ec101a00 msgr2=0x7fd4ec103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.111+0000 7fd4f4373700 1 -- 172.21.15.116:0/3003654938 shutdown_connections 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.111+0000 7fd4f4373700 1 -- 172.21.15.116:0/3003654938 wait complete. 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.112+0000 7fd4f4373700 1 Processor -- start 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.112+0000 7fd4f4373700 1 -- start start 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.112+0000 7fd4f4373700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.112+0000 7fd4f4373700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec19a5f0 con 0x7fd4ec106420 2024-03-04T15:54:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.112+0000 7fd4f210f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.112+0000 7fd4f210f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec19a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42570/0 (socket says 172.21.15.116:42570) 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.112+0000 7fd4f210f700 1 -- 172.21.15.116:0/360286059 learned_addr learned my addr 172.21.15.116:0/360286059 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.112+0000 7fd4f210f700 1 -- 172.21.15.116:0/360286059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e8009550 con 0x7fd4ec106420 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.113+0000 7fd4f210f700 1 --2- 172.21.15.116:0/360286059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec19a0b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8006af0 tx=0x7fd4e8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.113+0000 7fd4e37fe700 1 -- 172.21.15.116:0/360286059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e8003b90 con 0x7fd4ec106420 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.113+0000 7fd4e37fe700 1 -- 172.21.15.116:0/360286059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd4e801a8a0 con 0x7fd4ec106420 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.113+0000 7fd4f4373700 1 -- 172.21.15.116:0/360286059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ec19a7f0 con 0x7fd4ec106420 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.113+0000 7fd4e37fe700 1 -- 172.21.15.116:0/360286059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e8019880 con 0x7fd4ec106420 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.113+0000 7fd4f4373700 1 -- 172.21.15.116:0/360286059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ec19ac90 con 0x7fd4ec106420 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.113+0000 7fd4e37fe700 1 -- 172.21.15.116:0/360286059 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fd4e8019af0 con 0x7fd4ec106420 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.114+0000 7fd4f4373700 1 -- 172.21.15.116:0/360286059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ec0623c0 con 0x7fd4ec106420 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.114+0000 7fd4e37fe700 1 --2- 172.21.15.116:0/360286059 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd4d8038330 0x7fd4d803a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.114+0000 7fd4e37fe700 1 -- 172.21.15.116:0/360286059 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd4e804c670 con 0x7fd4ec106420 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.114+0000 7fd4f190e700 1 --2- 172.21.15.116:0/360286059 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd4d8038330 0x7fd4d803a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.114+0000 7fd4f190e700 1 --2- 172.21.15.116:0/360286059 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd4d8038330 0x7fd4d803a7f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc006fd0 tx=0x7fd4dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.118+0000 7fd4e37fe700 1 -- 172.21.15.116:0/360286059 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd4e802b750 con 0x7fd4ec106420 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.310+0000 7fd4f4373700 1 -- 172.21.15.116:0/360286059 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4ec10a820 con 0x7fd4d8038330 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.311+0000 7fd4e37fe700 1 -- 172.21.15.116:0/360286059 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fd4ec10a820 con 0x7fd4d8038330 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.314+0000 7fd4f4373700 1 -- 172.21.15.116:0/360286059 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd4d8038330 msgr2=0x7fd4d803a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.314+0000 7fd4f4373700 1 --2- 172.21.15.116:0/360286059 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd4d8038330 0x7fd4d803a7f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc006fd0 tx=0x7fd4dc006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.314+0000 7fd4f4373700 1 -- 172.21.15.116:0/360286059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 msgr2=0x7fd4ec19a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.314+0000 7fd4f4373700 1 --2- 172.21.15.116:0/360286059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec19a0b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8006af0 tx=0x7fd4e8011a90 comp rx=0 tx=0).stop 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.314+0000 7fd4f4373700 1 -- 172.21.15.116:0/360286059 shutdown_connections 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.314+0000 7fd4f4373700 1 --2- 172.21.15.116:0/360286059 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fd4d8038330 0x7fd4d803a7f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.314+0000 7fd4f4373700 1 --2- 172.21.15.116:0/360286059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec106420 0x7fd4ec19a0b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.314+0000 7fd4f4373700 1 -- 172.21.15.116:0/360286059 >> 172.21.15.116:0/360286059 conn(0x7fd4ec101a00 msgr2=0x7fd4ec1027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:27.825 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.314+0000 7fd4f4373700 1 -- 172.21.15.116:0/360286059 shutdown_connections 2024-03-04T15:54:27.825 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:27.314+0000 7fd4f4373700 1 -- 172.21.15.116:0/360286059 wait complete. 2024-03-04T15:54:27.825 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:54:27.825 INFO:teuthology.orchestra.run.smithi116.stdout:Adding key to root@localhost authorized_keys... 2024-03-04T15:54:27.825 INFO:teuthology.orchestra.run.smithi116.stdout:Adding host smithi116... 2024-03-04T15:54:28.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:28 smithi116 ceph-mon[29120]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:29.401 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:29 smithi116 ceph-mon[29120]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:30.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:30 smithi116 ceph-mon[29120]: Deploying cephadm binary to smithi116 2024-03-04T15:54:31.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Added host 'smithi116' with addr '172.21.15.116' 2024-03-04T15:54:31.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.215+0000 7f58ab5e6700 1 Processor -- start 2024-03-04T15:54:31.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.215+0000 7f58ab5e6700 1 -- start start 2024-03-04T15:54:31.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.215+0000 7f58ab5e6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a41092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:31.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.215+0000 7f58ab5e6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4079e60 con 0x7f58a4106f00 2024-03-04T15:54:31.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.215+0000 7f58a9382700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a41092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:31.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.215+0000 7f58a9382700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a41092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42578/0 (socket says 172.21.15.116:42578) 2024-03-04T15:54:31.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.215+0000 7f58a9382700 1 -- 172.21.15.116:0/2959502851 learned_addr learned my addr 172.21.15.116:0/2959502851 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:31.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.216+0000 7f58a9382700 1 -- 172.21.15.116:0/2959502851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a4109830 con 0x7f58a4106f00 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.216+0000 7f58a9382700 1 --2- 172.21.15.116:0/2959502851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a41092f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009e80 tx=0x7f58a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4f7cd447dd3cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.216+0000 7f589bfff700 1 -- 172.21.15.116:0/2959502851 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a000e070 con 0x7f58a4106f00 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.216+0000 7f589bfff700 1 -- 172.21.15.116:0/2959502851 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f58a0003e90 con 0x7f58a4106f00 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.216+0000 7f589bfff700 1 -- 172.21.15.116:0/2959502851 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a0004380 con 0x7f58a4106f00 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.217+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2959502851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 msgr2=0x7f58a41092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.217+0000 7f58ab5e6700 1 --2- 172.21.15.116:0/2959502851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a41092f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009e80 tx=0x7f58a000b330 comp rx=0 tx=0).stop 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.217+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2959502851 shutdown_connections 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.217+0000 7f58ab5e6700 1 --2- 172.21.15.116:0/2959502851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a41092f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.217+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2959502851 >> 172.21.15.116:0/2959502851 conn(0x7f58a41008b0 msgr2=0x7f58a4102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.217+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2959502851 shutdown_connections 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.217+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2959502851 wait complete. 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.217+0000 7f58ab5e6700 1 Processor -- start 2024-03-04T15:54:31.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.217+0000 7f58ab5e6700 1 -- start start 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f58ab5e6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f58ab5e6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a419a610 con 0x7f58a4106f00 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f58a9382700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f58a9382700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42586/0 (socket says 172.21.15.116:42586) 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f58a9382700 1 -- 172.21.15.116:0/2003375868 learned_addr learned my addr 172.21.15.116:0/2003375868 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f58a9382700 1 -- 172.21.15.116:0/2003375868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a0009b30 con 0x7f58a4106f00 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f58a9382700 1 --2- 172.21.15.116:0/2003375868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a419a0d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58a000b7b0 tx=0x7f58a0004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f589a7fc700 1 -- 172.21.15.116:0/2003375868 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a000e040 con 0x7f58a4106f00 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f589a7fc700 1 -- 172.21.15.116:0/2003375868 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f58a0003c30 con 0x7f58a4106f00 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2003375868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a419a810 con 0x7f58a4106f00 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f589a7fc700 1 -- 172.21.15.116:0/2003375868 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a001b740 con 0x7f58a4106f00 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.218+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2003375868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a419ad10 con 0x7f58a4106f00 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.219+0000 7f589a7fc700 1 -- 172.21.15.116:0/2003375868 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f58a0019040 con 0x7f58a4106f00 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.219+0000 7f589a7fc700 1 --2- 172.21.15.116:0/2003375868 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5890038320 0x7f589003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:31.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.219+0000 7f589a7fc700 1 -- 172.21.15.116:0/2003375868 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f58a004bc40 con 0x7f58a4106f00 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.219+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2003375868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a410a550 con 0x7f58a4106f00 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.219+0000 7f58a8b81700 1 --2- 172.21.15.116:0/2003375868 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5890038320 0x7f589003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.220+0000 7f58a8b81700 1 --2- 172.21.15.116:0/2003375868 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5890038320 0x7f589003a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5894006fd0 tx=0x7f5894006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.224+0000 7f589a7fc700 1 -- 172.21.15.116:0/2003375868 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f58a0012c30 con 0x7f58a4106f00 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:28.417+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2003375868 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}) v1 -- 0x7f58a4101da0 con 0x7f5890038320 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.991+0000 7f589a7fc700 1 -- 172.21.15.116:0/2003375868 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f58a4101da0 con 0x7f5890038320 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.993+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2003375868 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5890038320 msgr2=0x7f589003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.993+0000 7f58ab5e6700 1 --2- 172.21.15.116:0/2003375868 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5890038320 0x7f589003a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5894006fd0 tx=0x7f5894006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.993+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2003375868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 msgr2=0x7f58a419a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.993+0000 7f58ab5e6700 1 --2- 172.21.15.116:0/2003375868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a419a0d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58a000b7b0 tx=0x7f58a0004920 comp rx=0 tx=0).stop 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.993+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2003375868 shutdown_connections 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.993+0000 7f58ab5e6700 1 --2- 172.21.15.116:0/2003375868 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5890038320 0x7f589003a7e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.993+0000 7f58ab5e6700 1 --2- 172.21.15.116:0/2003375868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a4106f00 0x7f58a419a0d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:31.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.993+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2003375868 >> 172.21.15.116:0/2003375868 conn(0x7f58a41008b0 msgr2=0x7f58a4101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:31.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.993+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2003375868 shutdown_connections 2024-03-04T15:54:31.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:30.993+0000 7f58ab5e6700 1 -- 172.21.15.116:0/2003375868 wait complete. 2024-03-04T15:54:31.592 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying mon service with default placement... 2024-03-04T15:54:32.166 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:31 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:32.166 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:31 smithi116 ceph-mon[29120]: Added host smithi116 2024-03-04T15:54:32.166 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:31 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:54:33.309 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:33 smithi116 ceph-mon[29120]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:33.309 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:33 smithi116 ceph-mon[29120]: Saving service mon spec with placement count:5 2024-03-04T15:54:33.309 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:33 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:33.309 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:33 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:33.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-04T15:54:33.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.981+0000 7fbb9399a700 1 Processor -- start 2024-03-04T15:54:33.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.981+0000 7fbb9399a700 1 -- start start 2024-03-04T15:54:33.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.981+0000 7fbb9399a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c105770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:33.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.981+0000 7fbb9399a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8c105d40 con 0x7fbb8c105350 2024-03-04T15:54:33.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.982+0000 7fbb91736700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c105770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:33.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.982+0000 7fbb91736700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c105770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42598/0 (socket says 172.21.15.116:42598) 2024-03-04T15:54:33.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.982+0000 7fbb91736700 1 -- 172.21.15.116:0/1500765290 learned_addr learned my addr 172.21.15.116:0/1500765290 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:33.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.982+0000 7fbb91736700 1 -- 172.21.15.116:0/1500765290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c106550 con 0x7fbb8c105350 2024-03-04T15:54:33.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.982+0000 7fbb91736700 1 --2- 172.21.15.116:0/1500765290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c105770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbb880098a0 tx=0x7fbb88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad1bf5a8d8cd3c9c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:33.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.982+0000 7fbb83fff700 1 -- 172.21.15.116:0/1500765290 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb88004180 con 0x7fbb8c105350 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.983+0000 7fbb83fff700 1 -- 172.21.15.116:0/1500765290 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbb880042e0 con 0x7fbb8c105350 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.983+0000 7fbb83fff700 1 -- 172.21.15.116:0/1500765290 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb88011a50 con 0x7fbb8c105350 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.983+0000 7fbb9399a700 1 -- 172.21.15.116:0/1500765290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 msgr2=0x7fbb8c105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.983+0000 7fbb9399a700 1 --2- 172.21.15.116:0/1500765290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c105770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbb880098a0 tx=0x7fbb88009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.983+0000 7fbb9399a700 1 -- 172.21.15.116:0/1500765290 shutdown_connections 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.983+0000 7fbb9399a700 1 --2- 172.21.15.116:0/1500765290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c105770 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.983+0000 7fbb9399a700 1 -- 172.21.15.116:0/1500765290 >> 172.21.15.116:0/1500765290 conn(0x7fbb8c1008d0 msgr2=0x7fbb8c102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.983+0000 7fbb9399a700 1 -- 172.21.15.116:0/1500765290 shutdown_connections 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.983+0000 7fbb9399a700 1 -- 172.21.15.116:0/1500765290 wait complete. 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.984+0000 7fbb9399a700 1 Processor -- start 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.984+0000 7fbb9399a700 1 -- start start 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.984+0000 7fbb9399a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c079b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.984+0000 7fbb9399a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8c19ce60 con 0x7fbb8c105350 2024-03-04T15:54:33.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.984+0000 7fbb91736700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c079b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.984+0000 7fbb91736700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c079b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42600/0 (socket says 172.21.15.116:42600) 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.984+0000 7fbb91736700 1 -- 172.21.15.116:0/333917925 learned_addr learned my addr 172.21.15.116:0/333917925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.985+0000 7fbb91736700 1 -- 172.21.15.116:0/333917925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb88009550 con 0x7fbb8c105350 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.985+0000 7fbb91736700 1 --2- 172.21.15.116:0/333917925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c079b60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbb88003de0 tx=0x7fbb88011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.985+0000 7fbb827fc700 1 -- 172.21.15.116:0/333917925 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb88003930 con 0x7fbb8c105350 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.985+0000 7fbb827fc700 1 -- 172.21.15.116:0/333917925 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbb88003a90 con 0x7fbb8c105350 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.985+0000 7fbb827fc700 1 -- 172.21.15.116:0/333917925 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb880196c0 con 0x7fbb8c105350 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.985+0000 7fbb9399a700 1 -- 172.21.15.116:0/333917925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8c19cfa0 con 0x7fbb8c105350 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.985+0000 7fbb9399a700 1 -- 172.21.15.116:0/333917925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8c19d440 con 0x7fbb8c105350 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.986+0000 7fbb827fc700 1 -- 172.21.15.116:0/333917925 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fbb88019930 con 0x7fbb8c105350 2024-03-04T15:54:33.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.986+0000 7fbb9399a700 1 -- 172.21.15.116:0/333917925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8c0623c0 con 0x7fbb8c105350 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.986+0000 7fbb827fc700 1 --2- 172.21.15.116:0/333917925 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbb78038370 0x7fbb7803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.986+0000 7fbb827fc700 1 -- 172.21.15.116:0/333917925 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbb8804c4e0 con 0x7fbb8c105350 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.986+0000 7fbb90f35700 1 --2- 172.21.15.116:0/333917925 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbb78038370 0x7fbb7803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.987+0000 7fbb90f35700 1 --2- 172.21.15.116:0/333917925 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbb78038370 0x7fbb7803a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c006fd0 tx=0x7fbb7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:31.992+0000 7fbb827fc700 1 -- 172.21.15.116:0/333917925 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbb88010560 con 0x7fbb8c105350 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.198+0000 7fbb9399a700 1 -- 172.21.15.116:0/333917925 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb8c102bc0 con 0x7fbb78038370 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.203+0000 7fbb827fc700 1 -- 172.21.15.116:0/333917925 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fbb8c102bc0 con 0x7fbb78038370 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.205+0000 7fbb9399a700 1 -- 172.21.15.116:0/333917925 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbb78038370 msgr2=0x7fbb7803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.205+0000 7fbb9399a700 1 --2- 172.21.15.116:0/333917925 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbb78038370 0x7fbb7803a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c006fd0 tx=0x7fbb7c006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.205+0000 7fbb9399a700 1 -- 172.21.15.116:0/333917925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 msgr2=0x7fbb8c079b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.205+0000 7fbb9399a700 1 --2- 172.21.15.116:0/333917925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c079b60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbb88003de0 tx=0x7fbb88011ab0 comp rx=0 tx=0).stop 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.205+0000 7fbb9399a700 1 -- 172.21.15.116:0/333917925 shutdown_connections 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.205+0000 7fbb9399a700 1 --2- 172.21.15.116:0/333917925 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbb78038370 0x7fbb7803a830 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:33.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.205+0000 7fbb9399a700 1 --2- 172.21.15.116:0/333917925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb8c105350 0x7fbb8c079b60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:33.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.205+0000 7fbb9399a700 1 -- 172.21.15.116:0/333917925 >> 172.21.15.116:0/333917925 conn(0x7fbb8c1008d0 msgr2=0x7fbb8c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:33.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.205+0000 7fbb9399a700 1 -- 172.21.15.116:0/333917925 shutdown_connections 2024-03-04T15:54:33.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:32.205+0000 7fbb9399a700 1 -- 172.21.15.116:0/333917925 wait complete. 2024-03-04T15:54:33.607 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying mgr service with default placement... 2024-03-04T15:54:34.686 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:34 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:34.686 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:34 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.006+0000 7fea7a0b9700 1 Processor -- start 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.006+0000 7fea7a0b9700 1 -- start start 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.006+0000 7fea7a0b9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea74105750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.006+0000 7fea7a0b9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea74105d20 con 0x7fea74105330 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.007+0000 7fea737fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea74105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.007+0000 7fea737fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea74105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42608/0 (socket says 172.21.15.116:42608) 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.007+0000 7fea737fe700 1 -- 172.21.15.116:0/1658457173 learned_addr learned my addr 172.21.15.116:0/1658457173 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.007+0000 7fea737fe700 1 -- 172.21.15.116:0/1658457173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea74106530 con 0x7fea74105330 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.007+0000 7fea737fe700 1 --2- 172.21.15.116:0/1658457173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea74105750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea640098a0 tx=0x7fea64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a62dd3817c8ed16 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:35.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.007+0000 7fea727fc700 1 -- 172.21.15.116:0/1658457173 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea64004180 con 0x7fea74105330 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.007+0000 7fea727fc700 1 -- 172.21.15.116:0/1658457173 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fea640042e0 con 0x7fea74105330 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.007+0000 7fea727fc700 1 -- 172.21.15.116:0/1658457173 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea64011af0 con 0x7fea74105330 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.008+0000 7fea7a0b9700 1 -- 172.21.15.116:0/1658457173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 msgr2=0x7fea74105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.008+0000 7fea7a0b9700 1 --2- 172.21.15.116:0/1658457173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea74105750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea640098a0 tx=0x7fea64009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.008+0000 7fea7a0b9700 1 -- 172.21.15.116:0/1658457173 shutdown_connections 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.008+0000 7fea7a0b9700 1 --2- 172.21.15.116:0/1658457173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea74105750 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.008+0000 7fea7a0b9700 1 -- 172.21.15.116:0/1658457173 >> 172.21.15.116:0/1658457173 conn(0x7fea741008b0 msgr2=0x7fea74102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.008+0000 7fea7a0b9700 1 -- 172.21.15.116:0/1658457173 shutdown_connections 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.008+0000 7fea7a0b9700 1 -- 172.21.15.116:0/1658457173 wait complete. 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.009+0000 7fea7a0b9700 1 Processor -- start 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.009+0000 7fea7a0b9700 1 -- start start 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.009+0000 7fea7a0b9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea7419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.009+0000 7fea7a0b9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7419a6f0 con 0x7fea74105330 2024-03-04T15:54:35.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.009+0000 7fea737fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea7419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.009+0000 7fea737fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea7419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42624/0 (socket says 172.21.15.116:42624) 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.009+0000 7fea737fe700 1 -- 172.21.15.116:0/38004082 learned_addr learned my addr 172.21.15.116:0/38004082 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.009+0000 7fea737fe700 1 -- 172.21.15.116:0/38004082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea64009550 con 0x7fea74105330 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.009+0000 7fea737fe700 1 --2- 172.21.15.116:0/38004082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea7419a1b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea64011670 tx=0x7fea64011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.010+0000 7fea70ff9700 1 -- 172.21.15.116:0/38004082 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea640039f0 con 0x7fea74105330 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.010+0000 7fea70ff9700 1 -- 172.21.15.116:0/38004082 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fea64003b50 con 0x7fea74105330 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.010+0000 7fea7a0b9700 1 -- 172.21.15.116:0/38004082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea7419a8f0 con 0x7fea74105330 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.010+0000 7fea70ff9700 1 -- 172.21.15.116:0/38004082 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea640247c0 con 0x7fea74105330 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.010+0000 7fea7a0b9700 1 -- 172.21.15.116:0/38004082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea7419ad90 con 0x7fea74105330 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.011+0000 7fea70ff9700 1 -- 172.21.15.116:0/38004082 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fea6402d3f0 con 0x7fea74105330 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.011+0000 7fea7a0b9700 1 -- 172.21.15.116:0/38004082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea740623c0 con 0x7fea74105330 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.011+0000 7fea70ff9700 1 --2- 172.21.15.116:0/38004082 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fea5c038320 0x7fea5c03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.011+0000 7fea70ff9700 1 -- 172.21.15.116:0/38004082 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fea6404c3c0 con 0x7fea74105330 2024-03-04T15:54:35.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.011+0000 7fea72ffd700 1 --2- 172.21.15.116:0/38004082 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fea5c038320 0x7fea5c03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:35.396 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.011+0000 7fea72ffd700 1 --2- 172.21.15.116:0/38004082 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fea5c038320 0x7fea5c03a7e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fea68006fd0 tx=0x7fea68006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:35.396 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.016+0000 7fea70ff9700 1 -- 172.21.15.116:0/38004082 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fea64010970 con 0x7fea74105330 2024-03-04T15:54:35.396 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.232+0000 7fea7a0b9700 1 -- 172.21.15.116:0/38004082 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fea74102c50 con 0x7fea5c038320 2024-03-04T15:54:35.396 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.236+0000 7fea70ff9700 1 -- 172.21.15.116:0/38004082 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fea74102c50 con 0x7fea5c038320 2024-03-04T15:54:35.396 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.238+0000 7fea7a0b9700 1 -- 172.21.15.116:0/38004082 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fea5c038320 msgr2=0x7fea5c03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:35.396 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.238+0000 7fea7a0b9700 1 --2- 172.21.15.116:0/38004082 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fea5c038320 0x7fea5c03a7e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fea68006fd0 tx=0x7fea68006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:35.396 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.238+0000 7fea7a0b9700 1 -- 172.21.15.116:0/38004082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 msgr2=0x7fea7419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:35.397 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.238+0000 7fea7a0b9700 1 --2- 172.21.15.116:0/38004082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea7419a1b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea64011670 tx=0x7fea64011b50 comp rx=0 tx=0).stop 2024-03-04T15:54:35.397 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.238+0000 7fea7a0b9700 1 -- 172.21.15.116:0/38004082 shutdown_connections 2024-03-04T15:54:35.397 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.238+0000 7fea7a0b9700 1 --2- 172.21.15.116:0/38004082 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fea5c038320 0x7fea5c03a7e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:35.397 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.238+0000 7fea7a0b9700 1 --2- 172.21.15.116:0/38004082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea74105330 0x7fea7419a1b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:35.397 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.238+0000 7fea7a0b9700 1 -- 172.21.15.116:0/38004082 >> 172.21.15.116:0/38004082 conn(0x7fea741008b0 msgr2=0x7fea74102030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:35.397 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.238+0000 7fea7a0b9700 1 -- 172.21.15.116:0/38004082 shutdown_connections 2024-03-04T15:54:35.397 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:34.238+0000 7fea7a0b9700 1 -- 172.21.15.116:0/38004082 wait complete. 2024-03-04T15:54:35.397 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying crash service with default placement... 2024-03-04T15:54:35.713 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:35 smithi116 ceph-mon[29120]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:35.713 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:35 smithi116 ceph-mon[29120]: Saving service mgr spec with placement count:2 2024-03-04T15:54:35.713 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:35 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:36.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-04T15:54:36.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.813+0000 7f322c62c700 1 Processor -- start 2024-03-04T15:54:36.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.813+0000 7f322c62c700 1 -- start start 2024-03-04T15:54:36.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.813+0000 7f322c62c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f3224109310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.813+0000 7f322c62c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3224079e60 con 0x7f3224106f20 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.813+0000 7f322a3c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f3224109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.813+0000 7f322a3c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f3224109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50990/0 (socket says 172.21.15.116:50990) 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.813+0000 7f322a3c8700 1 -- 172.21.15.116:0/29003174 learned_addr learned my addr 172.21.15.116:0/29003174 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.814+0000 7f322a3c8700 1 -- 172.21.15.116:0/29003174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224109850 con 0x7f3224106f20 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.814+0000 7f322a3c8700 1 --2- 172.21.15.116:0/29003174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f3224109310 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32200098a0 tx=0x7f3220009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd3393e4b5a9b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.814+0000 7f32293c6700 1 -- 172.21.15.116:0/29003174 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3220004180 con 0x7f3224106f20 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.814+0000 7f32293c6700 1 -- 172.21.15.116:0/29003174 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f32200042e0 con 0x7f3224106f20 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.814+0000 7f32293c6700 1 -- 172.21.15.116:0/29003174 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3220011af0 con 0x7f3224106f20 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 -- 172.21.15.116:0/29003174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 msgr2=0x7f3224109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 --2- 172.21.15.116:0/29003174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f3224109310 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32200098a0 tx=0x7f3220009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 -- 172.21.15.116:0/29003174 shutdown_connections 2024-03-04T15:54:36.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 --2- 172.21.15.116:0/29003174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f3224109310 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 -- 172.21.15.116:0/29003174 >> 172.21.15.116:0/29003174 conn(0x7f32241008d0 msgr2=0x7f3224102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 -- 172.21.15.116:0/29003174 shutdown_connections 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 -- 172.21.15.116:0/29003174 wait complete. 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 Processor -- start 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 -- start start 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f322419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.815+0000 7f322c62c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322419a650 con 0x7f3224106f20 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.816+0000 7f322a3c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f322419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.816+0000 7f322a3c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f322419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50992/0 (socket says 172.21.15.116:50992) 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.816+0000 7f322a3c8700 1 -- 172.21.15.116:0/3891410940 learned_addr learned my addr 172.21.15.116:0/3891410940 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.816+0000 7f322a3c8700 1 -- 172.21.15.116:0/3891410940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3220009550 con 0x7f3224106f20 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.816+0000 7f322a3c8700 1 --2- 172.21.15.116:0/3891410940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f322419a110 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3220011670 tx=0x7f3220011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.816+0000 7f321b7fe700 1 -- 172.21.15.116:0/3891410940 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3220003a40 con 0x7f3224106f20 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.816+0000 7f322c62c700 1 -- 172.21.15.116:0/3891410940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322419a850 con 0x7f3224106f20 2024-03-04T15:54:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.816+0000 7f321b7fe700 1 -- 172.21.15.116:0/3891410940 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3220003ba0 con 0x7f3224106f20 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.816+0000 7f321b7fe700 1 -- 172.21.15.116:0/3891410940 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3220024820 con 0x7f3224106f20 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.816+0000 7f322c62c700 1 -- 172.21.15.116:0/3891410940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322419acf0 con 0x7f3224106f20 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.817+0000 7f321b7fe700 1 -- 172.21.15.116:0/3891410940 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f322001ac70 con 0x7f3224106f20 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.817+0000 7f322c62c700 1 -- 172.21.15.116:0/3891410940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322410a5e0 con 0x7f3224106f20 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.817+0000 7f321b7fe700 1 --2- 172.21.15.116:0/3891410940 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f3210038330 0x7f321003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.817+0000 7f321b7fe700 1 -- 172.21.15.116:0/3891410940 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f322004c290 con 0x7f3224106f20 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.817+0000 7f3229bc7700 1 --2- 172.21.15.116:0/3891410940 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f3210038330 0x7f321003a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.818+0000 7f3229bc7700 1 --2- 172.21.15.116:0/3891410940 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f3210038330 0x7f321003a7f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3214006fd0 tx=0x7f3214006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:35.826+0000 7f321b7fe700 1 -- 172.21.15.116:0/3891410940 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3220018760 con 0x7f3224106f20 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.037+0000 7f322c62c700 1 -- 172.21.15.116:0/3891410940 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f3224101db0 con 0x7f3210038330 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.042+0000 7f321b7fe700 1 -- 172.21.15.116:0/3891410940 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f3224101db0 con 0x7f3210038330 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.044+0000 7f322c62c700 1 -- 172.21.15.116:0/3891410940 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f3210038330 msgr2=0x7f321003a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.044+0000 7f322c62c700 1 --2- 172.21.15.116:0/3891410940 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f3210038330 0x7f321003a7f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3214006fd0 tx=0x7f3214006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:36.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.044+0000 7f322c62c700 1 -- 172.21.15.116:0/3891410940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 msgr2=0x7f322419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:36.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.044+0000 7f322c62c700 1 --2- 172.21.15.116:0/3891410940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f322419a110 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3220011670 tx=0x7f3220011aa0 comp rx=0 tx=0).stop 2024-03-04T15:54:36.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.044+0000 7f322c62c700 1 -- 172.21.15.116:0/3891410940 shutdown_connections 2024-03-04T15:54:36.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.044+0000 7f322c62c700 1 --2- 172.21.15.116:0/3891410940 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f3210038330 0x7f321003a7f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:36.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.044+0000 7f322c62c700 1 --2- 172.21.15.116:0/3891410940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3224106f20 0x7f322419a110 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:36.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.044+0000 7f322c62c700 1 -- 172.21.15.116:0/3891410940 >> 172.21.15.116:0/3891410940 conn(0x7f32241008d0 msgr2=0x7f32241016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:36.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.044+0000 7f322c62c700 1 -- 172.21.15.116:0/3891410940 shutdown_connections 2024-03-04T15:54:36.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.044+0000 7f322c62c700 1 -- 172.21.15.116:0/3891410940 wait complete. 2024-03-04T15:54:36.520 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying ceph-exporter service with default placement... 2024-03-04T15:54:36.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:36 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:36.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:36 smithi116 ceph-mon[29120]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:36.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:36 smithi116 ceph-mon[29120]: Saving service crash spec with placement * 2024-03-04T15:54:36.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:36 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:38.427 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:38 smithi116 ceph-mon[29120]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:38.427 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:38 smithi116 ceph-mon[29120]: Saving service ceph-exporter spec with placement * 2024-03-04T15:54:38.427 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:38 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:38.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-03-04T15:54:38.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.915+0000 7f2b64053700 1 Processor -- start 2024-03-04T15:54:38.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.916+0000 7f2b64053700 1 -- start start 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.916+0000 7f2b64053700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c105130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.916+0000 7f2b64053700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c079e40 con 0x7f2b5c104d10 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.916+0000 7f2b61def700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c105130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.916+0000 7f2b61def700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c105130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50998/0 (socket says 172.21.15.116:50998) 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.916+0000 7f2b61def700 1 -- 172.21.15.116:0/1137838498 learned_addr learned my addr 172.21.15.116:0/1137838498 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.917+0000 7f2b61def700 1 -- 172.21.15.116:0/1137838498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c105670 con 0x7f2b5c104d10 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.917+0000 7f2b61def700 1 --2- 172.21.15.116:0/1137838498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c105130 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0098a0 tx=0x7f2b4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fdd2345feff5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.917+0000 7f2b60ded700 1 -- 172.21.15.116:0/1137838498 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b4c004180 con 0x7f2b5c104d10 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.917+0000 7f2b60ded700 1 -- 172.21.15.116:0/1137838498 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2b4c0042e0 con 0x7f2b5c104d10 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.917+0000 7f2b60ded700 1 -- 172.21.15.116:0/1137838498 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b4c011af0 con 0x7f2b5c104d10 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 -- 172.21.15.116:0/1137838498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 msgr2=0x7f2b5c105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 --2- 172.21.15.116:0/1137838498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c105130 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0098a0 tx=0x7f2b4c009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 -- 172.21.15.116:0/1137838498 shutdown_connections 2024-03-04T15:54:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 --2- 172.21.15.116:0/1137838498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c105130 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 -- 172.21.15.116:0/1137838498 >> 172.21.15.116:0/1137838498 conn(0x7f2b5c1008b0 msgr2=0x7f2b5c102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 -- 172.21.15.116:0/1137838498 shutdown_connections 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 -- 172.21.15.116:0/1137838498 wait complete. 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 Processor -- start 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 -- start start 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.918+0000 7f2b64053700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c19c870 con 0x7f2b5c104d10 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.919+0000 7f2b61def700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.919+0000 7f2b61def700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c19c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51004/0 (socket says 172.21.15.116:51004) 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.919+0000 7f2b61def700 1 -- 172.21.15.116:0/3159595649 learned_addr learned my addr 172.21.15.116:0/3159595649 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.919+0000 7f2b61def700 1 -- 172.21.15.116:0/3159595649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c009550 con 0x7f2b5c104d10 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.919+0000 7f2b61def700 1 --2- 172.21.15.116:0/3159595649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c19c330 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c011670 tx=0x7f2b4c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.919+0000 7f2b52ffd700 1 -- 172.21.15.116:0/3159595649 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b4c003a40 con 0x7f2b5c104d10 2024-03-04T15:54:38.608 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.919+0000 7f2b64053700 1 -- 172.21.15.116:0/3159595649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c19ca70 con 0x7f2b5c104d10 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.919+0000 7f2b52ffd700 1 -- 172.21.15.116:0/3159595649 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2b4c003ba0 con 0x7f2b5c104d10 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.919+0000 7f2b52ffd700 1 -- 172.21.15.116:0/3159595649 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b4c024790 con 0x7f2b5c104d10 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.919+0000 7f2b64053700 1 -- 172.21.15.116:0/3159595649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c19cf10 con 0x7f2b5c104d10 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.920+0000 7f2b52ffd700 1 -- 172.21.15.116:0/3159595649 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f2b4c01a430 con 0x7f2b5c104d10 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.920+0000 7f2b64053700 1 -- 172.21.15.116:0/3159595649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c195c00 con 0x7f2b5c104d10 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.920+0000 7f2b52ffd700 1 --2- 172.21.15.116:0/3159595649 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f2b48038320 0x7f2b4803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.920+0000 7f2b52ffd700 1 -- 172.21.15.116:0/3159595649 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2b4c04c520 con 0x7f2b5c104d10 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.920+0000 7f2b615ee700 1 --2- 172.21.15.116:0/3159595649 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f2b48038320 0x7f2b4803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.921+0000 7f2b615ee700 1 --2- 172.21.15.116:0/3159595649 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f2b48038320 0x7f2b4803a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2b58006fd0 tx=0x7f2b58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:36.925+0000 7f2b52ffd700 1 -- 172.21.15.116:0/3159595649 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2b4c018b40 con 0x7f2b5c104d10 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.141+0000 7f2b64053700 1 -- 172.21.15.116:0/3159595649 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b5c0611d0 con 0x7f2b48038320 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.146+0000 7f2b52ffd700 1 -- 172.21.15.116:0/3159595649 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f2b5c0611d0 con 0x7f2b48038320 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.148+0000 7f2b64053700 1 -- 172.21.15.116:0/3159595649 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f2b48038320 msgr2=0x7f2b4803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:38.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.148+0000 7f2b64053700 1 --2- 172.21.15.116:0/3159595649 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f2b48038320 0x7f2b4803a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2b58006fd0 tx=0x7f2b58006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:38.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.148+0000 7f2b64053700 1 -- 172.21.15.116:0/3159595649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 msgr2=0x7f2b5c19c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:38.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.148+0000 7f2b64053700 1 --2- 172.21.15.116:0/3159595649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c19c330 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c011670 tx=0x7f2b4c011b50 comp rx=0 tx=0).stop 2024-03-04T15:54:38.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.148+0000 7f2b64053700 1 -- 172.21.15.116:0/3159595649 shutdown_connections 2024-03-04T15:54:38.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.149+0000 7f2b64053700 1 --2- 172.21.15.116:0/3159595649 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f2b48038320 0x7f2b4803a7e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:38.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.149+0000 7f2b64053700 1 --2- 172.21.15.116:0/3159595649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b5c104d10 0x7f2b5c19c330 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:38.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.149+0000 7f2b64053700 1 -- 172.21.15.116:0/3159595649 >> 172.21.15.116:0/3159595649 conn(0x7f2b5c1008b0 msgr2=0x7f2b5c1908d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:38.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.149+0000 7f2b64053700 1 -- 172.21.15.116:0/3159595649 shutdown_connections 2024-03-04T15:54:38.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:37.149+0000 7f2b64053700 1 -- 172.21.15.116:0/3159595649 wait complete. 2024-03-04T15:54:38.610 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying prometheus service with default placement... 2024-03-04T15:54:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-04T15:54:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.994+0000 7fbd58739700 1 Processor -- start 2024-03-04T15:54:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.994+0000 7fbd58739700 1 -- start start 2024-03-04T15:54:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.995+0000 7fbd58739700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50104b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.995+0000 7fbd58739700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd50105120 con 0x7fbd50104730 2024-03-04T15:54:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.995+0000 7fbd564d5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50104b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.995+0000 7fbd564d5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50104b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51014/0 (socket says 172.21.15.116:51014) 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.995+0000 7fbd564d5700 1 -- 172.21.15.116:0/3029975155 learned_addr learned my addr 172.21.15.116:0/3029975155 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.995+0000 7fbd564d5700 1 -- 172.21.15.116:0/3029975155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd501076c0 con 0x7fbd50104730 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.995+0000 7fbd564d5700 1 --2- 172.21.15.116:0/3029975155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50104b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0098a0 tx=0x7fbd4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30f2b50658f2992 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.996+0000 7fbd554d3700 1 -- 172.21.15.116:0/3029975155 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd4c004180 con 0x7fbd50104730 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.996+0000 7fbd554d3700 1 -- 172.21.15.116:0/3029975155 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbd4c0042e0 con 0x7fbd50104730 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.996+0000 7fbd554d3700 1 -- 172.21.15.116:0/3029975155 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd4c011af0 con 0x7fbd50104730 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.996+0000 7fbd58739700 1 -- 172.21.15.116:0/3029975155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 msgr2=0x7fbd50104b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.996+0000 7fbd58739700 1 --2- 172.21.15.116:0/3029975155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50104b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0098a0 tx=0x7fbd4c009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.996+0000 7fbd58739700 1 -- 172.21.15.116:0/3029975155 shutdown_connections 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.996+0000 7fbd58739700 1 --2- 172.21.15.116:0/3029975155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50104b50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.996+0000 7fbd58739700 1 -- 172.21.15.116:0/3029975155 >> 172.21.15.116:0/3029975155 conn(0x7fbd50100040 msgr2=0x7fbd501024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.996+0000 7fbd58739700 1 -- 172.21.15.116:0/3029975155 shutdown_connections 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.996+0000 7fbd58739700 1 -- 172.21.15.116:0/3029975155 wait complete. 2024-03-04T15:54:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.997+0000 7fbd58739700 1 Processor -- start 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.997+0000 7fbd58739700 1 -- start start 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.997+0000 7fbd58739700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50107020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.997+0000 7fbd58739700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd50107560 con 0x7fbd50104730 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.997+0000 7fbd564d5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.997+0000 7fbd564d5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50107020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51020/0 (socket says 172.21.15.116:51020) 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.997+0000 7fbd564d5700 1 -- 172.21.15.116:0/2607688231 learned_addr learned my addr 172.21.15.116:0/2607688231 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.997+0000 7fbd564d5700 1 -- 172.21.15.116:0/2607688231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c009550 con 0x7fbd50104730 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.998+0000 7fbd564d5700 1 --2- 172.21.15.116:0/2607688231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50107020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c011670 tx=0x7fbd4c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.998+0000 7fbd477fe700 1 -- 172.21.15.116:0/2607688231 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd4c0039c0 con 0x7fbd50104730 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.998+0000 7fbd477fe700 1 -- 172.21.15.116:0/2607688231 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbd4c003b20 con 0x7fbd50104730 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.998+0000 7fbd58739700 1 -- 172.21.15.116:0/2607688231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd50105670 con 0x7fbd50104730 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.998+0000 7fbd477fe700 1 -- 172.21.15.116:0/2607688231 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd4c0247f0 con 0x7fbd50104730 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.998+0000 7fbd58739700 1 -- 172.21.15.116:0/2607688231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd50105b10 con 0x7fbd50104730 2024-03-04T15:54:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.999+0000 7fbd58739700 1 -- 172.21.15.116:0/2607688231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd500623c0 con 0x7fbd50104730 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.999+0000 7fbd477fe700 1 -- 172.21.15.116:0/2607688231 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fbd4c01ac70 con 0x7fbd50104730 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.999+0000 7fbd477fe700 1 --2- 172.21.15.116:0/2607688231 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbd3c038330 0x7fbd3c03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.999+0000 7fbd477fe700 1 -- 172.21.15.116:0/2607688231 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbd4c04c270 con 0x7fbd50104730 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.999+0000 7fbd55cd4700 1 --2- 172.21.15.116:0/2607688231 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbd3c038330 0x7fbd3c03a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:38.999+0000 7fbd55cd4700 1 --2- 172.21.15.116:0/2607688231 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbd3c038330 0x7fbd3c03a7f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbd40006fd0 tx=0x7fbd40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.004+0000 7fbd477fe700 1 -- 172.21.15.116:0/2607688231 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbd4c018740 con 0x7fbd50104730 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.204+0000 7fbd58739700 1 -- 172.21.15.116:0/2607688231 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd5010a8c0 con 0x7fbd3c038330 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.208+0000 7fbd477fe700 1 -- 172.21.15.116:0/2607688231 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fbd5010a8c0 con 0x7fbd3c038330 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.210+0000 7fbd58739700 1 -- 172.21.15.116:0/2607688231 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbd3c038330 msgr2=0x7fbd3c03a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.210+0000 7fbd58739700 1 --2- 172.21.15.116:0/2607688231 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbd3c038330 0x7fbd3c03a7f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbd40006fd0 tx=0x7fbd40006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.210+0000 7fbd58739700 1 -- 172.21.15.116:0/2607688231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 msgr2=0x7fbd50107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.210+0000 7fbd58739700 1 --2- 172.21.15.116:0/2607688231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50107020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c011670 tx=0x7fbd4c011aa0 comp rx=0 tx=0).stop 2024-03-04T15:54:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.211+0000 7fbd58739700 1 -- 172.21.15.116:0/2607688231 shutdown_connections 2024-03-04T15:54:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.211+0000 7fbd58739700 1 --2- 172.21.15.116:0/2607688231 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fbd3c038330 0x7fbd3c03a7f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.211+0000 7fbd58739700 1 --2- 172.21.15.116:0/2607688231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd50104730 0x7fbd50107020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.211+0000 7fbd58739700 1 -- 172.21.15.116:0/2607688231 >> 172.21.15.116:0/2607688231 conn(0x7fbd50100040 msgr2=0x7fbd50101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.211+0000 7fbd58739700 1 -- 172.21.15.116:0/2607688231 shutdown_connections 2024-03-04T15:54:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:39.211+0000 7fbd58739700 1 -- 172.21.15.116:0/2607688231 wait complete. 2024-03-04T15:54:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying grafana service with default placement... 2024-03-04T15:54:40.431 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:40 smithi116 ceph-mon[29120]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:40.431 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:40 smithi116 ceph-mon[29120]: Saving service prometheus spec with placement count:1 2024-03-04T15:54:40.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:40 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:40.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-04T15:54:40.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.236+0000 7f5d14719700 1 Processor -- start 2024-03-04T15:54:40.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.236+0000 7f5d14719700 1 -- start start 2024-03-04T15:54:40.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.237+0000 7f5d14719700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c105150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.237+0000 7f5d14719700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c079e40 con 0x7f5d0c104d30 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.237+0000 7f5d124b5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c105150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.237+0000 7f5d124b5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c105150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51032/0 (socket says 172.21.15.116:51032) 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.237+0000 7f5d124b5700 1 -- 172.21.15.116:0/4105126917 learned_addr learned my addr 172.21.15.116:0/4105126917 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.237+0000 7f5d124b5700 1 -- 172.21.15.116:0/4105126917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c105690 con 0x7f5d0c104d30 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.237+0000 7f5d124b5700 1 --2- 172.21.15.116:0/4105126917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c105150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5d080098a0 tx=0x7f5d08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ba8c7df1d1c279 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.237+0000 7f5d114b3700 1 -- 172.21.15.116:0/4105126917 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d08004180 con 0x7f5d0c104d30 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.237+0000 7f5d114b3700 1 -- 172.21.15.116:0/4105126917 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5d080042e0 con 0x7f5d0c104d30 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.237+0000 7f5d114b3700 1 -- 172.21.15.116:0/4105126917 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d08011af0 con 0x7f5d0c104d30 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.238+0000 7f5d14719700 1 -- 172.21.15.116:0/4105126917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 msgr2=0x7f5d0c105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.238+0000 7f5d14719700 1 --2- 172.21.15.116:0/4105126917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c105150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5d080098a0 tx=0x7f5d08009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.238+0000 7f5d14719700 1 -- 172.21.15.116:0/4105126917 shutdown_connections 2024-03-04T15:54:40.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.238+0000 7f5d14719700 1 --2- 172.21.15.116:0/4105126917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c105150 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.238+0000 7f5d14719700 1 -- 172.21.15.116:0/4105126917 >> 172.21.15.116:0/4105126917 conn(0x7f5d0c1008d0 msgr2=0x7f5d0c102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.238+0000 7f5d14719700 1 -- 172.21.15.116:0/4105126917 shutdown_connections 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.238+0000 7f5d14719700 1 -- 172.21.15.116:0/4105126917 wait complete. 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.239+0000 7f5d14719700 1 Processor -- start 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.239+0000 7f5d14719700 1 -- start start 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.239+0000 7f5d14719700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.239+0000 7f5d14719700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c19c870 con 0x7f5d0c104d30 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.239+0000 7f5d124b5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.239+0000 7f5d124b5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c19c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51048/0 (socket says 172.21.15.116:51048) 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.239+0000 7f5d124b5700 1 -- 172.21.15.116:0/3230830009 learned_addr learned my addr 172.21.15.116:0/3230830009 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.239+0000 7f5d124b5700 1 -- 172.21.15.116:0/3230830009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d08009550 con 0x7f5d0c104d30 2024-03-04T15:54:40.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.239+0000 7f5d124b5700 1 --2- 172.21.15.116:0/3230830009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c19c330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5d08011670 tx=0x7f5d08011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.239+0000 7f5d037fe700 1 -- 172.21.15.116:0/3230830009 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d08003a40 con 0x7f5d0c104d30 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.240+0000 7f5d037fe700 1 -- 172.21.15.116:0/3230830009 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5d08003ba0 con 0x7f5d0c104d30 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.240+0000 7f5d14719700 1 -- 172.21.15.116:0/3230830009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0c19ca70 con 0x7f5d0c104d30 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.240+0000 7f5d037fe700 1 -- 172.21.15.116:0/3230830009 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d08024820 con 0x7f5d0c104d30 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.240+0000 7f5d14719700 1 -- 172.21.15.116:0/3230830009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0c19cf10 con 0x7f5d0c104d30 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.240+0000 7f5d037fe700 1 -- 172.21.15.116:0/3230830009 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f5d08024ae0 con 0x7f5d0c104d30 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.240+0000 7f5d14719700 1 -- 172.21.15.116:0/3230830009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d0c195bb0 con 0x7f5d0c104d30 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.241+0000 7f5d037fe700 1 --2- 172.21.15.116:0/3230830009 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5cf80382e0 0x7f5cf803a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.241+0000 7f5d037fe700 1 -- 172.21.15.116:0/3230830009 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5d0804c300 con 0x7f5d0c104d30 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.241+0000 7f5d11cb4700 1 --2- 172.21.15.116:0/3230830009 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5cf80382e0 0x7f5cf803a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.241+0000 7f5d11cb4700 1 --2- 172.21.15.116:0/3230830009 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5cf80382e0 0x7f5cf803a7a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc006fd0 tx=0x7f5cfc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.245+0000 7f5d037fe700 1 -- 172.21.15.116:0/3230830009 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5d08024d90 con 0x7f5d0c104d30 2024-03-04T15:54:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.450+0000 7f5d14719700 1 -- 172.21.15.116:0/3230830009 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d0c0611d0 con 0x7f5cf80382e0 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.456+0000 7f5d037fe700 1 -- 172.21.15.116:0/3230830009 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5d0c0611d0 con 0x7f5cf80382e0 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.458+0000 7f5d14719700 1 -- 172.21.15.116:0/3230830009 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5cf80382e0 msgr2=0x7f5cf803a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.458+0000 7f5d14719700 1 --2- 172.21.15.116:0/3230830009 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5cf80382e0 0x7f5cf803a7a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc006fd0 tx=0x7f5cfc006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.458+0000 7f5d14719700 1 -- 172.21.15.116:0/3230830009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 msgr2=0x7f5d0c19c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.458+0000 7f5d14719700 1 --2- 172.21.15.116:0/3230830009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c19c330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5d08011670 tx=0x7f5d08011aa0 comp rx=0 tx=0).stop 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.458+0000 7f5d14719700 1 -- 172.21.15.116:0/3230830009 shutdown_connections 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.458+0000 7f5d14719700 1 --2- 172.21.15.116:0/3230830009 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f5cf80382e0 0x7f5cf803a7a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.458+0000 7f5d14719700 1 --2- 172.21.15.116:0/3230830009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0c104d30 0x7f5d0c19c330 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.458+0000 7f5d14719700 1 -- 172.21.15.116:0/3230830009 >> 172.21.15.116:0/3230830009 conn(0x7f5d0c1008d0 msgr2=0x7f5d0c1908f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.458+0000 7f5d14719700 1 -- 172.21.15.116:0/3230830009 shutdown_connections 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:40.458+0000 7f5d14719700 1 -- 172.21.15.116:0/3230830009 wait complete. 2024-03-04T15:54:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying node-exporter service with default placement... 2024-03-04T15:54:41.611 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:41 smithi116 ceph-mon[29120]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:41.611 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:41 smithi116 ceph-mon[29120]: Saving service grafana spec with placement count:1 2024-03-04T15:54:41.611 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:41 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.390+0000 7f77afb36700 1 Processor -- start 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.390+0000 7f77afb36700 1 -- start start 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.390+0000 7f77afb36700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a8106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.390+0000 7f77afb36700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a8106e20 con 0x7f77a8106430 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.391+0000 7f77ad8d2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a8106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.391+0000 7f77ad8d2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a8106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51050/0 (socket says 172.21.15.116:51050) 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.391+0000 7f77ad8d2700 1 -- 172.21.15.116:0/187744448 learned_addr learned my addr 172.21.15.116:0/187744448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.391+0000 7f77ad8d2700 1 -- 172.21.15.116:0/187744448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a8107630 con 0x7f77a8106430 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.391+0000 7f77ad8d2700 1 --2- 172.21.15.116:0/187744448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a8106850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7798009e80 tx=0x7f779800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2611fa76345df5d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:42.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.391+0000 7f77ac8d0700 1 -- 172.21.15.116:0/187744448 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f779800e070 con 0x7f77a8106430 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.391+0000 7f77ac8d0700 1 -- 172.21.15.116:0/187744448 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f779800b8c0 con 0x7f77a8106430 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.391+0000 7f77ac8d0700 1 -- 172.21.15.116:0/187744448 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77980044d0 con 0x7f77a8106430 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.392+0000 7f77afb36700 1 -- 172.21.15.116:0/187744448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 msgr2=0x7f77a8106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.392+0000 7f77afb36700 1 --2- 172.21.15.116:0/187744448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a8106850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7798009e80 tx=0x7f779800b330 comp rx=0 tx=0).stop 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.392+0000 7f77afb36700 1 -- 172.21.15.116:0/187744448 shutdown_connections 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.392+0000 7f77afb36700 1 --2- 172.21.15.116:0/187744448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a8106850 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.392+0000 7f77afb36700 1 -- 172.21.15.116:0/187744448 >> 172.21.15.116:0/187744448 conn(0x7f77a81019d0 msgr2=0x7f77a8103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.392+0000 7f77afb36700 1 -- 172.21.15.116:0/187744448 shutdown_connections 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.392+0000 7f77afb36700 1 -- 172.21.15.116:0/187744448 wait complete. 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.392+0000 7f77afb36700 1 Processor -- start 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.392+0000 7f77afb36700 1 -- start start 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77afb36700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a807bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77afb36700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a807c1a0 con 0x7f77a8106430 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77ad8d2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a807bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77ad8d2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a807bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51064/0 (socket says 172.21.15.116:51064) 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77ad8d2700 1 -- 172.21.15.116:0/641022748 learned_addr learned my addr 172.21.15.116:0/641022748 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77ad8d2700 1 -- 172.21.15.116:0/641022748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798009b30 con 0x7f77a8106430 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77ad8d2700 1 --2- 172.21.15.116:0/641022748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a807bc60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7798004940 tx=0x7f7798004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77a6ffd700 1 -- 172.21.15.116:0/641022748 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f779800e040 con 0x7f77a8106430 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77afb36700 1 -- 172.21.15.116:0/641022748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a807a2b0 con 0x7f77a8106430 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77a6ffd700 1 -- 172.21.15.116:0/641022748 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f779800bc40 con 0x7f77a8106430 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77a6ffd700 1 -- 172.21.15.116:0/641022748 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77980126e0 con 0x7f77a8106430 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.393+0000 7f77afb36700 1 -- 172.21.15.116:0/641022748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a807a7b0 con 0x7f77a8106430 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.394+0000 7f77a6ffd700 1 -- 172.21.15.116:0/641022748 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f7798019040 con 0x7f77a8106430 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.394+0000 7f77afb36700 1 -- 172.21.15.116:0/641022748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a80623c0 con 0x7f77a8106430 2024-03-04T15:54:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.394+0000 7f77a6ffd700 1 --2- 172.21.15.116:0/641022748 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f7790038320 0x7f779003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.394+0000 7f77a6ffd700 1 -- 172.21.15.116:0/641022748 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f779804be80 con 0x7f77a8106430 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.394+0000 7f77ad0d1700 1 --2- 172.21.15.116:0/641022748 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f7790038320 0x7f779003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.395+0000 7f77ad0d1700 1 --2- 172.21.15.116:0/641022748 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f7790038320 0x7f779003a7e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f779c006fd0 tx=0x7f779c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.399+0000 7f77a6ffd700 1 -- 172.21.15.116:0/641022748 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f779802cd20 con 0x7f77a8106430 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.602+0000 7f77afb36700 1 -- 172.21.15.116:0/641022748 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f77a8102ea0 con 0x7f7790038320 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.606+0000 7f77a6ffd700 1 -- 172.21.15.116:0/641022748 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f77a8102ea0 con 0x7f7790038320 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.608+0000 7f77afb36700 1 -- 172.21.15.116:0/641022748 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f7790038320 msgr2=0x7f779003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.608+0000 7f77afb36700 1 --2- 172.21.15.116:0/641022748 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f7790038320 0x7f779003a7e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f779c006fd0 tx=0x7f779c006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.608+0000 7f77afb36700 1 -- 172.21.15.116:0/641022748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 msgr2=0x7f77a807bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.608+0000 7f77afb36700 1 --2- 172.21.15.116:0/641022748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a807bc60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7798004940 tx=0x7f7798004a20 comp rx=0 tx=0).stop 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.608+0000 7f77afb36700 1 -- 172.21.15.116:0/641022748 shutdown_connections 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.608+0000 7f77afb36700 1 --2- 172.21.15.116:0/641022748 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f7790038320 0x7f779003a7e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.608+0000 7f77afb36700 1 --2- 172.21.15.116:0/641022748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a8106430 0x7f77a807bc60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.608+0000 7f77afb36700 1 -- 172.21.15.116:0/641022748 >> 172.21.15.116:0/641022748 conn(0x7f77a81019d0 msgr2=0x7f77a8102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:42.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.608+0000 7f77afb36700 1 -- 172.21.15.116:0/641022748 shutdown_connections 2024-03-04T15:54:42.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:41.608+0000 7f77afb36700 1 -- 172.21.15.116:0/641022748 wait complete. 2024-03-04T15:54:42.202 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying alertmanager service with default placement... 2024-03-04T15:54:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:42 smithi116 ceph-mon[29120]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:54:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:42 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:43.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-04T15:54:43.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.600+0000 7fe044c98700 1 Processor -- start 2024-03-04T15:54:43.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.600+0000 7fe044c98700 1 -- start start 2024-03-04T15:54:43.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.600+0000 7fe044c98700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe040104b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:43.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.600+0000 7fe044c98700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0401050f0 con 0x7fe040104700 2024-03-04T15:54:43.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.601+0000 7fe03e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe040104b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:43.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.601+0000 7fe03e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe040104b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51072/0 (socket says 172.21.15.116:51072) 2024-03-04T15:54:43.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.601+0000 7fe03e59c700 1 -- 172.21.15.116:0/3724557280 learned_addr learned my addr 172.21.15.116:0/3724557280 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.601+0000 7fe03e59c700 1 -- 172.21.15.116:0/3724557280 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe040105230 con 0x7fe040104700 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.601+0000 7fe03e59c700 1 --2- 172.21.15.116:0/3724557280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe040104b20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe0300098a0 tx=0x7fe030009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a5ae0d51f8b88e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.601+0000 7fe03dd9b700 1 -- 172.21.15.116:0/3724557280 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe030004180 con 0x7fe040104700 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.601+0000 7fe03dd9b700 1 -- 172.21.15.116:0/3724557280 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe0300042e0 con 0x7fe040104700 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.601+0000 7fe03dd9b700 1 -- 172.21.15.116:0/3724557280 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe030011af0 con 0x7fe040104700 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.602+0000 7fe044c98700 1 -- 172.21.15.116:0/3724557280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 msgr2=0x7fe040104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.602+0000 7fe044c98700 1 --2- 172.21.15.116:0/3724557280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe040104b20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe0300098a0 tx=0x7fe030009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.602+0000 7fe044c98700 1 -- 172.21.15.116:0/3724557280 shutdown_connections 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.602+0000 7fe044c98700 1 --2- 172.21.15.116:0/3724557280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe040104b20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.602+0000 7fe044c98700 1 -- 172.21.15.116:0/3724557280 >> 172.21.15.116:0/3724557280 conn(0x7fe040100040 msgr2=0x7fe0401024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.602+0000 7fe044c98700 1 -- 172.21.15.116:0/3724557280 shutdown_connections 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.602+0000 7fe044c98700 1 -- 172.21.15.116:0/3724557280 wait complete. 2024-03-04T15:54:43.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.602+0000 7fe044c98700 1 Processor -- start 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe044c98700 1 -- start start 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe044c98700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe04019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe044c98700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04019a6f0 con 0x7fe040104700 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe03e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe04019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe03e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe04019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51086/0 (socket says 172.21.15.116:51086) 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe03e59c700 1 -- 172.21.15.116:0/1693384542 learned_addr learned my addr 172.21.15.116:0/1693384542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe03e59c700 1 -- 172.21.15.116:0/1693384542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe030009550 con 0x7fe040104700 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe03e59c700 1 --2- 172.21.15.116:0/1693384542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe04019a1b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe030011670 tx=0x7fe030011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe0377fe700 1 -- 172.21.15.116:0/1693384542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0300039c0 con 0x7fe040104700 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe044c98700 1 -- 172.21.15.116:0/1693384542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04019a8f0 con 0x7fe040104700 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe0377fe700 1 -- 172.21.15.116:0/1693384542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe030003b20 con 0x7fe040104700 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe0377fe700 1 -- 172.21.15.116:0/1693384542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0300247f0 con 0x7fe040104700 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.603+0000 7fe044c98700 1 -- 172.21.15.116:0/1693384542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04019ad90 con 0x7fe040104700 2024-03-04T15:54:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.604+0000 7fe0377fe700 1 -- 172.21.15.116:0/1693384542 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fe03001a430 con 0x7fe040104700 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.604+0000 7fe0377fe700 1 --2- 172.21.15.116:0/1693384542 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe02c0382e0 0x7fe02c03a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.604+0000 7fe0377fe700 1 -- 172.21.15.116:0/1693384542 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe03004c2e0 con 0x7fe040104700 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.604+0000 7fe044c98700 1 -- 172.21.15.116:0/1693384542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0400623c0 con 0x7fe040104700 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.605+0000 7fe037fff700 1 --2- 172.21.15.116:0/1693384542 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe02c0382e0 0x7fe02c03a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.605+0000 7fe037fff700 1 --2- 172.21.15.116:0/1693384542 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe02c0382e0 0x7fe02c03a7a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe028006fd0 tx=0x7fe028006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.609+0000 7fe0377fe700 1 -- 172.21.15.116:0/1693384542 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe03001a6e0 con 0x7fe040104700 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.813+0000 7fe044c98700 1 -- 172.21.15.116:0/1693384542 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fe040102390 con 0x7fe02c0382e0 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.818+0000 7fe0377fe700 1 -- 172.21.15.116:0/1693384542 <== mgr.14120 v2:172.21.15.116:6800/1340152843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fe040102390 con 0x7fe02c0382e0 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.820+0000 7fe044c98700 1 -- 172.21.15.116:0/1693384542 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe02c0382e0 msgr2=0x7fe02c03a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.820+0000 7fe044c98700 1 --2- 172.21.15.116:0/1693384542 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe02c0382e0 0x7fe02c03a7a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe028006fd0 tx=0x7fe028006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.820+0000 7fe044c98700 1 -- 172.21.15.116:0/1693384542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 msgr2=0x7fe04019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.820+0000 7fe044c98700 1 --2- 172.21.15.116:0/1693384542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe04019a1b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe030011670 tx=0x7fe030011aa0 comp rx=0 tx=0).stop 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.820+0000 7fe044c98700 1 -- 172.21.15.116:0/1693384542 shutdown_connections 2024-03-04T15:54:43.382 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.820+0000 7fe044c98700 1 --2- 172.21.15.116:0/1693384542 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fe02c0382e0 0x7fe02c03a7a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:43.383 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.820+0000 7fe044c98700 1 --2- 172.21.15.116:0/1693384542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe040104700 0x7fe04019a1b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:43.383 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.820+0000 7fe044c98700 1 -- 172.21.15.116:0/1693384542 >> 172.21.15.116:0/1693384542 conn(0x7fe040100040 msgr2=0x7fe040101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:43.383 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.820+0000 7fe044c98700 1 -- 172.21.15.116:0/1693384542 shutdown_connections 2024-03-04T15:54:43.383 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:42.820+0000 7fe044c98700 1 -- 172.21.15.116:0/1693384542 wait complete. 2024-03-04T15:54:43.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:43 smithi116 ceph-mon[29120]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:43.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:43 smithi116 ceph-mon[29120]: Saving service node-exporter spec with placement * 2024-03-04T15:54:43.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:43 smithi116 ceph-mon[29120]: from='mgr.14120 172.21.15.116:0/4225898151' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:44.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.774+0000 7f169e05e700 1 Processor -- start 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.774+0000 7f169e05e700 1 -- start start 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.775+0000 7f169e05e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f1698106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.775+0000 7f169e05e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1698106e20 con 0x7f1698106430 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.775+0000 7f16977fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f1698106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.775+0000 7f16977fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f1698106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51098/0 (socket says 172.21.15.116:51098) 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.775+0000 7f16977fe700 1 -- 172.21.15.116:0/4109577022 learned_addr learned my addr 172.21.15.116:0/4109577022 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.775+0000 7f16977fe700 1 -- 172.21.15.116:0/4109577022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1698107630 con 0x7f1698106430 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.775+0000 7f16977fe700 1 --2- 172.21.15.116:0/4109577022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f1698106850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f16880098a0 tx=0x7f1688009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756ecad823d4eb23 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.775+0000 7f1696ffd700 1 -- 172.21.15.116:0/4109577022 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1688004180 con 0x7f1698106430 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.776+0000 7f1696ffd700 1 -- 172.21.15.116:0/4109577022 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f16880042e0 con 0x7f1698106430 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.776+0000 7f1696ffd700 1 -- 172.21.15.116:0/4109577022 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1688011af0 con 0x7f1698106430 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.776+0000 7f169e05e700 1 -- 172.21.15.116:0/4109577022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 msgr2=0x7f1698106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:44.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.776+0000 7f169e05e700 1 --2- 172.21.15.116:0/4109577022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f1698106850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f16880098a0 tx=0x7f1688009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.776+0000 7f169e05e700 1 -- 172.21.15.116:0/4109577022 shutdown_connections 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.776+0000 7f169e05e700 1 --2- 172.21.15.116:0/4109577022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f1698106850 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.776+0000 7f169e05e700 1 -- 172.21.15.116:0/4109577022 >> 172.21.15.116:0/4109577022 conn(0x7f16981019b0 msgr2=0x7f1698103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.776+0000 7f169e05e700 1 -- 172.21.15.116:0/4109577022 shutdown_connections 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.776+0000 7f169e05e700 1 -- 172.21.15.116:0/4109577022 wait complete. 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.777+0000 7f169e05e700 1 Processor -- start 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.777+0000 7f169e05e700 1 -- start start 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.777+0000 7f169e05e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f169819c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.777+0000 7f169e05e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169819c900 con 0x7f1698106430 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.777+0000 7f16977fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f169819c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.777+0000 7f16977fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f169819c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51106/0 (socket says 172.21.15.116:51106) 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.777+0000 7f16977fe700 1 -- 172.21.15.116:0/2044320289 learned_addr learned my addr 172.21.15.116:0/2044320289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.777+0000 7f16977fe700 1 -- 172.21.15.116:0/2044320289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1688009550 con 0x7f1698106430 2024-03-04T15:54:44.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.777+0000 7f16977fe700 1 --2- 172.21.15.116:0/2044320289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f169819c3c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1688011670 tx=0x7f1688011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.778+0000 7f16957fa700 1 -- 172.21.15.116:0/2044320289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16880039f0 con 0x7f1698106430 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.778+0000 7f16957fa700 1 -- 172.21.15.116:0/2044320289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1688003b50 con 0x7f1698106430 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.778+0000 7f169e05e700 1 -- 172.21.15.116:0/2044320289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169819cb00 con 0x7f1698106430 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.778+0000 7f16957fa700 1 -- 172.21.15.116:0/2044320289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16880247c0 con 0x7f1698106430 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.778+0000 7f169e05e700 1 -- 172.21.15.116:0/2044320289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169819cfa0 con 0x7f1698106430 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.778+0000 7f16957fa700 1 -- 172.21.15.116:0/2044320289 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f168802d3f0 con 0x7f1698106430 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.779+0000 7f16957fa700 1 --2- 172.21.15.116:0/2044320289 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f1684038320 0x7f168403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.779+0000 7f16957fa700 1 -- 172.21.15.116:0/2044320289 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f168804c3b0 con 0x7f1698106430 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.779+0000 7f169e05e700 1 -- 172.21.15.116:0/2044320289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16980623c0 con 0x7f1698106430 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.779+0000 7f168effd700 1 --2- 172.21.15.116:0/2044320289 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f1684038320 0x7f168403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.779+0000 7f168effd700 1 --2- 172.21.15.116:0/2044320289 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f1684038320 0x7f168403a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1680006fd0 tx=0x7f1680006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.784+0000 7f16957fa700 1 -- 172.21.15.116:0/2044320289 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f168802d6f0 con 0x7f1698106430 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.969+0000 7f169e05e700 1 -- 172.21.15.116:0/2044320289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f169810a860 con 0x7f1698106430 2024-03-04T15:54:44.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.975+0000 7f16957fa700 1 -- 172.21.15.116:0/2044320289 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1688020220 con 0x7f1698106430 2024-03-04T15:54:44.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.977+0000 7f169e05e700 1 -- 172.21.15.116:0/2044320289 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f1684038320 msgr2=0x7f168403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:44.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.977+0000 7f169e05e700 1 --2- 172.21.15.116:0/2044320289 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f1684038320 0x7f168403a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1680006fd0 tx=0x7f1680006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:44.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.977+0000 7f169e05e700 1 -- 172.21.15.116:0/2044320289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 msgr2=0x7f169819c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:44.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.977+0000 7f169e05e700 1 --2- 172.21.15.116:0/2044320289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f169819c3c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1688011670 tx=0x7f1688011b50 comp rx=0 tx=0).stop 2024-03-04T15:54:44.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.977+0000 7f169e05e700 1 -- 172.21.15.116:0/2044320289 shutdown_connections 2024-03-04T15:54:44.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.977+0000 7f169e05e700 1 --2- 172.21.15.116:0/2044320289 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f1684038320 0x7f168403a7e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:44.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.977+0000 7f169e05e700 1 --2- 172.21.15.116:0/2044320289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1698106430 0x7f169819c3c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:44.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.977+0000 7f169e05e700 1 -- 172.21.15.116:0/2044320289 >> 172.21.15.116:0/2044320289 conn(0x7f16981019b0 msgr2=0x7f16981030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:44.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.977+0000 7f169e05e700 1 -- 172.21.15.116:0/2044320289 shutdown_connections 2024-03-04T15:54:44.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:43.977+0000 7f169e05e700 1 -- 172.21.15.116:0/2044320289 wait complete. 2024-03-04T15:54:44.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:44 smithi116 ceph-mon[29120]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:44.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:44 smithi116 ceph-mon[29120]: Saving service alertmanager spec with placement count:1 2024-03-04T15:54:44.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:44 smithi116 ceph-mon[29120]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:54:44.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:44 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2044320289' entity='client.admin' 2024-03-04T15:54:45.749 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.977+0000 7fa731fa5700 1 Processor -- start 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.977+0000 7fa731fa5700 1 -- start start 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.977+0000 7fa731fa5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c104b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.977+0000 7fa731fa5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c105120 con 0x7fa72c104730 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.977+0000 7fa72b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c104b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.977+0000 7fa72b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c104b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60904/0 (socket says 172.21.15.116:60904) 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.977+0000 7fa72b7fe700 1 -- 172.21.15.116:0/1111386037 learned_addr learned my addr 172.21.15.116:0/1111386037 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.977+0000 7fa72b7fe700 1 -- 172.21.15.116:0/1111386037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c1076c0 con 0x7fa72c104730 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.977+0000 7fa72b7fe700 1 --2- 172.21.15.116:0/1111386037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c104b50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa71c0098a0 tx=0x7fa71c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d51319d65dd45ef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.978+0000 7fa72a7fc700 1 -- 172.21.15.116:0/1111386037 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa71c004180 con 0x7fa72c104730 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.978+0000 7fa72a7fc700 1 -- 172.21.15.116:0/1111386037 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa71c0042e0 con 0x7fa72c104730 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.978+0000 7fa72a7fc700 1 -- 172.21.15.116:0/1111386037 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa71c011af0 con 0x7fa72c104730 2024-03-04T15:54:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.978+0000 7fa731fa5700 1 -- 172.21.15.116:0/1111386037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 msgr2=0x7fa72c104b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.978+0000 7fa731fa5700 1 --2- 172.21.15.116:0/1111386037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c104b50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa71c0098a0 tx=0x7fa71c009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.978+0000 7fa731fa5700 1 -- 172.21.15.116:0/1111386037 shutdown_connections 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.978+0000 7fa731fa5700 1 --2- 172.21.15.116:0/1111386037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c104b50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.978+0000 7fa731fa5700 1 -- 172.21.15.116:0/1111386037 >> 172.21.15.116:0/1111386037 conn(0x7fa72c100040 msgr2=0x7fa72c1024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.978+0000 7fa731fa5700 1 -- 172.21.15.116:0/1111386037 shutdown_connections 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.978+0000 7fa731fa5700 1 -- 172.21.15.116:0/1111386037 wait complete. 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.979+0000 7fa731fa5700 1 Processor -- start 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.979+0000 7fa731fa5700 1 -- start start 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.979+0000 7fa731fa5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c107020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.979+0000 7fa731fa5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c107560 con 0x7fa72c104730 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.979+0000 7fa72b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.979+0000 7fa72b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c107020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60914/0 (socket says 172.21.15.116:60914) 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.979+0000 7fa72b7fe700 1 -- 172.21.15.116:0/3184105036 learned_addr learned my addr 172.21.15.116:0/3184105036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.979+0000 7fa72b7fe700 1 -- 172.21.15.116:0/3184105036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71c009550 con 0x7fa72c104730 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.980+0000 7fa72b7fe700 1 --2- 172.21.15.116:0/3184105036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c107020 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa71c011670 tx=0x7fa71c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.980+0000 7fa728ff9700 1 -- 172.21.15.116:0/3184105036 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa71c0039f0 con 0x7fa72c104730 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.980+0000 7fa731fa5700 1 -- 172.21.15.116:0/3184105036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72c105670 con 0x7fa72c104730 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.980+0000 7fa728ff9700 1 -- 172.21.15.116:0/3184105036 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa71c003b50 con 0x7fa72c104730 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.980+0000 7fa728ff9700 1 -- 172.21.15.116:0/3184105036 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa71c0247c0 con 0x7fa72c104730 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.980+0000 7fa731fa5700 1 -- 172.21.15.116:0/3184105036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72c105b10 con 0x7fa72c104730 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.981+0000 7fa728ff9700 1 -- 172.21.15.116:0/3184105036 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa71c02d3f0 con 0x7fa72c104730 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.981+0000 7fa728ff9700 1 --2- 172.21.15.116:0/3184105036 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa714038320 0x7fa71403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.981+0000 7fa731fa5700 1 -- 172.21.15.116:0/3184105036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72c0623c0 con 0x7fa72c104730 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.981+0000 7fa728ff9700 1 -- 172.21.15.116:0/3184105036 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa71c04c3a0 con 0x7fa72c104730 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.981+0000 7fa72affd700 1 --2- 172.21.15.116:0/3184105036 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa714038320 0x7fa71403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.981+0000 7fa72affd700 1 --2- 172.21.15.116:0/3184105036 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa714038320 0x7fa71403a7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa720006fd0 tx=0x7fa720006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:45.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:44.986+0000 7fa728ff9700 1 -- 172.21.15.116:0/3184105036 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa71c010740 con 0x7fa72c104730 2024-03-04T15:54:45.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.171+0000 7fa731fa5700 1 -- 172.21.15.116:0/3184105036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fa72c106420 con 0x7fa72c104730 2024-03-04T15:54:45.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.174+0000 7fa728ff9700 1 -- 172.21.15.116:0/3184105036 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fa71c00f980 con 0x7fa72c104730 2024-03-04T15:54:45.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.177+0000 7fa731fa5700 1 -- 172.21.15.116:0/3184105036 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa714038320 msgr2=0x7fa71403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:45.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.177+0000 7fa731fa5700 1 --2- 172.21.15.116:0/3184105036 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa714038320 0x7fa71403a7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa720006fd0 tx=0x7fa720006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:45.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.177+0000 7fa731fa5700 1 -- 172.21.15.116:0/3184105036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 msgr2=0x7fa72c107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:45.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.177+0000 7fa731fa5700 1 --2- 172.21.15.116:0/3184105036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c107020 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa71c011670 tx=0x7fa71c011b50 comp rx=0 tx=0).stop 2024-03-04T15:54:45.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.177+0000 7fa731fa5700 1 -- 172.21.15.116:0/3184105036 shutdown_connections 2024-03-04T15:54:45.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.177+0000 7fa731fa5700 1 --2- 172.21.15.116:0/3184105036 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fa714038320 0x7fa71403a7e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:45.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.177+0000 7fa731fa5700 1 --2- 172.21.15.116:0/3184105036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa72c104730 0x7fa72c107020 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:45.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.177+0000 7fa731fa5700 1 -- 172.21.15.116:0/3184105036 >> 172.21.15.116:0/3184105036 conn(0x7fa72c100040 msgr2=0x7fa72c101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:45.754 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.177+0000 7fa731fa5700 1 -- 172.21.15.116:0/3184105036 shutdown_connections 2024-03-04T15:54:45.754 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:45.177+0000 7fa731fa5700 1 -- 172.21.15.116:0/3184105036 wait complete. 2024-03-04T15:54:45.754 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling the dashboard module... 2024-03-04T15:54:46.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:46 smithi116 ceph-mon[29120]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:54:46.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:46 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3184105036' entity='client.admin' 2024-03-04T15:54:47.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:47 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1011045391' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-04T15:54:47.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.225+0000 7ff98c700700 1 Processor -- start 2024-03-04T15:54:47.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.225+0000 7ff98c700700 1 -- start start 2024-03-04T15:54:47.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.226+0000 7ff98c700700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff984106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.226+0000 7ff98c700700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff984106e10 con 0x7ff984106420 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.226+0000 7ff98a49c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff984106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.226+0000 7ff98a49c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff984106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60928/0 (socket says 172.21.15.116:60928) 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.226+0000 7ff98a49c700 1 -- 172.21.15.116:0/2672825399 learned_addr learned my addr 172.21.15.116:0/2672825399 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.226+0000 7ff98a49c700 1 -- 172.21.15.116:0/2672825399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff984107620 con 0x7ff984106420 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.226+0000 7ff98a49c700 1 --2- 172.21.15.116:0/2672825399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff984106840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff9800098a0 tx=0x7ff980009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0f4270bf6bd7cf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.227+0000 7ff98949a700 1 -- 172.21.15.116:0/2672825399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff980004180 con 0x7ff984106420 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.227+0000 7ff98949a700 1 -- 172.21.15.116:0/2672825399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff9800042e0 con 0x7ff984106420 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.227+0000 7ff98949a700 1 -- 172.21.15.116:0/2672825399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff980011af0 con 0x7ff984106420 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.227+0000 7ff98c700700 1 -- 172.21.15.116:0/2672825399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 msgr2=0x7ff984106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.227+0000 7ff98c700700 1 --2- 172.21.15.116:0/2672825399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff984106840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff9800098a0 tx=0x7ff980009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.227+0000 7ff98c700700 1 -- 172.21.15.116:0/2672825399 shutdown_connections 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.227+0000 7ff98c700700 1 --2- 172.21.15.116:0/2672825399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff984106840 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:47.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.227+0000 7ff98c700700 1 -- 172.21.15.116:0/2672825399 >> 172.21.15.116:0/2672825399 conn(0x7ff984101a00 msgr2=0x7ff984103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.227+0000 7ff98c700700 1 -- 172.21.15.116:0/2672825399 shutdown_connections 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.227+0000 7ff98c700700 1 -- 172.21.15.116:0/2672825399 wait complete. 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.228+0000 7ff98c700700 1 Processor -- start 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.228+0000 7ff98c700700 1 -- start start 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.228+0000 7ff98c700700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff98419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.228+0000 7ff98c700700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98419a620 con 0x7ff984106420 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.228+0000 7ff98a49c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff98419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.228+0000 7ff98a49c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff98419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60938/0 (socket says 172.21.15.116:60938) 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.228+0000 7ff98a49c700 1 -- 172.21.15.116:0/1011045391 learned_addr learned my addr 172.21.15.116:0/1011045391 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.228+0000 7ff98a49c700 1 -- 172.21.15.116:0/1011045391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff980009550 con 0x7ff984106420 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.229+0000 7ff98a49c700 1 --2- 172.21.15.116:0/1011045391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff98419a0e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff980006ba0 tx=0x7ff980011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.229+0000 7ff9777fe700 1 -- 172.21.15.116:0/1011045391 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff980003b00 con 0x7ff984106420 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.229+0000 7ff9777fe700 1 -- 172.21.15.116:0/1011045391 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff98001a8e0 con 0x7ff984106420 2024-03-04T15:54:47.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.229+0000 7ff9777fe700 1 -- 172.21.15.116:0/1011045391 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff980019990 con 0x7ff984106420 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.229+0000 7ff98c700700 1 -- 172.21.15.116:0/1011045391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98419a820 con 0x7ff984106420 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.229+0000 7ff98c700700 1 -- 172.21.15.116:0/1011045391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98419acc0 con 0x7ff984106420 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.229+0000 7ff9777fe700 1 -- 172.21.15.116:0/1011045391 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff980019c00 con 0x7ff984106420 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.230+0000 7ff98c700700 1 -- 172.21.15.116:0/1011045391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9840623c0 con 0x7ff984106420 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.230+0000 7ff9777fe700 1 --2- 172.21.15.116:0/1011045391 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7ff970038320 0x7ff97003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.230+0000 7ff9777fe700 1 -- 172.21.15.116:0/1011045391 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff98004c6d0 con 0x7ff984106420 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.230+0000 7ff989c9b700 1 --2- 172.21.15.116:0/1011045391 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7ff970038320 0x7ff97003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.230+0000 7ff989c9b700 1 --2- 172.21.15.116:0/1011045391 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7ff970038320 0x7ff97003a7e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff978006fd0 tx=0x7ff978006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.234+0000 7ff9777fe700 1 -- 172.21.15.116:0/1011045391 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff98002d440 con 0x7ff984106420 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:46.474+0000 7ff98c700700 1 -- 172.21.15.116:0/1011045391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff98410a820 con 0x7ff984106420 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.180+0000 7ff9777fe700 1 -- 172.21.15.116:0/1011045391 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7ff98002c3a0 con 0x7ff984106420 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.182+0000 7ff9777fe700 1 -- 172.21.15.116:0/1011045391 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff980023760 con 0x7ff984106420 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.185+0000 7ff98c700700 1 -- 172.21.15.116:0/1011045391 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7ff970038320 msgr2=0x7ff97003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.185+0000 7ff98c700700 1 --2- 172.21.15.116:0/1011045391 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7ff970038320 0x7ff97003a7e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff978006fd0 tx=0x7ff978006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:47.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.185+0000 7ff98c700700 1 -- 172.21.15.116:0/1011045391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 msgr2=0x7ff98419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:47.771 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.185+0000 7ff98c700700 1 --2- 172.21.15.116:0/1011045391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff98419a0e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff980006ba0 tx=0x7ff980011b50 comp rx=0 tx=0).stop 2024-03-04T15:54:47.771 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.186+0000 7ff98c700700 1 -- 172.21.15.116:0/1011045391 shutdown_connections 2024-03-04T15:54:47.771 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.186+0000 7ff98c700700 1 --2- 172.21.15.116:0/1011045391 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7ff970038320 0x7ff97003a7e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:47.771 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.186+0000 7ff98c700700 1 --2- 172.21.15.116:0/1011045391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff984106420 0x7ff98419a0e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:47.771 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.186+0000 7ff98c700700 1 -- 172.21.15.116:0/1011045391 >> 172.21.15.116:0/1011045391 conn(0x7ff984101a00 msgr2=0x7ff9841027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:47.771 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.186+0000 7ff98c700700 1 -- 172.21.15.116:0/1011045391 shutdown_connections 2024-03-04T15:54:47.771 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:47.186+0000 7ff98c700700 1 -- 172.21.15.116:0/1011045391 wait complete. 2024-03-04T15:54:48.416 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:48 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1011045391' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-04T15:54:48.416 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:48 smithi116 ceph-mon[29120]: mgrmap e9: smithi116.jwdhop(active, since 26s) 2024-03-04T15:54:49.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:54:49.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-04T15:54:49.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:54:49.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "smithi116.jwdhop", 2024-03-04T15:54:49.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T15:54:49.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:54:49.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.208+0000 7f839b20c700 1 Processor -- start 2024-03-04T15:54:49.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.208+0000 7f839b20c700 1 -- start start 2024-03-04T15:54:49.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.208+0000 7f839b20c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f8394104b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:49.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.208+0000 7f839b20c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8394105120 con 0x7f8394104730 2024-03-04T15:54:49.094 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.208+0000 7f8398fa8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f8394104b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:49.094 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.208+0000 7f8398fa8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f8394104b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60970/0 (socket says 172.21.15.116:60970) 2024-03-04T15:54:49.094 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.208+0000 7f8398fa8700 1 -- 172.21.15.116:0/1105651392 learned_addr learned my addr 172.21.15.116:0/1105651392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:49.094 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.208+0000 7f8398fa8700 1 -- 172.21.15.116:0/1105651392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83941076c0 con 0x7f8394104730 2024-03-04T15:54:49.094 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.209+0000 7f8398fa8700 1 --2- 172.21.15.116:0/1105651392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f8394104b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f83880098a0 tx=0x7f8388009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d70dd4483f1986 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:49.095 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.209+0000 7f83937fe700 1 -- 172.21.15.116:0/1105651392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8388004180 con 0x7f8394104730 2024-03-04T15:54:49.095 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.209+0000 7f83937fe700 1 -- 172.21.15.116:0/1105651392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f83880042e0 con 0x7f8394104730 2024-03-04T15:54:49.095 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.209+0000 7f83937fe700 1 -- 172.21.15.116:0/1105651392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8388011af0 con 0x7f8394104730 2024-03-04T15:54:49.095 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.209+0000 7f839b20c700 1 -- 172.21.15.116:0/1105651392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 msgr2=0x7f8394104b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:49.095 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.209+0000 7f839b20c700 1 --2- 172.21.15.116:0/1105651392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f8394104b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f83880098a0 tx=0x7f8388009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:49.095 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.210+0000 7f839b20c700 1 -- 172.21.15.116:0/1105651392 shutdown_connections 2024-03-04T15:54:49.096 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.210+0000 7f839b20c700 1 --2- 172.21.15.116:0/1105651392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f8394104b50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:49.096 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.210+0000 7f839b20c700 1 -- 172.21.15.116:0/1105651392 >> 172.21.15.116:0/1105651392 conn(0x7f8394100040 msgr2=0x7f83941024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:49.096 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.210+0000 7f839b20c700 1 -- 172.21.15.116:0/1105651392 shutdown_connections 2024-03-04T15:54:49.096 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.210+0000 7f839b20c700 1 -- 172.21.15.116:0/1105651392 wait complete. 2024-03-04T15:54:49.096 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.210+0000 7f839b20c700 1 Processor -- start 2024-03-04T15:54:49.096 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.210+0000 7f839b20c700 1 -- start start 2024-03-04T15:54:49.097 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f839b20c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f839419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:49.097 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f839b20c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839419a6f0 con 0x7f8394104730 2024-03-04T15:54:49.097 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f8398fa8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f839419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:49.097 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f8398fa8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f839419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60986/0 (socket says 172.21.15.116:60986) 2024-03-04T15:54:49.097 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f8398fa8700 1 -- 172.21.15.116:0/3193520978 learned_addr learned my addr 172.21.15.116:0/3193520978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:49.097 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f8398fa8700 1 -- 172.21.15.116:0/3193520978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8388009550 con 0x7f8394104730 2024-03-04T15:54:49.097 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f8398fa8700 1 --2- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f839419a1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8388011670 tx=0x7f8388011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:49.097 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f8391ffb700 1 -- 172.21.15.116:0/3193520978 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83880039c0 con 0x7f8394104730 2024-03-04T15:54:49.098 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f8391ffb700 1 -- 172.21.15.116:0/3193520978 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8388003b20 con 0x7f8394104730 2024-03-04T15:54:49.098 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f839b20c700 1 -- 172.21.15.116:0/3193520978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839419a8f0 con 0x7f8394104730 2024-03-04T15:54:49.098 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f8391ffb700 1 -- 172.21.15.116:0/3193520978 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83880247f0 con 0x7f8394104730 2024-03-04T15:54:49.098 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.211+0000 7f839b20c700 1 -- 172.21.15.116:0/3193520978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839419ad90 con 0x7f8394104730 2024-03-04T15:54:49.098 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.212+0000 7f8391ffb700 1 -- 172.21.15.116:0/3193520978 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f8388024a60 con 0x7f8394104730 2024-03-04T15:54:49.098 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.213+0000 7f8391ffb700 1 --2- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f8384038380 0x7f838403a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:49.098 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.213+0000 7f839b20c700 1 -- 172.21.15.116:0/3193520978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83940623c0 con 0x7f8394104730 2024-03-04T15:54:49.098 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.213+0000 7f8393fff700 1 -- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f8384038380 msgr2=0x7f838403a840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1340152843 2024-03-04T15:54:49.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.213+0000 7f8391ffb700 1 -- 172.21.15.116:0/3193520978 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f838804d310 con 0x7f8394104730 2024-03-04T15:54:49.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.213+0000 7f8393fff700 1 --2- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f8384038380 0x7f838403a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T15:54:49.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.217+0000 7f8391ffb700 1 -- 172.21.15.116:0/3193520978 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8388022310 con 0x7f8394104730 2024-03-04T15:54:49.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.414+0000 7f8393fff700 1 -- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f8384038380 msgr2=0x7f838403a840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1340152843 2024-03-04T15:54:49.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.414+0000 7f8393fff700 1 --2- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f8384038380 0x7f838403a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T15:54:49.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.466+0000 7f839b20c700 1 -- 172.21.15.116:0/3193520978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f839410a8f0 con 0x7f8394104730 2024-03-04T15:54:49.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.467+0000 7f8391ffb700 1 -- 172.21.15.116:0/3193520978 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f838800a0d0 con 0x7f8394104730 2024-03-04T15:54:49.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.469+0000 7f839b20c700 1 -- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f8384038380 msgr2=0x7f838403a840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T15:54:49.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.469+0000 7f839b20c700 1 --2- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f8384038380 0x7f838403a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:49.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.469+0000 7f839b20c700 1 -- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 msgr2=0x7f839419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:49.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.469+0000 7f839b20c700 1 --2- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f839419a1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8388011670 tx=0x7f8388011aa0 comp rx=0 tx=0).stop 2024-03-04T15:54:49.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.469+0000 7f839b20c700 1 -- 172.21.15.116:0/3193520978 shutdown_connections 2024-03-04T15:54:49.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.469+0000 7f839b20c700 1 --2- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7f8384038380 0x7f838403a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:49.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.469+0000 7f839b20c700 1 --2- 172.21.15.116:0/3193520978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8394104730 0x7f839419a1b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:49.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.469+0000 7f839b20c700 1 -- 172.21.15.116:0/3193520978 >> 172.21.15.116:0/3193520978 conn(0x7f8394100040 msgr2=0x7f83941017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:49.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.469+0000 7f839b20c700 1 -- 172.21.15.116:0/3193520978 shutdown_connections 2024-03-04T15:54:49.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:48.469+0000 7f839b20c700 1 -- 172.21.15.116:0/3193520978 wait complete. 2024-03-04T15:54:49.101 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-04T15:54:49.101 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 9... 2024-03-04T15:54:49.370 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:49 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3193520978' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T15:54:54.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: Active manager daemon smithi116.jwdhop restarted 2024-03-04T15:54:54.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: Activating manager daemon smithi116.jwdhop 2024-03-04T15:54:54.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: osdmap e3: 0 total, 0 up, 0 in 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: mgrmap e10: smithi116.jwdhop(active, starting, since 0.00375318s) 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi116.jwdhop", "id": "smithi116.jwdhop"}]: dispatch 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: Manager daemon smithi116.jwdhop is now available 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:54:54.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:54 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/trash_purge_schedule"}]: dispatch 2024-03-04T15:54:55.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:54:55.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-04T15:54:55.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.507+0000 7fcb4307f700 1 Processor -- start 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.507+0000 7fcb4307f700 1 -- start start 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.507+0000 7fcb4307f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c105120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.507+0000 7fcb4307f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c074620 con 0x7fcb3c104d00 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.507+0000 7fcb40e1b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c105120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.507+0000 7fcb40e1b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c105120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32768/0 (socket says 172.21.15.116:32768) 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.507+0000 7fcb40e1b700 1 -- 172.21.15.116:0/4050871725 learned_addr learned my addr 172.21.15.116:0/4050871725 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.508+0000 7fcb40e1b700 1 -- 172.21.15.116:0/4050871725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c105660 con 0x7fcb3c104d00 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.508+0000 7fcb40e1b700 1 --2- 172.21.15.116:0/4050871725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c105120 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcb300098a0 tx=0x7fcb30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bdd772741ffc148 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.508+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/4050871725 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb30004180 con 0x7fcb3c104d00 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.508+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/4050871725 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcb300042e0 con 0x7fcb3c104d00 2024-03-04T15:54:55.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.508+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/4050871725 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb30011af0 con 0x7fcb3c104d00 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.509+0000 7fcb4307f700 1 -- 172.21.15.116:0/4050871725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 msgr2=0x7fcb3c105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.509+0000 7fcb4307f700 1 --2- 172.21.15.116:0/4050871725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c105120 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcb300098a0 tx=0x7fcb30009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.509+0000 7fcb4307f700 1 -- 172.21.15.116:0/4050871725 shutdown_connections 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.509+0000 7fcb4307f700 1 --2- 172.21.15.116:0/4050871725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c105120 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.509+0000 7fcb4307f700 1 -- 172.21.15.116:0/4050871725 >> 172.21.15.116:0/4050871725 conn(0x7fcb3c1008c0 msgr2=0x7fcb3c102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.509+0000 7fcb4307f700 1 -- 172.21.15.116:0/4050871725 shutdown_connections 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.509+0000 7fcb4307f700 1 -- 172.21.15.116:0/4050871725 wait complete. 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.509+0000 7fcb4307f700 1 Processor -- start 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.509+0000 7fcb4307f700 1 -- start start 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.509+0000 7fcb4307f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c19c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb4307f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c19c880 con 0x7fcb3c104d00 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb40e1b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c19c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb40e1b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c19c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32780/0 (socket says 172.21.15.116:32780) 2024-03-04T15:54:55.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb40e1b700 1 -- 172.21.15.116:0/3495494563 learned_addr learned my addr 172.21.15.116:0/3495494563 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb40e1b700 1 -- 172.21.15.116:0/3495494563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30009550 con 0x7fcb3c104d00 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb40e1b700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c19c340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcb30011670 tx=0x7fcb30011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb30003a40 con 0x7fcb3c104d00 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb4307f700 1 -- 172.21.15.116:0/3495494563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3c19ca80 con 0x7fcb3c104d00 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcb30003ba0 con 0x7fcb3c104d00 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb30024820 con 0x7fcb3c104d00 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.510+0000 7fcb4307f700 1 -- 172.21.15.116:0/3495494563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c19cf20 con 0x7fcb3c104d00 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.511+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fcb30024ae0 con 0x7fcb3c104d00 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.512+0000 7fcb39ffb700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 0x7fcb2c03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.512+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 --> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb2c03af00 con 0x7fcb2c038330 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.512+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcb3004c580 con 0x7fcb3c104d00 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.512+0000 7fcb3bfff700 1 -- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 msgr2=0x7fcb2c03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1340152843 2024-03-04T15:54:55.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.512+0000 7fcb3bfff700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 0x7fcb2c03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.713+0000 7fcb3bfff700 1 -- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 msgr2=0x7fcb2c03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1340152843 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:49.713+0000 7fcb3bfff700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 0x7fcb2c03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:50.114+0000 7fcb3bfff700 1 -- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 msgr2=0x7fcb2c03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1340152843 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:50.114+0000 7fcb3bfff700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 0x7fcb2c03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:50.915+0000 7fcb3bfff700 1 -- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 msgr2=0x7fcb2c03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1340152843 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:50.915+0000 7fcb3bfff700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 0x7fcb2c03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:52.516+0000 7fcb3bfff700 1 -- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 msgr2=0x7fcb2c03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1340152843 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:52.516+0000 7fcb3bfff700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 0x7fcb2c03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:53.932+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 10) v1 ==== 45063+0+0 (secure 0 0 0) 0x7fcb3001ac70 con 0x7fcb3c104d00 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:53.932+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 msgr2=0x7fcb2c03a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:53.932+0000 7fcb39ffb700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 0x7fcb2c03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.933+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7fcb30023c20 con 0x7fcb3c104d00 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.933+0000 7fcb39ffb700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fcb2c03bdf0 0x7fcb2c03e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.933+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 --> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb2c03af00 con 0x7fcb2c03bdf0 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.933+0000 7fcb3bfff700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fcb2c03bdf0 0x7fcb2c03e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.934+0000 7fcb3bfff700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fcb2c03bdf0 0x7fcb2c03e1e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcb28003a10 tx=0x7fcb280092b0 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.934+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 <== mgr.14164 v2:172.21.15.116:6800/2020358480 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fcb2c03af00 con 0x7fcb2c03bdf0 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.941+0000 7fcb4307f700 1 -- 172.21.15.116:0/3495494563 --> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcb3c19d680 con 0x7fcb2c03bdf0 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.941+0000 7fcb39ffb700 1 -- 172.21.15.116:0/3495494563 <== mgr.14164 v2:172.21.15.116:6800/2020358480 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fcb3c19d680 con 0x7fcb2c03bdf0 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 -- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fcb2c03bdf0 msgr2=0x7fcb2c03e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fcb2c03bdf0 0x7fcb2c03e1e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcb28003a10 tx=0x7fcb280092b0 comp rx=0 tx=0).stop 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 -- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 msgr2=0x7fcb3c19c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c19c340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcb30011670 tx=0x7fcb30011aa0 comp rx=0 tx=0).stop 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 -- 172.21.15.116:0/3495494563 shutdown_connections 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fcb2c03bdf0 0x7fcb2c03e1e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c104d00 0x7fcb3c19c340 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 --2- 172.21.15.116:0/3495494563 >> [v2:172.21.15.116:6800/1340152843,v1:172.21.15.116:6801/1340152843] conn(0x7fcb2c038330 0x7fcb2c03a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:55.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 -- 172.21.15.116:0/3495494563 >> 172.21.15.116:0/3495494563 conn(0x7fcb3c1008c0 msgr2=0x7fcb3c1908c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:55.715 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 -- 172.21.15.116:0/3495494563 shutdown_connections 2024-03-04T15:54:55.715 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:54.942+0000 7fcb4307f700 1 -- 172.21.15.116:0/3495494563 wait complete. 2024-03-04T15:54:55.715 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 9 is available 2024-03-04T15:54:55.715 INFO:teuthology.orchestra.run.smithi116.stdout:Generating a dashboard self-signed certificate... 2024-03-04T15:54:55.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:55 smithi116 ceph-mon[29120]: [04/Mar/2024:15:54:54] ENGINE Bus STARTING 2024-03-04T15:54:55.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:55 smithi116 ceph-mon[29120]: [04/Mar/2024:15:54:54] ENGINE Serving on https://172.21.15.116:7150 2024-03-04T15:54:55.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:55 smithi116 ceph-mon[29120]: [04/Mar/2024:15:54:54] ENGINE Serving on http://172.21.15.116:8765 2024-03-04T15:54:55.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:55 smithi116 ceph-mon[29120]: [04/Mar/2024:15:54:54] ENGINE Bus STARTED 2024-03-04T15:54:55.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:55 smithi116 ceph-mon[29120]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T15:54:55.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:55 smithi116 ceph-mon[29120]: mgrmap e11: smithi116.jwdhop(active, since 1.0053s) 2024-03-04T15:54:55.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:55 smithi116 ceph-mon[29120]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T15:54:55.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:55 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:56.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-04T15:54:56.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.169+0000 7f544a2b4700 1 Processor -- start 2024-03-04T15:54:56.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.169+0000 7f544a2b4700 1 -- start start 2024-03-04T15:54:56.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.169+0000 7f544a2b4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f54441092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.169+0000 7f544a2b4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5444074640 con 0x7f5444106ef0 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.169+0000 7f5443fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f54441092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.169+0000 7f5443fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f54441092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51736/0 (socket says 172.21.15.116:51736) 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.169+0000 7f5443fff700 1 -- 172.21.15.116:0/2463001605 learned_addr learned my addr 172.21.15.116:0/2463001605 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.170+0000 7f5443fff700 1 -- 172.21.15.116:0/2463001605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5444109820 con 0x7f5444106ef0 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.170+0000 7f5443fff700 1 --2- 172.21.15.116:0/2463001605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f54441092e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f54380098a0 tx=0x7f5438009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3887d02fd6cb7b29 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.170+0000 7f5442ffd700 1 -- 172.21.15.116:0/2463001605 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438004180 con 0x7f5444106ef0 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.170+0000 7f5442ffd700 1 -- 172.21.15.116:0/2463001605 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f54380042e0 con 0x7f5444106ef0 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.170+0000 7f5442ffd700 1 -- 172.21.15.116:0/2463001605 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438011af0 con 0x7f5444106ef0 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.171+0000 7f544a2b4700 1 -- 172.21.15.116:0/2463001605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 msgr2=0x7f54441092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.171+0000 7f544a2b4700 1 --2- 172.21.15.116:0/2463001605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f54441092e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f54380098a0 tx=0x7f5438009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.171+0000 7f544a2b4700 1 -- 172.21.15.116:0/2463001605 shutdown_connections 2024-03-04T15:54:56.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.171+0000 7f544a2b4700 1 --2- 172.21.15.116:0/2463001605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f54441092e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.171+0000 7f544a2b4700 1 -- 172.21.15.116:0/2463001605 >> 172.21.15.116:0/2463001605 conn(0x7f54441008c0 msgr2=0x7f5444102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.171+0000 7f544a2b4700 1 -- 172.21.15.116:0/2463001605 shutdown_connections 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.171+0000 7f544a2b4700 1 -- 172.21.15.116:0/2463001605 wait complete. 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.171+0000 7f544a2b4700 1 Processor -- start 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.172+0000 7f544a2b4700 1 -- start start 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.172+0000 7f544a2b4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f544419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.172+0000 7f544a2b4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544419a6a0 con 0x7f5444106ef0 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.172+0000 7f5443fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f544419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.172+0000 7f5443fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f544419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51746/0 (socket says 172.21.15.116:51746) 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.172+0000 7f5443fff700 1 -- 172.21.15.116:0/2198318264 learned_addr learned my addr 172.21.15.116:0/2198318264 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.172+0000 7f5443fff700 1 -- 172.21.15.116:0/2198318264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5438009550 con 0x7f5444106ef0 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.172+0000 7f5443fff700 1 --2- 172.21.15.116:0/2198318264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f544419a160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5438011670 tx=0x7f5438011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.173+0000 7f54417fa700 1 -- 172.21.15.116:0/2198318264 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438003a40 con 0x7f5444106ef0 2024-03-04T15:54:56.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.173+0000 7f54417fa700 1 -- 172.21.15.116:0/2198318264 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5438003ba0 con 0x7f5444106ef0 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.173+0000 7f54417fa700 1 -- 172.21.15.116:0/2198318264 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438024820 con 0x7f5444106ef0 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.173+0000 7f544a2b4700 1 -- 172.21.15.116:0/2198318264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544419a8a0 con 0x7f5444106ef0 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.173+0000 7f544a2b4700 1 -- 172.21.15.116:0/2198318264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544419ad40 con 0x7f5444106ef0 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.174+0000 7f54417fa700 1 -- 172.21.15.116:0/2198318264 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f5438024a90 con 0x7f5444106ef0 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.174+0000 7f544a2b4700 1 -- 172.21.15.116:0/2198318264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544410a5b0 con 0x7f5444106ef0 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.174+0000 7f54417fa700 1 --2- 172.21.15.116:0/2198318264 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f542c038260 0x7f542c03a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.174+0000 7f54417fa700 1 -- 172.21.15.116:0/2198318264 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f543804c230 con 0x7f5444106ef0 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.174+0000 7f54437fe700 1 --2- 172.21.15.116:0/2198318264 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f542c038260 0x7f542c03a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.175+0000 7f54437fe700 1 --2- 172.21.15.116:0/2198318264 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f542c038260 0x7f542c03a720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5434006fd0 tx=0x7f5434006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.179+0000 7f54417fa700 1 -- 172.21.15.116:0/2198318264 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5438022310 con 0x7f5444106ef0 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.382+0000 7f544a2b4700 1 -- 172.21.15.116:0/2198318264 --> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f5444102bc0 con 0x7f542c038260 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.459+0000 7f54417fa700 1 -- 172.21.15.116:0/2198318264 <== mgr.14164 v2:172.21.15.116:6800/2020358480 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f5444102bc0 con 0x7f542c038260 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.461+0000 7f544a2b4700 1 -- 172.21.15.116:0/2198318264 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f542c038260 msgr2=0x7f542c03a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:56.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.461+0000 7f544a2b4700 1 --2- 172.21.15.116:0/2198318264 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f542c038260 0x7f542c03a720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5434006fd0 tx=0x7f5434006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:56.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.461+0000 7f544a2b4700 1 -- 172.21.15.116:0/2198318264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 msgr2=0x7f544419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:56.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.461+0000 7f544a2b4700 1 --2- 172.21.15.116:0/2198318264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f544419a160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5438011670 tx=0x7f5438011aa0 comp rx=0 tx=0).stop 2024-03-04T15:54:56.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.461+0000 7f544a2b4700 1 -- 172.21.15.116:0/2198318264 shutdown_connections 2024-03-04T15:54:56.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.461+0000 7f544a2b4700 1 --2- 172.21.15.116:0/2198318264 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f542c038260 0x7f542c03a720 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:56.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.461+0000 7f544a2b4700 1 --2- 172.21.15.116:0/2198318264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5444106ef0 0x7f544419a160 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:56.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.461+0000 7f544a2b4700 1 -- 172.21.15.116:0/2198318264 >> 172.21.15.116:0/2198318264 conn(0x7f54441008c0 msgr2=0x7f5444101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:56.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.461+0000 7f544a2b4700 1 -- 172.21.15.116:0/2198318264 shutdown_connections 2024-03-04T15:54:56.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:56.461+0000 7f544a2b4700 1 -- 172.21.15.116:0/2198318264 wait complete. 2024-03-04T15:54:56.943 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial admin user... 2024-03-04T15:54:57.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:57 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:57.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:57 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:57.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:57 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:57.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:57 smithi116 ceph-mon[29120]: mgrmap e12: smithi116.jwdhop(active, since 3s) 2024-03-04T15:54:58.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:58 smithi116 ceph-mon[29120]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:54:58.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:58 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:54:58.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$6JHyngSfw3NUHWvshbIjqO19n703NNj3Vl6Rac0L.4HO9i7S3tkCS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709567697, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-04T15:54:58.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.360+0000 7f06e2660700 1 Processor -- start 2024-03-04T15:54:58.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.360+0000 7f06e2660700 1 -- start start 2024-03-04T15:54:58.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.360+0000 7f06e2660700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:58.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.360+0000 7f06e2660700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc106e10 con 0x7f06dc106420 2024-03-04T15:54:58.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.360+0000 7f06dbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:58.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.360+0000 7f06dbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51750/0 (socket says 172.21.15.116:51750) 2024-03-04T15:54:58.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.360+0000 7f06dbfff700 1 -- 172.21.15.116:0/662588981 learned_addr learned my addr 172.21.15.116:0/662588981 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:58.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.360+0000 7f06dbfff700 1 -- 172.21.15.116:0/662588981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc107620 con 0x7f06dc106420 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.360+0000 7f06dbfff700 1 --2- 172.21.15.116:0/662588981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc106840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0098a0 tx=0x7f06cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3697932158507cc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.361+0000 7f06daffd700 1 -- 172.21.15.116:0/662588981 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06cc004180 con 0x7f06dc106420 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.361+0000 7f06daffd700 1 -- 172.21.15.116:0/662588981 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f06cc0042e0 con 0x7f06dc106420 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.361+0000 7f06daffd700 1 -- 172.21.15.116:0/662588981 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06cc011af0 con 0x7f06dc106420 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.361+0000 7f06e2660700 1 -- 172.21.15.116:0/662588981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 msgr2=0x7f06dc106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.361+0000 7f06e2660700 1 --2- 172.21.15.116:0/662588981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc106840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0098a0 tx=0x7f06cc009bb0 comp rx=0 tx=0).stop 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.361+0000 7f06e2660700 1 -- 172.21.15.116:0/662588981 shutdown_connections 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.361+0000 7f06e2660700 1 --2- 172.21.15.116:0/662588981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc106840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.361+0000 7f06e2660700 1 -- 172.21.15.116:0/662588981 >> 172.21.15.116:0/662588981 conn(0x7f06dc101a00 msgr2=0x7f06dc103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.361+0000 7f06e2660700 1 -- 172.21.15.116:0/662588981 shutdown_connections 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.361+0000 7f06e2660700 1 -- 172.21.15.116:0/662588981 wait complete. 2024-03-04T15:54:58.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.362+0000 7f06e2660700 1 Processor -- start 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.362+0000 7f06e2660700 1 -- start start 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.362+0000 7f06e2660700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.362+0000 7f06e2660700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc19a670 con 0x7f06dc106420 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.362+0000 7f06dbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.362+0000 7f06dbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51766/0 (socket says 172.21.15.116:51766) 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.362+0000 7f06dbfff700 1 -- 172.21.15.116:0/4113140993 learned_addr learned my addr 172.21.15.116:0/4113140993 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.362+0000 7f06dbfff700 1 -- 172.21.15.116:0/4113140993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc009550 con 0x7f06dc106420 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.363+0000 7f06dbfff700 1 --2- 172.21.15.116:0/4113140993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc19a130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f06cc000c00 tx=0x7f06cc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.363+0000 7f06d97fa700 1 -- 172.21.15.116:0/4113140993 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06cc003a70 con 0x7f06dc106420 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.363+0000 7f06e2660700 1 -- 172.21.15.116:0/4113140993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06dc19a870 con 0x7f06dc106420 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.363+0000 7f06d97fa700 1 -- 172.21.15.116:0/4113140993 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f06cc003bd0 con 0x7f06dc106420 2024-03-04T15:54:58.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.363+0000 7f06d97fa700 1 -- 172.21.15.116:0/4113140993 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06cc0238c0 con 0x7f06dc106420 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.363+0000 7f06e2660700 1 -- 172.21.15.116:0/4113140993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06dc19ad10 con 0x7f06dc106420 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.364+0000 7f06d97fa700 1 -- 172.21.15.116:0/4113140993 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f06cc023b80 con 0x7f06dc106420 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.364+0000 7f06d97fa700 1 --2- 172.21.15.116:0/4113140993 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f06c4038370 0x7f06c403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.364+0000 7f06d97fa700 1 -- 172.21.15.116:0/4113140993 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f06cc04c2d0 con 0x7f06dc106420 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.364+0000 7f06e2660700 1 -- 172.21.15.116:0/4113140993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06dc0623c0 con 0x7f06dc106420 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.364+0000 7f06db7fe700 1 --2- 172.21.15.116:0/4113140993 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f06c4038370 0x7f06c403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.364+0000 7f06db7fe700 1 --2- 172.21.15.116:0/4113140993 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f06c4038370 0x7f06c403a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06d0006fd0 tx=0x7f06d0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.369+0000 7f06d97fa700 1 -- 172.21.15.116:0/4113140993 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f06cc010d10 con 0x7f06dc106420 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.581+0000 7f06e2660700 1 -- 172.21.15.116:0/4113140993 --> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f06dc19db50 con 0x7f06c4038370 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.805+0000 7f06d97fa700 1 -- 172.21.15.116:0/4113140993 <== mgr.14164 v2:172.21.15.116:6800/2020358480 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f06dc19db50 con 0x7f06c4038370 2024-03-04T15:54:58.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.808+0000 7f06e2660700 1 -- 172.21.15.116:0/4113140993 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f06c4038370 msgr2=0x7f06c403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:58.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.808+0000 7f06e2660700 1 --2- 172.21.15.116:0/4113140993 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f06c4038370 0x7f06c403a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06d0006fd0 tx=0x7f06d0006e40 comp rx=0 tx=0).stop 2024-03-04T15:54:58.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.808+0000 7f06e2660700 1 -- 172.21.15.116:0/4113140993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 msgr2=0x7f06dc19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:54:58.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.808+0000 7f06e2660700 1 --2- 172.21.15.116:0/4113140993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc19a130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f06cc000c00 tx=0x7f06cc011b50 comp rx=0 tx=0).stop 2024-03-04T15:54:58.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.808+0000 7f06e2660700 1 -- 172.21.15.116:0/4113140993 shutdown_connections 2024-03-04T15:54:58.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.808+0000 7f06e2660700 1 --2- 172.21.15.116:0/4113140993 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f06c4038370 0x7f06c403a830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:58.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.808+0000 7f06e2660700 1 --2- 172.21.15.116:0/4113140993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc106420 0x7f06dc19a130 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:54:58.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.808+0000 7f06e2660700 1 -- 172.21.15.116:0/4113140993 >> 172.21.15.116:0/4113140993 conn(0x7f06dc101a00 msgr2=0x7f06dc1030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:54:58.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.808+0000 7f06e2660700 1 -- 172.21.15.116:0/4113140993 shutdown_connections 2024-03-04T15:54:58.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:57.808+0000 7f06e2660700 1 -- 172.21.15.116:0/4113140993 wait complete. 2024-03-04T15:54:58.840 INFO:teuthology.orchestra.run.smithi116.stdout:Fetching dashboard port number... 2024-03-04T15:54:59.446 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:54:59 smithi116 ceph-mon[29120]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:55:00.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 8443 2024-03-04T15:55:00.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.245+0000 7fe43d044700 1 Processor -- start 2024-03-04T15:55:00.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.245+0000 7fe43d044700 1 -- start start 2024-03-04T15:55:00.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.246+0000 7fe43d044700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe438105750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:00.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.246+0000 7fe43d044700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe438105d20 con 0x7fe438105330 2024-03-04T15:55:00.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.246+0000 7fe436d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe438105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:00.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.246+0000 7fe436d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe438105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51776/0 (socket says 172.21.15.116:51776) 2024-03-04T15:55:00.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.246+0000 7fe436d9d700 1 -- 172.21.15.116:0/2082827416 learned_addr learned my addr 172.21.15.116:0/2082827416 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:00.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.246+0000 7fe436d9d700 1 -- 172.21.15.116:0/2082827416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe438106530 con 0x7fe438105330 2024-03-04T15:55:00.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.246+0000 7fe436d9d700 1 --2- 172.21.15.116:0/2082827416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe438105750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe4280098a0 tx=0x7fe428009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c6e2251536767f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.246+0000 7fe435d9b700 1 -- 172.21.15.116:0/2082827416 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe428004180 con 0x7fe438105330 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.246+0000 7fe435d9b700 1 -- 172.21.15.116:0/2082827416 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe4280042e0 con 0x7fe438105330 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.246+0000 7fe435d9b700 1 -- 172.21.15.116:0/2082827416 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe428011af0 con 0x7fe438105330 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.247+0000 7fe43d044700 1 -- 172.21.15.116:0/2082827416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 msgr2=0x7fe438105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.247+0000 7fe43d044700 1 --2- 172.21.15.116:0/2082827416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe438105750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe4280098a0 tx=0x7fe428009bb0 comp rx=0 tx=0).stop 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.247+0000 7fe43d044700 1 -- 172.21.15.116:0/2082827416 shutdown_connections 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.247+0000 7fe43d044700 1 --2- 172.21.15.116:0/2082827416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe438105750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.247+0000 7fe43d044700 1 -- 172.21.15.116:0/2082827416 >> 172.21.15.116:0/2082827416 conn(0x7fe4381008b0 msgr2=0x7fe438102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.247+0000 7fe43d044700 1 -- 172.21.15.116:0/2082827416 shutdown_connections 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.247+0000 7fe43d044700 1 -- 172.21.15.116:0/2082827416 wait complete. 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.248+0000 7fe43d044700 1 Processor -- start 2024-03-04T15:55:00.035 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.248+0000 7fe43d044700 1 -- start start 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.248+0000 7fe43d044700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe43819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.248+0000 7fe43d044700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43819a6d0 con 0x7fe438105330 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.248+0000 7fe436d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe43819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.248+0000 7fe436d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe43819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51784/0 (socket says 172.21.15.116:51784) 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.248+0000 7fe436d9d700 1 -- 172.21.15.116:0/927004195 learned_addr learned my addr 172.21.15.116:0/927004195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.248+0000 7fe436d9d700 1 -- 172.21.15.116:0/927004195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe428009550 con 0x7fe438105330 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.249+0000 7fe436d9d700 1 --2- 172.21.15.116:0/927004195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe43819a190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe428011670 tx=0x7fe428011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.249+0000 7fe41ffff700 1 -- 172.21.15.116:0/927004195 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4280039f0 con 0x7fe438105330 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.249+0000 7fe43d044700 1 -- 172.21.15.116:0/927004195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43819a8d0 con 0x7fe438105330 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.249+0000 7fe41ffff700 1 -- 172.21.15.116:0/927004195 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe428003b50 con 0x7fe438105330 2024-03-04T15:55:00.036 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.249+0000 7fe41ffff700 1 -- 172.21.15.116:0/927004195 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4280247c0 con 0x7fe438105330 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.249+0000 7fe43d044700 1 -- 172.21.15.116:0/927004195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43819ad70 con 0x7fe438105330 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.250+0000 7fe41ffff700 1 -- 172.21.15.116:0/927004195 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fe42802d3f0 con 0x7fe438105330 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.250+0000 7fe41ffff700 1 --2- 172.21.15.116:0/927004195 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe420038370 0x7fe42003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.250+0000 7fe41ffff700 1 -- 172.21.15.116:0/927004195 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe42804d4c0 con 0x7fe438105330 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.250+0000 7fe43659c700 1 --2- 172.21.15.116:0/927004195 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe420038370 0x7fe42003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.250+0000 7fe43d044700 1 -- 172.21.15.116:0/927004195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4380623c0 con 0x7fe438105330 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.250+0000 7fe43659c700 1 --2- 172.21.15.116:0/927004195 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe420038370 0x7fe42003a830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe42c006fd0 tx=0x7fe42c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.255+0000 7fe41ffff700 1 -- 172.21.15.116:0/927004195 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe42802d740 con 0x7fe438105330 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.444+0000 7fe43d044700 1 -- 172.21.15.116:0/927004195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fe438109730 con 0x7fe438105330 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.444+0000 7fe41ffff700 1 -- 172.21.15.116:0/927004195 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fe4280107f0 con 0x7fe438105330 2024-03-04T15:55:00.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.446+0000 7fe43d044700 1 -- 172.21.15.116:0/927004195 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe420038370 msgr2=0x7fe42003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:00.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.446+0000 7fe43d044700 1 --2- 172.21.15.116:0/927004195 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe420038370 0x7fe42003a830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe42c006fd0 tx=0x7fe42c006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:00.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.446+0000 7fe43d044700 1 -- 172.21.15.116:0/927004195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 msgr2=0x7fe43819a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:00.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.446+0000 7fe43d044700 1 --2- 172.21.15.116:0/927004195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe43819a190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe428011670 tx=0x7fe428011b50 comp rx=0 tx=0).stop 2024-03-04T15:55:00.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.446+0000 7fe43d044700 1 -- 172.21.15.116:0/927004195 shutdown_connections 2024-03-04T15:55:00.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.446+0000 7fe43d044700 1 --2- 172.21.15.116:0/927004195 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe420038370 0x7fe42003a830 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:00.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.446+0000 7fe43d044700 1 --2- 172.21.15.116:0/927004195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438105330 0x7fe43819a190 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:00.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.446+0000 7fe43d044700 1 -- 172.21.15.116:0/927004195 >> 172.21.15.116:0/927004195 conn(0x7fe4381008b0 msgr2=0x7fe4380798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:00.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.447+0000 7fe43d044700 1 -- 172.21.15.116:0/927004195 shutdown_connections 2024-03-04T15:55:00.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:54:59.447+0000 7fe43d044700 1 -- 172.21.15.116:0/927004195 wait complete. 2024-03-04T15:55:00.039 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:55:00.039 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-04T15:55:00.046 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:55:00.046 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T15:55:00.046 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-04T15:55:00.046 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-04T15:55:00.048 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph Dashboard is now available at: 2024-03-04T15:55:00.048 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:00.048 INFO:teuthology.orchestra.run.smithi116.stdout: URL: https://smithi116.front.sepia.ceph.com:8443/ 2024-03-04T15:55:00.048 INFO:teuthology.orchestra.run.smithi116.stdout: User: admin 2024-03-04T15:55:00.048 INFO:teuthology.orchestra.run.smithi116.stdout: Password: cgo6tgrnlm 2024-03-04T15:55:00.048 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:00.049 INFO:teuthology.orchestra.run.smithi116.stdout:Saving cluster configuration to /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config directory 2024-03-04T15:55:00.049 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling autotune for osd_memory_target 2024-03-04T15:55:00.344 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:00 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/927004195' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.443+0000 7f64cd320700 1 Processor -- start 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.443+0000 7f64cd320700 1 -- start start 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.443+0000 7f64cd320700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c81092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.443+0000 7f64cd320700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c8074640 con 0x7f64c8106ef0 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.443+0000 7f64c6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c81092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.443+0000 7f64c6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c81092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51790/0 (socket says 172.21.15.116:51790) 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.443+0000 7f64c6ffd700 1 -- 172.21.15.116:0/4197953170 learned_addr learned my addr 172.21.15.116:0/4197953170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.443+0000 7f64c6ffd700 1 -- 172.21.15.116:0/4197953170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c8109820 con 0x7f64c8106ef0 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.443+0000 7f64c6ffd700 1 --2- 172.21.15.116:0/4197953170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c81092e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f64b80098a0 tx=0x7f64b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83f8170019b28ae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.444+0000 7f64c5ffb700 1 -- 172.21.15.116:0/4197953170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64b8004180 con 0x7f64c8106ef0 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.444+0000 7f64c5ffb700 1 -- 172.21.15.116:0/4197953170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f64b80042e0 con 0x7f64c8106ef0 2024-03-04T15:55:01.172 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.444+0000 7f64c5ffb700 1 -- 172.21.15.116:0/4197953170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64b8011af0 con 0x7f64c8106ef0 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.444+0000 7f64cd320700 1 -- 172.21.15.116:0/4197953170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 msgr2=0x7f64c81092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.444+0000 7f64cd320700 1 --2- 172.21.15.116:0/4197953170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c81092e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f64b80098a0 tx=0x7f64b8009bb0 comp rx=0 tx=0).stop 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.444+0000 7f64cd320700 1 -- 172.21.15.116:0/4197953170 shutdown_connections 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.444+0000 7f64cd320700 1 --2- 172.21.15.116:0/4197953170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c81092e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.444+0000 7f64cd320700 1 -- 172.21.15.116:0/4197953170 >> 172.21.15.116:0/4197953170 conn(0x7f64c81008c0 msgr2=0x7f64c8102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.444+0000 7f64cd320700 1 -- 172.21.15.116:0/4197953170 shutdown_connections 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.444+0000 7f64cd320700 1 -- 172.21.15.116:0/4197953170 wait complete. 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.445+0000 7f64cd320700 1 Processor -- start 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.445+0000 7f64cd320700 1 -- start start 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.445+0000 7f64cd320700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c8106850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.445+0000 7f64cd320700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c8106d90 con 0x7f64c8106ef0 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.445+0000 7f64c6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c8106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.445+0000 7f64c6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c8106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51802/0 (socket says 172.21.15.116:51802) 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.445+0000 7f64c6ffd700 1 -- 172.21.15.116:0/2814696356 learned_addr learned my addr 172.21.15.116:0/2814696356 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.446+0000 7f64c6ffd700 1 -- 172.21.15.116:0/2814696356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8009550 con 0x7f64c8106ef0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.446+0000 7f64c6ffd700 1 --2- 172.21.15.116:0/2814696356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c8106850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f64b8011670 tx=0x7f64b8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.446+0000 7f64affff700 1 -- 172.21.15.116:0/2814696356 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64b8003a40 con 0x7f64c8106ef0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.446+0000 7f64affff700 1 -- 172.21.15.116:0/2814696356 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f64b8003ba0 con 0x7f64c8106ef0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.446+0000 7f64cd320700 1 -- 172.21.15.116:0/2814696356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c8104ea0 con 0x7f64c8106ef0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.446+0000 7f64affff700 1 -- 172.21.15.116:0/2814696356 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64b8024820 con 0x7f64c8106ef0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.446+0000 7f64cd320700 1 -- 172.21.15.116:0/2814696356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c8105340 con 0x7f64c8106ef0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.447+0000 7f64affff700 1 -- 172.21.15.116:0/2814696356 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f64b8024ae0 con 0x7f64c8106ef0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.447+0000 7f64affff700 1 --2- 172.21.15.116:0/2814696356 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f64b00383d0 0x7f64b003a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.447+0000 7f64affff700 1 -- 172.21.15.116:0/2814696356 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f64b804c520 con 0x7f64c8106ef0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.447+0000 7f64c67fc700 1 --2- 172.21.15.116:0/2814696356 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f64b00383d0 0x7f64b003a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.447+0000 7f64cd320700 1 -- 172.21.15.116:0/2814696356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c810a5b0 con 0x7f64c8106ef0 2024-03-04T15:55:01.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.447+0000 7f64c67fc700 1 --2- 172.21.15.116:0/2814696356 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f64b00383d0 0x7f64b003a890 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f64bc006fd0 tx=0x7f64bc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.452+0000 7f64affff700 1 -- 172.21.15.116:0/2814696356 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f64b8022720 con 0x7f64c8106ef0 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.640+0000 7f64cd320700 1 -- 172.21.15.116:0/2814696356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f64c810a7a0 con 0x7f64c8106ef0 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.640+0000 7f64affff700 1 -- 172.21.15.116:0/2814696356 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f64b8049030 con 0x7f64c8106ef0 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.643+0000 7f64cd320700 1 -- 172.21.15.116:0/2814696356 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f64b00383d0 msgr2=0x7f64b003a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.643+0000 7f64cd320700 1 --2- 172.21.15.116:0/2814696356 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f64b00383d0 0x7f64b003a890 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f64bc006fd0 tx=0x7f64bc006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.643+0000 7f64cd320700 1 -- 172.21.15.116:0/2814696356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 msgr2=0x7f64c8106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.643+0000 7f64cd320700 1 --2- 172.21.15.116:0/2814696356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c8106850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f64b8011670 tx=0x7f64b8011aa0 comp rx=0 tx=0).stop 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.643+0000 7f64cd320700 1 -- 172.21.15.116:0/2814696356 shutdown_connections 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.643+0000 7f64cd320700 1 --2- 172.21.15.116:0/2814696356 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f64b00383d0 0x7f64b003a890 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.643+0000 7f64cd320700 1 --2- 172.21.15.116:0/2814696356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64c8106ef0 0x7f64c8106850 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.643+0000 7f64cd320700 1 -- 172.21.15.116:0/2814696356 >> 172.21.15.116:0/2814696356 conn(0x7f64c81008c0 msgr2=0x7f64c8101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.643+0000 7f64cd320700 1 -- 172.21.15.116:0/2814696356 shutdown_connections 2024-03-04T15:55:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:00.643+0000 7f64cd320700 1 -- 172.21.15.116:0/2814696356 wait complete. 2024-03-04T15:55:01.458 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:01 smithi116 ceph-mon[29120]: mgrmap e13: smithi116.jwdhop(active, since 6s) 2024-03-04T15:55:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.554+0000 7f152a651700 1 Processor -- start 2024-03-04T15:55:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.554+0000 7f152a651700 1 -- start start 2024-03-04T15:55:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.554+0000 7f152a651700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152407a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.554+0000 7f152a651700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152407a8c0 con 0x7f152407be90 2024-03-04T15:55:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.555+0000 7f1523fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152407a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:02.504 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.555+0000 7f1523fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152407a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51816/0 (socket says 172.21.15.116:51816) 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.555+0000 7f1523fff700 1 -- 172.21.15.116:0/941526938 learned_addr learned my addr 172.21.15.116:0/941526938 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.555+0000 7f1523fff700 1 -- 172.21.15.116:0/941526938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152407aa00 con 0x7f152407be90 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.555+0000 7f1523fff700 1 --2- 172.21.15.116:0/941526938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152407a2f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f15140098a0 tx=0x7f1514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8673622e689a8266 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.555+0000 7f1522ffd700 1 -- 172.21.15.116:0/941526938 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1514004180 con 0x7f152407be90 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.555+0000 7f1522ffd700 1 -- 172.21.15.116:0/941526938 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f15140042e0 con 0x7f152407be90 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.555+0000 7f1522ffd700 1 -- 172.21.15.116:0/941526938 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1514011af0 con 0x7f152407be90 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.556+0000 7f152a651700 1 -- 172.21.15.116:0/941526938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 msgr2=0x7f152407a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.556+0000 7f152a651700 1 --2- 172.21.15.116:0/941526938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152407a2f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f15140098a0 tx=0x7f1514009bb0 comp rx=0 tx=0).stop 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.556+0000 7f152a651700 1 -- 172.21.15.116:0/941526938 shutdown_connections 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.556+0000 7f152a651700 1 --2- 172.21.15.116:0/941526938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152407a2f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.556+0000 7f152a651700 1 -- 172.21.15.116:0/941526938 >> 172.21.15.116:0/941526938 conn(0x7f1524101a00 msgr2=0x7f1524103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.556+0000 7f152a651700 1 -- 172.21.15.116:0/941526938 shutdown_connections 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.556+0000 7f152a651700 1 -- 172.21.15.116:0/941526938 wait complete. 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.556+0000 7f152a651700 1 Processor -- start 2024-03-04T15:55:02.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.556+0000 7f152a651700 1 -- start start 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f152a651700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152419c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f152a651700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152419c800 con 0x7f152407be90 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f1523fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152419c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f1523fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152419c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51826/0 (socket says 172.21.15.116:51826) 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f1523fff700 1 -- 172.21.15.116:0/3885143978 learned_addr learned my addr 172.21.15.116:0/3885143978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f1523fff700 1 -- 172.21.15.116:0/3885143978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1514009550 con 0x7f152407be90 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f1523fff700 1 --2- 172.21.15.116:0/3885143978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152419c2c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1514011ad0 tx=0x7f1514011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f15217fa700 1 -- 172.21.15.116:0/3885143978 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1514003ab0 con 0x7f152407be90 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f15217fa700 1 -- 172.21.15.116:0/3885143978 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1514003c10 con 0x7f152407be90 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f152a651700 1 -- 172.21.15.116:0/3885143978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152419ca00 con 0x7f152407be90 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f15217fa700 1 -- 172.21.15.116:0/3885143978 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15140198e0 con 0x7f152407be90 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.557+0000 7f152a651700 1 -- 172.21.15.116:0/3885143978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152419cea0 con 0x7f152407be90 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.558+0000 7f15217fa700 1 -- 172.21.15.116:0/3885143978 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f1514019ba0 con 0x7f152407be90 2024-03-04T15:55:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.558+0000 7f152a651700 1 -- 172.21.15.116:0/3885143978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152410c800 con 0x7f152407be90 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.558+0000 7f15217fa700 1 --2- 172.21.15.116:0/3885143978 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f150c038410 0x7f150c03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.558+0000 7f15217fa700 1 -- 172.21.15.116:0/3885143978 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f151404c340 con 0x7f152407be90 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.559+0000 7f15237fe700 1 --2- 172.21.15.116:0/3885143978 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f150c038410 0x7f150c03a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.559+0000 7f15237fe700 1 --2- 172.21.15.116:0/3885143978 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f150c038410 0x7f150c03a8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1518006fd0 tx=0x7f1518006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.563+0000 7f15217fa700 1 -- 172.21.15.116:0/3885143978 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f151402b300 con 0x7f152407be90 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.857+0000 7f152a651700 1 -- 172.21.15.116:0/3885143978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f152419fce0 con 0x7f152407be90 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.860+0000 7f15217fa700 1 -- 172.21.15.116:0/3885143978 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f1514010b70 con 0x7f152407be90 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.862+0000 7f152a651700 1 -- 172.21.15.116:0/3885143978 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f150c038410 msgr2=0x7f150c03a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.862+0000 7f152a651700 1 --2- 172.21.15.116:0/3885143978 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f150c038410 0x7f150c03a8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1518006fd0 tx=0x7f1518006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.862+0000 7f152a651700 1 -- 172.21.15.116:0/3885143978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 msgr2=0x7f152419c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.862+0000 7f152a651700 1 --2- 172.21.15.116:0/3885143978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152419c2c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1514011ad0 tx=0x7f1514011bb0 comp rx=0 tx=0).stop 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.862+0000 7f152a651700 1 -- 172.21.15.116:0/3885143978 shutdown_connections 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.862+0000 7f152a651700 1 --2- 172.21.15.116:0/3885143978 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f150c038410 0x7f150c03a8d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.862+0000 7f152a651700 1 --2- 172.21.15.116:0/3885143978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152407be90 0x7f152419c2c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.862+0000 7f152a651700 1 -- 172.21.15.116:0/3885143978 >> 172.21.15.116:0/3885143978 conn(0x7f1524101a00 msgr2=0x7f1524102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.862+0000 7f152a651700 1 -- 172.21.15.116:0/3885143978 shutdown_connections 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T15:55:01.862+0000 7f152a651700 1 -- 172.21.15.116:0/3885143978 wait complete. 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:Or, if you are only running a single cluster on this host: 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: ceph telemetry on 2024-03-04T15:55:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:For more information see: 2024-03-04T15:55:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-04T15:55:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:02.509 INFO:teuthology.orchestra.run.smithi116.stdout:Bootstrap complete. 2024-03-04T15:55:02.534 INFO:tasks.cephadm:Fetching config... 2024-03-04T15:55:02.534 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:55:02.534 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-04T15:55:02.549 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-04T15:55:02.549 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:55:02.549 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-04T15:55:02.604 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-04T15:55:02.604 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:55:02.604 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/keyring of=/dev/stdout 2024-03-04T15:55:02.671 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-04T15:55:02.671 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:55:02.672 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-04T15:55:02.727 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-04T15:55:02.727 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/TEcKv95RWBJzAgjyMTHoBBcVhuqcve6m4cv3aesWwtjWrsVWrPk634jq5sl9THVBkVfPlRVm3ilmahHMRIW8gFURCzYRSLVQwNiDswu+vFjGHafIv4QOg1T564eg4fEJ5xA9AGEqcaJT4B4rtyncNR7d0H0sNKDP5d50wh8RiE+71VsUR0keD+hbjvBKjzzlCNw443tWQNB0HKUoKfke4neWfQUgIjBNfmMKOxKROwFr4wdQYfG2/D8SQ2HLIOB99/aH3Wof+Pod5JWARFY546tTzfBoBtgnX1Z5RgbvYYyi+WDfNg6X00qDojOSp6tJo2KR+PFP80mHu6AUApPvRRMqDUZl+EFtKJxZ9Rj04KYPfzBmN/AdeksMRiesaY7wG4iMMlBirNpa1xBDo7wL5O5x/kHOj/+puoity0mW5S/USyGnhh7CuZ+NPaXbbmGd8tUnoP062e4au5AtfFG++UwOWffZ3c9lEokBMDEhW59Xsi7jS439J4DTzTatKtk= ceph-4960653c-da3f-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:55:02.806 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/TEcKv95RWBJzAgjyMTHoBBcVhuqcve6m4cv3aesWwtjWrsVWrPk634jq5sl9THVBkVfPlRVm3ilmahHMRIW8gFURCzYRSLVQwNiDswu+vFjGHafIv4QOg1T564eg4fEJ5xA9AGEqcaJT4B4rtyncNR7d0H0sNKDP5d50wh8RiE+71VsUR0keD+hbjvBKjzzlCNw443tWQNB0HKUoKfke4neWfQUgIjBNfmMKOxKROwFr4wdQYfG2/D8SQ2HLIOB99/aH3Wof+Pod5JWARFY546tTzfBoBtgnX1Z5RgbvYYyi+WDfNg6X00qDojOSp6tJo2KR+PFP80mHu6AUApPvRRMqDUZl+EFtKJxZ9Rj04KYPfzBmN/AdeksMRiesaY7wG4iMMlBirNpa1xBDo7wL5O5x/kHOj/+puoity0mW5S/USyGnhh7CuZ+NPaXbbmGd8tUnoP062e4au5AtfFG++UwOWffZ3c9lEokBMDEhW59Xsi7jS439J4DTzTatKtk= ceph-4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:55:02.820 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/TEcKv95RWBJzAgjyMTHoBBcVhuqcve6m4cv3aesWwtjWrsVWrPk634jq5sl9THVBkVfPlRVm3ilmahHMRIW8gFURCzYRSLVQwNiDswu+vFjGHafIv4QOg1T564eg4fEJ5xA9AGEqcaJT4B4rtyncNR7d0H0sNKDP5d50wh8RiE+71VsUR0keD+hbjvBKjzzlCNw443tWQNB0HKUoKfke4neWfQUgIjBNfmMKOxKROwFr4wdQYfG2/D8SQ2HLIOB99/aH3Wof+Pod5JWARFY546tTzfBoBtgnX1Z5RgbvYYyi+WDfNg6X00qDojOSp6tJo2KR+PFP80mHu6AUApPvRRMqDUZl+EFtKJxZ9Rj04KYPfzBmN/AdeksMRiesaY7wG4iMMlBirNpa1xBDo7wL5O5x/kHOj/+puoity0mW5S/USyGnhh7CuZ+NPaXbbmGd8tUnoP062e4au5AtfFG++UwOWffZ3c9lEokBMDEhW59Xsi7jS439J4DTzTatKtk= ceph-4960653c-da3f-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:55:02.865 INFO:teuthology.orchestra.run.smithi152.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/TEcKv95RWBJzAgjyMTHoBBcVhuqcve6m4cv3aesWwtjWrsVWrPk634jq5sl9THVBkVfPlRVm3ilmahHMRIW8gFURCzYRSLVQwNiDswu+vFjGHafIv4QOg1T564eg4fEJ5xA9AGEqcaJT4B4rtyncNR7d0H0sNKDP5d50wh8RiE+71VsUR0keD+hbjvBKjzzlCNw443tWQNB0HKUoKfke4neWfQUgIjBNfmMKOxKROwFr4wdQYfG2/D8SQ2HLIOB99/aH3Wof+Pod5JWARFY546tTzfBoBtgnX1Z5RgbvYYyi+WDfNg6X00qDojOSp6tJo2KR+PFP80mHu6AUApPvRRMqDUZl+EFtKJxZ9Rj04KYPfzBmN/AdeksMRiesaY7wG4iMMlBirNpa1xBDo7wL5O5x/kHOj/+puoity0mW5S/USyGnhh7CuZ+NPaXbbmGd8tUnoP062e4au5AtfFG++UwOWffZ3c9lEokBMDEhW59Xsi7jS439J4DTzTatKtk= ceph-4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:55:02.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-04T15:55:03.082 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:02 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3885143978' entity='client.admin' 2024-03-04T15:55:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:55:05.599 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:05 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:05.599 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:05 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:05.599 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:05 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:55:05.599 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:05 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:05.599 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:05 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T15:55:05.600 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:05 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-04T15:55:05.600 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:05 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:55:05.600 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:05 smithi116 ceph-mon[29120]: Deploying daemon ceph-exporter.smithi116 on smithi116 2024-03-04T15:55:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.626+0000 7fc155cee700 1 -- 172.21.15.116:0/4022575325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150101e90 msgr2=0x7fc1501022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.626+0000 7fc155cee700 1 --2- 172.21.15.116:0/4022575325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150101e90 0x7fc1501022b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc140009910 tx=0x7fc140009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.626+0000 7fc155cee700 1 -- 172.21.15.116:0/4022575325 shutdown_connections 2024-03-04T15:55:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.626+0000 7fc155cee700 1 --2- 172.21.15.116:0/4022575325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150101e90 0x7fc1501022b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.626+0000 7fc155cee700 1 -- 172.21.15.116:0/4022575325 >> 172.21.15.116:0/4022575325 conn(0x7fc1500fa4b0 msgr2=0x7fc1500fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.626+0000 7fc155cee700 1 -- 172.21.15.116:0/4022575325 shutdown_connections 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.626+0000 7fc155cee700 1 -- 172.21.15.116:0/4022575325 wait complete. 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.627+0000 7fc155cee700 1 Processor -- start 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.627+0000 7fc155cee700 1 -- start start 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.627+0000 7fc155cee700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150101e90 0x7fc150101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.627+0000 7fc155cee700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc150101ad0 con 0x7fc150101e90 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.627+0000 7fc14f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150101e90 0x7fc150101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.627+0000 7fc14f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150101e90 0x7fc150101590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47400/0 (socket says 172.21.15.116:47400) 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.627+0000 7fc14f7fe700 1 -- 172.21.15.116:0/2504686776 learned_addr learned my addr 172.21.15.116:0/2504686776 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.628+0000 7fc14f7fe700 1 -- 172.21.15.116:0/2504686776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1400095f0 con 0x7fc150101e90 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.628+0000 7fc14f7fe700 1 --2- 172.21.15.116:0/2504686776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150101e90 0x7fc150101590 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc140005b40 tx=0x7fc140004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.628+0000 7fc14cff9700 1 -- 172.21.15.116:0/2504686776 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14001d070 con 0x7fc150101e90 2024-03-04T15:55:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.628+0000 7fc14cff9700 1 -- 172.21.15.116:0/2504686776 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc14000be10 con 0x7fc150101e90 2024-03-04T15:55:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.628+0000 7fc155cee700 1 -- 172.21.15.116:0/2504686776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1500ffbe0 con 0x7fc150101e90 2024-03-04T15:55:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.628+0000 7fc14cff9700 1 -- 172.21.15.116:0/2504686776 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14000f460 con 0x7fc150101e90 2024-03-04T15:55:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.628+0000 7fc155cee700 1 -- 172.21.15.116:0/2504686776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc150100080 con 0x7fc150101e90 2024-03-04T15:55:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.629+0000 7fc14cff9700 1 -- 172.21.15.116:0/2504686776 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fc14000f5c0 con 0x7fc150101e90 2024-03-04T15:55:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.629+0000 7fc14cff9700 1 --2- 172.21.15.116:0/2504686776 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fc138038410 0x7fc13803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.629+0000 7fc155cee700 1 -- 172.21.15.116:0/2504686776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1500623c0 con 0x7fc150101e90 2024-03-04T15:55:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.629+0000 7fc14cff9700 1 -- 172.21.15.116:0/2504686776 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc14004c400 con 0x7fc150101e90 2024-03-04T15:55:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.629+0000 7fc14effd700 1 --2- 172.21.15.116:0/2504686776 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fc138038410 0x7fc13803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.629+0000 7fc14effd700 1 --2- 172.21.15.116:0/2504686776 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fc138038410 0x7fc13803a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc144006fd0 tx=0x7fc144006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.634+0000 7fc14cff9700 1 -- 172.21.15.116:0/2504686776 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc14002c440 con 0x7fc150101e90 2024-03-04T15:55:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.829+0000 7fc155cee700 1 -- 172.21.15.116:0/2504686776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fc15018f2e0 con 0x7fc150101e90 2024-03-04T15:55:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.834+0000 7fc14cff9700 1 -- 172.21.15.116:0/2504686776 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc140027070 con 0x7fc150101e90 2024-03-04T15:55:05.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.836+0000 7fc155cee700 1 -- 172.21.15.116:0/2504686776 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fc138038410 msgr2=0x7fc13803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:05.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.836+0000 7fc155cee700 1 --2- 172.21.15.116:0/2504686776 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fc138038410 0x7fc13803a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc144006fd0 tx=0x7fc144006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:05.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.836+0000 7fc155cee700 1 -- 172.21.15.116:0/2504686776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150101e90 msgr2=0x7fc150101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.836+0000 7fc155cee700 1 --2- 172.21.15.116:0/2504686776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150101e90 0x7fc150101590 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc140005b40 tx=0x7fc140004740 comp rx=0 tx=0).stop 2024-03-04T15:55:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.836+0000 7fc155cee700 1 -- 172.21.15.116:0/2504686776 shutdown_connections 2024-03-04T15:55:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.836+0000 7fc155cee700 1 --2- 172.21.15.116:0/2504686776 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fc138038410 0x7fc13803a8d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.836+0000 7fc155cee700 1 --2- 172.21.15.116:0/2504686776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150101e90 0x7fc150101590 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.836+0000 7fc155cee700 1 -- 172.21.15.116:0/2504686776 >> 172.21.15.116:0/2504686776 conn(0x7fc1500fa4b0 msgr2=0x7fc1500fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.836+0000 7fc155cee700 1 -- 172.21.15.116:0/2504686776 shutdown_connections 2024-03-04T15:55:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:05.836+0000 7fc155cee700 1 -- 172.21.15.116:0/2504686776 wait complete. 2024-03-04T15:55:06.914 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:06 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2504686776' entity='client.admin' 2024-03-04T15:55:06.934 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-04T15:55:06.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-04T15:55:07.971 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:55:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.625+0000 7f786e286700 1 -- 172.21.15.116:0/745502316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78680ff7a0 msgr2=0x7f78680ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.625+0000 7f786e286700 1 --2- 172.21.15.116:0/745502316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78680ff7a0 0x7f78680ffbc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7858009910 tx=0x7f7858009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.625+0000 7f786e286700 1 -- 172.21.15.116:0/745502316 shutdown_connections 2024-03-04T15:55:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.625+0000 7f786e286700 1 --2- 172.21.15.116:0/745502316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78680ff7a0 0x7f78680ffbc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.625+0000 7f786e286700 1 -- 172.21.15.116:0/745502316 >> 172.21.15.116:0/745502316 conn(0x7f78680fad20 msgr2=0x7f78680fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.625+0000 7f786e286700 1 -- 172.21.15.116:0/745502316 shutdown_connections 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.625+0000 7f786e286700 1 -- 172.21.15.116:0/745502316 wait complete. 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.626+0000 7f786e286700 1 Processor -- start 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.626+0000 7f786e286700 1 -- start start 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.626+0000 7f786e286700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78680ff7a0 0x7f78681948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.626+0000 7f786e286700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7868194de0 con 0x7f78680ff7a0 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.626+0000 7f7867fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78680ff7a0 0x7f78681948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.626+0000 7f7867fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78680ff7a0 0x7f78681948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47430/0 (socket says 172.21.15.116:47430) 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.626+0000 7f7867fff700 1 -- 172.21.15.116:0/262573370 learned_addr learned my addr 172.21.15.116:0/262573370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.627+0000 7f7867fff700 1 -- 172.21.15.116:0/262573370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78580095f0 con 0x7f78680ff7a0 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.627+0000 7f7867fff700 1 --2- 172.21.15.116:0/262573370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78680ff7a0 0x7f78681948a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7858005950 tx=0x7f7858009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.627+0000 7f78657fa700 1 -- 172.21.15.116:0/262573370 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f785801d070 con 0x7f78680ff7a0 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.627+0000 7f78657fa700 1 -- 172.21.15.116:0/262573370 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f785800be10 con 0x7f78680ff7a0 2024-03-04T15:55:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.627+0000 7f786e286700 1 -- 172.21.15.116:0/262573370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7868194fe0 con 0x7f78680ff7a0 2024-03-04T15:55:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.627+0000 7f78657fa700 1 -- 172.21.15.116:0/262573370 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f785800f460 con 0x7f78680ff7a0 2024-03-04T15:55:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.627+0000 7f786e286700 1 -- 172.21.15.116:0/262573370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7868195480 con 0x7f78680ff7a0 2024-03-04T15:55:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.628+0000 7f78657fa700 1 -- 172.21.15.116:0/262573370 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f785800f690 con 0x7f78680ff7a0 2024-03-04T15:55:09.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.628+0000 7f786e286700 1 -- 172.21.15.116:0/262573370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78681055e0 con 0x7f78680ff7a0 2024-03-04T15:55:09.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.628+0000 7f78657fa700 1 --2- 172.21.15.116:0/262573370 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f785003a5d0 0x7f785003ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:09.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.628+0000 7f78657fa700 1 -- 172.21.15.116:0/262573370 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f785804c400 con 0x7f78680ff7a0 2024-03-04T15:55:09.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.628+0000 7f78677fe700 1 --2- 172.21.15.116:0/262573370 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f785003a5d0 0x7f785003ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:09.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.629+0000 7f78677fe700 1 --2- 172.21.15.116:0/262573370 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f785003a5d0 0x7f785003ca90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f785c006fd0 tx=0x7f785c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.633+0000 7f78657fa700 1 -- 172.21.15.116:0/262573370 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f785802a360 con 0x7f78680ff7a0 2024-03-04T15:55:09.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.834+0000 7f786e286700 1 -- 172.21.15.116:0/262573370 --> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f7868067fa0 con 0x7f785003a5d0 2024-03-04T15:55:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.838+0000 7f78657fa700 1 -- 172.21.15.116:0/262573370 <== mgr.14164 v2:172.21.15.116:6800/2020358480 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7868067fa0 con 0x7f785003a5d0 2024-03-04T15:55:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.840+0000 7f786e286700 1 -- 172.21.15.116:0/262573370 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f785003a5d0 msgr2=0x7f785003ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.840+0000 7f786e286700 1 --2- 172.21.15.116:0/262573370 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f785003a5d0 0x7f785003ca90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f785c006fd0 tx=0x7f785c006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.840+0000 7f786e286700 1 -- 172.21.15.116:0/262573370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78680ff7a0 msgr2=0x7f78681948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.840+0000 7f786e286700 1 --2- 172.21.15.116:0/262573370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78680ff7a0 0x7f78681948a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7858005950 tx=0x7f7858009e60 comp rx=0 tx=0).stop 2024-03-04T15:55:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.840+0000 7f786e286700 1 -- 172.21.15.116:0/262573370 shutdown_connections 2024-03-04T15:55:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.840+0000 7f786e286700 1 --2- 172.21.15.116:0/262573370 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f785003a5d0 0x7f785003ca90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.840+0000 7f786e286700 1 --2- 172.21.15.116:0/262573370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78680ff7a0 0x7f78681948a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.840+0000 7f786e286700 1 -- 172.21.15.116:0/262573370 >> 172.21.15.116:0/262573370 conn(0x7f78680fad20 msgr2=0x7f78680fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.840+0000 7f786e286700 1 -- 172.21.15.116:0/262573370 shutdown_connections 2024-03-04T15:55:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:09.840+0000 7f786e286700 1 -- 172.21.15.116:0/262573370 wait complete. 2024-03-04T15:55:10.441 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi152 2024-03-04T15:55:10.442 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:55:10.442 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.conf 2024-03-04T15:55:10.462 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:55:10.463 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:55:10.520 INFO:tasks.cephadm:Adding host smithi152 to orchestrator... 2024-03-04T15:55:10.521 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph orch host add smithi152 2024-03-04T15:55:10.735 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:55:11.173 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:10 smithi116 ceph-mon[29120]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:55:11.173 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:10 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:11.173 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:10 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:11.173 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:10 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:11.173 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:10 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:11.174 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:10 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:11.174 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:10 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T15:55:11.174 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:10 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-04T15:55:11.174 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:10 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:55:12.137 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:11 smithi116 ceph-mon[29120]: Deploying daemon crash.smithi116 on smithi116 2024-03-04T15:55:12.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.237+0000 7f67feef7700 1 -- 172.21.15.116:0/4075097665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67f80ff790 msgr2=0x7f67f80ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:12.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.237+0000 7f67feef7700 1 --2- 172.21.15.116:0/4075097665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67f80ff790 0x7f67f80ffbb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009910 tx=0x7f67ec009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:12.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.237+0000 7f67feef7700 1 -- 172.21.15.116:0/4075097665 shutdown_connections 2024-03-04T15:55:12.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.237+0000 7f67feef7700 1 --2- 172.21.15.116:0/4075097665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67f80ff790 0x7f67f80ffbb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:12.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.237+0000 7f67feef7700 1 -- 172.21.15.116:0/4075097665 >> 172.21.15.116:0/4075097665 conn(0x7f67f80fad30 msgr2=0x7f67f80fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:12.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.237+0000 7f67feef7700 1 -- 172.21.15.116:0/4075097665 shutdown_connections 2024-03-04T15:55:12.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.237+0000 7f67feef7700 1 -- 172.21.15.116:0/4075097665 wait complete. 2024-03-04T15:55:12.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.237+0000 7f67feef7700 1 Processor -- start 2024-03-04T15:55:12.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.237+0000 7f67feef7700 1 -- start start 2024-03-04T15:55:12.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67feef7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67f80ff790 0x7f67f81948c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:12.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67feef7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f8194e00 con 0x7f67f80ff790 2024-03-04T15:55:12.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67fcc93700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67f80ff790 0x7f67f81948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:12.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67fcc93700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67f80ff790 0x7f67f81948c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47452/0 (socket says 172.21.15.116:47452) 2024-03-04T15:55:12.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67fcc93700 1 -- 172.21.15.116:0/1775741419 learned_addr learned my addr 172.21.15.116:0/1775741419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:12.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67fcc93700 1 -- 172.21.15.116:0/1775741419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec0095f0 con 0x7f67f80ff790 2024-03-04T15:55:12.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67fcc93700 1 --2- 172.21.15.116:0/1775741419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67f80ff790 0x7f67f81948c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f67ec005b40 tx=0x7f67ec004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:12.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67f5ffb700 1 -- 172.21.15.116:0/1775741419 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67ec01d070 con 0x7f67f80ff790 2024-03-04T15:55:12.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67feef7700 1 -- 172.21.15.116:0/1775741419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f8195000 con 0x7f67f80ff790 2024-03-04T15:55:12.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67f5ffb700 1 -- 172.21.15.116:0/1775741419 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f67ec00be10 con 0x7f67f80ff790 2024-03-04T15:55:12.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67f5ffb700 1 -- 172.21.15.116:0/1775741419 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67ec00f460 con 0x7f67f80ff790 2024-03-04T15:55:12.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.238+0000 7f67feef7700 1 -- 172.21.15.116:0/1775741419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f81954a0 con 0x7f67f80ff790 2024-03-04T15:55:12.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.239+0000 7f67f5ffb700 1 -- 172.21.15.116:0/1775741419 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f67ec00f6d0 con 0x7f67f80ff790 2024-03-04T15:55:12.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.239+0000 7f67feef7700 1 -- 172.21.15.116:0/1775741419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f818e3b0 con 0x7f67f80ff790 2024-03-04T15:55:12.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.239+0000 7f67f5ffb700 1 --2- 172.21.15.116:0/1775741419 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f67e80383c0 0x7f67e803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:12.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.239+0000 7f67f5ffb700 1 -- 172.21.15.116:0/1775741419 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f67ec04c3d0 con 0x7f67f80ff790 2024-03-04T15:55:12.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.240+0000 7f67f7fff700 1 --2- 172.21.15.116:0/1775741419 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f67e80383c0 0x7f67e803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:12.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.240+0000 7f67f7fff700 1 --2- 172.21.15.116:0/1775741419 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f67e80383c0 0x7f67e803a880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f67e4006fd0 tx=0x7f67e4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:12.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.244+0000 7f67f5ffb700 1 -- 172.21.15.116:0/1775741419 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f67ec02a360 con 0x7f67f80ff790 2024-03-04T15:55:12.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:12.441+0000 7f67feef7700 1 -- 172.21.15.116:0/1775741419 --> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}) v1 -- 0x7f67f80611d0 con 0x7f67e80383c0 2024-03-04T15:55:14.257 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:14 smithi116 ceph-mon[29120]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:55:14.257 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:14 smithi116 ceph-mon[29120]: Deploying cephadm binary to smithi152 2024-03-04T15:55:14.257 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:14 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:15.114 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi152' with addr '172.21.15.152' 2024-03-04T15:55:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.110+0000 7f67f5ffb700 1 -- 172.21.15.116:0/1775741419 <== mgr.14164 v2:172.21.15.116:6800/2020358480 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f67f80611d0 con 0x7f67e80383c0 2024-03-04T15:55:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.113+0000 7f67feef7700 1 -- 172.21.15.116:0/1775741419 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f67e80383c0 msgr2=0x7f67e803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.113+0000 7f67feef7700 1 --2- 172.21.15.116:0/1775741419 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f67e80383c0 0x7f67e803a880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f67e4006fd0 tx=0x7f67e4006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.113+0000 7f67feef7700 1 -- 172.21.15.116:0/1775741419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67f80ff790 msgr2=0x7f67f81948c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.113+0000 7f67feef7700 1 --2- 172.21.15.116:0/1775741419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67f80ff790 0x7f67f81948c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f67ec005b40 tx=0x7f67ec004740 comp rx=0 tx=0).stop 2024-03-04T15:55:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.113+0000 7f67feef7700 1 -- 172.21.15.116:0/1775741419 shutdown_connections 2024-03-04T15:55:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.113+0000 7f67feef7700 1 --2- 172.21.15.116:0/1775741419 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f67e80383c0 0x7f67e803a880 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.113+0000 7f67feef7700 1 --2- 172.21.15.116:0/1775741419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67f80ff790 0x7f67f81948c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.113+0000 7f67feef7700 1 -- 172.21.15.116:0/1775741419 >> 172.21.15.116:0/1775741419 conn(0x7f67f80fad30 msgr2=0x7f67f80fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.113+0000 7f67feef7700 1 -- 172.21.15.116:0/1775741419 shutdown_connections 2024-03-04T15:55:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:15.113+0000 7f67feef7700 1 -- 172.21.15.116:0/1775741419 wait complete. 2024-03-04T15:55:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:15 smithi116 ceph-mon[29120]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:15 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:15 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:15 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:15.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:15 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:15.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:15 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:15.650 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T15:55:15.867 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:55:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:16 smithi116 ceph-mon[29120]: Deploying daemon node-exporter.smithi116 on smithi116 2024-03-04T15:55:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:16 smithi116 ceph-mon[29120]: Added host smithi152 2024-03-04T15:55:17.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:17 smithi116 ceph-mon[29120]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:18.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.224+0000 7fd874749700 1 -- 172.21.15.116:0/3165855313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c0feb50 msgr2=0x7fd86c0fef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:18.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.224+0000 7fd874749700 1 --2- 172.21.15.116:0/3165855313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c0feb50 0x7fd86c0fef70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd868009910 tx=0x7fd868009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:18.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.224+0000 7fd874749700 1 -- 172.21.15.116:0/3165855313 shutdown_connections 2024-03-04T15:55:18.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.224+0000 7fd874749700 1 --2- 172.21.15.116:0/3165855313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c0feb50 0x7fd86c0fef70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:18.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.224+0000 7fd874749700 1 -- 172.21.15.116:0/3165855313 >> 172.21.15.116:0/3165855313 conn(0x7fd86c0fa4b0 msgr2=0x7fd86c0fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:18.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.224+0000 7fd874749700 1 -- 172.21.15.116:0/3165855313 shutdown_connections 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.224+0000 7fd874749700 1 -- 172.21.15.116:0/3165855313 wait complete. 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.225+0000 7fd874749700 1 Processor -- start 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.225+0000 7fd874749700 1 -- start start 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.225+0000 7fd874749700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c0feb50 0x7fd86c1959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.225+0000 7fd874749700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c195f00 con 0x7fd86c0feb50 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.225+0000 7fd8724e5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c0feb50 0x7fd86c1959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.225+0000 7fd8724e5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c0feb50 0x7fd86c1959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57358/0 (socket says 172.21.15.116:57358) 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.225+0000 7fd8724e5700 1 -- 172.21.15.116:0/3639785024 learned_addr learned my addr 172.21.15.116:0/3639785024 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.226+0000 7fd8724e5700 1 -- 172.21.15.116:0/3639785024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8680095f0 con 0x7fd86c0feb50 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.226+0000 7fd8724e5700 1 --2- 172.21.15.116:0/3639785024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c0feb50 0x7fd86c1959c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd868005b40 tx=0x7fd868004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.226+0000 7fd8637fe700 1 -- 172.21.15.116:0/3639785024 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd86801d070 con 0x7fd86c0feb50 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.226+0000 7fd8637fe700 1 -- 172.21.15.116:0/3639785024 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd86800be10 con 0x7fd86c0feb50 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.226+0000 7fd874749700 1 -- 172.21.15.116:0/3639785024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c196100 con 0x7fd86c0feb50 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.226+0000 7fd8637fe700 1 -- 172.21.15.116:0/3639785024 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd86800f460 con 0x7fd86c0feb50 2024-03-04T15:55:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.226+0000 7fd874749700 1 -- 172.21.15.116:0/3639785024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c1965a0 con 0x7fd86c0feb50 2024-03-04T15:55:18.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.227+0000 7fd8637fe700 1 -- 172.21.15.116:0/3639785024 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fd86800b960 con 0x7fd86c0feb50 2024-03-04T15:55:18.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.227+0000 7fd8637fe700 1 --2- 172.21.15.116:0/3639785024 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd8580383c0 0x7fd85803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:18.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.227+0000 7fd874749700 1 -- 172.21.15.116:0/3639785024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c18f4c0 con 0x7fd86c0feb50 2024-03-04T15:55:18.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.227+0000 7fd8637fe700 1 -- 172.21.15.116:0/3639785024 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd86804c1f0 con 0x7fd86c0feb50 2024-03-04T15:55:18.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.227+0000 7fd871ce4700 1 --2- 172.21.15.116:0/3639785024 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd8580383c0 0x7fd85803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:18.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.227+0000 7fd871ce4700 1 --2- 172.21.15.116:0/3639785024 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd8580383c0 0x7fd85803a880 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd85c006fd0 tx=0x7fd85c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:18.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.232+0000 7fd8637fe700 1 -- 172.21.15.116:0/3639785024 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd86802bba0 con 0x7fd86c0feb50 2024-03-04T15:55:18.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.427+0000 7fd874749700 1 -- 172.21.15.116:0/3639785024 --> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd86c0ff680 con 0x7fd8580383c0 2024-03-04T15:55:18.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.428+0000 7fd8637fe700 1 -- 172.21.15.116:0/3639785024 <== mgr.14164 v2:172.21.15.116:6800/2020358480 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fd86c0ff680 con 0x7fd8580383c0 2024-03-04T15:55:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:55:18.430 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.152", "hostname": "smithi152", "labels": [], "status": ""}] 2024-03-04T15:55:18.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.430+0000 7fd874749700 1 -- 172.21.15.116:0/3639785024 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd8580383c0 msgr2=0x7fd85803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:18.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.430+0000 7fd874749700 1 --2- 172.21.15.116:0/3639785024 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd8580383c0 0x7fd85803a880 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd85c006fd0 tx=0x7fd85c006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:18.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.430+0000 7fd874749700 1 -- 172.21.15.116:0/3639785024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c0feb50 msgr2=0x7fd86c1959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:18.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.430+0000 7fd874749700 1 --2- 172.21.15.116:0/3639785024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c0feb50 0x7fd86c1959c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd868005b40 tx=0x7fd868004740 comp rx=0 tx=0).stop 2024-03-04T15:55:18.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.430+0000 7fd874749700 1 -- 172.21.15.116:0/3639785024 shutdown_connections 2024-03-04T15:55:18.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.430+0000 7fd874749700 1 --2- 172.21.15.116:0/3639785024 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd8580383c0 0x7fd85803a880 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:18.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.430+0000 7fd874749700 1 --2- 172.21.15.116:0/3639785024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c0feb50 0x7fd86c1959c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:18.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.430+0000 7fd874749700 1 -- 172.21.15.116:0/3639785024 >> 172.21.15.116:0/3639785024 conn(0x7fd86c0fa4b0 msgr2=0x7fd86c0fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:18.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.430+0000 7fd874749700 1 -- 172.21.15.116:0/3639785024 shutdown_connections 2024-03-04T15:55:18.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:18.430+0000 7fd874749700 1 -- 172.21.15.116:0/3639785024 wait complete. 2024-03-04T15:55:19.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:19 smithi116 ceph-mon[29120]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:19.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:19 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:19.533 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-04T15:55:19.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd crush tunables default 2024-03-04T15:55:19.756 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:55:20.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:20 smithi116 ceph-mon[29120]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:55:20.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:20 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:20.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:20 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:20.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:20 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:20.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:20 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:21 smithi116 ceph-mon[29120]: Deploying daemon alertmanager.smithi116 on smithi116 2024-03-04T15:55:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:21 smithi116 ceph-mon[29120]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:22.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.364+0000 7f1bcfbac700 1 -- 172.21.15.116:0/300080773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bc80ff7a0 msgr2=0x7f1bc80ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:22.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.364+0000 7f1bcfbac700 1 --2- 172.21.15.116:0/300080773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bc80ff7a0 0x7f1bc80ffbc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009910 tx=0x7f1bc4009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:22.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.364+0000 7f1bcfbac700 1 -- 172.21.15.116:0/300080773 shutdown_connections 2024-03-04T15:55:22.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.364+0000 7f1bcfbac700 1 --2- 172.21.15.116:0/300080773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bc80ff7a0 0x7f1bc80ffbc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:22.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.364+0000 7f1bcfbac700 1 -- 172.21.15.116:0/300080773 >> 172.21.15.116:0/300080773 conn(0x7f1bc80fad20 msgr2=0x7f1bc80fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.364+0000 7f1bcfbac700 1 -- 172.21.15.116:0/300080773 shutdown_connections 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.364+0000 7f1bcfbac700 1 -- 172.21.15.116:0/300080773 wait complete. 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.365+0000 7f1bcfbac700 1 Processor -- start 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.365+0000 7f1bcfbac700 1 -- start start 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.365+0000 7f1bcfbac700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bc80ff7a0 0x7f1bc80fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.365+0000 7f1bcfbac700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc80fd220 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.365+0000 7f1bcd948700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bc80ff7a0 0x7f1bc80fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.365+0000 7f1bcd948700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bc80ff7a0 0x7f1bc80fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57374/0 (socket says 172.21.15.116:57374) 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.365+0000 7f1bcd948700 1 -- 172.21.15.116:0/739443994 learned_addr learned my addr 172.21.15.116:0/739443994 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.365+0000 7f1bcd948700 1 -- 172.21.15.116:0/739443994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc40095f0 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.366+0000 7f1bcd948700 1 --2- 172.21.15.116:0/739443994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bc80ff7a0 0x7f1bc80fcce0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4005b40 tx=0x7f1bc4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.366+0000 7f1bbeffd700 1 -- 172.21.15.116:0/739443994 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bc401d070 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.366+0000 7f1bbeffd700 1 -- 172.21.15.116:0/739443994 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1bc400be10 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.366+0000 7f1bcfbac700 1 -- 172.21.15.116:0/739443994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc80fb330 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.366+0000 7f1bbeffd700 1 -- 172.21.15.116:0/739443994 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bc400f460 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.366+0000 7f1bcfbac700 1 -- 172.21.15.116:0/739443994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc80fb7d0 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.366+0000 7f1bbeffd700 1 -- 172.21.15.116:0/739443994 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f1bc400f6d0 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.367+0000 7f1bbeffd700 1 --2- 172.21.15.116:0/739443994 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f1bb40383c0 0x7f1bb403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.367+0000 7f1bbeffd700 1 -- 172.21.15.116:0/739443994 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1bc404c510 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.367+0000 7f1bcd147700 1 --2- 172.21.15.116:0/739443994 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f1bb40383c0 0x7f1bb403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.367+0000 7f1bcfbac700 1 -- 172.21.15.116:0/739443994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc80623c0 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.367+0000 7f1bcd147700 1 --2- 172.21.15.116:0/739443994 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f1bb40383c0 0x7f1bb403a880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8006fd0 tx=0x7f1bb8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.372+0000 7f1bbeffd700 1 -- 172.21.15.116:0/739443994 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1bc4027070 con 0x7f1bc80ff7a0 2024-03-04T15:55:22.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:22.576+0000 7f1bcfbac700 1 -- 172.21.15.116:0/739443994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f1bc819b9e0 con 0x7f1bc80ff7a0 2024-03-04T15:55:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.234+0000 7f1bbeffd700 1 -- 172.21.15.116:0/739443994 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1bc402c440 con 0x7f1bc80ff7a0 2024-03-04T15:55:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.236+0000 7f1bcfbac700 1 -- 172.21.15.116:0/739443994 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f1bb40383c0 msgr2=0x7f1bb403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.236+0000 7f1bcfbac700 1 --2- 172.21.15.116:0/739443994 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f1bb40383c0 0x7f1bb403a880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8006fd0 tx=0x7f1bb8006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.236+0000 7f1bcfbac700 1 -- 172.21.15.116:0/739443994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bc80ff7a0 msgr2=0x7f1bc80fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.236+0000 7f1bcfbac700 1 --2- 172.21.15.116:0/739443994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bc80ff7a0 0x7f1bc80fcce0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4005b40 tx=0x7f1bc4004740 comp rx=0 tx=0).stop 2024-03-04T15:55:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.237+0000 7f1bcfbac700 1 -- 172.21.15.116:0/739443994 shutdown_connections 2024-03-04T15:55:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.237+0000 7f1bcfbac700 1 --2- 172.21.15.116:0/739443994 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f1bb40383c0 0x7f1bb403a880 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.237+0000 7f1bcfbac700 1 --2- 172.21.15.116:0/739443994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bc80ff7a0 0x7f1bc80fcce0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.237+0000 7f1bcfbac700 1 -- 172.21.15.116:0/739443994 >> 172.21.15.116:0/739443994 conn(0x7f1bc80fad20 msgr2=0x7f1bc8068810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.237+0000 7f1bcfbac700 1 -- 172.21.15.116:0/739443994 shutdown_connections 2024-03-04T15:55:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:55:23.237+0000 7f1bcfbac700 1 -- 172.21.15.116:0/739443994 wait complete. 2024-03-04T15:55:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-03-04T15:55:23.612 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:23 smithi116 ceph-mon[29120]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:23.612 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:23 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/739443994' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:55:23.958 INFO:tasks.cephadm:Adding mon.smithi116 on smithi116 2024-03-04T15:55:23.958 INFO:tasks.cephadm:Adding mon.smithi152 on smithi152 2024-03-04T15:55:23.958 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph orch apply mon '2;smithi116:172.21.15.116=smithi116;smithi152:172.21.15.152=smithi152' 2024-03-04T15:55:24.160 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:24.207 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:24.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:24 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/739443994' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:55:24.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:24 smithi116 ceph-mon[29120]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:55:24.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:24 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:25.585 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:25 smithi116 ceph-mon[29120]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:26.108 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.105+0000 7ff079137700 1 -- 172.21.15.152:0/1025764076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740ff790 msgr2=0x7ff0740ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:26.108 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.105+0000 7ff079137700 1 --2- 172.21.15.152:0/1025764076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740ff790 0x7ff0740ffbb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff068009910 tx=0x7ff068009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:26.108 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.105+0000 7ff079137700 1 -- 172.21.15.152:0/1025764076 shutdown_connections 2024-03-04T15:55:26.108 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.105+0000 7ff079137700 1 --2- 172.21.15.152:0/1025764076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740ff790 0x7ff0740ffbb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:26.108 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.105+0000 7ff079137700 1 -- 172.21.15.152:0/1025764076 >> 172.21.15.152:0/1025764076 conn(0x7ff0740fad10 msgr2=0x7ff0740fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:26.108 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.105+0000 7ff079137700 1 -- 172.21.15.152:0/1025764076 shutdown_connections 2024-03-04T15:55:26.108 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.105+0000 7ff079137700 1 -- 172.21.15.152:0/1025764076 wait complete. 2024-03-04T15:55:26.108 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.106+0000 7ff079137700 1 Processor -- start 2024-03-04T15:55:26.108 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.106+0000 7ff079137700 1 -- start start 2024-03-04T15:55:26.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.106+0000 7ff079137700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740ff790 0x7ff0741959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:26.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.106+0000 7ff079137700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074195ee0 con 0x7ff0740ff790 2024-03-04T15:55:26.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.106+0000 7ff072d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740ff790 0x7ff0741959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:26.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.106+0000 7ff072d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740ff790 0x7ff0741959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:34782/0 (socket says 172.21.15.152:34782) 2024-03-04T15:55:26.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.106+0000 7ff072d9d700 1 -- 172.21.15.152:0/1306611981 learned_addr learned my addr 172.21.15.152:0/1306611981 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:55:26.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.106+0000 7ff072d9d700 1 -- 172.21.15.152:0/1306611981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0680095f0 con 0x7ff0740ff790 2024-03-04T15:55:26.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.107+0000 7ff072d9d700 1 --2- 172.21.15.152:0/1306611981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740ff790 0x7ff0741959a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff068005b40 tx=0x7ff068004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:26.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.107+0000 7ff05bfff700 1 -- 172.21.15.152:0/1306611981 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06801d070 con 0x7ff0740ff790 2024-03-04T15:55:26.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.107+0000 7ff05bfff700 1 -- 172.21.15.152:0/1306611981 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff06800be10 con 0x7ff0740ff790 2024-03-04T15:55:26.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.107+0000 7ff079137700 1 -- 172.21.15.152:0/1306611981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0741960e0 con 0x7ff0740ff790 2024-03-04T15:55:26.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.107+0000 7ff05bfff700 1 -- 172.21.15.152:0/1306611981 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06800f460 con 0x7ff0740ff790 2024-03-04T15:55:26.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.107+0000 7ff079137700 1 -- 172.21.15.152:0/1306611981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff074196580 con 0x7ff0740ff790 2024-03-04T15:55:26.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.108+0000 7ff05bfff700 1 -- 172.21.15.152:0/1306611981 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7ff06800f6d0 con 0x7ff0740ff790 2024-03-04T15:55:26.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.108+0000 7ff079137700 1 -- 172.21.15.152:0/1306611981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07418f3c0 con 0x7ff0740ff790 2024-03-04T15:55:26.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.108+0000 7ff05bfff700 1 --2- 172.21.15.152:0/1306611981 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7ff05c0383c0 0x7ff05c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:26.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.108+0000 7ff05bfff700 1 -- 172.21.15.152:0/1306611981 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff06804c5a0 con 0x7ff0740ff790 2024-03-04T15:55:26.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.108+0000 7ff07259c700 1 --2- 172.21.15.152:0/1306611981 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7ff05c0383c0 0x7ff05c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:26.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.109+0000 7ff07259c700 1 --2- 172.21.15.152:0/1306611981 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7ff05c0383c0 0x7ff05c03a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff064006fd0 tx=0x7ff064006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:26.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.113+0000 7ff05bfff700 1 -- 172.21.15.152:0/1306611981 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff06802a360 con 0x7ff0740ff790 2024-03-04T15:55:26.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.311+0000 7ff079137700 1 -- 172.21.15.152:0/1306611981 --> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi116:172.21.15.116=smithi116;smithi152:172.21.15.152=smithi152", "target": ["mon-mgr", ""]}) v1 -- 0x7ff074061340 con 0x7ff05c0383c0 2024-03-04T15:55:26.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.317+0000 7ff05bfff700 1 -- 172.21.15.152:0/1306611981 <== mgr.14164 v2:172.21.15.116:6800/2020358480 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff074061340 con 0x7ff05c0383c0 2024-03-04T15:55:26.319 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mon update... 2024-03-04T15:55:26.321 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.319+0000 7ff079137700 1 -- 172.21.15.152:0/1306611981 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7ff05c0383c0 msgr2=0x7ff05c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:26.321 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.319+0000 7ff079137700 1 --2- 172.21.15.152:0/1306611981 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7ff05c0383c0 0x7ff05c03a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff064006fd0 tx=0x7ff064006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:26.321 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.319+0000 7ff079137700 1 -- 172.21.15.152:0/1306611981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740ff790 msgr2=0x7ff0741959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:26.322 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.319+0000 7ff079137700 1 --2- 172.21.15.152:0/1306611981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740ff790 0x7ff0741959a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff068005b40 tx=0x7ff068004740 comp rx=0 tx=0).stop 2024-03-04T15:55:26.322 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.319+0000 7ff079137700 1 -- 172.21.15.152:0/1306611981 shutdown_connections 2024-03-04T15:55:26.322 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.319+0000 7ff079137700 1 --2- 172.21.15.152:0/1306611981 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7ff05c0383c0 0x7ff05c03a880 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:26.322 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.319+0000 7ff079137700 1 --2- 172.21.15.152:0/1306611981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740ff790 0x7ff0741959a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:26.322 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.319+0000 7ff079137700 1 -- 172.21.15.152:0/1306611981 >> 172.21.15.152:0/1306611981 conn(0x7ff0740fad10 msgr2=0x7ff074069140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:26.322 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.319+0000 7ff079137700 1 -- 172.21.15.152:0/1306611981 shutdown_connections 2024-03-04T15:55:26.322 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:26.319+0000 7ff079137700 1 -- 172.21.15.152:0/1306611981 wait complete. 2024-03-04T15:55:26.823 DEBUG:teuthology.orchestra.run.smithi152:mon.smithi152> sudo journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi152.service 2024-03-04T15:55:26.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:55:26.826 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:55:27.055 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:27.102 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:27.406 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:27 smithi116 ceph-mon[29120]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:27.406 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:27 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:28.452 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:28 smithi116 ceph-mon[29120]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi116:172.21.15.116=smithi116;smithi152:172.21.15.152=smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:55:28.452 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:28 smithi116 ceph-mon[29120]: Saving service mon spec with placement smithi116:172.21.15.116=smithi116;smithi152:172.21.15.152=smithi152;count:2 2024-03-04T15:55:28.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 -- 172.21.15.152:0/3974923578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d0101e70 msgr2=0x7f01d0102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:28.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 --2- 172.21.15.152:0/3974923578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d0101e70 0x7f01d0102290 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009910 tx=0x7f01c4009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:28.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 -- 172.21.15.152:0/3974923578 shutdown_connections 2024-03-04T15:55:28.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 --2- 172.21.15.152:0/3974923578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d0101e70 0x7f01d0102290 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:28.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 -- 172.21.15.152:0/3974923578 >> 172.21.15.152:0/3974923578 conn(0x7f01d00fa4b0 msgr2=0x7f01d00fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:28.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 -- 172.21.15.152:0/3974923578 shutdown_connections 2024-03-04T15:55:28.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 -- 172.21.15.152:0/3974923578 wait complete. 2024-03-04T15:55:28.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 Processor -- start 2024-03-04T15:55:28.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 -- start start 2024-03-04T15:55:28.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d0101e70 0x7f01d01959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:28.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.527+0000 7f01d6dc2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0195ef0 con 0x7f01d0101e70 2024-03-04T15:55:28.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.528+0000 7f01d4b5e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d0101e70 0x7f01d01959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:28.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.528+0000 7f01d4b5e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d0101e70 0x7f01d01959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:34808/0 (socket says 172.21.15.152:34808) 2024-03-04T15:55:28.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.528+0000 7f01d4b5e700 1 -- 172.21.15.152:0/1215662654 learned_addr learned my addr 172.21.15.152:0/1215662654 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:55:28.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.528+0000 7f01d4b5e700 1 -- 172.21.15.152:0/1215662654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c40095f0 con 0x7f01d0101e70 2024-03-04T15:55:28.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.528+0000 7f01d4b5e700 1 --2- 172.21.15.152:0/1215662654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d0101e70 0x7f01d01959b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005b40 tx=0x7f01c4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:28.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.528+0000 7f01cdffb700 1 -- 172.21.15.152:0/1215662654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c401d070 con 0x7f01d0101e70 2024-03-04T15:55:28.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.528+0000 7f01cdffb700 1 -- 172.21.15.152:0/1215662654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f01c400be10 con 0x7f01d0101e70 2024-03-04T15:55:28.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.528+0000 7f01d6dc2700 1 -- 172.21.15.152:0/1215662654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d01960f0 con 0x7f01d0101e70 2024-03-04T15:55:28.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.528+0000 7f01cdffb700 1 -- 172.21.15.152:0/1215662654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c400f460 con 0x7f01d0101e70 2024-03-04T15:55:28.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.528+0000 7f01d6dc2700 1 -- 172.21.15.152:0/1215662654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d0196590 con 0x7f01d0101e70 2024-03-04T15:55:28.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.529+0000 7f01cdffb700 1 -- 172.21.15.152:0/1215662654 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f01c400b960 con 0x7f01d0101e70 2024-03-04T15:55:28.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.529+0000 7f01d6dc2700 1 -- 172.21.15.152:0/1215662654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d018f4b0 con 0x7f01d0101e70 2024-03-04T15:55:28.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.529+0000 7f01cdffb700 1 --2- 172.21.15.152:0/1215662654 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f01c0038410 0x7f01c003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:28.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.529+0000 7f01cdffb700 1 -- 172.21.15.152:0/1215662654 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f01c404c510 con 0x7f01d0101e70 2024-03-04T15:55:28.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.530+0000 7f01cffff700 1 --2- 172.21.15.152:0/1215662654 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f01c0038410 0x7f01c003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:28.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.530+0000 7f01cffff700 1 --2- 172.21.15.152:0/1215662654 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f01c0038410 0x7f01c003a8d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f01bc006fd0 tx=0x7f01bc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:28.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.534+0000 7f01cdffb700 1 -- 172.21.15.152:0/1215662654 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f01c402bb60 con 0x7f01d0101e70 2024-03-04T15:55:28.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.798+0000 7f01d6dc2700 1 -- 172.21.15.152:0/1215662654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01d002cc70 con 0x7f01d0101e70 2024-03-04T15:55:28.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.799+0000 7f01cdffb700 1 -- 172.21.15.152:0/1215662654 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f01c4027030 con 0x7f01d0101e70 2024-03-04T15:55:28.801 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:55:28.801 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:55:28.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.801+0000 7f01d6dc2700 1 -- 172.21.15.152:0/1215662654 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f01c0038410 msgr2=0x7f01c003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:28.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.801+0000 7f01d6dc2700 1 --2- 172.21.15.152:0/1215662654 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f01c0038410 0x7f01c003a8d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f01bc006fd0 tx=0x7f01bc006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:28.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.801+0000 7f01d6dc2700 1 -- 172.21.15.152:0/1215662654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d0101e70 msgr2=0x7f01d01959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:28.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.801+0000 7f01d6dc2700 1 --2- 172.21.15.152:0/1215662654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d0101e70 0x7f01d01959b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005b40 tx=0x7f01c4004740 comp rx=0 tx=0).stop 2024-03-04T15:55:28.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.801+0000 7f01d6dc2700 1 -- 172.21.15.152:0/1215662654 shutdown_connections 2024-03-04T15:55:28.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.801+0000 7f01d6dc2700 1 --2- 172.21.15.152:0/1215662654 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f01c0038410 0x7f01c003a8d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:28.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.801+0000 7f01d6dc2700 1 --2- 172.21.15.152:0/1215662654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d0101e70 0x7f01d01959b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:28.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.801+0000 7f01d6dc2700 1 -- 172.21.15.152:0/1215662654 >> 172.21.15.152:0/1215662654 conn(0x7f01d00fa4b0 msgr2=0x7f01d00fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:28.804 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.801+0000 7f01d6dc2700 1 -- 172.21.15.152:0/1215662654 shutdown_connections 2024-03-04T15:55:28.804 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:28.801+0000 7f01d6dc2700 1 -- 172.21.15.152:0/1215662654 wait complete. 2024-03-04T15:55:28.804 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:55:29.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:29 smithi116 ceph-mon[29120]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:29.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:29 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/1215662654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:55:30.395 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:55:30.395 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:55:30.604 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:30.650 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:30.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:30.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:30.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:30.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:30.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:30.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:30.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T15:55:30.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:30.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:30.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:55:30.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:55:30.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:30.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: Deploying daemon grafana.smithi116 on smithi116 2024-03-04T15:55:30.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:30 smithi116 ceph-mon[29120]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:32.085 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.081+0000 7fd4b6853700 1 -- 172.21.15.152:0/1736080500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b01013b0 msgr2=0x7fd4b01037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:32.085 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.081+0000 7fd4b6853700 1 --2- 172.21.15.152:0/1736080500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b01013b0 0x7fd4b01037a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0009910 tx=0x7fd4a0009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:32.085 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.081+0000 7fd4b6853700 1 -- 172.21.15.152:0/1736080500 shutdown_connections 2024-03-04T15:55:32.085 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.081+0000 7fd4b6853700 1 --2- 172.21.15.152:0/1736080500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b01013b0 0x7fd4b01037a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:32.085 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.081+0000 7fd4b6853700 1 -- 172.21.15.152:0/1736080500 >> 172.21.15.152:0/1736080500 conn(0x7fd4b00fad20 msgr2=0x7fd4b00fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:32.085 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.081+0000 7fd4b6853700 1 -- 172.21.15.152:0/1736080500 shutdown_connections 2024-03-04T15:55:32.085 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.081+0000 7fd4b6853700 1 -- 172.21.15.152:0/1736080500 wait complete. 2024-03-04T15:55:32.085 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.082+0000 7fd4b6853700 1 Processor -- start 2024-03-04T15:55:32.085 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.082+0000 7fd4b6853700 1 -- start start 2024-03-04T15:55:32.085 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.082+0000 7fd4b6853700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b01013b0 0x7fd4b0195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.082+0000 7fd4b6853700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0195ec0 con 0x7fd4b01013b0 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.082+0000 7fd4affff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b01013b0 0x7fd4b0195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.082+0000 7fd4affff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b01013b0 0x7fd4b0195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:34816/0 (socket says 172.21.15.152:34816) 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.082+0000 7fd4affff700 1 -- 172.21.15.152:0/54508404 learned_addr learned my addr 172.21.15.152:0/54508404 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.082+0000 7fd4affff700 1 -- 172.21.15.152:0/54508404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a00095f0 con 0x7fd4b01013b0 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.083+0000 7fd4affff700 1 --2- 172.21.15.152:0/54508404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b01013b0 0x7fd4b0195980 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0005b40 tx=0x7fd4a0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.083+0000 7fd4ad7fa700 1 -- 172.21.15.152:0/54508404 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4a001d070 con 0x7fd4b01013b0 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.083+0000 7fd4b6853700 1 -- 172.21.15.152:0/54508404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b01960c0 con 0x7fd4b01013b0 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.083+0000 7fd4ad7fa700 1 -- 172.21.15.152:0/54508404 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd4a000be10 con 0x7fd4b01013b0 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.083+0000 7fd4ad7fa700 1 -- 172.21.15.152:0/54508404 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4a000f460 con 0x7fd4b01013b0 2024-03-04T15:55:32.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.083+0000 7fd4b6853700 1 -- 172.21.15.152:0/54508404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b0196560 con 0x7fd4b01013b0 2024-03-04T15:55:32.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.084+0000 7fd4b6853700 1 -- 172.21.15.152:0/54508404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b018f4c0 con 0x7fd4b01013b0 2024-03-04T15:55:32.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.084+0000 7fd4ad7fa700 1 -- 172.21.15.152:0/54508404 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fd4a00227e0 con 0x7fd4b01013b0 2024-03-04T15:55:32.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.084+0000 7fd4ad7fa700 1 --2- 172.21.15.152:0/54508404 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd4980383c0 0x7fd49803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:32.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.084+0000 7fd4ad7fa700 1 -- 172.21.15.152:0/54508404 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd4a004c580 con 0x7fd4b01013b0 2024-03-04T15:55:32.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.084+0000 7fd4af7fe700 1 --2- 172.21.15.152:0/54508404 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd4980383c0 0x7fd49803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:32.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.085+0000 7fd4af7fe700 1 --2- 172.21.15.152:0/54508404 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd4980383c0 0x7fd49803a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4006fd0 tx=0x7fd4a4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:32.091 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.089+0000 7fd4ad7fa700 1 -- 172.21.15.152:0/54508404 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd4a002bba0 con 0x7fd4b01013b0 2024-03-04T15:55:32.358 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.355+0000 7fd4b6853700 1 -- 172.21.15.152:0/54508404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4b002cc70 con 0x7fd4b01013b0 2024-03-04T15:55:32.358 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.356+0000 7fd4ad7fa700 1 -- 172.21.15.152:0/54508404 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fd4a0027020 con 0x7fd4b01013b0 2024-03-04T15:55:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:55:32.359 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:55:32.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.358+0000 7fd4b6853700 1 -- 172.21.15.152:0/54508404 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd4980383c0 msgr2=0x7fd49803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:32.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.358+0000 7fd4b6853700 1 --2- 172.21.15.152:0/54508404 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd4980383c0 0x7fd49803a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4006fd0 tx=0x7fd4a4006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:32.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.358+0000 7fd4b6853700 1 -- 172.21.15.152:0/54508404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b01013b0 msgr2=0x7fd4b0195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:32.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.358+0000 7fd4b6853700 1 --2- 172.21.15.152:0/54508404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b01013b0 0x7fd4b0195980 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0005b40 tx=0x7fd4a0004740 comp rx=0 tx=0).stop 2024-03-04T15:55:32.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.358+0000 7fd4b6853700 1 -- 172.21.15.152:0/54508404 shutdown_connections 2024-03-04T15:55:32.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.358+0000 7fd4b6853700 1 --2- 172.21.15.152:0/54508404 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fd4980383c0 0x7fd49803a880 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:32.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.358+0000 7fd4b6853700 1 --2- 172.21.15.152:0/54508404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b01013b0 0x7fd4b0195980 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:32.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.358+0000 7fd4b6853700 1 -- 172.21.15.152:0/54508404 >> 172.21.15.152:0/54508404 conn(0x7fd4b00fad20 msgr2=0x7fd4b00fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:32.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.358+0000 7fd4b6853700 1 -- 172.21.15.152:0/54508404 shutdown_connections 2024-03-04T15:55:32.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:32.358+0000 7fd4b6853700 1 -- 172.21.15.152:0/54508404 wait complete. 2024-03-04T15:55:32.362 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:55:33.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:32 smithi116 ceph-mon[29120]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:33.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:32 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/54508404' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:55:33.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:55:33.959 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:55:34.162 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:34.207 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:35.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:35 smithi116 ceph-mon[29120]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:35.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:35 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:35.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.794+0000 7fefa4698700 1 -- 172.21.15.152:0/210494389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef9c0ff7a0 msgr2=0x7fef9c0ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:35.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.794+0000 7fefa4698700 1 --2- 172.21.15.152:0/210494389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef9c0ff7a0 0x7fef9c0ffbc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fef8c009910 tx=0x7fef8c009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:35.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.794+0000 7fefa4698700 1 -- 172.21.15.152:0/210494389 shutdown_connections 2024-03-04T15:55:35.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.794+0000 7fefa4698700 1 --2- 172.21.15.152:0/210494389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef9c0ff7a0 0x7fef9c0ffbc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:35.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.794+0000 7fefa4698700 1 -- 172.21.15.152:0/210494389 >> 172.21.15.152:0/210494389 conn(0x7fef9c0fad20 msgr2=0x7fef9c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:35.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.794+0000 7fefa4698700 1 -- 172.21.15.152:0/210494389 shutdown_connections 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.794+0000 7fefa4698700 1 -- 172.21.15.152:0/210494389 wait complete. 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.795+0000 7fefa4698700 1 Processor -- start 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.795+0000 7fefa4698700 1 -- start start 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.795+0000 7fefa4698700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef9c0ff7a0 0x7fef9c069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.795+0000 7fefa4698700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c198710 con 0x7fef9c0ff7a0 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.795+0000 7fefa2434700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef9c0ff7a0 0x7fef9c069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.795+0000 7fefa2434700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef9c0ff7a0 0x7fef9c069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:56650/0 (socket says 172.21.15.152:56650) 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.795+0000 7fefa2434700 1 -- 172.21.15.152:0/1809394663 learned_addr learned my addr 172.21.15.152:0/1809394663 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.795+0000 7fefa2434700 1 -- 172.21.15.152:0/1809394663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c0095f0 con 0x7fef9c0ff7a0 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.795+0000 7fefa2434700 1 --2- 172.21.15.152:0/1809394663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef9c0ff7a0 0x7fef9c069390 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fef8c005b40 tx=0x7fef8c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.796+0000 7fef937fe700 1 -- 172.21.15.152:0/1809394663 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef8c01d070 con 0x7fef9c0ff7a0 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.796+0000 7fef937fe700 1 -- 172.21.15.152:0/1809394663 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fef8c00be10 con 0x7fef9c0ff7a0 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.796+0000 7fefa4698700 1 -- 172.21.15.152:0/1809394663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c198850 con 0x7fef9c0ff7a0 2024-03-04T15:55:35.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.796+0000 7fef937fe700 1 -- 172.21.15.152:0/1809394663 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef8c00f460 con 0x7fef9c0ff7a0 2024-03-04T15:55:35.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.796+0000 7fefa4698700 1 -- 172.21.15.152:0/1809394663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c198cf0 con 0x7fef9c0ff7a0 2024-03-04T15:55:35.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.797+0000 7fef937fe700 1 -- 172.21.15.152:0/1809394663 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fef8c00b960 con 0x7fef9c0ff7a0 2024-03-04T15:55:35.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.797+0000 7fefa4698700 1 -- 172.21.15.152:0/1809394663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c18e3c0 con 0x7fef9c0ff7a0 2024-03-04T15:55:35.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.797+0000 7fef937fe700 1 --2- 172.21.15.152:0/1809394663 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fef880383c0 0x7fef8803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:35.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.797+0000 7fef937fe700 1 -- 172.21.15.152:0/1809394663 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fef8c04c3c0 con 0x7fef9c0ff7a0 2024-03-04T15:55:35.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.797+0000 7fefa1c33700 1 --2- 172.21.15.152:0/1809394663 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fef880383c0 0x7fef8803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:35.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.797+0000 7fefa1c33700 1 --2- 172.21.15.152:0/1809394663 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fef880383c0 0x7fef8803a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fef98006fd0 tx=0x7fef98006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:35.804 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:35.802+0000 7fef937fe700 1 -- 172.21.15.152:0/1809394663 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fef8c02a360 con 0x7fef9c0ff7a0 2024-03-04T15:55:36.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.064+0000 7fefa4698700 1 -- 172.21.15.152:0/1809394663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef9c02cc70 con 0x7fef9c0ff7a0 2024-03-04T15:55:36.067 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.065+0000 7fef937fe700 1 -- 172.21.15.152:0/1809394663 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fef8c027030 con 0x7fef9c0ff7a0 2024-03-04T15:55:36.067 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:55:36.067 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:55:36.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.067+0000 7fefa4698700 1 -- 172.21.15.152:0/1809394663 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fef880383c0 msgr2=0x7fef8803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:36.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.067+0000 7fefa4698700 1 --2- 172.21.15.152:0/1809394663 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fef880383c0 0x7fef8803a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fef98006fd0 tx=0x7fef98006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:36.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.067+0000 7fefa4698700 1 -- 172.21.15.152:0/1809394663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef9c0ff7a0 msgr2=0x7fef9c069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:36.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.067+0000 7fefa4698700 1 --2- 172.21.15.152:0/1809394663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef9c0ff7a0 0x7fef9c069390 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fef8c005b40 tx=0x7fef8c004740 comp rx=0 tx=0).stop 2024-03-04T15:55:36.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.067+0000 7fefa4698700 1 -- 172.21.15.152:0/1809394663 shutdown_connections 2024-03-04T15:55:36.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.067+0000 7fefa4698700 1 --2- 172.21.15.152:0/1809394663 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fef880383c0 0x7fef8803a880 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:36.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.067+0000 7fefa4698700 1 --2- 172.21.15.152:0/1809394663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef9c0ff7a0 0x7fef9c069390 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:36.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.067+0000 7fefa4698700 1 -- 172.21.15.152:0/1809394663 >> 172.21.15.152:0/1809394663 conn(0x7fef9c0fad20 msgr2=0x7fef9c0fba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:36.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.067+0000 7fefa4698700 1 -- 172.21.15.152:0/1809394663 shutdown_connections 2024-03-04T15:55:36.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:36.067+0000 7fefa4698700 1 -- 172.21.15.152:0/1809394663 wait complete. 2024-03-04T15:55:36.070 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:55:37.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:37 smithi116 ceph-mon[29120]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:37.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:37 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/1809394663' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:55:37.664 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:55:37.664 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:55:37.869 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:37.916 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:39.442 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.439+0000 7fee2e278700 1 -- 172.21.15.152:0/1208438495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee281008a0 msgr2=0x7fee28100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:39.442 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.439+0000 7fee2e278700 1 --2- 172.21.15.152:0/1208438495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee281008a0 0x7fee28100cc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fee18009910 tx=0x7fee18009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:39.442 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.439+0000 7fee2e278700 1 -- 172.21.15.152:0/1208438495 shutdown_connections 2024-03-04T15:55:39.442 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.439+0000 7fee2e278700 1 --2- 172.21.15.152:0/1208438495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee281008a0 0x7fee28100cc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:39.442 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.439+0000 7fee2e278700 1 -- 172.21.15.152:0/1208438495 >> 172.21.15.152:0/1208438495 conn(0x7fee280fbe60 msgr2=0x7fee280fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:39.442 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.439+0000 7fee2e278700 1 -- 172.21.15.152:0/1208438495 shutdown_connections 2024-03-04T15:55:39.442 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.439+0000 7fee2e278700 1 -- 172.21.15.152:0/1208438495 wait complete. 2024-03-04T15:55:39.443 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.439+0000 7fee2e278700 1 Processor -- start 2024-03-04T15:55:39.443 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.439+0000 7fee2e278700 1 -- start start 2024-03-04T15:55:39.443 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee2e278700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee281008a0 0x7fee28195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:39.443 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee2e278700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee28195e90 con 0x7fee281008a0 2024-03-04T15:55:39.443 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee27fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee281008a0 0x7fee28195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:39.443 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee27fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee281008a0 0x7fee28195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:56678/0 (socket says 172.21.15.152:56678) 2024-03-04T15:55:39.443 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee27fff700 1 -- 172.21.15.152:0/3316760287 learned_addr learned my addr 172.21.15.152:0/3316760287 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:55:39.443 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee27fff700 1 -- 172.21.15.152:0/3316760287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee180095f0 con 0x7fee281008a0 2024-03-04T15:55:39.443 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee27fff700 1 --2- 172.21.15.152:0/3316760287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee281008a0 0x7fee28195950 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fee18005b40 tx=0x7fee18004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:39.448 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee257fa700 1 -- 172.21.15.152:0/3316760287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee1801d070 con 0x7fee281008a0 2024-03-04T15:55:39.448 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee257fa700 1 -- 172.21.15.152:0/3316760287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fee1800be10 con 0x7fee281008a0 2024-03-04T15:55:39.448 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee257fa700 1 -- 172.21.15.152:0/3316760287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee1800f460 con 0x7fee281008a0 2024-03-04T15:55:39.448 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.440+0000 7fee2e278700 1 -- 172.21.15.152:0/3316760287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee28196090 con 0x7fee281008a0 2024-03-04T15:55:39.448 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.441+0000 7fee2e278700 1 -- 172.21.15.152:0/3316760287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee28196530 con 0x7fee281008a0 2024-03-04T15:55:39.448 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.441+0000 7fee2e278700 1 -- 172.21.15.152:0/3316760287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2818f4c0 con 0x7fee281008a0 2024-03-04T15:55:39.448 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.442+0000 7fee257fa700 1 -- 172.21.15.152:0/3316760287 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fee180227e0 con 0x7fee281008a0 2024-03-04T15:55:39.448 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.442+0000 7fee257fa700 1 --2- 172.21.15.152:0/3316760287 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fee100383c0 0x7fee1003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:39.449 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.442+0000 7fee257fa700 1 -- 172.21.15.152:0/3316760287 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fee1804c4a0 con 0x7fee281008a0 2024-03-04T15:55:39.449 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.442+0000 7fee277fe700 1 --2- 172.21.15.152:0/3316760287 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fee100383c0 0x7fee1003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:39.449 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.442+0000 7fee277fe700 1 --2- 172.21.15.152:0/3316760287 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fee100383c0 0x7fee1003a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fee1c006fd0 tx=0x7fee1c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:39.449 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.447+0000 7fee257fa700 1 -- 172.21.15.152:0/3316760287 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fee1802bba0 con 0x7fee281008a0 2024-03-04T15:55:39.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:39 smithi116 ceph-mon[29120]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:39.716 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.714+0000 7fee2e278700 1 -- 172.21.15.152:0/3316760287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee2802cc70 con 0x7fee281008a0 2024-03-04T15:55:39.716 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.714+0000 7fee257fa700 1 -- 172.21.15.152:0/3316760287 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fee18027020 con 0x7fee281008a0 2024-03-04T15:55:39.717 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:55:39.717 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:55:39.718 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.716+0000 7fee2e278700 1 -- 172.21.15.152:0/3316760287 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fee100383c0 msgr2=0x7fee1003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:39.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.716+0000 7fee2e278700 1 --2- 172.21.15.152:0/3316760287 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fee100383c0 0x7fee1003a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fee1c006fd0 tx=0x7fee1c006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:39.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.716+0000 7fee2e278700 1 -- 172.21.15.152:0/3316760287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee281008a0 msgr2=0x7fee28195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:39.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.716+0000 7fee2e278700 1 --2- 172.21.15.152:0/3316760287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee281008a0 0x7fee28195950 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fee18005b40 tx=0x7fee18004740 comp rx=0 tx=0).stop 2024-03-04T15:55:39.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.716+0000 7fee2e278700 1 -- 172.21.15.152:0/3316760287 shutdown_connections 2024-03-04T15:55:39.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.716+0000 7fee2e278700 1 --2- 172.21.15.152:0/3316760287 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fee100383c0 0x7fee1003a880 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:39.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.716+0000 7fee2e278700 1 --2- 172.21.15.152:0/3316760287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee281008a0 0x7fee28195950 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:39.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.716+0000 7fee2e278700 1 -- 172.21.15.152:0/3316760287 >> 172.21.15.152:0/3316760287 conn(0x7fee280fbe60 msgr2=0x7fee280fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:39.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.716+0000 7fee2e278700 1 -- 172.21.15.152:0/3316760287 shutdown_connections 2024-03-04T15:55:39.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:39.716+0000 7fee2e278700 1 -- 172.21.15.152:0/3316760287 wait complete. 2024-03-04T15:55:39.719 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:55:40.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:40 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/3316760287' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:55:41.428 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:55:41.429 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:55:41.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:41 smithi116 ceph-mon[29120]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:41.634 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:41.680 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:43.257 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.255+0000 7fab6a071700 1 -- 172.21.15.152:0/1302883524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab640ff7a0 msgr2=0x7fab640ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:43.257 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.255+0000 7fab6a071700 1 --2- 172.21.15.152:0/1302883524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab640ff7a0 0x7fab640ffbc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fab58009910 tx=0x7fab58009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:43.258 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.255+0000 7fab6a071700 1 -- 172.21.15.152:0/1302883524 shutdown_connections 2024-03-04T15:55:43.258 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.255+0000 7fab6a071700 1 --2- 172.21.15.152:0/1302883524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab640ff7a0 0x7fab640ffbc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:43.258 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.255+0000 7fab6a071700 1 -- 172.21.15.152:0/1302883524 >> 172.21.15.152:0/1302883524 conn(0x7fab640fad20 msgr2=0x7fab640fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:43.258 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.255+0000 7fab6a071700 1 -- 172.21.15.152:0/1302883524 shutdown_connections 2024-03-04T15:55:43.258 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.255+0000 7fab6a071700 1 -- 172.21.15.152:0/1302883524 wait complete. 2024-03-04T15:55:43.259 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.256+0000 7fab6a071700 1 Processor -- start 2024-03-04T15:55:43.259 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.256+0000 7fab6a071700 1 -- start start 2024-03-04T15:55:43.259 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.256+0000 7fab6a071700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab640ff7a0 0x7fab641948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:43.259 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.256+0000 7fab6a071700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab64194de0 con 0x7fab640ff7a0 2024-03-04T15:55:43.260 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.256+0000 7fab637fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab640ff7a0 0x7fab641948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:43.260 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.257+0000 7fab637fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab640ff7a0 0x7fab641948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:56698/0 (socket says 172.21.15.152:56698) 2024-03-04T15:55:43.260 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.257+0000 7fab637fe700 1 -- 172.21.15.152:0/3008599623 learned_addr learned my addr 172.21.15.152:0/3008599623 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:55:43.260 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.257+0000 7fab637fe700 1 -- 172.21.15.152:0/3008599623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab580095f0 con 0x7fab640ff7a0 2024-03-04T15:55:43.260 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.257+0000 7fab637fe700 1 --2- 172.21.15.152:0/3008599623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab640ff7a0 0x7fab641948a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab58005b40 tx=0x7fab58004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:43.260 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.257+0000 7fab60ff9700 1 -- 172.21.15.152:0/3008599623 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab5801d070 con 0x7fab640ff7a0 2024-03-04T15:55:43.260 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.257+0000 7fab60ff9700 1 -- 172.21.15.152:0/3008599623 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fab5800be10 con 0x7fab640ff7a0 2024-03-04T15:55:43.260 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.257+0000 7fab60ff9700 1 -- 172.21.15.152:0/3008599623 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab5800f460 con 0x7fab640ff7a0 2024-03-04T15:55:43.260 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.257+0000 7fab6a071700 1 -- 172.21.15.152:0/3008599623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab64194fe0 con 0x7fab640ff7a0 2024-03-04T15:55:43.260 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.257+0000 7fab6a071700 1 -- 172.21.15.152:0/3008599623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab64195480 con 0x7fab640ff7a0 2024-03-04T15:55:43.261 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.258+0000 7fab60ff9700 1 -- 172.21.15.152:0/3008599623 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fab5800b960 con 0x7fab640ff7a0 2024-03-04T15:55:43.261 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.258+0000 7fab60ff9700 1 --2- 172.21.15.152:0/3008599623 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fab4c038410 0x7fab4c03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:43.261 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.258+0000 7fab6a071700 1 -- 172.21.15.152:0/3008599623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6418e3c0 con 0x7fab640ff7a0 2024-03-04T15:55:43.261 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.258+0000 7fab60ff9700 1 -- 172.21.15.152:0/3008599623 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fab5804c420 con 0x7fab640ff7a0 2024-03-04T15:55:43.261 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.258+0000 7fab62ffd700 1 --2- 172.21.15.152:0/3008599623 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fab4c038410 0x7fab4c03a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:43.261 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.259+0000 7fab62ffd700 1 --2- 172.21.15.152:0/3008599623 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fab4c038410 0x7fab4c03a8d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fab54006fd0 tx=0x7fab54006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:43.265 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.263+0000 7fab60ff9700 1 -- 172.21.15.152:0/3008599623 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fab58027070 con 0x7fab640ff7a0 2024-03-04T15:55:43.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:43 smithi116 ceph-mon[29120]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:43.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.525+0000 7fab6a071700 1 -- 172.21.15.152:0/3008599623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab6402cc70 con 0x7fab640ff7a0 2024-03-04T15:55:43.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.526+0000 7fab60ff9700 1 -- 172.21.15.152:0/3008599623 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fab5800f5c0 con 0x7fab640ff7a0 2024-03-04T15:55:43.529 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:55:43.529 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:55:43.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.528+0000 7fab6a071700 1 -- 172.21.15.152:0/3008599623 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fab4c038410 msgr2=0x7fab4c03a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:43.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.528+0000 7fab6a071700 1 --2- 172.21.15.152:0/3008599623 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fab4c038410 0x7fab4c03a8d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fab54006fd0 tx=0x7fab54006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:43.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.528+0000 7fab6a071700 1 -- 172.21.15.152:0/3008599623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab640ff7a0 msgr2=0x7fab641948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:43.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.528+0000 7fab6a071700 1 --2- 172.21.15.152:0/3008599623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab640ff7a0 0x7fab641948a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab58005b40 tx=0x7fab58004740 comp rx=0 tx=0).stop 2024-03-04T15:55:43.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.528+0000 7fab6a071700 1 -- 172.21.15.152:0/3008599623 shutdown_connections 2024-03-04T15:55:43.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.528+0000 7fab6a071700 1 --2- 172.21.15.152:0/3008599623 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fab4c038410 0x7fab4c03a8d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:43.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.528+0000 7fab6a071700 1 --2- 172.21.15.152:0/3008599623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab640ff7a0 0x7fab641948a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:43.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.528+0000 7fab6a071700 1 -- 172.21.15.152:0/3008599623 >> 172.21.15.152:0/3008599623 conn(0x7fab640fad20 msgr2=0x7fab640fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:43.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.528+0000 7fab6a071700 1 -- 172.21.15.152:0/3008599623 shutdown_connections 2024-03-04T15:55:43.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:43.528+0000 7fab6a071700 1 -- 172.21.15.152:0/3008599623 wait complete. 2024-03-04T15:55:43.532 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:55:44.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:44 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/3008599623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:55:45.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:55:45.057 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:55:45.265 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:45.312 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:45.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:45 smithi116 ceph-mon[29120]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:46.663 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.658+0000 7f88c2650700 1 -- 172.21.15.152:0/1353289099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88bc0feb50 msgr2=0x7f88bc0fef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:46.663 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.658+0000 7f88c2650700 1 --2- 172.21.15.152:0/1353289099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88bc0feb50 0x7f88bc0fef70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009910 tx=0x7f88ac009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.658+0000 7f88c2650700 1 -- 172.21.15.152:0/1353289099 shutdown_connections 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.658+0000 7f88c2650700 1 --2- 172.21.15.152:0/1353289099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88bc0feb50 0x7f88bc0fef70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.658+0000 7f88c2650700 1 -- 172.21.15.152:0/1353289099 >> 172.21.15.152:0/1353289099 conn(0x7f88bc0fa4b0 msgr2=0x7f88bc0fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.658+0000 7f88c2650700 1 -- 172.21.15.152:0/1353289099 shutdown_connections 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.658+0000 7f88c2650700 1 -- 172.21.15.152:0/1353289099 wait complete. 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.659+0000 7f88c2650700 1 Processor -- start 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.659+0000 7f88c2650700 1 -- start start 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.659+0000 7f88c2650700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88bc0feb50 0x7f88bc1959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.659+0000 7f88c2650700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc195f00 con 0x7f88bc0feb50 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.659+0000 7f88bbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88bc0feb50 0x7f88bc1959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.659+0000 7f88bbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88bc0feb50 0x7f88bc1959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:42906/0 (socket says 172.21.15.152:42906) 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.659+0000 7f88bbfff700 1 -- 172.21.15.152:0/202822766 learned_addr learned my addr 172.21.15.152:0/202822766 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:55:46.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.660+0000 7f88bbfff700 1 -- 172.21.15.152:0/202822766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac0095f0 con 0x7f88bc0feb50 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.660+0000 7f88bbfff700 1 --2- 172.21.15.152:0/202822766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88bc0feb50 0x7f88bc1959c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f88ac005950 tx=0x7f88ac009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.660+0000 7f88b97fa700 1 -- 172.21.15.152:0/202822766 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88ac01d070 con 0x7f88bc0feb50 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.660+0000 7f88b97fa700 1 -- 172.21.15.152:0/202822766 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f88ac00be10 con 0x7f88bc0feb50 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.660+0000 7f88c2650700 1 -- 172.21.15.152:0/202822766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc196100 con 0x7f88bc0feb50 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.660+0000 7f88b97fa700 1 -- 172.21.15.152:0/202822766 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88ac00f460 con 0x7f88bc0feb50 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.660+0000 7f88c2650700 1 -- 172.21.15.152:0/202822766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc1965a0 con 0x7f88bc0feb50 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.661+0000 7f88b97fa700 1 -- 172.21.15.152:0/202822766 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f88ac00f690 con 0x7f88bc0feb50 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.661+0000 7f88b97fa700 1 --2- 172.21.15.152:0/202822766 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f88a40383c0 0x7f88a403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.661+0000 7f88b97fa700 1 -- 172.21.15.152:0/202822766 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f88ac04c330 con 0x7f88bc0feb50 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.661+0000 7f88c2650700 1 -- 172.21.15.152:0/202822766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc18f4c0 con 0x7f88bc0feb50 2024-03-04T15:55:46.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.661+0000 7f88bb7fe700 1 --2- 172.21.15.152:0/202822766 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f88a40383c0 0x7f88a403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:46.666 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.662+0000 7f88bb7fe700 1 --2- 172.21.15.152:0/202822766 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f88a40383c0 0x7f88a403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f88b0006fd0 tx=0x7f88b0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:46.668 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.666+0000 7f88b97fa700 1 -- 172.21.15.152:0/202822766 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f88ac027030 con 0x7f88bc0feb50 2024-03-04T15:55:46.932 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.930+0000 7f88c2650700 1 -- 172.21.15.152:0/202822766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88bc0ff680 con 0x7f88bc0feb50 2024-03-04T15:55:46.932 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.930+0000 7f88b97fa700 1 -- 172.21.15.152:0/202822766 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f88bc0ff680 con 0x7f88bc0feb50 2024-03-04T15:55:46.933 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:55:46.933 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:55:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.932+0000 7f88c2650700 1 -- 172.21.15.152:0/202822766 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f88a40383c0 msgr2=0x7f88a403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.932+0000 7f88c2650700 1 --2- 172.21.15.152:0/202822766 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f88a40383c0 0x7f88a403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f88b0006fd0 tx=0x7f88b0006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.933+0000 7f88c2650700 1 -- 172.21.15.152:0/202822766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88bc0feb50 msgr2=0x7f88bc1959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.933+0000 7f88c2650700 1 --2- 172.21.15.152:0/202822766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88bc0feb50 0x7f88bc1959c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f88ac005950 tx=0x7f88ac009e60 comp rx=0 tx=0).stop 2024-03-04T15:55:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.933+0000 7f88c2650700 1 -- 172.21.15.152:0/202822766 shutdown_connections 2024-03-04T15:55:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.933+0000 7f88c2650700 1 --2- 172.21.15.152:0/202822766 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f88a40383c0 0x7f88a403a880 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.933+0000 7f88c2650700 1 --2- 172.21.15.152:0/202822766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88bc0feb50 0x7f88bc1959c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.933+0000 7f88c2650700 1 -- 172.21.15.152:0/202822766 >> 172.21.15.152:0/202822766 conn(0x7f88bc0fa4b0 msgr2=0x7f88bc0fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.933+0000 7f88c2650700 1 -- 172.21.15.152:0/202822766 shutdown_connections 2024-03-04T15:55:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:46.933+0000 7f88c2650700 1 -- 172.21.15.152:0/202822766 wait complete. 2024-03-04T15:55:46.936 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:55:47.204 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:46 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:47.204 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:46 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:47.204 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:46 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:47.204 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:46 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:47.204 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:46 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:47.204 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:46 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:47.204 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:46 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:47.204 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:46 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:47.204 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:46 smithi116 ceph-mon[29120]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:47.204 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:46 smithi116 ceph-mon[29120]: Deploying daemon prometheus.smithi116 on smithi116 2024-03-04T15:55:48.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:47 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/202822766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:55:48.462 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:55:48.462 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:55:48.666 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:48.713 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:49.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:49 smithi116 ceph-mon[29120]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:49.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:49 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:55:50.202 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 -- 172.21.15.152:0/3535471488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c8074dc0 msgr2=0x7f05c8073220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:50.202 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 --2- 172.21.15.152:0/3535471488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c8074dc0 0x7f05c8073220 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009910 tx=0x7f05bc009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:50.202 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 -- 172.21.15.152:0/3535471488 shutdown_connections 2024-03-04T15:55:50.202 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 --2- 172.21.15.152:0/3535471488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c8074dc0 0x7f05c8073220 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 -- 172.21.15.152:0/3535471488 >> 172.21.15.152:0/3535471488 conn(0x7f05c80fbe30 msgr2=0x7f05c80fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 -- 172.21.15.152:0/3535471488 shutdown_connections 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 -- 172.21.15.152:0/3535471488 wait complete. 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 Processor -- start 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 -- start start 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c8074dc0 0x7f05c819c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.199+0000 7f05ceb26700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c819c690 con 0x7f05c8074dc0 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.200+0000 7f05cc8c2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c8074dc0 0x7f05c819c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.200+0000 7f05cc8c2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c8074dc0 0x7f05c819c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:42920/0 (socket says 172.21.15.152:42920) 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.200+0000 7f05cc8c2700 1 -- 172.21.15.152:0/4041307649 learned_addr learned my addr 172.21.15.152:0/4041307649 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.200+0000 7f05cc8c2700 1 -- 172.21.15.152:0/4041307649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc0095f0 con 0x7f05c8074dc0 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.200+0000 7f05cc8c2700 1 --2- 172.21.15.152:0/4041307649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c8074dc0 0x7f05c819c150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f05bc005b40 tx=0x7f05bc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.200+0000 7f05c5ffb700 1 -- 172.21.15.152:0/4041307649 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05bc01d070 con 0x7f05c8074dc0 2024-03-04T15:55:50.203 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.200+0000 7f05c5ffb700 1 -- 172.21.15.152:0/4041307649 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f05bc00be10 con 0x7f05c8074dc0 2024-03-04T15:55:50.204 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.200+0000 7f05c5ffb700 1 -- 172.21.15.152:0/4041307649 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05bc00f460 con 0x7f05c8074dc0 2024-03-04T15:55:50.204 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.200+0000 7f05ceb26700 1 -- 172.21.15.152:0/4041307649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c81044f0 con 0x7f05c8074dc0 2024-03-04T15:55:50.204 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.200+0000 7f05ceb26700 1 -- 172.21.15.152:0/4041307649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c819cba0 con 0x7f05c8074dc0 2024-03-04T15:55:50.205 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.201+0000 7f05c5ffb700 1 -- 172.21.15.152:0/4041307649 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f05bc0227e0 con 0x7f05c8074dc0 2024-03-04T15:55:50.205 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.201+0000 7f05ceb26700 1 -- 172.21.15.152:0/4041307649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c8195c70 con 0x7f05c8074dc0 2024-03-04T15:55:50.205 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.201+0000 7f05c5ffb700 1 --2- 172.21.15.152:0/4041307649 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f05b8038410 0x7f05b803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:50.205 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.201+0000 7f05c5ffb700 1 -- 172.21.15.152:0/4041307649 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f05bc04c420 con 0x7f05c8074dc0 2024-03-04T15:55:50.205 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.202+0000 7f05c7fff700 1 --2- 172.21.15.152:0/4041307649 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f05b8038410 0x7f05b803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:50.205 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.202+0000 7f05c7fff700 1 --2- 172.21.15.152:0/4041307649 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f05b8038410 0x7f05b803a8d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f05b4006fd0 tx=0x7f05b4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:50.208 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.206+0000 7f05c5ffb700 1 -- 172.21.15.152:0/4041307649 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f05bc02ba00 con 0x7f05c8074dc0 2024-03-04T15:55:50.471 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.469+0000 7f05ceb26700 1 -- 172.21.15.152:0/4041307649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05c80623c0 con 0x7f05c8074dc0 2024-03-04T15:55:50.472 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.470+0000 7f05c5ffb700 1 -- 172.21.15.152:0/4041307649 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f05bc027030 con 0x7f05c8074dc0 2024-03-04T15:55:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:55:50.473 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:55:50.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.472+0000 7f05ceb26700 1 -- 172.21.15.152:0/4041307649 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f05b8038410 msgr2=0x7f05b803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:50.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.472+0000 7f05ceb26700 1 --2- 172.21.15.152:0/4041307649 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f05b8038410 0x7f05b803a8d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f05b4006fd0 tx=0x7f05b4006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:50.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.472+0000 7f05ceb26700 1 -- 172.21.15.152:0/4041307649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c8074dc0 msgr2=0x7f05c819c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:50.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.472+0000 7f05ceb26700 1 --2- 172.21.15.152:0/4041307649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c8074dc0 0x7f05c819c150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f05bc005b40 tx=0x7f05bc004740 comp rx=0 tx=0).stop 2024-03-04T15:55:50.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.472+0000 7f05ceb26700 1 -- 172.21.15.152:0/4041307649 shutdown_connections 2024-03-04T15:55:50.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.472+0000 7f05ceb26700 1 --2- 172.21.15.152:0/4041307649 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f05b8038410 0x7f05b803a8d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:50.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.472+0000 7f05ceb26700 1 --2- 172.21.15.152:0/4041307649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c8074dc0 0x7f05c819c150 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:50.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.472+0000 7f05ceb26700 1 -- 172.21.15.152:0/4041307649 >> 172.21.15.152:0/4041307649 conn(0x7f05c80fbe30 msgr2=0x7f05c80fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:50.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.472+0000 7f05ceb26700 1 -- 172.21.15.152:0/4041307649 shutdown_connections 2024-03-04T15:55:50.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:50.472+0000 7f05ceb26700 1 -- 172.21.15.152:0/4041307649 wait complete. 2024-03-04T15:55:50.475 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:55:51.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:51 smithi116 ceph-mon[29120]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:51.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:51 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/4041307649' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:55:52.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:55:52.058 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:55:52.259 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:52.306 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:53.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:53 smithi116 ceph-mon[29120]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:53.678 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.675+0000 7f72f03ba700 1 -- 172.21.15.152:0/1099579889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e81013f0 msgr2=0x7f72e81037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.675+0000 7f72f03ba700 1 --2- 172.21.15.152:0/1099579889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e81013f0 0x7f72e81037e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009910 tx=0x7f72d8009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.676+0000 7f72f03ba700 1 -- 172.21.15.152:0/1099579889 shutdown_connections 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.676+0000 7f72f03ba700 1 --2- 172.21.15.152:0/1099579889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e81013f0 0x7f72e81037e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.676+0000 7f72f03ba700 1 -- 172.21.15.152:0/1099579889 >> 172.21.15.152:0/1099579889 conn(0x7f72e80fad20 msgr2=0x7f72e80fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.676+0000 7f72f03ba700 1 -- 172.21.15.152:0/1099579889 shutdown_connections 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.676+0000 7f72f03ba700 1 -- 172.21.15.152:0/1099579889 wait complete. 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.676+0000 7f72f03ba700 1 Processor -- start 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.676+0000 7f72f03ba700 1 -- start start 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.676+0000 7f72f03ba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e81013f0 0x7f72e8195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.676+0000 7f72f03ba700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e8195ec0 con 0x7f72e81013f0 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.677+0000 7f72ee156700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e81013f0 0x7f72e8195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:53.679 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.677+0000 7f72ee156700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e81013f0 0x7f72e8195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:42942/0 (socket says 172.21.15.152:42942) 2024-03-04T15:55:53.680 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.677+0000 7f72ee156700 1 -- 172.21.15.152:0/2233829469 learned_addr learned my addr 172.21.15.152:0/2233829469 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:55:53.680 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.677+0000 7f72ee156700 1 -- 172.21.15.152:0/2233829469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d80095f0 con 0x7f72e81013f0 2024-03-04T15:55:53.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.677+0000 7f72ee156700 1 --2- 172.21.15.152:0/2233829469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e81013f0 0x7f72e8195980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f72d8005b40 tx=0x7f72d8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:53.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.677+0000 7f72df7fe700 1 -- 172.21.15.152:0/2233829469 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72d801d070 con 0x7f72e81013f0 2024-03-04T15:55:53.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.677+0000 7f72f03ba700 1 -- 172.21.15.152:0/2233829469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e81960c0 con 0x7f72e81013f0 2024-03-04T15:55:53.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.677+0000 7f72df7fe700 1 -- 172.21.15.152:0/2233829469 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f72d800be10 con 0x7f72e81013f0 2024-03-04T15:55:53.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.677+0000 7f72df7fe700 1 -- 172.21.15.152:0/2233829469 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72d800f460 con 0x7f72e81013f0 2024-03-04T15:55:53.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.677+0000 7f72f03ba700 1 -- 172.21.15.152:0/2233829469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e8196560 con 0x7f72e81013f0 2024-03-04T15:55:53.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.678+0000 7f72f03ba700 1 -- 172.21.15.152:0/2233829469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e818f4d0 con 0x7f72e81013f0 2024-03-04T15:55:53.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.678+0000 7f72df7fe700 1 -- 172.21.15.152:0/2233829469 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f72d80227e0 con 0x7f72e81013f0 2024-03-04T15:55:53.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.678+0000 7f72df7fe700 1 --2- 172.21.15.152:0/2233829469 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f72d40383c0 0x7f72d403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:53.682 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.679+0000 7f72df7fe700 1 -- 172.21.15.152:0/2233829469 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f72d804c4b0 con 0x7f72e81013f0 2024-03-04T15:55:53.682 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.679+0000 7f72ed955700 1 --2- 172.21.15.152:0/2233829469 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f72d40383c0 0x7f72d403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:53.682 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.679+0000 7f72ed955700 1 --2- 172.21.15.152:0/2233829469 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f72d40383c0 0x7f72d403a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f72e4006fd0 tx=0x7f72e4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:53.685 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.683+0000 7f72df7fe700 1 -- 172.21.15.152:0/2233829469 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f72d802bba0 con 0x7f72e81013f0 2024-03-04T15:55:53.953 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.951+0000 7f72f03ba700 1 -- 172.21.15.152:0/2233829469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72e802cc70 con 0x7f72e81013f0 2024-03-04T15:55:53.953 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.951+0000 7f72df7fe700 1 -- 172.21.15.152:0/2233829469 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f72d8027030 con 0x7f72e81013f0 2024-03-04T15:55:53.953 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:55:53.954 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:55:53.955 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.953+0000 7f72f03ba700 1 -- 172.21.15.152:0/2233829469 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f72d40383c0 msgr2=0x7f72d403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:53.955 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.953+0000 7f72f03ba700 1 --2- 172.21.15.152:0/2233829469 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f72d40383c0 0x7f72d403a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f72e4006fd0 tx=0x7f72e4006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:53.955 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.953+0000 7f72f03ba700 1 -- 172.21.15.152:0/2233829469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e81013f0 msgr2=0x7f72e8195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:53.955 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.953+0000 7f72f03ba700 1 --2- 172.21.15.152:0/2233829469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e81013f0 0x7f72e8195980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f72d8005b40 tx=0x7f72d8004740 comp rx=0 tx=0).stop 2024-03-04T15:55:53.956 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.953+0000 7f72f03ba700 1 -- 172.21.15.152:0/2233829469 shutdown_connections 2024-03-04T15:55:53.956 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.953+0000 7f72f03ba700 1 --2- 172.21.15.152:0/2233829469 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7f72d40383c0 0x7f72d403a880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:53.956 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.953+0000 7f72f03ba700 1 --2- 172.21.15.152:0/2233829469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e81013f0 0x7f72e8195980 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:53.956 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.953+0000 7f72f03ba700 1 -- 172.21.15.152:0/2233829469 >> 172.21.15.152:0/2233829469 conn(0x7f72e80fad20 msgr2=0x7f72e80fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:53.956 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.953+0000 7f72f03ba700 1 -- 172.21.15.152:0/2233829469 shutdown_connections 2024-03-04T15:55:53.956 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:53.953+0000 7f72f03ba700 1 -- 172.21.15.152:0/2233829469 wait complete. 2024-03-04T15:55:53.956 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:55:54.377 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:54 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/2233829469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:55:55.392 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:55 smithi116 ceph-mon[29120]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:55.463 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:55:55.463 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:55:55.666 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:55.712 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:57.167 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.164+0000 7fe772549700 1 -- 172.21.15.152:0/4160855400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76c0ff7a0 msgr2=0x7fe76c0ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:57.167 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.164+0000 7fe772549700 1 --2- 172.21.15.152:0/4160855400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76c0ff7a0 0x7fe76c0ffbc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009910 tx=0x7fe75c009c20 comp rx=0 tx=0).stop 2024-03-04T15:55:57.167 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.164+0000 7fe772549700 1 -- 172.21.15.152:0/4160855400 shutdown_connections 2024-03-04T15:55:57.167 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.164+0000 7fe772549700 1 --2- 172.21.15.152:0/4160855400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76c0ff7a0 0x7fe76c0ffbc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:57.167 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.164+0000 7fe772549700 1 -- 172.21.15.152:0/4160855400 >> 172.21.15.152:0/4160855400 conn(0x7fe76c0fad20 msgr2=0x7fe76c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:57.167 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.164+0000 7fe772549700 1 -- 172.21.15.152:0/4160855400 shutdown_connections 2024-03-04T15:55:57.167 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.164+0000 7fe772549700 1 -- 172.21.15.152:0/4160855400 wait complete. 2024-03-04T15:55:57.167 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.165+0000 7fe772549700 1 Processor -- start 2024-03-04T15:55:57.167 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.165+0000 7fe772549700 1 -- start start 2024-03-04T15:55:57.169 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.165+0000 7fe772549700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76c0ff7a0 0x7fe76c0fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.165+0000 7fe772549700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0fd220 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.165+0000 7fe76bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76c0ff7a0 0x7fe76c0fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.165+0000 7fe76bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76c0ff7a0 0x7fe76c0fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:43956/0 (socket says 172.21.15.152:43956) 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.165+0000 7fe76bfff700 1 -- 172.21.15.152:0/3891853074 learned_addr learned my addr 172.21.15.152:0/3891853074 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.166+0000 7fe76bfff700 1 -- 172.21.15.152:0/3891853074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c0095f0 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.166+0000 7fe76bfff700 1 --2- 172.21.15.152:0/3891853074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76c0ff7a0 0x7fe76c0fcce0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe75c005b40 tx=0x7fe75c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.166+0000 7fe7697fa700 1 -- 172.21.15.152:0/3891853074 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe75c01d070 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.166+0000 7fe772549700 1 -- 172.21.15.152:0/3891853074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c0fb330 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.166+0000 7fe7697fa700 1 -- 172.21.15.152:0/3891853074 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe75c00be10 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.166+0000 7fe7697fa700 1 -- 172.21.15.152:0/3891853074 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe75c00f460 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.166+0000 7fe772549700 1 -- 172.21.15.152:0/3891853074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c0fb7d0 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.167+0000 7fe772549700 1 -- 172.21.15.152:0/3891853074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c0623c0 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.167+0000 7fe7697fa700 1 -- 172.21.15.152:0/3891853074 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fe75c0227e0 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.167+0000 7fe7697fa700 1 --2- 172.21.15.152:0/3891853074 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe7540383c0 0x7fe75403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:55:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.167+0000 7fe7697fa700 1 -- 172.21.15.152:0/3891853074 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe75c04c4c0 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.167+0000 7fe76b7fe700 1 --2- 172.21.15.152:0/3891853074 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe7540383c0 0x7fe75403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:55:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.168+0000 7fe76b7fe700 1 --2- 172.21.15.152:0/3891853074 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe7540383c0 0x7fe75403a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe760006fd0 tx=0x7fe760006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:55:57.174 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.172+0000 7fe7697fa700 1 -- 172.21.15.152:0/3891853074 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe75c02bba0 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.441 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.439+0000 7fe772549700 1 -- 172.21.15.152:0/3891853074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe76c18edb0 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.441 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.439+0000 7fe7697fa700 1 -- 172.21.15.152:0/3891853074 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe75c027020 con 0x7fe76c0ff7a0 2024-03-04T15:55:57.442 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:55:57.442 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:55:57.443 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.441+0000 7fe772549700 1 -- 172.21.15.152:0/3891853074 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe7540383c0 msgr2=0x7fe75403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:57.444 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.441+0000 7fe772549700 1 --2- 172.21.15.152:0/3891853074 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe7540383c0 0x7fe75403a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe760006fd0 tx=0x7fe760006e40 comp rx=0 tx=0).stop 2024-03-04T15:55:57.444 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.441+0000 7fe772549700 1 -- 172.21.15.152:0/3891853074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76c0ff7a0 msgr2=0x7fe76c0fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:55:57.444 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.441+0000 7fe772549700 1 --2- 172.21.15.152:0/3891853074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76c0ff7a0 0x7fe76c0fcce0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe75c005b40 tx=0x7fe75c004740 comp rx=0 tx=0).stop 2024-03-04T15:55:57.444 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.441+0000 7fe772549700 1 -- 172.21.15.152:0/3891853074 shutdown_connections 2024-03-04T15:55:57.444 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.441+0000 7fe772549700 1 --2- 172.21.15.152:0/3891853074 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe7540383c0 0x7fe75403a880 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:57.444 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.441+0000 7fe772549700 1 --2- 172.21.15.152:0/3891853074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76c0ff7a0 0x7fe76c0fcce0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:55:57.444 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.441+0000 7fe772549700 1 -- 172.21.15.152:0/3891853074 >> 172.21.15.152:0/3891853074 conn(0x7fe76c0fad20 msgr2=0x7fe76c068810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:55:57.444 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.441+0000 7fe772549700 1 -- 172.21.15.152:0/3891853074 shutdown_connections 2024-03-04T15:55:57.444 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:55:57.441+0000 7fe772549700 1 -- 172.21.15.152:0/3891853074 wait complete. 2024-03-04T15:55:57.445 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:55:57.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:57 smithi116 ceph-mon[29120]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:55:58.481 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:58 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/3891853074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:55:59.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:55:59.034 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:55:59.238 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:59.285 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:55:59.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:55:59 smithi116 ceph-mon[29120]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:00.634 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:00 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:00.634 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:00 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:00.634 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:00 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:00.634 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:00 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T15:56:00.634 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:00 smithi116 ceph-mon[29120]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:00.780 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.778+0000 7fe4fe005700 1 -- 172.21.15.152:0/2623287086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4f80ff7a0 msgr2=0x7fe4f80ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:00.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.778+0000 7fe4fe005700 1 --2- 172.21.15.152:0/2623287086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4f80ff7a0 0x7fe4f80ffbc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009910 tx=0x7fe4e8009c20 comp rx=0 tx=0).stop 2024-03-04T15:56:00.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.778+0000 7fe4fe005700 1 -- 172.21.15.152:0/2623287086 shutdown_connections 2024-03-04T15:56:00.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.778+0000 7fe4fe005700 1 --2- 172.21.15.152:0/2623287086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4f80ff7a0 0x7fe4f80ffbc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:00.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.778+0000 7fe4fe005700 1 -- 172.21.15.152:0/2623287086 >> 172.21.15.152:0/2623287086 conn(0x7fe4f80fad20 msgr2=0x7fe4f80fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:00.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.778+0000 7fe4fe005700 1 -- 172.21.15.152:0/2623287086 shutdown_connections 2024-03-04T15:56:00.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.778+0000 7fe4fe005700 1 -- 172.21.15.152:0/2623287086 wait complete. 2024-03-04T15:56:00.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.778+0000 7fe4fe005700 1 Processor -- start 2024-03-04T15:56:00.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.779+0000 7fe4fe005700 1 -- start start 2024-03-04T15:56:00.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.779+0000 7fe4fe005700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4f80ff7a0 0x7fe4f81948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:00.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.779+0000 7fe4fe005700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f8194de0 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.779+0000 7fe4f77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4f80ff7a0 0x7fe4f81948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:00.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.779+0000 7fe4f77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4f80ff7a0 0x7fe4f81948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:43978/0 (socket says 172.21.15.152:43978) 2024-03-04T15:56:00.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.779+0000 7fe4f77fe700 1 -- 172.21.15.152:0/1155051115 learned_addr learned my addr 172.21.15.152:0/1155051115 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:56:00.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.779+0000 7fe4f77fe700 1 -- 172.21.15.152:0/1155051115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e80095f0 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.779+0000 7fe4f77fe700 1 --2- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4f80ff7a0 0x7fe4f81948a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8005b40 tx=0x7fe4e8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:00.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.780+0000 7fe4f4ff9700 1 -- 172.21.15.152:0/1155051115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e801d070 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.780+0000 7fe4fe005700 1 -- 172.21.15.152:0/1155051115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f8194fe0 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.780+0000 7fe4f4ff9700 1 -- 172.21.15.152:0/1155051115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe4e800be10 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.780+0000 7fe4f4ff9700 1 -- 172.21.15.152:0/1155051115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e800f460 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.780+0000 7fe4fe005700 1 -- 172.21.15.152:0/1155051115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f8195480 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.780+0000 7fe4fe005700 1 -- 172.21.15.152:0/1155051115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f818e3c0 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.781+0000 7fe4f4ff9700 1 -- 172.21.15.152:0/1155051115 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 14) v1 ==== 45354+0+0 (secure 0 0 0) 0x7fe4e80227e0 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.781+0000 7fe4f4ff9700 1 --2- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe4e0038410 0x7fe4e003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:00.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.781+0000 7fe4f4ff9700 1 -- 172.21.15.152:0/1155051115 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe4e804c4d0 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.781+0000 7fe4f6ffd700 1 -- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe4e0038410 msgr2=0x7fe4e003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2020358480 2024-03-04T15:56:00.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.781+0000 7fe4f6ffd700 1 --2- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe4e0038410 0x7fe4e003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T15:56:00.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.790+0000 7fe4f4ff9700 1 -- 172.21.15.152:0/1155051115 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe4e802bba0 con 0x7fe4f80ff7a0 2024-03-04T15:56:00.984 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.982+0000 7fe4f6ffd700 1 -- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe4e0038410 msgr2=0x7fe4e003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2020358480 2024-03-04T15:56:00.984 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:00.982+0000 7fe4f6ffd700 1 --2- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe4e0038410 0x7fe4e003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T15:56:01.057 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.055+0000 7fe4fe005700 1 -- 172.21.15.152:0/1155051115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4f802cc70 con 0x7fe4f80ff7a0 2024-03-04T15:56:01.058 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.056+0000 7fe4f4ff9700 1 -- 172.21.15.152:0/1155051115 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe4e8027070 con 0x7fe4f80ff7a0 2024-03-04T15:56:01.058 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:56:01.058 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:56:01.060 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.058+0000 7fe4fe005700 1 -- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe4e0038410 msgr2=0x7fe4e003a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T15:56:01.060 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.058+0000 7fe4fe005700 1 --2- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe4e0038410 0x7fe4e003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:01.060 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.058+0000 7fe4fe005700 1 -- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4f80ff7a0 msgr2=0x7fe4f81948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:01.060 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.058+0000 7fe4fe005700 1 --2- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4f80ff7a0 0x7fe4f81948a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8005b40 tx=0x7fe4e8004740 comp rx=0 tx=0).stop 2024-03-04T15:56:01.060 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.058+0000 7fe4fe005700 1 -- 172.21.15.152:0/1155051115 shutdown_connections 2024-03-04T15:56:01.060 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.058+0000 7fe4fe005700 1 --2- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fe4e0038410 0x7fe4e003a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:01.060 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.058+0000 7fe4fe005700 1 --2- 172.21.15.152:0/1155051115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4f80ff7a0 0x7fe4f81948a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:01.061 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.058+0000 7fe4fe005700 1 -- 172.21.15.152:0/1155051115 >> 172.21.15.152:0/1155051115 conn(0x7fe4f80fad20 msgr2=0x7fe4f80fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:01.061 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.058+0000 7fe4fe005700 1 -- 172.21.15.152:0/1155051115 shutdown_connections 2024-03-04T15:56:01.061 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:01.058+0000 7fe4fe005700 1 -- 172.21.15.152:0/1155051115 wait complete. 2024-03-04T15:56:01.061 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:56:01.677 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:01 smithi116 ceph-mon[29120]: from='mgr.14164 172.21.15.116:0/4065190273' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T15:56:01.677 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:01 smithi116 ceph-mon[29120]: mgrmap e14: smithi116.jwdhop(active, since 66s) 2024-03-04T15:56:01.677 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:01 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/1155051115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:56:02.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:56:02.648 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:56:02.851 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:56:02.898 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:56:04.293 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.278+0000 7fbb22b28700 1 -- 172.21.15.152:0/3365892455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c0ff7a0 msgr2=0x7fbb1c0ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:04.293 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.278+0000 7fbb22b28700 1 --2- 172.21.15.152:0/3365892455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c0ff7a0 0x7fbb1c0ffbc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400b0a0 tx=0x7fbb1400b3b0 comp rx=0 tx=0).stop 2024-03-04T15:56:04.293 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.278+0000 7fbb22b28700 1 -- 172.21.15.152:0/3365892455 shutdown_connections 2024-03-04T15:56:04.293 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.278+0000 7fbb22b28700 1 --2- 172.21.15.152:0/3365892455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c0ff7a0 0x7fbb1c0ffbc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:04.293 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.278+0000 7fbb22b28700 1 -- 172.21.15.152:0/3365892455 >> 172.21.15.152:0/3365892455 conn(0x7fbb1c0fad20 msgr2=0x7fbb1c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:04.293 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.278+0000 7fbb22b28700 1 -- 172.21.15.152:0/3365892455 shutdown_connections 2024-03-04T15:56:04.293 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.278+0000 7fbb22b28700 1 -- 172.21.15.152:0/3365892455 wait complete. 2024-03-04T15:56:04.293 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.279+0000 7fbb22b28700 1 Processor -- start 2024-03-04T15:56:04.293 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.279+0000 7fbb22b28700 1 -- start start 2024-03-04T15:56:04.293 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.279+0000 7fbb22b28700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c0ff7a0 0x7fbb1c069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.279+0000 7fbb22b28700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c196500 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.279+0000 7fbb208c4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c0ff7a0 0x7fbb1c069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.279+0000 7fbb208c4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c0ff7a0 0x7fbb1c069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:43992/0 (socket says 172.21.15.152:43992) 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.279+0000 7fbb208c4700 1 -- 172.21.15.152:0/1957828827 learned_addr learned my addr 172.21.15.152:0/1957828827 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.279+0000 7fbb208c4700 1 -- 172.21.15.152:0/1957828827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb14009d00 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.279+0000 7fbb208c4700 1 --2- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c0ff7a0 0x7fbb1c069390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400b830 tx=0x7fbb1400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.280+0000 7fbb19ffb700 1 -- 172.21.15.152:0/1957828827 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1400e040 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.280+0000 7fbb19ffb700 1 -- 172.21.15.152:0/1957828827 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbb14012af0 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.280+0000 7fbb22b28700 1 -- 172.21.15.152:0/1957828827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1c19ab10 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.280+0000 7fbb19ffb700 1 -- 172.21.15.152:0/1957828827 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1401d9f0 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.280+0000 7fbb22b28700 1 -- 172.21.15.152:0/1957828827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c19ae70 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.281+0000 7fbb19ffb700 1 -- 172.21.15.152:0/1957828827 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 14) v1 ==== 45354+0+0 (secure 0 0 0) 0x7fbb1401dcb0 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.281+0000 7fbb22b28700 1 -- 172.21.15.152:0/1957828827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1c18e3c0 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.294 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.281+0000 7fbb19ffb700 1 --2- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fbb08038410 0x7fbb0803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:04.295 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.281+0000 7fbb19ffb700 1 -- 172.21.15.152:0/1957828827 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbb1404ca00 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.295 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.281+0000 7fbb1bfff700 1 -- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fbb08038410 msgr2=0x7fbb0803a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2020358480 2024-03-04T15:56:04.295 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.281+0000 7fbb1bfff700 1 --2- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fbb08038410 0x7fbb0803a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T15:56:04.295 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.286+0000 7fbb19ffb700 1 -- 172.21.15.152:0/1957828827 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbb1401c3e0 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.484 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.482+0000 7fbb1bfff700 1 -- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fbb08038410 msgr2=0x7fbb0803a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2020358480 2024-03-04T15:56:04.484 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.482+0000 7fbb1bfff700 1 --2- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fbb08038410 0x7fbb0803a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T15:56:04.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.550+0000 7fbb22b28700 1 -- 172.21.15.152:0/1957828827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb1c02cc70 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.553 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.551+0000 7fbb19ffb700 1 -- 172.21.15.152:0/1957828827 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fbb1401c3e0 con 0x7fbb1c0ff7a0 2024-03-04T15:56:04.553 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:56:04.553 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:56:04.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.553+0000 7fbb22b28700 1 -- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fbb08038410 msgr2=0x7fbb0803a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T15:56:04.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.553+0000 7fbb22b28700 1 --2- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fbb08038410 0x7fbb0803a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:04.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.553+0000 7fbb22b28700 1 -- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c0ff7a0 msgr2=0x7fbb1c069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:04.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.553+0000 7fbb22b28700 1 --2- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c0ff7a0 0x7fbb1c069390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400b830 tx=0x7fbb1400bd40 comp rx=0 tx=0).stop 2024-03-04T15:56:04.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.553+0000 7fbb22b28700 1 -- 172.21.15.152:0/1957828827 shutdown_connections 2024-03-04T15:56:04.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.553+0000 7fbb22b28700 1 --2- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:6800/2020358480,v1:172.21.15.116:6801/2020358480] conn(0x7fbb08038410 0x7fbb0803a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:04.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.553+0000 7fbb22b28700 1 --2- 172.21.15.152:0/1957828827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c0ff7a0 0x7fbb1c069390 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:04.556 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.553+0000 7fbb22b28700 1 -- 172.21.15.152:0/1957828827 >> 172.21.15.152:0/1957828827 conn(0x7fbb1c0fad20 msgr2=0x7fbb1c0fba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:04.556 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.553+0000 7fbb22b28700 1 -- 172.21.15.152:0/1957828827 shutdown_connections 2024-03-04T15:56:04.556 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:04.553+0000 7fbb22b28700 1 -- 172.21.15.152:0/1957828827 wait complete. 2024-03-04T15:56:04.556 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:56:04.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:04 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/1957828827' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:56:06.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:56:06.078 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:56:06.284 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:56:06.331 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:56:07.339 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: Active manager daemon smithi116.jwdhop restarted 2024-03-04T15:56:07.339 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: Activating manager daemon smithi116.jwdhop 2024-03-04T15:56:07.339 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: osdmap e5: 0 total, 0 up, 0 in 2024-03-04T15:56:07.339 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: mgrmap e15: smithi116.jwdhop(active, starting, since 0.00318461s) 2024-03-04T15:56:07.340 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T15:56:07.340 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi116.jwdhop", "id": "smithi116.jwdhop"}]: dispatch 2024-03-04T15:56:07.340 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:56:07.340 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:56:07.340 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:56:07.340 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: Manager daemon smithi116.jwdhop is now available 2024-03-04T15:56:07.340 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:07.340 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:07.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.916+0000 7f79cc787700 1 -- 172.21.15.152:0/588403853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4100890 msgr2=0x7f79c4100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:07.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.916+0000 7f79cc787700 1 --2- 172.21.15.152:0/588403853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4100890 0x7f79c4100cb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009910 tx=0x7f79b4009c20 comp rx=0 tx=0).stop 2024-03-04T15:56:07.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.916+0000 7f79cc787700 1 -- 172.21.15.152:0/588403853 shutdown_connections 2024-03-04T15:56:07.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.916+0000 7f79cc787700 1 --2- 172.21.15.152:0/588403853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4100890 0x7f79c4100cb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.916+0000 7f79cc787700 1 -- 172.21.15.152:0/588403853 >> 172.21.15.152:0/588403853 conn(0x7f79c40fbe30 msgr2=0x7f79c40fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.916+0000 7f79cc787700 1 -- 172.21.15.152:0/588403853 shutdown_connections 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.916+0000 7f79cc787700 1 -- 172.21.15.152:0/588403853 wait complete. 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.917+0000 7f79cc787700 1 Processor -- start 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.917+0000 7f79cc787700 1 -- start start 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.917+0000 7f79cc787700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4100890 0x7f79c4074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.917+0000 7f79cc787700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c4075070 con 0x7f79c4100890 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.917+0000 7f79ca523700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4100890 0x7f79c4074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.917+0000 7f79ca523700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4100890 0x7f79c4074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:40064/0 (socket says 172.21.15.152:40064) 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.917+0000 7f79ca523700 1 -- 172.21.15.152:0/271476449 learned_addr learned my addr 172.21.15.152:0/271476449 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.917+0000 7f79ca523700 1 -- 172.21.15.152:0/271476449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b40095f0 con 0x7f79c4100890 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.917+0000 7f79ca523700 1 --2- 172.21.15.152:0/271476449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4100890 0x7f79c4074b30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f79b4005b40 tx=0x7f79b4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.918+0000 7f79bb7fe700 1 -- 172.21.15.152:0/271476449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b401d070 con 0x7f79c4100890 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.918+0000 7f79cc787700 1 -- 172.21.15.152:0/271476449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c4073180 con 0x7f79c4100890 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.918+0000 7f79bb7fe700 1 -- 172.21.15.152:0/271476449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f79b400be10 con 0x7f79c4100890 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.918+0000 7f79bb7fe700 1 -- 172.21.15.152:0/271476449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b400f460 con 0x7f79c4100890 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.918+0000 7f79cc787700 1 -- 172.21.15.152:0/271476449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c4073620 con 0x7f79c4100890 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.918+0000 7f79bb7fe700 1 -- 172.21.15.152:0/271476449 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 15) v1 ==== 45077+0+0 (secure 0 0 0) 0x7f79b400f6d0 con 0x7f79c4100890 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.919+0000 7f79cc787700 1 -- 172.21.15.152:0/271476449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c40623c0 con 0x7f79c4100890 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.919+0000 7f79bb7fe700 1 -- 172.21.15.152:0/271476449 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f79b404d1b0 con 0x7f79c4100890 2024-03-04T15:56:07.925 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:07.924+0000 7f79bb7fe700 1 -- 172.21.15.152:0/271476449 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f79b400fdf0 con 0x7f79c4100890 2024-03-04T15:56:08.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.049+0000 7f79bb7fe700 1 -- 172.21.15.152:0/271476449 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 16) v1 ==== 45204+0+0 (secure 0 0 0) 0x7f79b40227e0 con 0x7f79c4100890 2024-03-04T15:56:08.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.049+0000 7f79bb7fe700 1 --2- 172.21.15.152:0/271476449 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f79b0039440 0x7f79b003b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:08.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.050+0000 7f79c9d22700 1 --2- 172.21.15.152:0/271476449 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f79b0039440 0x7f79b003b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:08.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.050+0000 7f79c9d22700 1 --2- 172.21.15.152:0/271476449 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f79b0039440 0x7f79b003b830 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f79c0006fd0 tx=0x7f79c0006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:08.223 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.220+0000 7f79cc787700 1 -- 172.21.15.152:0/271476449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79c418ee90 con 0x7f79c4100890 2024-03-04T15:56:08.223 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.221+0000 7f79bb7fe700 1 -- 172.21.15.152:0/271476449 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f79b4027090 con 0x7f79c4100890 2024-03-04T15:56:08.223 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:56:08.223 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:56:08.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.223+0000 7f79cc787700 1 -- 172.21.15.152:0/271476449 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f79b0039440 msgr2=0x7f79b003b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:08.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.223+0000 7f79cc787700 1 --2- 172.21.15.152:0/271476449 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f79b0039440 0x7f79b003b830 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f79c0006fd0 tx=0x7f79c0006e40 comp rx=0 tx=0).stop 2024-03-04T15:56:08.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.223+0000 7f79cc787700 1 -- 172.21.15.152:0/271476449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4100890 msgr2=0x7f79c4074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:08.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.223+0000 7f79cc787700 1 --2- 172.21.15.152:0/271476449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4100890 0x7f79c4074b30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f79b4005b40 tx=0x7f79b4004740 comp rx=0 tx=0).stop 2024-03-04T15:56:08.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.223+0000 7f79cc787700 1 -- 172.21.15.152:0/271476449 shutdown_connections 2024-03-04T15:56:08.226 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.223+0000 7f79cc787700 1 --2- 172.21.15.152:0/271476449 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f79b0039440 0x7f79b003b830 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:08.226 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.223+0000 7f79cc787700 1 --2- 172.21.15.152:0/271476449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4100890 0x7f79c4074b30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:08.226 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.223+0000 7f79cc787700 1 -- 172.21.15.152:0/271476449 >> 172.21.15.152:0/271476449 conn(0x7f79c40fbe30 msgr2=0x7f79c40fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:08.226 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.223+0000 7f79cc787700 1 -- 172.21.15.152:0/271476449 shutdown_connections 2024-03-04T15:56:08.226 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:08.223+0000 7f79cc787700 1 -- 172.21.15.152:0/271476449 wait complete. 2024-03-04T15:56:08.226 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:56:08.333 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:08 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:08.333 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:08 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:56:08.333 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:08 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:56:08.333 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:08 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/trash_purge_schedule"}]: dispatch 2024-03-04T15:56:08.334 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:08 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:08.334 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:08 smithi116 ceph-mon[29120]: mgrmap e16: smithi116.jwdhop(active, since 1.00491s) 2024-03-04T15:56:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:09 smithi116 ceph-mon[29120]: [04/Mar/2024:15:56:07] ENGINE Bus STARTING 2024-03-04T15:56:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:09 smithi116 ceph-mon[29120]: [04/Mar/2024:15:56:07] ENGINE Serving on http://172.21.15.116:8765 2024-03-04T15:56:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:09 smithi116 ceph-mon[29120]: [04/Mar/2024:15:56:08] ENGINE Serving on https://172.21.15.116:7150 2024-03-04T15:56:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:09 smithi116 ceph-mon[29120]: [04/Mar/2024:15:56:08] ENGINE Bus STARTED 2024-03-04T15:56:09.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:09 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:09.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:09 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/271476449' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:56:09.834 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:56:09.834 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:56:10.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:10 smithi116 ceph-mon[29120]: mgrmap e17: smithi116.jwdhop(active, since 2s) 2024-03-04T15:56:10.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:10 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:11.143 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:56:11.202 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:56:12.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:12.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:12.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:12.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:12.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:56:13.276 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.946+0000 7ffb1c954700 1 -- 172.21.15.152:0/3401881467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb18100890 msgr2=0x7ffb18100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:13.276 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.946+0000 7ffb1c954700 1 --2- 172.21.15.152:0/3401881467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb18100890 0x7ffb18100cb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009910 tx=0x7ffb08009c20 comp rx=0 tx=0).stop 2024-03-04T15:56:13.277 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.946+0000 7ffb1c954700 1 -- 172.21.15.152:0/3401881467 shutdown_connections 2024-03-04T15:56:13.277 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.946+0000 7ffb1c954700 1 --2- 172.21.15.152:0/3401881467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb18100890 0x7ffb18100cb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:13.277 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.946+0000 7ffb1c954700 1 -- 172.21.15.152:0/3401881467 >> 172.21.15.152:0/3401881467 conn(0x7ffb180fbe30 msgr2=0x7ffb180fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:13.277 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.946+0000 7ffb1c954700 1 -- 172.21.15.152:0/3401881467 shutdown_connections 2024-03-04T15:56:13.277 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.946+0000 7ffb1c954700 1 -- 172.21.15.152:0/3401881467 wait complete. 2024-03-04T15:56:13.277 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.947+0000 7ffb1c954700 1 Processor -- start 2024-03-04T15:56:13.277 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.947+0000 7ffb1c954700 1 -- start start 2024-03-04T15:56:13.277 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.947+0000 7ffb1c954700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb18100890 0x7ffb181959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:13.277 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.947+0000 7ffb1c954700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18195f00 con 0x7ffb18100890 2024-03-04T15:56:13.277 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.948+0000 7ffb1659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb18100890 0x7ffb181959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:13.278 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:56:13.278 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:56:13.278 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.948+0000 7ffb1659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb18100890 0x7ffb181959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:40068/0 (socket says 172.21.15.152:40068) 2024-03-04T15:56:13.278 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.948+0000 7ffb1659c700 1 -- 172.21.15.152:0/551234961 learned_addr learned my addr 172.21.15.152:0/551234961 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:56:13.278 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.948+0000 7ffb1659c700 1 -- 172.21.15.152:0/551234961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb080095f0 con 0x7ffb18100890 2024-03-04T15:56:13.278 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.948+0000 7ffb1659c700 1 --2- 172.21.15.152:0/551234961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb18100890 0x7ffb181959c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffb08005b40 tx=0x7ffb08004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:13.278 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.948+0000 7ffb077fe700 1 -- 172.21.15.152:0/551234961 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb0801d070 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.948+0000 7ffb077fe700 1 -- 172.21.15.152:0/551234961 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffb0800be10 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.948+0000 7ffb1c954700 1 -- 172.21.15.152:0/551234961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb18196100 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.948+0000 7ffb077fe700 1 -- 172.21.15.152:0/551234961 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb0800f460 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.948+0000 7ffb1c954700 1 -- 172.21.15.152:0/551234961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb181965a0 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.949+0000 7ffb077fe700 1 -- 172.21.15.152:0/551234961 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7ffb0800f6d0 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.949+0000 7ffb1c954700 1 -- 172.21.15.152:0/551234961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1818f4b0 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.949+0000 7ffb077fe700 1 --2- 172.21.15.152:0/551234961 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffb000384b0 0x7ffb0003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.950+0000 7ffb077fe700 1 -- 172.21.15.152:0/551234961 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ffb0804c660 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.950+0000 7ffb15d9b700 1 --2- 172.21.15.152:0/551234961 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffb000384b0 0x7ffb0003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.950+0000 7ffb15d9b700 1 --2- 172.21.15.152:0/551234961 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffb000384b0 0x7ffb0003a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c006fd0 tx=0x7ffb0c006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:12.955+0000 7ffb077fe700 1 -- 172.21.15.152:0/551234961 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffb0802a360 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.224+0000 7ffb1c954700 1 -- 172.21.15.152:0/551234961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb1802cc70 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.224+0000 7ffb077fe700 1 -- 172.21.15.152:0/551234961 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ffb08027030 con 0x7ffb18100890 2024-03-04T15:56:13.279 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.226+0000 7ffb1c954700 1 -- 172.21.15.152:0/551234961 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffb000384b0 msgr2=0x7ffb0003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:13.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.226+0000 7ffb1c954700 1 --2- 172.21.15.152:0/551234961 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffb000384b0 0x7ffb0003a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c006fd0 tx=0x7ffb0c006e40 comp rx=0 tx=0).stop 2024-03-04T15:56:13.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.226+0000 7ffb1c954700 1 -- 172.21.15.152:0/551234961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb18100890 msgr2=0x7ffb181959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:13.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.226+0000 7ffb1c954700 1 --2- 172.21.15.152:0/551234961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb18100890 0x7ffb181959c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffb08005b40 tx=0x7ffb08004740 comp rx=0 tx=0).stop 2024-03-04T15:56:13.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.226+0000 7ffb1c954700 1 -- 172.21.15.152:0/551234961 shutdown_connections 2024-03-04T15:56:13.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.226+0000 7ffb1c954700 1 --2- 172.21.15.152:0/551234961 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffb000384b0 0x7ffb0003a970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:13.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.226+0000 7ffb1c954700 1 --2- 172.21.15.152:0/551234961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb18100890 0x7ffb181959c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:13.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.226+0000 7ffb1c954700 1 -- 172.21.15.152:0/551234961 >> 172.21.15.152:0/551234961 conn(0x7ffb180fbe30 msgr2=0x7ffb180fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:13.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.227+0000 7ffb1c954700 1 -- 172.21.15.152:0/551234961 shutdown_connections 2024-03-04T15:56:13.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:13.227+0000 7ffb1c954700 1 -- 172.21.15.152:0/551234961 wait complete. 2024-03-04T15:56:13.280 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:56:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:14.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:56:14.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:14.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:56:14.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:14 smithi116 ceph-mon[29120]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T15:56:14.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:14 smithi116 ceph-mon[29120]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T15:56:14.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:14 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/551234961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:56:14.955 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:56:14.956 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:56:15.337 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:56:15.658 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:56:15.658 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:15.659 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:15 smithi116 ceph-mon[29120]: Deploying daemon ceph-exporter.smithi152 on smithi152 2024-03-04T15:56:17.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.541+0000 7f1bd54ed700 1 -- 172.21.15.152:0/3255189145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0106830 msgr2=0x7f1bd0106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:17.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.541+0000 7f1bd54ed700 1 --2- 172.21.15.152:0/3255189145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0106830 0x7f1bd0106c10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009910 tx=0x7f1bc4009c20 comp rx=0 tx=0).stop 2024-03-04T15:56:17.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.541+0000 7f1bd54ed700 1 -- 172.21.15.152:0/3255189145 shutdown_connections 2024-03-04T15:56:17.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.541+0000 7f1bd54ed700 1 --2- 172.21.15.152:0/3255189145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0106830 0x7f1bd0106c10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:17.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.541+0000 7f1bd54ed700 1 -- 172.21.15.152:0/3255189145 >> 172.21.15.152:0/3255189145 conn(0x7f1bd00fc380 msgr2=0x7f1bd00fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:17.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.541+0000 7f1bd54ed700 1 -- 172.21.15.152:0/3255189145 shutdown_connections 2024-03-04T15:56:17.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.541+0000 7f1bd54ed700 1 -- 172.21.15.152:0/3255189145 wait complete. 2024-03-04T15:56:17.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.541+0000 7f1bd54ed700 1 Processor -- start 2024-03-04T15:56:17.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.542+0000 7f1bd54ed700 1 -- start start 2024-03-04T15:56:17.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.542+0000 7f1bd54ed700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0106830 0x7f1bd019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:17.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.542+0000 7f1bd54ed700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd019bff0 con 0x7f1bd0106830 2024-03-04T15:56:17.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.542+0000 7f1bceffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0106830 0x7f1bd019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:17.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.542+0000 7f1bceffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0106830 0x7f1bd019bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:58250/0 (socket says 172.21.15.152:58250) 2024-03-04T15:56:17.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.542+0000 7f1bceffd700 1 -- 172.21.15.152:0/2962281688 learned_addr learned my addr 172.21.15.152:0/2962281688 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:56:17.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.542+0000 7f1bceffd700 1 -- 172.21.15.152:0/2962281688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc40095f0 con 0x7f1bd0106830 2024-03-04T15:56:17.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.543+0000 7f1bceffd700 1 --2- 172.21.15.152:0/2962281688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0106830 0x7f1bd019bab0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4005b40 tx=0x7f1bc4004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:17.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.543+0000 7f1bb7fff700 1 -- 172.21.15.152:0/2962281688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bc401d070 con 0x7f1bd0106830 2024-03-04T15:56:17.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.543+0000 7f1bd54ed700 1 -- 172.21.15.152:0/2962281688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd019c1f0 con 0x7f1bd0106830 2024-03-04T15:56:17.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.543+0000 7f1bb7fff700 1 -- 172.21.15.152:0/2962281688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1bc40044d0 con 0x7f1bd0106830 2024-03-04T15:56:17.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.543+0000 7f1bb7fff700 1 -- 172.21.15.152:0/2962281688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bc4003910 con 0x7f1bd0106830 2024-03-04T15:56:17.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.543+0000 7f1bd54ed700 1 -- 172.21.15.152:0/2962281688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd0195b30 con 0x7f1bd0106830 2024-03-04T15:56:17.548 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.544+0000 7f1bb7fff700 1 -- 172.21.15.152:0/2962281688 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f1bc4003b90 con 0x7f1bd0106830 2024-03-04T15:56:17.548 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.544+0000 7f1bd54ed700 1 -- 172.21.15.152:0/2962281688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd00623c0 con 0x7f1bd0106830 2024-03-04T15:56:17.548 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.544+0000 7f1bb7fff700 1 --2- 172.21.15.152:0/2962281688 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1bb80384b0 0x7f1bb803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:17.548 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.544+0000 7f1bb7fff700 1 -- 172.21.15.152:0/2962281688 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f1bc404c4c0 con 0x7f1bd0106830 2024-03-04T15:56:17.548 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.544+0000 7f1bce7fc700 1 --2- 172.21.15.152:0/2962281688 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1bb80384b0 0x7f1bb803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:17.548 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.545+0000 7f1bce7fc700 1 --2- 172.21.15.152:0/2962281688 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1bb80384b0 0x7f1bb803a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0006fd0 tx=0x7f1bc0006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:17.551 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.549+0000 7f1bb7fff700 1 -- 172.21.15.152:0/2962281688 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1bc402a540 con 0x7f1bd0106830 2024-03-04T15:56:17.824 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.822+0000 7f1bd54ed700 1 -- 172.21.15.152:0/2962281688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1bd01964f0 con 0x7f1bd0106830 2024-03-04T15:56:17.830 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.828+0000 7f1bb7fff700 1 -- 172.21.15.152:0/2962281688 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f1bc402a360 con 0x7f1bd0106830 2024-03-04T15:56:17.830 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:56:17.830 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:56:17.832 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.830+0000 7f1bd54ed700 1 -- 172.21.15.152:0/2962281688 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1bb80384b0 msgr2=0x7f1bb803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:17.832 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.830+0000 7f1bd54ed700 1 --2- 172.21.15.152:0/2962281688 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1bb80384b0 0x7f1bb803a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0006fd0 tx=0x7f1bc0006e40 comp rx=0 tx=0).stop 2024-03-04T15:56:17.832 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.830+0000 7f1bd54ed700 1 -- 172.21.15.152:0/2962281688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0106830 msgr2=0x7f1bd019bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:17.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.830+0000 7f1bd54ed700 1 --2- 172.21.15.152:0/2962281688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0106830 0x7f1bd019bab0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4005b40 tx=0x7f1bc4004c80 comp rx=0 tx=0).stop 2024-03-04T15:56:17.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.830+0000 7f1bd54ed700 1 -- 172.21.15.152:0/2962281688 shutdown_connections 2024-03-04T15:56:17.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.830+0000 7f1bd54ed700 1 --2- 172.21.15.152:0/2962281688 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1bb80384b0 0x7f1bb803a970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:17.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.830+0000 7f1bd54ed700 1 --2- 172.21.15.152:0/2962281688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0106830 0x7f1bd019bab0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:17.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.830+0000 7f1bd54ed700 1 -- 172.21.15.152:0/2962281688 >> 172.21.15.152:0/2962281688 conn(0x7f1bd00fc380 msgr2=0x7f1bd0108c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:17.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.831+0000 7f1bd54ed700 1 -- 172.21.15.152:0/2962281688 shutdown_connections 2024-03-04T15:56:17.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:17.831+0000 7f1bd54ed700 1 -- 172.21.15.152:0/2962281688 wait complete. 2024-03-04T15:56:17.833 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:56:18.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:17 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/2962281688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:56:19.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:19.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:19.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:19.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:19.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T15:56:19.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-04T15:56:19.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:19.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:19 smithi116 ceph-mon[29120]: Deploying daemon crash.smithi152 on smithi152 2024-03-04T15:56:19.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:56:19.491 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:56:19.701 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:56:21.337 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.334+0000 7f622f4e2700 1 -- 172.21.15.152:0/1657290175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228105820 msgr2=0x7f6228105c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:21.337 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.334+0000 7f622f4e2700 1 --2- 172.21.15.152:0/1657290175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228105820 0x7f6228105c00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6218009910 tx=0x7f6218009c20 comp rx=0 tx=0).stop 2024-03-04T15:56:21.337 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.334+0000 7f622f4e2700 1 -- 172.21.15.152:0/1657290175 shutdown_connections 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.334+0000 7f622f4e2700 1 --2- 172.21.15.152:0/1657290175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228105820 0x7f6228105c00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.334+0000 7f622f4e2700 1 -- 172.21.15.152:0/1657290175 >> 172.21.15.152:0/1657290175 conn(0x7f6228075e80 msgr2=0x7f6228076290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.334+0000 7f622f4e2700 1 -- 172.21.15.152:0/1657290175 shutdown_connections 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.334+0000 7f622f4e2700 1 -- 172.21.15.152:0/1657290175 wait complete. 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.334+0000 7f622f4e2700 1 Processor -- start 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.334+0000 7f622f4e2700 1 -- start start 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f622f4e2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228105820 0x7f6228197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f622f4e2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62281984a0 con 0x7f6228105820 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f622d27e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228105820 0x7f6228197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f622d27e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228105820 0x7f6228197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:58294/0 (socket says 172.21.15.152:58294) 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f622d27e700 1 -- 172.21.15.152:0/778034938 learned_addr learned my addr 172.21.15.152:0/778034938 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f622d27e700 1 -- 172.21.15.152:0/778034938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180095f0 con 0x7f6228105820 2024-03-04T15:56:21.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f622d27e700 1 --2- 172.21.15.152:0/778034938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228105820 0x7f6228197f60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6218005b40 tx=0x7f6218004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:21.339 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f621e7fc700 1 -- 172.21.15.152:0/778034938 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f621801d070 con 0x7f6228105820 2024-03-04T15:56:21.339 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f622f4e2700 1 -- 172.21.15.152:0/778034938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62281986a0 con 0x7f6228105820 2024-03-04T15:56:21.339 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f621e7fc700 1 -- 172.21.15.152:0/778034938 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f62180044d0 con 0x7f6228105820 2024-03-04T15:56:21.339 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.335+0000 7f621e7fc700 1 -- 172.21.15.152:0/778034938 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6218003910 con 0x7f6228105820 2024-03-04T15:56:21.339 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.336+0000 7f622f4e2700 1 -- 172.21.15.152:0/778034938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6228198b20 con 0x7f6228105820 2024-03-04T15:56:21.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.336+0000 7f622f4e2700 1 -- 172.21.15.152:0/778034938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62280623c0 con 0x7f6228105820 2024-03-04T15:56:21.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.336+0000 7f621e7fc700 1 -- 172.21.15.152:0/778034938 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f621800b960 con 0x7f6228105820 2024-03-04T15:56:21.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.337+0000 7f621e7fc700 1 --2- 172.21.15.152:0/778034938 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f62140384b0 0x7f621403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:21.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.337+0000 7f621e7fc700 1 -- 172.21.15.152:0/778034938 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f621804c2d0 con 0x7f6228105820 2024-03-04T15:56:21.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.337+0000 7f622ca7d700 1 --2- 172.21.15.152:0/778034938 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f62140384b0 0x7f621403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:21.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.337+0000 7f622ca7d700 1 --2- 172.21.15.152:0/778034938 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f62140384b0 0x7f621403a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6224006fd0 tx=0x7f6224006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:21.345 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.342+0000 7f621e7fc700 1 -- 172.21.15.152:0/778034938 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f621802a540 con 0x7f6228105820 2024-03-04T15:56:21.618 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.616+0000 7f622f4e2700 1 -- 172.21.15.152:0/778034938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62281992c0 con 0x7f6228105820 2024-03-04T15:56:21.619 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:56:21.619 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:56:21.619 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.616+0000 7f621e7fc700 1 -- 172.21.15.152:0/778034938 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f621802a360 con 0x7f6228105820 2024-03-04T15:56:21.620 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.618+0000 7f622f4e2700 1 -- 172.21.15.152:0/778034938 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f62140384b0 msgr2=0x7f621403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:21.620 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.618+0000 7f622f4e2700 1 --2- 172.21.15.152:0/778034938 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f62140384b0 0x7f621403a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6224006fd0 tx=0x7f6224006e40 comp rx=0 tx=0).stop 2024-03-04T15:56:21.620 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.618+0000 7f622f4e2700 1 -- 172.21.15.152:0/778034938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228105820 msgr2=0x7f6228197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:21.620 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.618+0000 7f622f4e2700 1 --2- 172.21.15.152:0/778034938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228105820 0x7f6228197f60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6218005b40 tx=0x7f6218004c80 comp rx=0 tx=0).stop 2024-03-04T15:56:21.620 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.618+0000 7f622f4e2700 1 -- 172.21.15.152:0/778034938 shutdown_connections 2024-03-04T15:56:21.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.618+0000 7f622f4e2700 1 --2- 172.21.15.152:0/778034938 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f62140384b0 0x7f621403a970 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:21.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.618+0000 7f622f4e2700 1 --2- 172.21.15.152:0/778034938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228105820 0x7f6228197f60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:21.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.618+0000 7f622f4e2700 1 -- 172.21.15.152:0/778034938 >> 172.21.15.152:0/778034938 conn(0x7f6228075e80 msgr2=0x7f62280fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:21.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.618+0000 7f622f4e2700 1 -- 172.21.15.152:0/778034938 shutdown_connections 2024-03-04T15:56:21.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:21.618+0000 7f622f4e2700 1 -- 172.21.15.152:0/778034938 wait complete. 2024-03-04T15:56:21.621 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:56:22.980 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:22.980 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:22.980 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:22 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/778034938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:56:22.980 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:22.980 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:22.980 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:22 smithi116 ceph-mon[29120]: Deploying daemon node-exporter.smithi152 on smithi152 2024-03-04T15:56:22.980 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:22.980 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:56:23.224 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:56:23.224 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:56:23.428 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.797+0000 7fa81eee5700 1 -- 172.21.15.152:0/1822850921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818106830 msgr2=0x7fa818106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.797+0000 7fa81eee5700 1 --2- 172.21.15.152:0/1822850921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818106830 0x7fa818106c10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009910 tx=0x7fa80c009c20 comp rx=0 tx=0).stop 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.798+0000 7fa81eee5700 1 -- 172.21.15.152:0/1822850921 shutdown_connections 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.798+0000 7fa81eee5700 1 --2- 172.21.15.152:0/1822850921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818106830 0x7fa818106c10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.798+0000 7fa81eee5700 1 -- 172.21.15.152:0/1822850921 >> 172.21.15.152:0/1822850921 conn(0x7fa8180fc380 msgr2=0x7fa8180fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.798+0000 7fa81eee5700 1 -- 172.21.15.152:0/1822850921 shutdown_connections 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.798+0000 7fa81eee5700 1 -- 172.21.15.152:0/1822850921 wait complete. 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.798+0000 7fa81eee5700 1 Processor -- start 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.798+0000 7fa81eee5700 1 -- start start 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.798+0000 7fa81eee5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818106830 0x7fa818195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:25.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.798+0000 7fa81eee5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8181962c0 con 0x7fa818106830 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.799+0000 7fa81cc81700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818106830 0x7fa818195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.799+0000 7fa81cc81700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818106830 0x7fa818195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:37634/0 (socket says 172.21.15.152:37634) 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.799+0000 7fa81cc81700 1 -- 172.21.15.152:0/4170598202 learned_addr learned my addr 172.21.15.152:0/4170598202 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.799+0000 7fa81cc81700 1 -- 172.21.15.152:0/4170598202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c0095f0 con 0x7fa818106830 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.799+0000 7fa81cc81700 1 --2- 172.21.15.152:0/4170598202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818106830 0x7fa818195d80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa80c005b40 tx=0x7fa80c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.799+0000 7fa815ffb700 1 -- 172.21.15.152:0/4170598202 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa80c01d070 con 0x7fa818106830 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.799+0000 7fa81eee5700 1 -- 172.21.15.152:0/4170598202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8181964c0 con 0x7fa818106830 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.799+0000 7fa815ffb700 1 -- 172.21.15.152:0/4170598202 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa80c0044d0 con 0x7fa818106830 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.799+0000 7fa815ffb700 1 -- 172.21.15.152:0/4170598202 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa80c003910 con 0x7fa818106830 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.799+0000 7fa81eee5700 1 -- 172.21.15.152:0/4170598202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa818196940 con 0x7fa818106830 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.800+0000 7fa81eee5700 1 -- 172.21.15.152:0/4170598202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8180623c0 con 0x7fa818106830 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.800+0000 7fa815ffb700 1 -- 172.21.15.152:0/4170598202 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7fa80c00b960 con 0x7fa818106830 2024-03-04T15:56:25.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.800+0000 7fa815ffb700 1 --2- 172.21.15.152:0/4170598202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa8080384b0 0x7fa80803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:25.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.801+0000 7fa815ffb700 1 -- 172.21.15.152:0/4170598202 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa80c04c5b0 con 0x7fa818106830 2024-03-04T15:56:25.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.801+0000 7fa817fff700 1 --2- 172.21.15.152:0/4170598202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa8080384b0 0x7fa80803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:25.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.801+0000 7fa817fff700 1 --2- 172.21.15.152:0/4170598202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa8080384b0 0x7fa80803a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa804006fd0 tx=0x7fa804006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:25.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:25.806+0000 7fa815ffb700 1 -- 172.21.15.152:0/4170598202 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa80c02a540 con 0x7fa818106830 2024-03-04T15:56:26.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.077+0000 7fa81eee5700 1 -- 172.21.15.152:0/4170598202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8181045c0 con 0x7fa818106830 2024-03-04T15:56:26.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.078+0000 7fa815ffb700 1 -- 172.21.15.152:0/4170598202 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa80c02a360 con 0x7fa818106830 2024-03-04T15:56:26.080 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:56:26.080 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:56:26.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.080+0000 7fa81eee5700 1 -- 172.21.15.152:0/4170598202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa8080384b0 msgr2=0x7fa80803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:26.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.080+0000 7fa81eee5700 1 --2- 172.21.15.152:0/4170598202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa8080384b0 0x7fa80803a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa804006fd0 tx=0x7fa804006e40 comp rx=0 tx=0).stop 2024-03-04T15:56:26.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.080+0000 7fa81eee5700 1 -- 172.21.15.152:0/4170598202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818106830 msgr2=0x7fa818195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:26.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.080+0000 7fa81eee5700 1 --2- 172.21.15.152:0/4170598202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818106830 0x7fa818195d80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa80c005b40 tx=0x7fa80c004c80 comp rx=0 tx=0).stop 2024-03-04T15:56:26.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.080+0000 7fa81eee5700 1 -- 172.21.15.152:0/4170598202 shutdown_connections 2024-03-04T15:56:26.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.080+0000 7fa81eee5700 1 --2- 172.21.15.152:0/4170598202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa8080384b0 0x7fa80803a970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:26.083 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.080+0000 7fa81eee5700 1 --2- 172.21.15.152:0/4170598202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818106830 0x7fa818195d80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:26.083 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.080+0000 7fa81eee5700 1 -- 172.21.15.152:0/4170598202 >> 172.21.15.152:0/4170598202 conn(0x7fa8180fc380 msgr2=0x7fa818108bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:26.083 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.080+0000 7fa81eee5700 1 -- 172.21.15.152:0/4170598202 shutdown_connections 2024-03-04T15:56:26.083 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:26.080+0000 7fa81eee5700 1 -- 172.21.15.152:0/4170598202 wait complete. 2024-03-04T15:56:26.083 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:56:26.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:26 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/4170598202' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:56:27.717 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:56:27.717 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:56:28.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:28.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:28.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:28.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:28.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.aawieo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:56:28.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.aawieo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:56:28.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:56:28.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:28.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: Deploying daemon mgr.smithi152.aawieo on smithi152 2024-03-04T15:56:28.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:28.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:28.734 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:56:30.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:30 smithi116 ceph-mon[29120]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:30.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.715+0000 7f632e6df700 1 -- 172.21.15.152:0/2352467818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6328105780 msgr2=0x7f6328105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:30.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.715+0000 7f632e6df700 1 --2- 172.21.15.152:0/2352467818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6328105780 0x7f6328105b60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6318009910 tx=0x7f6318009c20 comp rx=0 tx=0).stop 2024-03-04T15:56:30.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.715+0000 7f632e6df700 1 -- 172.21.15.152:0/2352467818 shutdown_connections 2024-03-04T15:56:30.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.715+0000 7f632e6df700 1 --2- 172.21.15.152:0/2352467818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6328105780 0x7f6328105b60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.715+0000 7f632e6df700 1 -- 172.21.15.152:0/2352467818 >> 172.21.15.152:0/2352467818 conn(0x7f63280752b0 msgr2=0x7f63280756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.715+0000 7f632e6df700 1 -- 172.21.15.152:0/2352467818 shutdown_connections 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.715+0000 7f632e6df700 1 -- 172.21.15.152:0/2352467818 wait complete. 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.716+0000 7f632e6df700 1 Processor -- start 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.716+0000 7f632e6df700 1 -- start start 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.716+0000 7f632e6df700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6328105780 0x7f63281949f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.716+0000 7f632e6df700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6328068e80 con 0x7f6328105780 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.716+0000 7f6327fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6328105780 0x7f63281949f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.716+0000 7f6327fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6328105780 0x7f63281949f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:37678/0 (socket says 172.21.15.152:37678) 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.716+0000 7f6327fff700 1 -- 172.21.15.152:0/2136450471 learned_addr learned my addr 172.21.15.152:0/2136450471 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.717+0000 7f6327fff700 1 -- 172.21.15.152:0/2136450471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63180095f0 con 0x7f6328105780 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.717+0000 7f6327fff700 1 --2- 172.21.15.152:0/2136450471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6328105780 0x7f63281949f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6318005b40 tx=0x7f6318004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:30.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.717+0000 7f63257fa700 1 -- 172.21.15.152:0/2136450471 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f631801d070 con 0x7f6328105780 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.717+0000 7f632e6df700 1 -- 172.21.15.152:0/2136450471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6328194f30 con 0x7f6328105780 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.717+0000 7f63257fa700 1 -- 172.21.15.152:0/2136450471 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f63180044d0 con 0x7f6328105780 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.717+0000 7f63257fa700 1 -- 172.21.15.152:0/2136450471 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6318003930 con 0x7f6328105780 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.717+0000 7f632e6df700 1 -- 172.21.15.152:0/2136450471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63281952f0 con 0x7f6328105780 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.718+0000 7f63257fa700 1 -- 172.21.15.152:0/2136450471 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f6318003bf0 con 0x7f6328105780 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.719+0000 7f63257fa700 1 --2- 172.21.15.152:0/2136450471 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f63100384b0 0x7f631003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.719+0000 7f63257fa700 1 -- 172.21.15.152:0/2136450471 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f631804c3a0 con 0x7f6328105780 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.719+0000 7f632e6df700 1 -- 172.21.15.152:0/2136450471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63281039c0 con 0x7f6328105780 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.719+0000 7f63277fe700 1 --2- 172.21.15.152:0/2136450471 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f63100384b0 0x7f631003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.719+0000 7f63277fe700 1 --2- 172.21.15.152:0/2136450471 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f63100384b0 0x7f631003a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f631c006fd0 tx=0x7f631c006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:30.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:30.724+0000 7f63257fa700 1 -- 172.21.15.152:0/2136450471 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f631802a540 con 0x7f6328105780 2024-03-04T15:56:31.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.007+0000 7f632e6df700 1 -- 172.21.15.152:0/2136450471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6328195540 con 0x7f6328105780 2024-03-04T15:56:31.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.008+0000 7f63257fa700 1 -- 172.21.15.152:0/2136450471 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f631802a360 con 0x7f6328105780 2024-03-04T15:56:31.010 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:56:31.010 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:53:45.456905Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:56:31.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.010+0000 7f632e6df700 1 -- 172.21.15.152:0/2136450471 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f63100384b0 msgr2=0x7f631003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:31.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.010+0000 7f632e6df700 1 --2- 172.21.15.152:0/2136450471 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f63100384b0 0x7f631003a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f631c006fd0 tx=0x7f631c006e40 comp rx=0 tx=0).stop 2024-03-04T15:56:31.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.010+0000 7f632e6df700 1 -- 172.21.15.152:0/2136450471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6328105780 msgr2=0x7f63281949f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:31.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.010+0000 7f632e6df700 1 --2- 172.21.15.152:0/2136450471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6328105780 0x7f63281949f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6318005b40 tx=0x7f6318004c80 comp rx=0 tx=0).stop 2024-03-04T15:56:31.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.010+0000 7f632e6df700 1 -- 172.21.15.152:0/2136450471 shutdown_connections 2024-03-04T15:56:31.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.010+0000 7f632e6df700 1 --2- 172.21.15.152:0/2136450471 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f63100384b0 0x7f631003a970 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:31.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.010+0000 7f632e6df700 1 --2- 172.21.15.152:0/2136450471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6328105780 0x7f63281949f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:31.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.010+0000 7f632e6df700 1 -- 172.21.15.152:0/2136450471 >> 172.21.15.152:0/2136450471 conn(0x7f63280752b0 msgr2=0x7f63280fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:31.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.010+0000 7f632e6df700 1 -- 172.21.15.152:0/2136450471 shutdown_connections 2024-03-04T15:56:31.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:31.010+0000 7f632e6df700 1 -- 172.21.15.152:0/2136450471 wait complete. 2024-03-04T15:56:31.013 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T15:56:31.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:31 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:31.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:31 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:31.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:31 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:31.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:31 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:31.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:31 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:56:31.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:31 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:31.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:31 smithi116 ceph-mon[29120]: Deploying daemon mon.smithi152 on smithi152 2024-03-04T15:56:31.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:31 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/2136450471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:56:31.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:31 smithi116 ceph-mon[29120]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:33.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:33 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:34.182 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-04T15:56:34.182 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:56:34.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:34 smithi116 ceph-mon[29120]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:34.666 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi152/config 2024-03-04T15:56:39.895 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.890+0000 7f55079d5700 1 -- 172.21.15.152:0/3375257389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54ec0035c0 msgr2=0x7f54ec005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:39.895 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.890+0000 7f55079d5700 1 --2- 172.21.15.152:0/3375257389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54ec0035c0 0x7f54ec005a50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5500076780 tx=0x7f54fc014ec0 comp rx=0 tx=0).stop 2024-03-04T15:56:39.895 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.890+0000 7f55079d5700 1 -- 172.21.15.152:0/3375257389 shutdown_connections 2024-03-04T15:56:39.895 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.890+0000 7f55079d5700 1 --2- 172.21.15.152:0/3375257389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54ec0035c0 0x7f54ec005a50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:39.895 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.890+0000 7f55079d5700 1 -- 172.21.15.152:0/3375257389 >> 172.21.15.152:0/3375257389 conn(0x7f5500075340 msgr2=0x7f5500075750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:39.895 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.891+0000 7f55079d5700 1 -- 172.21.15.152:0/3375257389 shutdown_connections 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.891+0000 7f55079d5700 1 -- 172.21.15.152:0/3375257389 wait complete. 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.891+0000 7f55079d5700 1 Processor -- start 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f55079d5700 1 -- start start 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f55079d5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54ec0035c0 0x7f5500194af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f55079d5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5500195030 0x7f55001954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f55079d5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5500069480 con 0x7f5500195030 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f55079d5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55001a2160 con 0x7f54ec0035c0 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5505771700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54ec0035c0 0x7f5500194af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5505771700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54ec0035c0 0x7f5500194af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:35402/0 (socket says 172.21.15.152:35402) 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5505771700 1 -- 172.21.15.152:0/1229287864 learned_addr learned my addr 172.21.15.152:0/1229287864 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:56:39.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5504f70700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5500195030 0x7f55001954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5505771700 1 -- 172.21.15.152:0/1229287864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54ec0035c0 msgr2=0x7f5500194af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5505771700 1 -- 172.21.15.152:0/1229287864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54ec0035c0 msgr2=0x7f5500194af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5505771700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54ec0035c0 0x7f5500194af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5505771700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54ec0035c0 0x7f5500194af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5504f70700 1 -- 172.21.15.152:0/1229287864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54ec0035c0 msgr2=0x7f5500194af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5504f70700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54ec0035c0 0x7f5500194af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.892+0000 7f5504f70700 1 -- 172.21.15.152:0/1229287864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc014910 con 0x7f5500195030 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.893+0000 7f5504f70700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5500195030 0x7f55001954b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f54f000b4d0 tx=0x7f54f000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.893+0000 7f54f67fc700 1 -- 172.21.15.152:0/1229287864 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54f0011800 con 0x7f5500195030 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.893+0000 7f55079d5700 1 -- 172.21.15.152:0/1229287864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55001a2300 con 0x7f5500195030 2024-03-04T15:56:39.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.893+0000 7f54f67fc700 1 -- 172.21.15.152:0/1229287864 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f54f0011e10 con 0x7f5500195030 2024-03-04T15:56:39.898 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.893+0000 7f54f67fc700 1 -- 172.21.15.152:0/1229287864 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54f000f2e0 con 0x7f5500195030 2024-03-04T15:56:39.898 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.893+0000 7f55079d5700 1 -- 172.21.15.152:0/1229287864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55001a2850 con 0x7f5500195030 2024-03-04T15:56:39.898 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.894+0000 7f55079d5700 1 -- 172.21.15.152:0/1229287864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55000623c0 con 0x7f5500195030 2024-03-04T15:56:39.899 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.894+0000 7f54f67fc700 1 -- 172.21.15.152:0/1229287864 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f54f000f5a0 con 0x7f5500195030 2024-03-04T15:56:39.899 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.895+0000 7f54f67fc700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54e406c5b0 0x7f54e406ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:39.899 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.895+0000 7f54f67fc700 1 -- 172.21.15.152:0/1229287864 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f54f008a350 con 0x7f5500195030 2024-03-04T15:56:39.899 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.895+0000 7f5505771700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54e406c5b0 0x7f54e406ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:39.899 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.895+0000 7f5505771700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54e406c5b0 0x7f54e406ea70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f54fc001eb0 tx=0x7f54fc014130 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:39.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:39.899+0000 7f54f67fc700 1 -- 172.21.15.152:0/1229287864 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f54f0059710 con 0x7f5500195030 2024-03-04T15:56:39.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T15:56:39.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: mon.smithi116 calling monitor election 2024-03-04T15:56:39.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:56:39.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:39.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: mon.smithi152 calling monitor election 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3444779396' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/crt"}]: dispatch 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: mon.smithi116 is new leader, mons smithi116,smithi152 in quorum (ranks 0,1) 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: monmap e2: 2 mons at {smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],smithi152=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: fsmap 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: osdmap e5: 0 total, 0 up, 0 in 2024-03-04T15:56:39.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: mgrmap e17: smithi116.jwdhop(active, since 32s) 2024-03-04T15:56:39.978 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: Standby manager daemon smithi152.aawieo started 2024-03-04T15:56:39.978 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3444779396' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:56:39.978 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: overall HEALTH_OK 2024-03-04T15:56:39.978 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3444779396' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/key"}]: dispatch 2024-03-04T15:56:39.978 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3444779396' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:56:39.978 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:39.978 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:40.184 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.182+0000 7f55079d5700 1 -- 172.21.15.152:0/1229287864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5500068ae0 con 0x7f5500195030 2024-03-04T15:56:40.185 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.183+0000 7f54f67fc700 1 -- 172.21.15.152:0/1229287864 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1026 (secure 0 0 0) 0x7f54f0015090 con 0x7f5500195030 2024-03-04T15:56:40.185 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T15:56:40.185 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":2,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:56:34.576097Z","created":"2024-03-04T15:53:45.456905Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi152","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T15:56:40.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.185+0000 7f55079d5700 1 -- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54e406c5b0 msgr2=0x7f54e406ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:40.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.185+0000 7f55079d5700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54e406c5b0 0x7f54e406ea70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f54fc001eb0 tx=0x7f54fc014130 comp rx=0 tx=0).stop 2024-03-04T15:56:40.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.185+0000 7f55079d5700 1 -- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5500195030 msgr2=0x7f55001954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:40.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.185+0000 7f55079d5700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5500195030 0x7f55001954b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f54f000b4d0 tx=0x7f54f000b890 comp rx=0 tx=0).stop 2024-03-04T15:56:40.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.185+0000 7f55079d5700 1 -- 172.21.15.152:0/1229287864 shutdown_connections 2024-03-04T15:56:40.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.185+0000 7f55079d5700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54ec0035c0 0x7f5500194af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:40.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.185+0000 7f55079d5700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54e406c5b0 0x7f54e406ea70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:40.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.185+0000 7f55079d5700 1 --2- 172.21.15.152:0/1229287864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5500195030 0x7f55001954b0 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f54f000b4d0 tx=0x7f54f000b890 comp rx=0 tx=0).stop 2024-03-04T15:56:40.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.185+0000 7f55079d5700 1 -- 172.21.15.152:0/1229287864 >> 172.21.15.152:0/1229287864 conn(0x7f5500075340 msgr2=0x7f550018dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:40.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.186+0000 7f55079d5700 1 -- 172.21.15.152:0/1229287864 shutdown_connections 2024-03-04T15:56:40.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:56:40.186+0000 7f55079d5700 1 -- 172.21.15.152:0/1229287864 wait complete. 2024-03-04T15:56:40.188 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 2 2024-03-04T15:56:40.842 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:40 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:56:40.863 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-04T15:56:40.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph config generate-minimal-conf 2024-03-04T15:56:40.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:40 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:40.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:40 smithi116 ceph-mon[29120]: mgrmap e18: smithi116.jwdhop(active, since 32s), standbys: smithi152.aawieo 2024-03-04T15:56:40.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:40 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:40.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:40 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi152.aawieo", "id": "smithi152.aawieo"}]: dispatch 2024-03-04T15:56:40.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:40 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:40.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:40 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:40.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:40 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/1229287864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:56:40.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:40 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:56:41.085 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:56:41.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:41 smithi116 ceph-mon[29120]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:41.981 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:41 smithi152 ceph-mon[36876]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:42.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.749+0000 7f3b648bc700 1 -- 172.21.15.116:0/811492550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b60106820 msgr2=0x7f3b60106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:42.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.749+0000 7f3b648bc700 1 --2- 172.21.15.116:0/811492550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b60106820 0x7f3b60106c00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3b50009910 tx=0x7f3b50009c20 comp rx=0 tx=0).stop 2024-03-04T15:56:42.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.750+0000 7f3b648bc700 1 -- 172.21.15.116:0/811492550 shutdown_connections 2024-03-04T15:56:42.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.750+0000 7f3b648bc700 1 --2- 172.21.15.116:0/811492550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b60106820 0x7f3b60106c00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:42.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.750+0000 7f3b648bc700 1 -- 172.21.15.116:0/811492550 >> 172.21.15.116:0/811492550 conn(0x7f3b600752b0 msgr2=0x7f3b600756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:42.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.750+0000 7f3b648bc700 1 -- 172.21.15.116:0/811492550 shutdown_connections 2024-03-04T15:56:42.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.750+0000 7f3b648bc700 1 -- 172.21.15.116:0/811492550 wait complete. 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.750+0000 7f3b648bc700 1 Processor -- start 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.750+0000 7f3b648bc700 1 -- start start 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b648bc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b60106820 0x7f3b60195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b648bc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b601963c0 0x7f3b6019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b648bc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b601969e0 con 0x7f3b60106820 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b648bc700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b60196b50 con 0x7f3b601963c0 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b60106820 0x7f3b60195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b60106820 0x7f3b60195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50174/0 (socket says 172.21.15.116:50174) 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5e59c700 1 -- 172.21.15.116:0/3245469210 learned_addr learned my addr 172.21.15.116:0/3245469210 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5dd9b700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b601963c0 0x7f3b6019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5dd9b700 1 -- 172.21.15.116:0/3245469210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b601963c0 msgr2=0x7f3b6019a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5dd9b700 1 -- 172.21.15.116:0/3245469210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b601963c0 msgr2=0x7f3b6019a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5dd9b700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b601963c0 0x7f3b6019a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5dd9b700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b601963c0 0x7f3b6019a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5e59c700 1 -- 172.21.15.116:0/3245469210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b601963c0 msgr2=0x7f3b6019a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5e59c700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b601963c0 0x7f3b6019a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5e59c700 1 -- 172.21.15.116:0/3245469210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b500095f0 con 0x7f3b60106820 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b5e59c700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b60106820 0x7f3b60195e80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3b50006010 tx=0x7f3b5000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.751+0000 7f3b4f7fe700 1 -- 172.21.15.116:0/3245469210 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b5001d070 con 0x7f3b60106820 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.752+0000 7f3b4f7fe700 1 -- 172.21.15.116:0/3245469210 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3b50005140 con 0x7f3b60106820 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.752+0000 7f3b648bc700 1 -- 172.21.15.116:0/3245469210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6019ae30 con 0x7f3b60106820 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.752+0000 7f3b4f7fe700 1 -- 172.21.15.116:0/3245469210 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b50017440 con 0x7f3b60106820 2024-03-04T15:56:42.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.752+0000 7f3b648bc700 1 -- 172.21.15.116:0/3245469210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6019b380 con 0x7f3b60106820 2024-03-04T15:56:42.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.753+0000 7f3b648bc700 1 -- 172.21.15.116:0/3245469210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b600623c0 con 0x7f3b60106820 2024-03-04T15:56:42.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.753+0000 7f3b4f7fe700 1 -- 172.21.15.116:0/3245469210 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f3b5000f460 con 0x7f3b60106820 2024-03-04T15:56:42.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.754+0000 7f3b4f7fe700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3b4806c5c0 0x7f3b4806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:42.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.754+0000 7f3b4f7fe700 1 -- 172.21.15.116:0/3245469210 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f3b5008ba60 con 0x7f3b60106820 2024-03-04T15:56:42.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.754+0000 7f3b5dd9b700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3b4806c5c0 0x7f3b4806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:42.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.754+0000 7f3b5dd9b700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3b4806c5c0 0x7f3b4806ea80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3b54009850 tx=0x7f3b54008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:42.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.758+0000 7f3b4f7fe700 1 -- 172.21.15.116:0/3245469210 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3b50057440 con 0x7f3b60106820 2024-03-04T15:56:42.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.956+0000 7f3b648bc700 1 -- 172.21.15.116:0/3245469210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3b6019b600 con 0x7f3b60106820 2024-03-04T15:56:42.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.956+0000 7f3b4f7fe700 1 -- 172.21.15.116:0/3245469210 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f3b50026100 con 0x7f3b60106820 2024-03-04T15:56:42.959 INFO:teuthology.orchestra.run.smithi116.stdout:# minimal ceph.conf for 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:42.959 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-03-04T15:56:42.959 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:42.960 INFO:teuthology.orchestra.run.smithi116.stdout: mon_host = [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2024-03-04T15:56:42.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.958+0000 7f3b648bc700 1 -- 172.21.15.116:0/3245469210 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3b4806c5c0 msgr2=0x7f3b4806ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:42.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.958+0000 7f3b648bc700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3b4806c5c0 0x7f3b4806ea80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3b54009850 tx=0x7f3b54008040 comp rx=0 tx=0).stop 2024-03-04T15:56:42.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.958+0000 7f3b648bc700 1 -- 172.21.15.116:0/3245469210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b60106820 msgr2=0x7f3b60195e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:42.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.958+0000 7f3b648bc700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b60106820 0x7f3b60195e80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3b50006010 tx=0x7f3b5000bda0 comp rx=0 tx=0).stop 2024-03-04T15:56:42.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.958+0000 7f3b648bc700 1 -- 172.21.15.116:0/3245469210 shutdown_connections 2024-03-04T15:56:42.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.958+0000 7f3b648bc700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3b4806c5c0 0x7f3b4806ea80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:42.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.958+0000 7f3b648bc700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b60106820 0x7f3b60195e80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:42.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.958+0000 7f3b648bc700 1 --2- 172.21.15.116:0/3245469210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b601963c0 0x7f3b6019a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:42.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.958+0000 7f3b648bc700 1 -- 172.21.15.116:0/3245469210 >> 172.21.15.116:0/3245469210 conn(0x7f3b600752b0 msgr2=0x7f3b600fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:42.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.959+0000 7f3b648bc700 1 -- 172.21.15.116:0/3245469210 shutdown_connections 2024-03-04T15:56:42.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:42.959+0000 7f3b648bc700 1 -- 172.21.15.116:0/3245469210 wait complete. 2024-03-04T15:56:43.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:43.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:56:43.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T15:56:43.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T15:56:43.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:56:43.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:56:43.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:43.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:43 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3245469210' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:43.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:43.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:56:43.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T15:56:43.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T15:56:43.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:43.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:56:43.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:56:43.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:43.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:43 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3245469210' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:43.599 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-04T15:56:43.599 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:56:43.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:56:43.628 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:56:43.628 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:56:43.697 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:56:43.697 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:56:43.731 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:56:43.731 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:56:43.802 INFO:tasks.cephadm:Deploying OSDs... 2024-03-04T15:56:43.802 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:56:43.802 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:56:43.819 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:56:43.819 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-04T15:56:43.874 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:56:43.874 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:56:43.875 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-04T15:56:43.875 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:56:43.875 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:56:43.875 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-04 15:55:55.789584670 +0000 2024-03-04T15:56:43.875 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-04 15:50:24.454921486 +0000 2024-03-04T15:56:43.875 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-04 15:50:24.454921486 +0000 2024-03-04T15:56:43.875 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-04 15:50:24.454921486 +0000 2024-03-04T15:56:43.875 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:56:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-04T15:56:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-04T15:56:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000200845 s, 2.5 MB/s 2024-03-04T15:56:43.944 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:56:44.000 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-04T15:56:44.055 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:56:44.056 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:56:44.056 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-04T15:56:44.056 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:56:44.056 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:56:44.056 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-04 15:55:55.789584670 +0000 2024-03-04T15:56:44.056 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-04 15:50:24.810915885 +0000 2024-03-04T15:56:44.056 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-04 15:50:24.810915885 +0000 2024-03-04T15:56:44.056 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-04 15:50:24.810915885 +0000 2024-03-04T15:56:44.056 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:56:44.122 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-04T15:56:44.122 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-04T15:56:44.122 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000196306 s, 2.6 MB/s 2024-03-04T15:56:44.124 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:56:44.180 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-04T15:56:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:56:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:56:44.237 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-04T15:56:44.237 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:56:44.237 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:56:44.237 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-04 15:55:55.789584670 +0000 2024-03-04T15:56:44.238 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-04 15:50:25.133910804 +0000 2024-03-04T15:56:44.238 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-04 15:50:25.133910804 +0000 2024-03-04T15:56:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-04 15:50:25.133910804 +0000 2024-03-04T15:56:44.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:56:44.302 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-04T15:56:44.302 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-04T15:56:44.302 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000171303 s, 3.0 MB/s 2024-03-04T15:56:44.303 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:56:44.359 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-04T15:56:44.415 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:56:44.415 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:56:44.415 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-04T15:56:44.415 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:56:44.415 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:56:44.415 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-04 15:55:55.789584670 +0000 2024-03-04T15:56:44.415 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-04 15:50:25.479905360 +0000 2024-03-04T15:56:44.416 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-04 15:50:25.479905360 +0000 2024-03-04T15:56:44.416 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-04 15:50:25.479905360 +0000 2024-03-04T15:56:44.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:56:44.480 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:44 smithi116 ceph-mon[29120]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:56:44.480 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:44 smithi116 ceph-mon[29120]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:56:44.481 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:44 smithi116 ceph-mon[29120]: Reconfiguring mon.smithi116 (unknown last config time)... 2024-03-04T15:56:44.481 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:44 smithi116 ceph-mon[29120]: Reconfiguring daemon mon.smithi116 on smithi116 2024-03-04T15:56:44.481 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:44 smithi116 ceph-mon[29120]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-04T15:56:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-04T15:56:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000168129 s, 3.0 MB/s 2024-03-04T15:56:44.484 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:56:44.539 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:56:44.539 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:56:44.558 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:56:44.558 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-03-04T15:56:44.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:44 smithi152 ceph-mon[36876]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:56:44.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:44 smithi152 ceph-mon[36876]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:56:44.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:44 smithi152 ceph-mon[36876]: Reconfiguring mon.smithi116 (unknown last config time)... 2024-03-04T15:56:44.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:44 smithi152 ceph-mon[36876]: Reconfiguring daemon mon.smithi116 on smithi116 2024-03-04T15:56:44.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:44 smithi152 ceph-mon[36876]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:44.616 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:56:44.616 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:56:44.616 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-04T15:56:44.616 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:56:44.616 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:56:44.616 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-04 15:56:33.947317194 +0000 2024-03-04T15:56:44.616 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-04 15:50:26.645065492 +0000 2024-03-04T15:56:44.616 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-04 15:50:26.645065492 +0000 2024-03-04T15:56:44.617 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-04 15:50:26.645065492 +0000 2024-03-04T15:56:44.617 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:56:44.687 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-04T15:56:44.687 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-04T15:56:44.687 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000294509 s, 1.7 MB/s 2024-03-04T15:56:44.688 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:56:44.748 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-03-04T15:56:44.806 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:56:44.806 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:56:44.806 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-04T15:56:44.806 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:56:44.806 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:56:44.806 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-04 15:56:33.948317184 +0000 2024-03-04T15:56:44.806 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-04 15:50:26.982061962 +0000 2024-03-04T15:56:44.806 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-04 15:50:26.982061962 +0000 2024-03-04T15:56:44.806 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-04 15:50:26.982061962 +0000 2024-03-04T15:56:44.807 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:56:44.876 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-04T15:56:44.876 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-04T15:56:44.877 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000206682 s, 2.5 MB/s 2024-03-04T15:56:44.878 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:56:44.936 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-03-04T15:56:44.993 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:56:44.993 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:56:44.993 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-04T15:56:44.993 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:56:44.993 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:56:44.993 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-04 15:56:33.948317184 +0000 2024-03-04T15:56:44.994 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-04 15:50:27.331058307 +0000 2024-03-04T15:56:44.994 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-04 15:50:27.331058307 +0000 2024-03-04T15:56:44.994 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-04 15:50:27.331058307 +0000 2024-03-04T15:56:44.994 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:56:45.061 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-04T15:56:45.061 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-04T15:56:45.061 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000225014 s, 2.3 MB/s 2024-03-04T15:56:45.062 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:56:45.119 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-03-04T15:56:45.175 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:56:45.175 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:56:45.175 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-04T15:56:45.175 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:56:45.175 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:56:45.176 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-04 15:56:33.949317174 +0000 2024-03-04T15:56:45.176 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-04 15:50:27.672054736 +0000 2024-03-04T15:56:45.176 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-04 15:50:27.672054736 +0000 2024-03-04T15:56:45.176 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-04 15:50:27.672054736 +0000 2024-03-04T15:56:45.176 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:56:45.242 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-04T15:56:45.242 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-04T15:56:45.242 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000191857 s, 2.7 MB/s 2024-03-04T15:56:45.244 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:56:45.300 INFO:tasks.cephadm:Deploying osd.0 on smithi116 with /dev/vg_nvme/lv_4... 2024-03-04T15:56:45.300 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T15:56:45.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:45 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:45.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:45 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:45.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:45 smithi116 ceph-mon[29120]: Reconfiguring mgr.smithi116.jwdhop (unknown last config time)... 2024-03-04T15:56:45.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:45 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.jwdhop", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:56:45.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:45 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:56:45.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:45 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:45.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:45 smithi116 ceph-mon[29120]: Reconfiguring daemon mgr.smithi116.jwdhop on smithi116 2024-03-04T15:56:45.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:45 smithi116 ceph-mon[29120]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:46.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:45 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:46.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:45 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:46.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:45 smithi152 ceph-mon[36876]: Reconfiguring mgr.smithi116.jwdhop (unknown last config time)... 2024-03-04T15:56:46.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:45 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.jwdhop", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:56:46.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:45 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:56:46.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:45 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:46.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:45 smithi152 ceph-mon[36876]: Reconfiguring daemon mgr.smithi116.jwdhop on smithi116 2024-03-04T15:56:46.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:45 smithi152 ceph-mon[36876]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:46.295 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:56:47.607 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:47 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:47.607 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:47 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:47.607 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:47 smithi116 ceph-mon[29120]: Reconfiguring ceph-exporter.smithi116 (monmap changed)... 2024-03-04T15:56:47.608 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:47 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T15:56:47.608 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:47 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:47.608 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:47 smithi116 ceph-mon[29120]: Reconfiguring daemon ceph-exporter.smithi116 on smithi116 2024-03-04T15:56:47.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:47 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:47.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:47 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:47.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:47 smithi152 ceph-mon[36876]: Reconfiguring ceph-exporter.smithi116 (monmap changed)... 2024-03-04T15:56:47.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:47 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T15:56:47.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:47 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:47.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:47 smithi152 ceph-mon[36876]: Reconfiguring daemon ceph-exporter.smithi116 on smithi116 2024-03-04T15:56:48.537 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:48 smithi116 ceph-mon[29120]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:48 smithi152 ceph-mon[36876]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T15:56:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T15:56:48.845 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-04T15:56:48.846 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-04T15:56:48.846 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011673 s, 898 MB/s 2024-03-04T15:56:48.846 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-04T15:56:49.592 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-03-04T15:56:50.209 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:56:50.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:50 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:50.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:50 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:50.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:50 smithi116 ceph-mon[29120]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:50.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:50 smithi116 ceph-mon[29120]: Reconfiguring crash.smithi116 (monmap changed)... 2024-03-04T15:56:50.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:50 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T15:56:50.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:50 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:50.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:50 smithi116 ceph-mon[29120]: Reconfiguring daemon crash.smithi116 on smithi116 2024-03-04T15:56:50.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:50 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:50.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:50 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:50.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:50 smithi152 ceph-mon[36876]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:50.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:50 smithi152 ceph-mon[36876]: Reconfiguring crash.smithi116 (monmap changed)... 2024-03-04T15:56:50.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:50 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T15:56:50.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:50 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:50.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:50 smithi152 ceph-mon[36876]: Reconfiguring daemon crash.smithi116 on smithi116 2024-03-04T15:56:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:52 smithi116 ceph-mon[29120]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:52 smithi116 ceph-mon[29120]: Reconfiguring alertmanager.smithi116 (dependencies changed)... 2024-03-04T15:56:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:52 smithi116 ceph-mon[29120]: Reconfiguring daemon alertmanager.smithi116 on smithi116 2024-03-04T15:56:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:56:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:52 smithi152 ceph-mon[36876]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:52 smithi152 ceph-mon[36876]: Reconfiguring alertmanager.smithi116 (dependencies changed)... 2024-03-04T15:56:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:52 smithi152 ceph-mon[36876]: Reconfiguring daemon alertmanager.smithi116 on smithi116 2024-03-04T15:56:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:56:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.957+0000 7fab7eb74700 1 -- 172.21.15.116:0/93247885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab78100bc0 msgr2=0x7fab78100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.957+0000 7fab7eb74700 1 --2- 172.21.15.116:0/93247885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab78100bc0 0x7fab78100fe0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009910 tx=0x7fab6c009c20 comp rx=0 tx=0).stop 2024-03-04T15:56:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.957+0000 7fab7eb74700 1 -- 172.21.15.116:0/93247885 shutdown_connections 2024-03-04T15:56:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.957+0000 7fab7eb74700 1 --2- 172.21.15.116:0/93247885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab78101dc0 0x7fab78102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.957+0000 7fab7eb74700 1 --2- 172.21.15.116:0/93247885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab78100bc0 0x7fab78100fe0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.957+0000 7fab7eb74700 1 -- 172.21.15.116:0/93247885 >> 172.21.15.116:0/93247885 conn(0x7fab780fc140 msgr2=0x7fab780fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.957+0000 7fab7eb74700 1 -- 172.21.15.116:0/93247885 shutdown_connections 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.957+0000 7fab7eb74700 1 -- 172.21.15.116:0/93247885 wait complete. 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7eb74700 1 Processor -- start 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7eb74700 1 -- start start 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7eb74700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab78100bc0 0x7fab78074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7eb74700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab78101dc0 0x7fab78073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7eb74700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab780736c0 con 0x7fab78100bc0 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7eb74700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab78073800 con 0x7fab78101dc0 2024-03-04T15:56:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7c910700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab78100bc0 0x7fab78074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7c910700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab78100bc0 0x7fab78074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38678/0 (socket says 172.21.15.116:38678) 2024-03-04T15:56:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7c910700 1 -- 172.21.15.116:0/3507650800 learned_addr learned my addr 172.21.15.116:0/3507650800 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:56:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7c910700 1 -- 172.21.15.116:0/3507650800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab78101dc0 msgr2=0x7fab78073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:56:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7c910700 1 --2- 172.21.15.116:0/3507650800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab78101dc0 0x7fab78073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:56:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7c910700 1 -- 172.21.15.116:0/3507650800 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c0095f0 con 0x7fab78100bc0 2024-03-04T15:56:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7c910700 1 --2- 172.21.15.116:0/3507650800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab78100bc0 0x7fab78074b30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00bda0 tx=0x7fab6c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab75ffb700 1 -- 172.21.15.116:0/3507650800 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6c01d070 con 0x7fab78100bc0 2024-03-04T15:56:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab75ffb700 1 -- 172.21.15.116:0/3507650800 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fab6c004c50 con 0x7fab78100bc0 2024-03-04T15:56:52.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab7eb74700 1 -- 172.21.15.116:0/3507650800 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab78073a80 con 0x7fab78100bc0 2024-03-04T15:56:52.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.958+0000 7fab75ffb700 1 -- 172.21.15.116:0/3507650800 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6c00f690 con 0x7fab78100bc0 2024-03-04T15:56:52.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.959+0000 7fab7eb74700 1 -- 172.21.15.116:0/3507650800 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab78073f70 con 0x7fab78100bc0 2024-03-04T15:56:52.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.960+0000 7fab7eb74700 1 -- 172.21.15.116:0/3507650800 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab78109720 con 0x7fab78100bc0 2024-03-04T15:56:52.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.960+0000 7fab75ffb700 1 -- 172.21.15.116:0/3507650800 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7fab6c022a20 con 0x7fab78100bc0 2024-03-04T15:56:52.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.960+0000 7fab75ffb700 1 --2- 172.21.15.116:0/3507650800 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fab6806c460 0x7fab6806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:56:52.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.961+0000 7fab75ffb700 1 -- 172.21.15.116:0/3507650800 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fab6c08bd30 con 0x7fab78100bc0 2024-03-04T15:56:52.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.961+0000 7fab77fff700 1 --2- 172.21.15.116:0/3507650800 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fab6806c460 0x7fab6806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:56:52.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.961+0000 7fab77fff700 1 --2- 172.21.15.116:0/3507650800 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fab6806c460 0x7fab6806e920 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab64005fd0 tx=0x7fab64005d00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:56:52.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:52.965+0000 7fab75ffb700 1 -- 172.21.15.116:0/3507650800 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fab6c05b1b0 con 0x7fab78100bc0 2024-03-04T15:56:53.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:56:53.164+0000 7fab7eb74700 1 -- 172.21.15.116:0/3507650800 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fab78106710 con 0x7fab6806c460 2024-03-04T15:56:53.578 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:53 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:56:53.578 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:53 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:56:53.578 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:53 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:53.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:53 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:56:53.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:53 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:56:53.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:53 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:54.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:54 smithi152 ceph-mon[36876]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:54.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:54 smithi152 ceph-mon[36876]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:54.701 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:54 smithi116 ceph-mon[29120]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:54.701 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:54 smithi116 ceph-mon[29120]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:56.363 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:56 smithi116 ceph-mon[29120]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:56.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:56 smithi152 ceph-mon[36876]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:58.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:58 smithi116 ceph-mon[29120]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:58.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:58 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:58.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:58 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:58 smithi152 ceph-mon[36876]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:56:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:58 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:58 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:56:59.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:59 smithi116 ceph-mon[29120]: Reconfiguring grafana.smithi116 (dependencies changed)... 2024-03-04T15:56:59.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:59 smithi116 ceph-mon[29120]: Reconfiguring daemon grafana.smithi116 on smithi116 2024-03-04T15:56:59.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:59 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/733388986' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d12966f-954e-4a09-bbbd-5b0f6a04d039"}]: dispatch 2024-03-04T15:56:59.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:59 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/733388986' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d12966f-954e-4a09-bbbd-5b0f6a04d039"}]': finished 2024-03-04T15:56:59.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:59 smithi116 ceph-mon[29120]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:56:59.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:56:59 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:56:59.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:59 smithi152 ceph-mon[36876]: Reconfiguring grafana.smithi116 (dependencies changed)... 2024-03-04T15:56:59.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:59 smithi152 ceph-mon[36876]: Reconfiguring daemon grafana.smithi116 on smithi116 2024-03-04T15:56:59.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:59 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/733388986' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d12966f-954e-4a09-bbbd-5b0f6a04d039"}]: dispatch 2024-03-04T15:56:59.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:59 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/733388986' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d12966f-954e-4a09-bbbd-5b0f6a04d039"}]': finished 2024-03-04T15:56:59.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:59 smithi152 ceph-mon[36876]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:56:59.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:56:59 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:00.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:00 smithi152 ceph-mon[36876]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:00.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:00 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2757364414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:57:00.717 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:00 smithi116 ceph-mon[29120]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:00.717 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:00 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2757364414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:57:02.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:02 smithi152 ceph-mon[36876]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:02.711 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:02 smithi116 ceph-mon[29120]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:04.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:04 smithi116 ceph-mon[29120]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:04.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:04 smithi152 ceph-mon[36876]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:06.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:06 smithi152 ceph-mon[36876]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:06.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:06 smithi116 ceph-mon[29120]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:57:07.649 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:57:08.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:08 smithi152 ceph-mon[36876]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:08.612 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:08 smithi116 ceph-mon[29120]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:09.575 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:09 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:09.575 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:09 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:09.575 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:09 smithi116 ceph-mon[29120]: Reconfiguring prometheus.smithi116 (dependencies changed)... 2024-03-04T15:57:09.575 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:09 smithi116 ceph-mon[29120]: Reconfiguring daemon prometheus.smithi116 on smithi116 2024-03-04T15:57:09.575 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:09 smithi116 ceph-mon[29120]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:09.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:09 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:09.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:09 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:09.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:09 smithi152 ceph-mon[36876]: Reconfiguring prometheus.smithi116 (dependencies changed)... 2024-03-04T15:57:09.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:09 smithi152 ceph-mon[36876]: Reconfiguring daemon prometheus.smithi116 on smithi116 2024-03-04T15:57:09.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:09 smithi152 ceph-mon[36876]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:10.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:10 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:57:10.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:10 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:10.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:10 smithi152 ceph-mon[36876]: Deploying daemon osd.0 on smithi116 2024-03-04T15:57:10.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:10 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:57:10.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:10 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:10.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:10 smithi116 ceph-mon[29120]: Deploying daemon osd.0 on smithi116 2024-03-04T15:57:11.707 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:11 smithi116 ceph-mon[29120]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:11 smithi152 ceph-mon[36876]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:14.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:14 smithi152 ceph-mon[36876]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:14.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:14 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:14.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:14 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:14.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:14 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T15:57:14.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:14 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:14.404 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:14 smithi116 ceph-mon[29120]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:14.404 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:14.404 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:14.404 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T15:57:14.404 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:15.350 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:15 smithi116 ceph-mon[29120]: Reconfiguring ceph-exporter.smithi152 (monmap changed)... 2024-03-04T15:57:15.350 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:15 smithi116 ceph-mon[29120]: Reconfiguring daemon ceph-exporter.smithi152 on smithi152 2024-03-04T15:57:15.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:15 smithi152 ceph-mon[36876]: Reconfiguring ceph-exporter.smithi152 (monmap changed)... 2024-03-04T15:57:15.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:15 smithi152 ceph-mon[36876]: Reconfiguring daemon ceph-exporter.smithi152 on smithi152 2024-03-04T15:57:16.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:16 smithi116 ceph-mon[29120]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:16.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:16.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:16.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T15:57:16.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:16.408 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:16 smithi152 ceph-mon[36876]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:16.408 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:16.408 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:16.408 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T15:57:16.408 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:17.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:17 smithi152 ceph-mon[36876]: Reconfiguring crash.smithi152 (monmap changed)... 2024-03-04T15:57:17.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:17 smithi152 ceph-mon[36876]: Reconfiguring daemon crash.smithi152 on smithi152 2024-03-04T15:57:17.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:17.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:17.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.aawieo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:57:17.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:57:17.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:17.430 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:17 smithi116 ceph-mon[29120]: Reconfiguring crash.smithi152 (monmap changed)... 2024-03-04T15:57:17.430 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:17 smithi116 ceph-mon[29120]: Reconfiguring daemon crash.smithi152 on smithi152 2024-03-04T15:57:17.430 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:17.430 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:17.430 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.aawieo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:57:17.430 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:57:17.431 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:18.457 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:18 smithi116 ceph-mon[29120]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:18.457 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:18 smithi116 ceph-mon[29120]: Reconfiguring mgr.smithi152.aawieo (monmap changed)... 2024-03-04T15:57:18.457 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:18 smithi116 ceph-mon[29120]: Reconfiguring daemon mgr.smithi152.aawieo on smithi152 2024-03-04T15:57:18.460 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:18 smithi152 ceph-mon[36876]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:18.460 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:18 smithi152 ceph-mon[36876]: Reconfiguring mgr.smithi152.aawieo (monmap changed)... 2024-03-04T15:57:18.460 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:18 smithi152 ceph-mon[36876]: Reconfiguring daemon mgr.smithi152.aawieo on smithi152 2024-03-04T15:57:19.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:19 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:19.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:19 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:19.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:19 smithi152 ceph-mon[36876]: Reconfiguring mon.smithi152 (monmap changed)... 2024-03-04T15:57:19.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:19 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:19.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:19 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:57:19.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:19 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:19.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:19 smithi152 ceph-mon[36876]: Reconfiguring daemon mon.smithi152 on smithi152 2024-03-04T15:57:19.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:19 smithi152 ceph-mon[36876]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:19.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:19 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:19.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:19 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:19.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:19.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:19.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:19 smithi116 ceph-mon[29120]: Reconfiguring mon.smithi152 (monmap changed)... 2024-03-04T15:57:19.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:19.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:57:19.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:19.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:19 smithi116 ceph-mon[29120]: Reconfiguring daemon mon.smithi152 on smithi152 2024-03-04T15:57:19.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:19 smithi116 ceph-mon[29120]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:19.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:19.842 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:19 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:20.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:20.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:20.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T15:57:20.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T15:57:20.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T15:57:20.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T15:57:20.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:20.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T15:57:20.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T15:57:20.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi116.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T15:57:20.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi116.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T15:57:20.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:20.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T15:57:20.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T15:57:20.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi116.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T15:57:20.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi116.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T15:57:20.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:20.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:21.317 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi116.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi116.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:21.318 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T15:57:21.319 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T15:57:21.319 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi116.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T15:57:21.319 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi116.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T15:57:21.319 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:21.319 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:22.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:21 smithi116 ceph-mon[29120]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:22.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:21 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:22.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:21 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:22.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:21 smithi152 ceph-mon[36876]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:22.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:21 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:22.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:21 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:22.399 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 0 on host 'smithi116' 2024-03-04T15:57:22.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.396+0000 7fab75ffb700 1 -- 172.21.15.116:0/3507650800 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fab78106710 con 0x7fab6806c460 2024-03-04T15:57:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 -- 172.21.15.116:0/3507650800 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fab6806c460 msgr2=0x7fab6806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:57:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 --2- 172.21.15.116:0/3507650800 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fab6806c460 0x7fab6806e920 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab64005fd0 tx=0x7fab64005d00 comp rx=0 tx=0).stop 2024-03-04T15:57:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 -- 172.21.15.116:0/3507650800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab78100bc0 msgr2=0x7fab78074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:57:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 --2- 172.21.15.116:0/3507650800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab78100bc0 0x7fab78074b30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00bda0 tx=0x7fab6c00be80 comp rx=0 tx=0).stop 2024-03-04T15:57:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 -- 172.21.15.116:0/3507650800 shutdown_connections 2024-03-04T15:57:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 --2- 172.21.15.116:0/3507650800 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fab6806c460 0x7fab6806e920 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:57:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 --2- 172.21.15.116:0/3507650800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab78100bc0 0x7fab78074b30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:57:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 --2- 172.21.15.116:0/3507650800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab78101dc0 0x7fab78073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:57:22.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 -- 172.21.15.116:0/3507650800 >> 172.21.15.116:0/3507650800 conn(0x7fab780fc140 msgr2=0x7fab78104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:57:22.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 -- 172.21.15.116:0/3507650800 shutdown_connections 2024-03-04T15:57:22.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:22.399+0000 7fab7eb74700 1 -- 172.21.15.116:0/3507650800 wait complete. 2024-03-04T15:57:23.074 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:57:23.074 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:23.074 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:23.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:57:23.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:23.360 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:23.369 DEBUG:teuthology.orchestra.run.smithi116:osd.0> sudo journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.0.service 2024-03-04T15:57:23.371 INFO:tasks.cephadm:Deploying osd.1 on smithi116 with /dev/vg_nvme/lv_3... 2024-03-04T15:57:23.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T15:57:24.126 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:57:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:24 smithi116 ceph-mon[29120]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:24 smithi116 ceph-mon[29120]: from='osd.0 [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:57:24.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:24 smithi152 ceph-mon[36876]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:24.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:24 smithi152 ceph-mon[36876]: from='osd.0 [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:57:25.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:25 smithi116 ceph-mon[29120]: from='osd.0 [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:57:25.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:25 smithi116 ceph-mon[29120]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:57:25.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:25 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:25.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:25 smithi116 ceph-mon[29120]: from='osd.0 [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T15:57:25.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:25 smithi116 ceph-mon[29120]: from='osd.0 [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T15:57:25.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:25 smithi116 ceph-mon[29120]: osdmap e8: 1 total, 0 up, 1 in 2024-03-04T15:57:25.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:25 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:25.477 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 15:57:25 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[48197]: 2024-03-04T15:57:25.241+0000 7fb9c2925700 -1 osd.0 0 waiting for initial osdmap 2024-03-04T15:57:25.477 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 15:57:25 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[48197]: 2024-03-04T15:57:25.247+0000 7fb9bd718700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:57:25.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:25 smithi152 ceph-mon[36876]: from='osd.0 [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:57:25.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:25 smithi152 ceph-mon[36876]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:57:25.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:25 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:25.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:25 smithi152 ceph-mon[36876]: from='osd.0 [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T15:57:25.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:25 smithi152 ceph-mon[36876]: from='osd.0 [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T15:57:25.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:25 smithi152 ceph-mon[36876]: osdmap e8: 1 total, 0 up, 1 in 2024-03-04T15:57:25.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:25 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:26 smithi152 ceph-mon[36876]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:26 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:26 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:26.679 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:26 smithi116 ceph-mon[29120]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:26.679 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:26 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:26.679 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:26 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T15:57:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T15:57:27.303 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-04T15:57:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-04T15:57:27.303 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138682 s, 756 MB/s 2024-03-04T15:57:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-04T15:57:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:27 smithi152 ceph-mon[36876]: purged_snaps scrub starts 2024-03-04T15:57:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:27 smithi152 ceph-mon[36876]: purged_snaps scrub ok 2024-03-04T15:57:27.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:27 smithi152 ceph-mon[36876]: osd.0 [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067] boot 2024-03-04T15:57:27.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:27 smithi152 ceph-mon[36876]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:57:27.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:27 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:27.704 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:27 smithi116 ceph-mon[29120]: purged_snaps scrub starts 2024-03-04T15:57:27.705 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:27 smithi116 ceph-mon[29120]: purged_snaps scrub ok 2024-03-04T15:57:27.705 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:27 smithi116 ceph-mon[29120]: osd.0 [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067] boot 2024-03-04T15:57:27.705 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:27 smithi116 ceph-mon[29120]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:57:27.705 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:28.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-03-04T15:57:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:57:28.582 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:28 smithi116 ceph-mon[29120]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:28.582 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:28 smithi116 ceph-mon[29120]: osdmap e10: 1 total, 1 up, 1 in 2024-03-04T15:57:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:28 smithi152 ceph-mon[36876]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:28 smithi152 ceph-mon[36876]: osdmap e10: 1 total, 1 up, 1 in 2024-03-04T15:57:29.623 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:29 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:29.623 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:29 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:29.623 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:29 smithi116 ceph-mon[29120]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:29.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:29 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:29.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:29 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:29.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:29 smithi152 ceph-mon[36876]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:30.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.315+0000 7f97b4178700 1 -- 172.21.15.116:0/4029299402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97ac101700 msgr2=0x7f97ac103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:57:30.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.315+0000 7f97b4178700 1 --2- 172.21.15.116:0/4029299402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97ac101700 0x7f97ac103af0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009910 tx=0x7f97a8009c20 comp rx=0 tx=0).stop 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.315+0000 7f97b4178700 1 -- 172.21.15.116:0/4029299402 shutdown_connections 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.315+0000 7f97b4178700 1 --2- 172.21.15.116:0/4029299402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97ac104030 0x7f97ac106420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.315+0000 7f97b4178700 1 --2- 172.21.15.116:0/4029299402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97ac101700 0x7f97ac103af0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.315+0000 7f97b4178700 1 -- 172.21.15.116:0/4029299402 >> 172.21.15.116:0/4029299402 conn(0x7f97ac0fb050 msgr2=0x7f97ac0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.315+0000 7f97b4178700 1 -- 172.21.15.116:0/4029299402 shutdown_connections 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.315+0000 7f97b4178700 1 -- 172.21.15.116:0/4029299402 wait complete. 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.316+0000 7f97b4178700 1 Processor -- start 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.316+0000 7f97b4178700 1 -- start start 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.316+0000 7f97b4178700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97ac101700 0x7f97ac101060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.316+0000 7f97b4178700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97ac104030 0x7f97ac0ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.316+0000 7f97b4178700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac0ffbf0 con 0x7f97ac101700 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.316+0000 7f97b4178700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac0ffd30 con 0x7f97ac104030 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.316+0000 7f97b1f14700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97ac101700 0x7f97ac101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:57:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97b1f14700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97ac101700 0x7f97ac101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47562/0 (socket says 172.21.15.116:47562) 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97b1f14700 1 -- 172.21.15.116:0/3439514496 learned_addr learned my addr 172.21.15.116:0/3439514496 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97b1713700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97ac104030 0x7f97ac0ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97b1f14700 1 -- 172.21.15.116:0/3439514496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97ac104030 msgr2=0x7f97ac0ff6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97b1f14700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97ac104030 0x7f97ac0ff6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97b1f14700 1 -- 172.21.15.116:0/3439514496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80095f0 con 0x7f97ac101700 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97b1f14700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97ac101700 0x7f97ac101060 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f97a8006010 tx=0x7f97a800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97a2ffd700 1 -- 172.21.15.116:0/3439514496 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97a801d070 con 0x7f97ac101700 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97a2ffd700 1 -- 172.21.15.116:0/3439514496 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f97a800f460 con 0x7f97ac101700 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97b4178700 1 -- 172.21.15.116:0/3439514496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ac0fffb0 con 0x7f97ac101700 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97a2ffd700 1 -- 172.21.15.116:0/3439514496 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97a8021640 con 0x7f97ac101700 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.317+0000 7f97b4178700 1 -- 172.21.15.116:0/3439514496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ac1004a0 con 0x7f97ac101700 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.318+0000 7f97b4178700 1 -- 172.21.15.116:0/3439514496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ac190570 con 0x7f97ac101700 2024-03-04T15:57:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.319+0000 7f97a2ffd700 1 -- 172.21.15.116:0/3439514496 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f97a800fab0 con 0x7f97ac101700 2024-03-04T15:57:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.319+0000 7f97a2ffd700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f979806c460 0x7f979806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:57:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.319+0000 7f97a2ffd700 1 -- 172.21.15.116:0/3439514496 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f97a808bc10 con 0x7f97ac101700 2024-03-04T15:57:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.319+0000 7f97b1713700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f979806c460 0x7f979806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:57:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.319+0000 7f97b1713700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f979806c460 0x7f979806e920 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f979c004630 tx=0x7f979c00b3c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:57:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.324+0000 7f97a2ffd700 1 -- 172.21.15.116:0/3439514496 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f97a805ad00 con 0x7f97ac101700 2024-03-04T15:57:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:30.524+0000 7f97b4178700 1 -- 172.21.15.116:0/3439514496 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f97ac0611d0 con 0x7f979806c460 2024-03-04T15:57:30.782 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:57:30.782 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:57:30.782 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:57:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:57:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:31.807 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:31 smithi116 ceph-mon[29120]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:31.807 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:31 smithi116 ceph-mon[29120]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:31.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:31 smithi152 ceph-mon[36876]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:31.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:31 smithi152 ceph-mon[36876]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:34.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:34 smithi116 ceph-mon[29120]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:34.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:34.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:34.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:57:34.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:34.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:34.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:34.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:34.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:34.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:34 smithi152 ceph-mon[36876]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:34.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:34.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:34.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:57:34.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:34.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:34.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:34.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:34.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:35.313 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:35 smithi152 ceph-mon[36876]: Detected new or changed devices on smithi116 2024-03-04T15:57:35.313 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:35 smithi152 ceph-mon[36876]: Adjusting osd_memory_target on smithi116 to 11840M 2024-03-04T15:57:35.313 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:35 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2331012254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f990720f-a19d-48c1-9545-81e511c2690b"}]: dispatch 2024-03-04T15:57:35.313 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:35 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2331012254' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f990720f-a19d-48c1-9545-81e511c2690b"}]': finished 2024-03-04T15:57:35.313 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:35 smithi152 ceph-mon[36876]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:57:35.313 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:35 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:57:35.575 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:35 smithi116 ceph-mon[29120]: Detected new or changed devices on smithi116 2024-03-04T15:57:35.575 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:35 smithi116 ceph-mon[29120]: Adjusting osd_memory_target on smithi116 to 11840M 2024-03-04T15:57:35.576 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:35 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2331012254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f990720f-a19d-48c1-9545-81e511c2690b"}]: dispatch 2024-03-04T15:57:35.576 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:35 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2331012254' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f990720f-a19d-48c1-9545-81e511c2690b"}]': finished 2024-03-04T15:57:35.576 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:35 smithi116 ceph-mon[29120]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:57:35.576 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:35 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:57:36.502 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:36 smithi116 ceph-mon[29120]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:36.502 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:36 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2528897631' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:57:36.502 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:36 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:36.502 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:36 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:36 smithi152 ceph-mon[36876]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:36 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2528897631' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:57:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:36 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:36 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:37.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:37 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:57:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:37 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:57:38.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:38 smithi152 ceph-mon[36876]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:38 smithi116 ceph-mon[29120]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:39.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:39.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:39.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:39.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:39.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:39.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:39 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:39.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:39 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:39.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:39 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:39.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:39 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:39.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:39 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:40.700 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:40 smithi116 ceph-mon[29120]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:40.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:40 smithi152 ceph-mon[36876]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:42.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:42 smithi116 ceph-mon[29120]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:42.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:42 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:57:42.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:42 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:42.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:42 smithi152 ceph-mon[36876]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:42.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:42 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:57:42.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:42 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:43.715 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:43 smithi116 ceph-mon[29120]: Deploying daemon osd.1 on smithi116 2024-03-04T15:57:43.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:43 smithi152 ceph-mon[36876]: Deploying daemon osd.1 on smithi116 2024-03-04T15:57:44.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:44 smithi152 ceph-mon[36876]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:44 smithi116 ceph-mon[29120]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:46.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:46 smithi116 ceph-mon[29120]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:46.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:46 smithi152 ceph-mon[36876]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:48.446 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:48 smithi116 ceph-mon[29120]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:48.446 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:48 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:48.446 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:48 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:48.446 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:48 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:48.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:48 smithi152 ceph-mon[36876]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:48.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:48 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:48.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:48 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:48.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:48 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:50.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:50 smithi116 ceph-mon[29120]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:50.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:50 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:50.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:50 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:50.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:50 smithi152 ceph-mon[36876]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:50.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:50 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:50.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:50 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:52.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.079+0000 7f97a2ffd700 1 -- 172.21.15.116:0/3439514496 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f97ac0611d0 con 0x7f979806c460 2024-03-04T15:57:52.081 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 1 on host 'smithi116' 2024-03-04T15:57:52.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.081+0000 7f97b4178700 1 -- 172.21.15.116:0/3439514496 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f979806c460 msgr2=0x7f979806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:57:52.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.081+0000 7f97b4178700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f979806c460 0x7f979806e920 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f979c004630 tx=0x7f979c00b3c0 comp rx=0 tx=0).stop 2024-03-04T15:57:52.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.081+0000 7f97b4178700 1 -- 172.21.15.116:0/3439514496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97ac101700 msgr2=0x7f97ac101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:57:52.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.081+0000 7f97b4178700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97ac101700 0x7f97ac101060 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f97a8006010 tx=0x7f97a800bd70 comp rx=0 tx=0).stop 2024-03-04T15:57:52.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.081+0000 7f97b4178700 1 -- 172.21.15.116:0/3439514496 shutdown_connections 2024-03-04T15:57:52.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.081+0000 7f97b4178700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f979806c460 0x7f979806e920 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:57:52.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.081+0000 7f97b4178700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97ac101700 0x7f97ac101060 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:57:52.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.081+0000 7f97b4178700 1 --2- 172.21.15.116:0/3439514496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97ac104030 0x7f97ac0ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:57:52.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.081+0000 7f97b4178700 1 -- 172.21.15.116:0/3439514496 >> 172.21.15.116:0/3439514496 conn(0x7f97ac0fb050 msgr2=0x7f97ac0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:57:52.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.081+0000 7f97b4178700 1 -- 172.21.15.116:0/3439514496 shutdown_connections 2024-03-04T15:57:52.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:57:52.082+0000 7f97b4178700 1 -- 172.21.15.116:0/3439514496 wait complete. 2024-03-04T15:57:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:52 smithi116 ceph-mon[29120]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:52 smithi116 ceph-mon[29120]: from='osd.1 [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:57:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:57:52.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:52 smithi152 ceph-mon[36876]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:52.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:52 smithi152 ceph-mon[36876]: from='osd.1 [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:57:52.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:52.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:52.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:57:53.268 DEBUG:teuthology.orchestra.run.smithi116:osd.1> sudo journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.1.service 2024-03-04T15:57:53.270 INFO:tasks.cephadm:Deploying osd.2 on smithi116 with /dev/vg_nvme/lv_2... 2024-03-04T15:57:53.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T15:57:53.542 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:53 smithi116 ceph-mon[29120]: from='osd.1 [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:57:53.542 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:53 smithi116 ceph-mon[29120]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:57:53.542 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:53 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:57:53.542 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:53 smithi116 ceph-mon[29120]: from='osd.1 [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T15:57:53.543 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 15:57:53 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[56141]: 2024-03-04T15:57:53.410+0000 7fa8bba67700 -1 osd.1 0 waiting for initial osdmap 2024-03-04T15:57:53.543 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 15:57:53 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[56141]: 2024-03-04T15:57:53.413+0000 7fa8b785c700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:57:53.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:53 smithi152 ceph-mon[36876]: from='osd.1 [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:57:53.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:53 smithi152 ceph-mon[36876]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:57:53.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:53 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:57:53.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:53 smithi152 ceph-mon[36876]: from='osd.1 [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T15:57:54.036 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:57:54.529 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:54 smithi116 ceph-mon[29120]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:54.529 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:54 smithi116 ceph-mon[29120]: from='osd.1 [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T15:57:54.529 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:54 smithi116 ceph-mon[29120]: osdmap e13: 2 total, 1 up, 2 in 2024-03-04T15:57:54.529 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:54 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:57:54.529 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:54 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:57:54.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:54 smithi152 ceph-mon[36876]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:54.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:54 smithi152 ceph-mon[36876]: from='osd.1 [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T15:57:54.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:54 smithi152 ceph-mon[36876]: osdmap e13: 2 total, 1 up, 2 in 2024-03-04T15:57:54.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:54 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:57:54.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:54 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:57:55.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:55 smithi116 ceph-mon[29120]: purged_snaps scrub starts 2024-03-04T15:57:55.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:55 smithi116 ceph-mon[29120]: purged_snaps scrub ok 2024-03-04T15:57:55.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:55 smithi116 ceph-mon[29120]: osd.1 [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008] boot 2024-03-04T15:57:55.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:55 smithi116 ceph-mon[29120]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:57:55.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:55 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:57:55.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:55 smithi152 ceph-mon[36876]: purged_snaps scrub starts 2024-03-04T15:57:55.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:55 smithi152 ceph-mon[36876]: purged_snaps scrub ok 2024-03-04T15:57:55.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:55 smithi152 ceph-mon[36876]: osd.1 [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008] boot 2024-03-04T15:57:55.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:55 smithi152 ceph-mon[36876]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:57:55.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:55 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:57:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:56 smithi116 ceph-mon[29120]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:56 smithi116 ceph-mon[29120]: osdmap e15: 2 total, 2 up, 2 in 2024-03-04T15:57:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:56 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:56 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:56.770 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T15:57:56.770 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T15:57:56.770 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-04T15:57:56.770 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-04T15:57:56.770 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127872 s, 820 MB/s 2024-03-04T15:57:56.770 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-04T15:57:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:56 smithi152 ceph-mon[36876]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:57:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:56 smithi152 ceph-mon[36876]: osdmap e15: 2 total, 2 up, 2 in 2024-03-04T15:57:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:56 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:56 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:57:57.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-03-04T15:57:58.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:57:58 smithi116 ceph-mon[29120]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:57:58.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:57:58 smithi152 ceph-mon[36876]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:57:58.923 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:58:00.665 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:00 smithi116 ceph-mon[29120]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:00 smithi152 ceph-mon[36876]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:01.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.231+0000 7fa432af8700 1 -- 172.21.15.116:0/3857203039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42c100bd0 msgr2=0x7fa42c100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:58:01.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.231+0000 7fa432af8700 1 --2- 172.21.15.116:0/3857203039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42c100bd0 0x7fa42c100ff0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa420009910 tx=0x7fa420009c20 comp rx=0 tx=0).stop 2024-03-04T15:58:01.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.231+0000 7fa432af8700 1 -- 172.21.15.116:0/3857203039 shutdown_connections 2024-03-04T15:58:01.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.231+0000 7fa432af8700 1 --2- 172.21.15.116:0/3857203039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa42c101dd0 0x7fa42c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:01.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.231+0000 7fa432af8700 1 --2- 172.21.15.116:0/3857203039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42c100bd0 0x7fa42c100ff0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:01.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.231+0000 7fa432af8700 1 -- 172.21.15.116:0/3857203039 >> 172.21.15.116:0/3857203039 conn(0x7fa42c0fc190 msgr2=0x7fa42c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:58:01.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.231+0000 7fa432af8700 1 -- 172.21.15.116:0/3857203039 shutdown_connections 2024-03-04T15:58:01.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.231+0000 7fa432af8700 1 -- 172.21.15.116:0/3857203039 wait complete. 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.232+0000 7fa432af8700 1 Processor -- start 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.232+0000 7fa432af8700 1 -- start start 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.232+0000 7fa432af8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa42c100bd0 0x7fa42c196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.232+0000 7fa432af8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42c101dd0 0x7fa42c1968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.232+0000 7fa432af8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c196ed0 con 0x7fa42c101dd0 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.232+0000 7fa432af8700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c197010 con 0x7fa42c100bd0 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.232+0000 7fa42bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42c101dd0 0x7fa42c1968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.232+0000 7fa42bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42c101dd0 0x7fa42c1968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60176/0 (socket says 172.21.15.116:60176) 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.232+0000 7fa42bfff700 1 -- 172.21.15.116:0/2217996339 learned_addr learned my addr 172.21.15.116:0/2217996339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.232+0000 7fa430894700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa42c100bd0 0x7fa42c196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.233+0000 7fa42bfff700 1 -- 172.21.15.116:0/2217996339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa42c100bd0 msgr2=0x7fa42c196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.233+0000 7fa42bfff700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa42c100bd0 0x7fa42c196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:01.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.233+0000 7fa42bfff700 1 -- 172.21.15.116:0/2217996339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4200095f0 con 0x7fa42c101dd0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.233+0000 7fa430894700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa42c100bd0 0x7fa42c196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.233+0000 7fa42bfff700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42c101dd0 0x7fa42c1968b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa41800d900 tx=0x7fa41800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.233+0000 7fa429ffb700 1 -- 172.21.15.116:0/2217996339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa418004a00 con 0x7fa42c101dd0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.233+0000 7fa429ffb700 1 -- 172.21.15.116:0/2217996339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa4180054d0 con 0x7fa42c101dd0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.233+0000 7fa432af8700 1 -- 172.21.15.116:0/2217996339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42c19bac0 con 0x7fa42c101dd0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.233+0000 7fa429ffb700 1 -- 172.21.15.116:0/2217996339 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa418009d60 con 0x7fa42c101dd0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.233+0000 7fa432af8700 1 -- 172.21.15.116:0/2217996339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42c19c010 con 0x7fa42c101dd0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.234+0000 7fa432af8700 1 -- 172.21.15.116:0/2217996339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa42c109750 con 0x7fa42c101dd0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.234+0000 7fa429ffb700 1 -- 172.21.15.116:0/2217996339 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7fa418005020 con 0x7fa42c101dd0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.235+0000 7fa429ffb700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa41c06c460 0x7fa41c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.235+0000 7fa429ffb700 1 -- 172.21.15.116:0/2217996339 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7fa41808acd0 con 0x7fa42c101dd0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.235+0000 7fa430894700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa41c06c460 0x7fa41c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:58:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.235+0000 7fa430894700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa41c06c460 0x7fa41c06e920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa420006010 tx=0x7fa420005a90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:58:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.239+0000 7fa429ffb700 1 -- 172.21.15.116:0/2217996339 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa418059c10 con 0x7fa42c101dd0 2024-03-04T15:58:01.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:01.460+0000 7fa432af8700 1 -- 172.21.15.116:0/2217996339 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa42c106720 con 0x7fa41c06c460 2024-03-04T15:58:02.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: Detected new or changed devices on smithi116 2024-03-04T15:58:02.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:02.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:02.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:02.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:02.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: Adjusting osd_memory_target on smithi116 to 5920M 2024-03-04T15:58:02.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:02.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:02.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:02.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:02.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:02.110 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:02.110 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:02.110 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:02.110 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:02.110 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:02.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: Detected new or changed devices on smithi116 2024-03-04T15:58:02.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:02.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:02.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: Adjusting osd_memory_target on smithi116 to 5920M 2024-03-04T15:58:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:02.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:02.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:02.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:02.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:04.340 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:04 smithi116 ceph-mon[29120]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:04.341 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:04 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:04.341 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:04 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:04.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:04 smithi152 ceph-mon[36876]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:04.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:04 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:04.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:04 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:06.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:06 smithi116 ceph-mon[29120]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:06.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:06 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/395651105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe3132c8-6b15-4a11-a884-7c6a09d42955"}]: dispatch 2024-03-04T15:58:06.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:06 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/395651105' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe3132c8-6b15-4a11-a884-7c6a09d42955"}]': finished 2024-03-04T15:58:06.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:06 smithi116 ceph-mon[29120]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:58:06.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:06 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:06.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:06 smithi152 ceph-mon[36876]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:06.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:06 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/395651105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe3132c8-6b15-4a11-a884-7c6a09d42955"}]: dispatch 2024-03-04T15:58:06.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:06 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/395651105' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe3132c8-6b15-4a11-a884-7c6a09d42955"}]': finished 2024-03-04T15:58:06.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:06 smithi152 ceph-mon[36876]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:58:06.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:06 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:07.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:07 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1424436444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:07.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:07.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:07.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:07.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:07.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:07.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:58:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:07 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/1424436444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:07.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:07.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:07.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:07.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:58:08.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:08 smithi152 ceph-mon[36876]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:08.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:08 smithi116 ceph-mon[29120]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:10.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:10 smithi116 ceph-mon[29120]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:10 smithi152 ceph-mon[36876]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:12.401 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:12 smithi116 ceph-mon[29120]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:12.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:12 smithi152 ceph-mon[36876]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:14 smithi116 ceph-mon[29120]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:58:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:14 smithi116 ceph-mon[29120]: Deploying daemon osd.2 on smithi116 2024-03-04T15:58:14.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:14 smithi152 ceph-mon[36876]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:14.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:14 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:58:14.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:14 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:14.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:14 smithi152 ceph-mon[36876]: Deploying daemon osd.2 on smithi116 2024-03-04T15:58:16.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:16 smithi116 ceph-mon[29120]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:16.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:16 smithi152 ceph-mon[36876]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:18.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:18 smithi152 ceph-mon[36876]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:18.563 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:18 smithi116 ceph-mon[29120]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:20.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:20 smithi116 ceph-mon[29120]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:20.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:20.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:20.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:20.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:20 smithi152 ceph-mon[36876]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:20.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:20.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:20.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:22.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:22 smithi116 ceph-mon[29120]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:22.252 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:58:22.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:22 smithi152 ceph-mon[36876]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:22.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:58:22.624 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 2 on host 'smithi116' 2024-03-04T15:58:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.622+0000 7fa429ffb700 1 -- 172.21.15.116:0/2217996339 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa42c106720 con 0x7fa41c06c460 2024-03-04T15:58:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.624+0000 7fa432af8700 1 -- 172.21.15.116:0/2217996339 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa41c06c460 msgr2=0x7fa41c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:58:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.624+0000 7fa432af8700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa41c06c460 0x7fa41c06e920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa420006010 tx=0x7fa420005a90 comp rx=0 tx=0).stop 2024-03-04T15:58:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.624+0000 7fa432af8700 1 -- 172.21.15.116:0/2217996339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42c101dd0 msgr2=0x7fa42c1968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:58:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.624+0000 7fa432af8700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42c101dd0 0x7fa42c1968b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa41800d900 tx=0x7fa41800dcc0 comp rx=0 tx=0).stop 2024-03-04T15:58:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.624+0000 7fa432af8700 1 -- 172.21.15.116:0/2217996339 shutdown_connections 2024-03-04T15:58:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.624+0000 7fa432af8700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa42c100bd0 0x7fa42c196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.624+0000 7fa432af8700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa41c06c460 0x7fa41c06e920 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.624+0000 7fa432af8700 1 --2- 172.21.15.116:0/2217996339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42c101dd0 0x7fa42c1968b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.624+0000 7fa432af8700 1 -- 172.21.15.116:0/2217996339 >> 172.21.15.116:0/2217996339 conn(0x7fa42c0fc190 msgr2=0x7fa42c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:58:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.624+0000 7fa432af8700 1 -- 172.21.15.116:0/2217996339 shutdown_connections 2024-03-04T15:58:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:58:22.625+0000 7fa432af8700 1 -- 172.21.15.116:0/2217996339 wait complete. 2024-03-04T15:58:23.294 DEBUG:teuthology.orchestra.run.smithi116:osd.2> sudo journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.2.service 2024-03-04T15:58:23.297 INFO:tasks.cephadm:Deploying osd.3 on smithi152 with /dev/vg_nvme/lv_4... 2024-03-04T15:58:23.297 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T15:58:23.516 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi152/config 2024-03-04T15:58:23.839 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:23 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:23.839 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:23 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:23.840 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:23 smithi152 ceph-mon[36876]: from='osd.2 [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:58:23.840 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:23 smithi152 ceph-mon[36876]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:23.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:23 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:23.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:23 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:23.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:23 smithi116 ceph-mon[29120]: from='osd.2 [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:58:23.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:23 smithi116 ceph-mon[29120]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:24.727 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:24 smithi152 ceph-mon[36876]: from='osd.2 [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:58:24.727 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:24 smithi152 ceph-mon[36876]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:58:24.727 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:24 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:24.727 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:24 smithi152 ceph-mon[36876]: from='osd.2 [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T15:58:24.965 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:24 smithi116 ceph-mon[29120]: from='osd.2 [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:58:24.965 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:24 smithi116 ceph-mon[29120]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:58:24.965 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:24 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:24.965 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:24 smithi116 ceph-mon[29120]: from='osd.2 [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T15:58:24.965 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 15:58:24 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[63952]: 2024-03-04T15:58:24.599+0000 7ff6888f8700 -1 osd.2 0 waiting for initial osdmap 2024-03-04T15:58:24.965 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 15:58:24 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[63952]: 2024-03-04T15:58:24.604+0000 7ff6826e9700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:58:25.231 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T15:58:25.231 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T15:58:25.231 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-04T15:58:25.231 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-04T15:58:25.231 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115793 s, 906 MB/s 2024-03-04T15:58:25.231 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-04T15:58:25.798 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:25 smithi152 ceph-mon[36876]: from='osd.2 [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T15:58:25.798 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:25 smithi152 ceph-mon[36876]: osdmap e18: 3 total, 2 up, 3 in 2024-03-04T15:58:25.798 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:25 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:25.798 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:25 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:25.799 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:25 smithi152 ceph-mon[36876]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:25.819 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_4 2024-03-04T15:58:25.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:25 smithi116 ceph-mon[29120]: from='osd.2 [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T15:58:25.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:25 smithi116 ceph-mon[29120]: osdmap e18: 3 total, 2 up, 3 in 2024-03-04T15:58:25.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:25 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:25.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:25 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:25.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:25 smithi116 ceph-mon[29120]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:26.025 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi152/config 2024-03-04T15:58:26.805 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:26 smithi152 ceph-mon[36876]: purged_snaps scrub starts 2024-03-04T15:58:26.805 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:26 smithi152 ceph-mon[36876]: purged_snaps scrub ok 2024-03-04T15:58:26.805 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:26 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:26.805 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:26 smithi152 ceph-mon[36876]: osd.2 [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941] boot 2024-03-04T15:58:26.805 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:26 smithi152 ceph-mon[36876]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:58:26.805 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:26 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:26.805 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:26 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:26.805 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:26 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:26.871 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:26 smithi116 ceph-mon[29120]: purged_snaps scrub starts 2024-03-04T15:58:26.871 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:26 smithi116 ceph-mon[29120]: purged_snaps scrub ok 2024-03-04T15:58:26.871 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:26 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:26.871 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:26 smithi116 ceph-mon[29120]: osd.2 [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941] boot 2024-03-04T15:58:26.871 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:26 smithi116 ceph-mon[29120]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:58:26.871 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:26 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:26.871 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:26 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:26.871 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:26 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:27.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.532+0000 7f7614b82700 1 -- 172.21.15.152:0/3017084242 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7610100bc0 msgr2=0x7f7610100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:58:27.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.532+0000 7f7614b82700 1 --2- 172.21.15.152:0/3017084242 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7610100bc0 0x7f7610100fe0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7600009910 tx=0x7f7600009c20 comp rx=0 tx=0).stop 2024-03-04T15:58:27.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.533+0000 7f7614b82700 1 -- 172.21.15.152:0/3017084242 shutdown_connections 2024-03-04T15:58:27.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.533+0000 7f7614b82700 1 --2- 172.21.15.152:0/3017084242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7610101dc0 0x7f7610102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:27.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.533+0000 7f7614b82700 1 --2- 172.21.15.152:0/3017084242 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7610100bc0 0x7f7610100fe0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.533+0000 7f7614b82700 1 -- 172.21.15.152:0/3017084242 >> 172.21.15.152:0/3017084242 conn(0x7f76100fc140 msgr2=0x7f76100fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.533+0000 7f7614b82700 1 -- 172.21.15.152:0/3017084242 shutdown_connections 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.533+0000 7f7614b82700 1 -- 172.21.15.152:0/3017084242 wait complete. 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.533+0000 7f7614b82700 1 Processor -- start 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.533+0000 7f7614b82700 1 -- start start 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.533+0000 7f7614b82700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7610100bc0 0x7f7610196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.533+0000 7f7614b82700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7610101dc0 0x7f76101968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f7614b82700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7610196ef0 con 0x7f7610100bc0 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f7614b82700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7610197030 con 0x7f7610101dc0 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f760dd9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7610101dc0 0x7f76101968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f760dd9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7610101dc0 0x7f76101968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:37218/0 (socket says 172.21.15.152:37218) 2024-03-04T15:58:27.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f760dd9b700 1 -- 172.21.15.152:0/2694570060 learned_addr learned my addr 172.21.15.152:0/2694570060 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f760e59c700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7610100bc0 0x7f7610196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f760dd9b700 1 -- 172.21.15.152:0/2694570060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7610100bc0 msgr2=0x7f7610196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f760dd9b700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7610100bc0 0x7f7610196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f760dd9b700 1 -- 172.21.15.152:0/2694570060 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76000095f0 con 0x7f7610101dc0 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f760dd9b700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7610101dc0 0x7f76101968d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f760400d8d0 tx=0x7f760400dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f75ff7fe700 1 -- 172.21.15.152:0/2694570060 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76040098a0 con 0x7f7610101dc0 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f75ff7fe700 1 -- 172.21.15.152:0/2694570060 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7604010460 con 0x7f7610101dc0 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f75ff7fe700 1 -- 172.21.15.152:0/2694570060 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f760400f5f0 con 0x7f7610101dc0 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f7614b82700 1 -- 172.21.15.152:0/2694570060 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761019bae0 con 0x7f7610101dc0 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.534+0000 7f7614b82700 1 -- 172.21.15.152:0/2694570060 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761019c030 con 0x7f7610101dc0 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.535+0000 7f7614b82700 1 -- 172.21.15.152:0/2694570060 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7610109720 con 0x7f7610101dc0 2024-03-04T15:58:27.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.536+0000 7f75ff7fe700 1 -- 172.21.15.152:0/2694570060 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f7604009a00 con 0x7f7610101dc0 2024-03-04T15:58:27.540 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.536+0000 7f75ff7fe700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f75f806c460 0x7f75f806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:58:27.540 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.536+0000 7f75ff7fe700 1 -- 172.21.15.152:0/2694570060 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2846+0+0 (secure 0 0 0) 0x7f760408aae0 con 0x7f7610101dc0 2024-03-04T15:58:27.540 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.537+0000 7f760e59c700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f75f806c460 0x7f75f806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:58:27.540 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.537+0000 7f760e59c700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f75f806c460 0x7f75f806e920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7600006010 tx=0x7f760000b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:58:27.542 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.541+0000 7f75ff7fe700 1 -- 172.21.15.152:0/2694570060 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f76040598f0 con 0x7f7610101dc0 2024-03-04T15:58:27.743 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:27.741+0000 7f7614b82700 1 -- 172.21.15.152:0/2694570060 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7610106710 con 0x7f75f806c460 2024-03-04T15:58:27.875 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:27 smithi116 ceph-mon[29120]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:58:27.875 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:27 smithi116 ceph-mon[29120]: pgmap v78: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:27.875 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:27 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:58:28.060 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:27 smithi152 ceph-mon[36876]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:58:28.060 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:27 smithi152 ceph-mon[36876]: pgmap v78: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:28.060 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:27 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:58:28.969 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:28 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T15:58:28.970 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:28 smithi116 ceph-mon[29120]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:58:28.970 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:28 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:58:28.970 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:28 smithi116 ceph-mon[29120]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:28.970 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:28 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:28.970 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:28 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:28.970 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:28 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:28.973 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:28 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T15:58:28.973 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:28 smithi152 ceph-mon[36876]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:58:28.973 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:28 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:58:28.974 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:28 smithi152 ceph-mon[36876]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:28.974 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:28 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:28.974 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:28 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:28.974 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:28 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:29.626 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68568]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:29.626 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68568]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:58:29.627 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68568]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:58:29.627 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68568]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:29.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:29 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:58:29.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:29 smithi116 ceph-mon[29120]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:58:29.954 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:29 smithi116 ceph-mon[29120]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:29.954 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68571]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:29.954 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68571]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:58:29.954 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68571]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:58:29.954 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68571]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:30.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:29 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:58:30.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:29 smithi152 ceph-mon[36876]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:58:30.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:29 smithi152 ceph-mon[36876]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:30.225 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68672]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:30.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68672]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:58:30.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68672]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:58:30.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 15:58:29 smithi116 sudo[68672]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:30.491 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68750]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:30.491 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:58:30.491 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68750]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:58:30.491 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68750]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:30.756 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68773]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:30.756 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68773]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:58:30.756 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68773]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:58:30.756 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68773]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: osdmap e23: 3 total, 3 up, 3 in 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: Detected new or changed devices on smithi116 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: Adjusting osd_memory_target on smithi116 to 3946M 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:30.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:30.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:30.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:30.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/465230316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15930a9b-f721-4c30-9c5b-bef5ef69777c"}]: dispatch 2024-03-04T15:58:30.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15930a9b-f721-4c30-9c5b-bef5ef69777c"}]: dispatch 2024-03-04T15:58:30.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15930a9b-f721-4c30-9c5b-bef5ef69777c"}]': finished 2024-03-04T15:58:30.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T15:58:30.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:30.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:30.763+0000 7f75ff7fe700 1 -- 172.21.15.152:0/2694570060 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7604056230 con 0x7f7610101dc0 2024-03-04T15:58:31.007 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68790]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:31.007 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68790]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:58:31.007 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68790]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:58:31.007 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 15:58:30 smithi116 sudo[68790]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:31.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: osdmap e23: 3 total, 3 up, 3 in 2024-03-04T15:58:31.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: Detected new or changed devices on smithi116 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: Adjusting osd_memory_target on smithi116 to 3946M 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='client.? 172.21.15.152:0/465230316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15930a9b-f721-4c30-9c5b-bef5ef69777c"}]: dispatch 2024-03-04T15:58:31.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15930a9b-f721-4c30-9c5b-bef5ef69777c"}]: dispatch 2024-03-04T15:58:31.110 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15930a9b-f721-4c30-9c5b-bef5ef69777c"}]': finished 2024-03-04T15:58:31.110 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T15:58:31.110 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:31.270 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 sudo[68836]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:31.271 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 sudo[68836]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:58:31.271 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 sudo[68836]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:58:31.271 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 sudo[68836]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 sudo[68897]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:31.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 sudo[68897]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:58:31.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 sudo[68897]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:58:31.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 sudo[68897]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:31.780 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 sudo[41094]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:31.781 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 sudo[41094]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:58:31.781 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 sudo[41094]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:58:31.781 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 sudo[41094]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:32.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 sudo[41281]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:32.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 sudo[41281]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:58:32.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 sudo[41281]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:58:32.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 sudo[41281]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:32.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 ceph-mon[36876]: from='client.? 172.21.15.152:0/3510864237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:32.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 ceph-mon[36876]: mgrmap e19: smithi116.jwdhop(active, since 2m), standbys: smithi152.aawieo 2024-03-04T15:58:32.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 ceph-mon[36876]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:32.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 ceph-mon[36876]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:32.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 ceph-mon[36876]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:32.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T15:58:32.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:58:32.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T15:58:32.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:31 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:58:32.144 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/3510864237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:32.144 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 ceph-mon[29120]: mgrmap e19: smithi116.jwdhop(active, since 2m), standbys: smithi152.aawieo 2024-03-04T15:58:32.145 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 ceph-mon[29120]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:32.145 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 ceph-mon[29120]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:32.145 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 ceph-mon[29120]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:32.145 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T15:58:32.145 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:58:32.145 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T15:58:32.145 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:31 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T15:58:32.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:32 smithi116 ceph-mon[29120]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:32.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:32 smithi116 ceph-mon[29120]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:32.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:32 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:33.067 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:32 smithi152 ceph-mon[36876]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:33.067 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:32 smithi152 ceph-mon[36876]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:33.067 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:32 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:34.475 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:34 smithi152 ceph-mon[36876]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:34.475 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:34.475 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:34.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:34 smithi116 ceph-mon[29120]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:34.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:34.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:35 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:35 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:35 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:35.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:35 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:35.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:35 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:35.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:35 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:36 smithi152 ceph-mon[36876]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:36.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:36 smithi116 ceph-mon[29120]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:37.356 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:37 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:58:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:37 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:58:38.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:38 smithi152 ceph-mon[36876]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-04T15:58:38.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:38 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T15:58:38.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:38 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:38.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:38 smithi152 ceph-mon[36876]: Deploying daemon osd.3 on smithi152 2024-03-04T15:58:38.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:38 smithi116 ceph-mon[29120]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-04T15:58:38.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:38 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T15:58:38.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:38 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:38.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:38 smithi116 ceph-mon[29120]: Deploying daemon osd.3 on smithi152 2024-03-04T15:58:40.528 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:40 smithi116 ceph-mon[29120]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-04T15:58:40.547 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:40 smithi152 ceph-mon[36876]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-04T15:58:42.536 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:42 smithi152 ceph-mon[36876]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-04T15:58:42.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:42 smithi116 ceph-mon[29120]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-04T15:58:44.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:44.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:44.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:44.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:43 smithi152 ceph-mon[36876]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-04T15:58:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:43 smithi116 ceph-mon[29120]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-04T15:58:46.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:45.999+0000 7f75ff7fe700 1 -- 172.21.15.152:0/2694570060 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7610106710 con 0x7f75f806c460 2024-03-04T15:58:46.000 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 3 on host 'smithi152' 2024-03-04T15:58:46.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 -- 172.21.15.152:0/2694570060 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f75f806c460 msgr2=0x7f75f806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:58:46.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f75f806c460 0x7f75f806e920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7600006010 tx=0x7f760000b540 comp rx=0 tx=0).stop 2024-03-04T15:58:46.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 -- 172.21.15.152:0/2694570060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7610101dc0 msgr2=0x7f76101968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:58:46.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7610101dc0 0x7f76101968d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f760400d8d0 tx=0x7f760400dbe0 comp rx=0 tx=0).stop 2024-03-04T15:58:46.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 -- 172.21.15.152:0/2694570060 shutdown_connections 2024-03-04T15:58:46.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f75f806c460 0x7f75f806e920 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:46.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7610100bc0 0x7f7610196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:46.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 --2- 172.21.15.152:0/2694570060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7610101dc0 0x7f76101968d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:46.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 -- 172.21.15.152:0/2694570060 >> 172.21.15.152:0/2694570060 conn(0x7f76100fc140 msgr2=0x7f7610104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:58:46.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 -- 172.21.15.152:0/2694570060 shutdown_connections 2024-03-04T15:58:46.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:46.002+0000 7f7614b82700 1 -- 172.21.15.152:0/2694570060 wait complete. 2024-03-04T15:58:46.311 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:46 smithi152 ceph-mon[36876]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-04T15:58:46.311 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:46 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:46.312 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:46 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:46.392 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:46 smithi116 ceph-mon[29120]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-04T15:58:46.392 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:46 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:46.392 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:46 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:47.163 DEBUG:teuthology.orchestra.run.smithi152:osd.3> sudo journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.3.service 2024-03-04T15:58:47.165 INFO:tasks.cephadm:Deploying osd.4 on smithi152 with /dev/vg_nvme/lv_3... 2024-03-04T15:58:47.165 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T15:58:47.415 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi152/config 2024-03-04T15:58:47.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:47 smithi116 ceph-mon[29120]: from='osd.3 [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T15:58:47.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:47 smithi116 ceph-mon[29120]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T15:58:47.536 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:47 smithi152 ceph-mon[36876]: from='osd.3 [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T15:58:47.536 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:47 smithi152 ceph-mon[36876]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T15:58:48.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:48 smithi116 ceph-mon[29120]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-04T15:58:48.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:48 smithi116 ceph-mon[29120]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T15:58:48.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:48 smithi116 ceph-mon[29120]: osdmap e25: 4 total, 3 up, 4 in 2024-03-04T15:58:48.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:48 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:48.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:48 smithi116 ceph-mon[29120]: from='osd.3 [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:58:48.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:48 smithi116 ceph-mon[29120]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:58:48.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:48 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:48.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:48 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:48.499 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:48 smithi152 ceph-mon[36876]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-04T15:58:48.499 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:48 smithi152 ceph-mon[36876]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T15:58:48.499 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:48 smithi152 ceph-mon[36876]: osdmap e25: 4 total, 3 up, 4 in 2024-03-04T15:58:48.499 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:48 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:48.499 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:48 smithi152 ceph-mon[36876]: from='osd.3 [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:58:48.499 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:48 smithi152 ceph-mon[36876]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:58:48.499 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:48 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:48.499 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:48 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:48.500 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 15:58:48 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[42907]: 2024-03-04T15:58:48.149+0000 7fbfcf6eb700 -1 osd.3 0 waiting for initial osdmap 2024-03-04T15:58:48.500 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 15:58:48 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[42907]: 2024-03-04T15:58:48.155+0000 7fbfc94dc700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:58:49.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:49 smithi152 ceph-mon[36876]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T15:58:49.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:49 smithi152 ceph-mon[36876]: osdmap e26: 4 total, 3 up, 4 in 2024-03-04T15:58:49.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:49 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:49.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:49 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:49.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:49 smithi152 ceph-mon[36876]: osd.3 [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679] boot 2024-03-04T15:58:49.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:49 smithi152 ceph-mon[36876]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T15:58:49.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:49 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:49 smithi116 ceph-mon[29120]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T15:58:49.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:49 smithi116 ceph-mon[29120]: osdmap e26: 4 total, 3 up, 4 in 2024-03-04T15:58:49.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:49 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:49.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:49 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:49.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:49 smithi116 ceph-mon[29120]: osd.3 [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679] boot 2024-03-04T15:58:49.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:49 smithi116 ceph-mon[29120]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T15:58:49.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:49 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:49.811 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T15:58:49.812 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T15:58:49.812 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-04T15:58:49.812 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-04T15:58:49.812 INFO:teuthology.orchestra.run.smithi152.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013167 s, 796 MB/s 2024-03-04T15:58:49.812 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 2024-03-04T15:58:49.812 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-04T15:58:50.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:50 smithi152 ceph-mon[36876]: purged_snaps scrub starts 2024-03-04T15:58:50.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:50 smithi152 ceph-mon[36876]: purged_snaps scrub ok 2024-03-04T15:58:50.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:50 smithi152 ceph-mon[36876]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:50.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:50 smithi152 ceph-mon[36876]: osdmap e28: 4 total, 4 up, 4 in 2024-03-04T15:58:50.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:50 smithi116 ceph-mon[29120]: purged_snaps scrub starts 2024-03-04T15:58:50.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:50 smithi116 ceph-mon[29120]: purged_snaps scrub ok 2024-03-04T15:58:50.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:50 smithi116 ceph-mon[29120]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:50.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:50 smithi116 ceph-mon[29120]: osdmap e28: 4 total, 4 up, 4 in 2024-03-04T15:58:51.093 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_3 2024-03-04T15:58:51.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:51 smithi152 ceph-mon[36876]: osdmap e29: 4 total, 4 up, 4 in 2024-03-04T15:58:51.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:51 smithi116 ceph-mon[29120]: osdmap e29: 4 total, 4 up, 4 in 2024-03-04T15:58:51.937 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi152/config 2024-03-04T15:58:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-03-04T15:58:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: osdmap e30: 4 total, 4 up, 4 in 2024-03-04T15:58:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:52.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:52.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:52.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:52.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:52.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:58:52.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:52.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:52.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:52.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: osdmap e30: 4 total, 4 up, 4 in 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:58:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:52.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:52.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:52.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:58:53.454 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.450+0000 7f8a4c975700 1 -- 172.21.15.152:0/2084485836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a48101dc0 msgr2=0x7f8a48102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:58:53.454 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.450+0000 7f8a4c975700 1 --2- 172.21.15.152:0/2084485836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a48101dc0 0x7f8a48102220 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009910 tx=0x7f8a30009c20 comp rx=0 tx=0).stop 2024-03-04T15:58:53.454 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.450+0000 7f8a4c975700 1 -- 172.21.15.152:0/2084485836 shutdown_connections 2024-03-04T15:58:53.454 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.450+0000 7f8a4c975700 1 --2- 172.21.15.152:0/2084485836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a48101dc0 0x7f8a48102220 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.450+0000 7f8a4c975700 1 --2- 172.21.15.152:0/2084485836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48100bc0 0x7f8a48100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.450+0000 7f8a4c975700 1 -- 172.21.15.152:0/2084485836 >> 172.21.15.152:0/2084485836 conn(0x7f8a480fc140 msgr2=0x7f8a480fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.450+0000 7f8a4c975700 1 -- 172.21.15.152:0/2084485836 shutdown_connections 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.450+0000 7f8a4c975700 1 -- 172.21.15.152:0/2084485836 wait complete. 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a4c975700 1 Processor -- start 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a4c975700 1 -- start start 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a4c975700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48100bc0 0x7f8a48196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a4c975700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a48101dc0 0x7f8a48196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a4c975700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a48196f70 con 0x7f8a48100bc0 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a4c975700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a481970b0 con 0x7f8a48101dc0 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a45d9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a48101dc0 0x7f8a48196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a45d9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a48101dc0 0x7f8a48196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:34986/0 (socket says 172.21.15.152:34986) 2024-03-04T15:58:53.455 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a45d9b700 1 -- 172.21.15.152:0/2455193757 learned_addr learned my addr 172.21.15.152:0/2455193757 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:58:53.456 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a4659c700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48100bc0 0x7f8a48196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:58:53.456 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a45d9b700 1 -- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48100bc0 msgr2=0x7f8a48196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:58:53.456 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a45d9b700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48100bc0 0x7f8a48196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:58:53.456 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a45d9b700 1 -- 172.21.15.152:0/2455193757 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a300095f0 con 0x7f8a48101dc0 2024-03-04T15:58:53.456 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a4659c700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48100bc0 0x7f8a48196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T15:58:53.456 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a45d9b700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a48101dc0 0x7f8a48196950 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8a30006010 tx=0x7f8a3000be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:58:53.456 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.451+0000 7f8a3f7fe700 1 -- 172.21.15.152:0/2455193757 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a3001d070 con 0x7f8a48101dc0 2024-03-04T15:58:53.456 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.452+0000 7f8a4c975700 1 -- 172.21.15.152:0/2455193757 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4819bb00 con 0x7f8a48101dc0 2024-03-04T15:58:53.456 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.452+0000 7f8a3f7fe700 1 -- 172.21.15.152:0/2455193757 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8a3000f460 con 0x7f8a48101dc0 2024-03-04T15:58:53.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.452+0000 7f8a3f7fe700 1 -- 172.21.15.152:0/2455193757 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a30021640 con 0x7f8a48101dc0 2024-03-04T15:58:53.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.452+0000 7f8a4c975700 1 -- 172.21.15.152:0/2455193757 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4819c070 con 0x7f8a48101dc0 2024-03-04T15:58:53.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.452+0000 7f8a4c975700 1 -- 172.21.15.152:0/2455193757 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a481097c0 con 0x7f8a48101dc0 2024-03-04T15:58:53.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.453+0000 7f8a3f7fe700 1 -- 172.21.15.152:0/2455193757 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8a3000fab0 con 0x7f8a48101dc0 2024-03-04T15:58:53.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.453+0000 7f8a3f7fe700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f8a3406c4e0 0x7f8a3406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:58:53.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.454+0000 7f8a3f7fe700 1 -- 172.21.15.152:0/2455193757 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f8a3008c610 con 0x7f8a48101dc0 2024-03-04T15:58:53.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.454+0000 7f8a4659c700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f8a3406c4e0 0x7f8a3406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:58:53.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.454+0000 7f8a4659c700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f8a3406c4e0 0x7f8a3406e9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8a48101c20 tx=0x7f8a400073c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:58:53.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.458+0000 7f8a3f7fe700 1 -- 172.21.15.152:0/2455193757 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8a3005b040 con 0x7f8a48101dc0 2024-03-04T15:58:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:53 smithi152 ceph-mon[36876]: Detected new or changed devices on smithi152 2024-03-04T15:58:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:53 smithi152 ceph-mon[36876]: Adjusting osd_memory_target on smithi152 to 14912M 2024-03-04T15:58:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:53 smithi152 ceph-mon[36876]: osdmap e31: 4 total, 4 up, 4 in 2024-03-04T15:58:53.668 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:58:53.667+0000 7f8a4c975700 1 -- 172.21.15.152:0/2455193757 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a48106710 con 0x7f8a3406c4e0 2024-03-04T15:58:53.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:53 smithi116 ceph-mon[29120]: Detected new or changed devices on smithi152 2024-03-04T15:58:53.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:53 smithi116 ceph-mon[29120]: Adjusting osd_memory_target on smithi152 to 14912M 2024-03-04T15:58:53.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:53 smithi116 ceph-mon[29120]: osdmap e31: 4 total, 4 up, 4 in 2024-03-04T15:58:54.470 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:54 smithi152 ceph-mon[36876]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T15:58:54.470 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:54 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:54.470 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:54 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:54.470 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:54 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:54.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:54 smithi116 ceph-mon[29120]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T15:58:54.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:54 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:54.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:54 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:54.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:54 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:55.415 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:55 smithi152 ceph-mon[36876]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:55.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:55 smithi116 ceph-mon[29120]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:56.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:56 smithi152 ceph-mon[36876]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-04T15:58:56.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:56 smithi152 ceph-mon[36876]: from='client.? 172.21.15.152:0/2756113830' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d201b6a-6a91-4a7e-984e-e8d7f0289793"}]: dispatch 2024-03-04T15:58:56.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:56 smithi152 ceph-mon[36876]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d201b6a-6a91-4a7e-984e-e8d7f0289793"}]: dispatch 2024-03-04T15:58:56.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:56 smithi152 ceph-mon[36876]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d201b6a-6a91-4a7e-984e-e8d7f0289793"}]': finished 2024-03-04T15:58:56.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:56 smithi152 ceph-mon[36876]: osdmap e32: 5 total, 4 up, 5 in 2024-03-04T15:58:56.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:56 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:58:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:56 smithi116 ceph-mon[29120]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-04T15:58:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:56 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/2756113830' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d201b6a-6a91-4a7e-984e-e8d7f0289793"}]: dispatch 2024-03-04T15:58:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:56 smithi116 ceph-mon[29120]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d201b6a-6a91-4a7e-984e-e8d7f0289793"}]: dispatch 2024-03-04T15:58:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:56 smithi116 ceph-mon[29120]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d201b6a-6a91-4a7e-984e-e8d7f0289793"}]': finished 2024-03-04T15:58:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:56 smithi116 ceph-mon[29120]: osdmap e32: 5 total, 4 up, 5 in 2024-03-04T15:58:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:56 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:58:57.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:57 smithi152 ceph-mon[36876]: from='client.? 172.21.15.152:0/3140915263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:57.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:57 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/3140915263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:58.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:58:58 smithi152 ceph-mon[36876]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:58:58.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:58:58 smithi116 ceph-mon[29120]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:00.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:00 smithi152 ceph-mon[36876]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:00.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:00 smithi116 ceph-mon[29120]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:02.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:02 smithi152 ceph-mon[36876]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:02.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:02 smithi116 ceph-mon[29120]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:04.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:04 smithi152 ceph-mon[36876]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:04.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:04 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T15:59:04.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:04 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:04.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:04 smithi152 ceph-mon[36876]: Deploying daemon osd.4 on smithi152 2024-03-04T15:59:04.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:04 smithi116 ceph-mon[29120]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:04.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:04 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T15:59:04.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:04 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:04.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:04 smithi116 ceph-mon[29120]: Deploying daemon osd.4 on smithi152 2024-03-04T15:59:06.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:06 smithi152 ceph-mon[36876]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:06.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:06 smithi116 ceph-mon[29120]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:59:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:59:08.435 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:08 smithi152 ceph-mon[36876]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:08 smithi116 ceph-mon[29120]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:09.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:09 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:09.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:09 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:09.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:09 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:09.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:09 smithi116 ceph-mon[29120]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:10.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:09 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:10.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:09 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:10.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:09 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:10.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:09 smithi152 ceph-mon[36876]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:12.347 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:12 smithi152 ceph-mon[36876]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:12.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:12 smithi116 ceph-mon[29120]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:12.687 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.685+0000 7f8a3f7fe700 1 -- 172.21.15.152:0/2455193757 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8a48106710 con 0x7f8a3406c4e0 2024-03-04T15:59:12.688 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 4 on host 'smithi152' 2024-03-04T15:59:12.689 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 -- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f8a3406c4e0 msgr2=0x7f8a3406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:12.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f8a3406c4e0 0x7f8a3406e9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8a48101c20 tx=0x7f8a400073c0 comp rx=0 tx=0).stop 2024-03-04T15:59:12.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 -- 172.21.15.152:0/2455193757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a48101dc0 msgr2=0x7f8a48196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:12.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a48101dc0 0x7f8a48196950 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8a30006010 tx=0x7f8a3000be20 comp rx=0 tx=0).stop 2024-03-04T15:59:12.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 -- 172.21.15.152:0/2455193757 shutdown_connections 2024-03-04T15:59:12.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f8a3406c4e0 0x7f8a3406e9a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:12.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48100bc0 0x7f8a48196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:12.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 --2- 172.21.15.152:0/2455193757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a48101dc0 0x7f8a48196950 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:12.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 -- 172.21.15.152:0/2455193757 >> 172.21.15.152:0/2455193757 conn(0x7f8a480fc140 msgr2=0x7f8a48104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:12.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 -- 172.21.15.152:0/2455193757 shutdown_connections 2024-03-04T15:59:12.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:12.688+0000 7f8a4c975700 1 -- 172.21.15.152:0/2455193757 wait complete. 2024-03-04T15:59:13.300 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:13 smithi152 ceph-mon[36876]: from='osd.4 [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T15:59:13.300 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:13 smithi152 ceph-mon[36876]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T15:59:13.300 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:13 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:13.300 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:13 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:13.300 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:13 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:13.300 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:13 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:13.321 DEBUG:teuthology.orchestra.run.smithi152:osd.4> sudo journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.4.service 2024-03-04T15:59:13.334 INFO:tasks.cephadm:Deploying osd.5 on smithi152 with /dev/vg_nvme/lv_2... 2024-03-04T15:59:13.334 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T15:59:13.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:13 smithi116 ceph-mon[29120]: from='osd.4 [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T15:59:13.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:13 smithi116 ceph-mon[29120]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T15:59:13.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:13 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:13.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:13 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:13.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:13 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:13.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:13 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:14.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:14 smithi116 ceph-mon[29120]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:14 smithi116 ceph-mon[29120]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T15:59:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:14 smithi116 ceph-mon[29120]: osdmap e33: 5 total, 4 up, 5 in 2024-03-04T15:59:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:14 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:14 smithi116 ceph-mon[29120]: from='osd.4 [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:59:14.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:14 smithi116 ceph-mon[29120]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:59:14.485 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:14 smithi152 ceph-mon[36876]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:14.485 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:14 smithi152 ceph-mon[36876]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T15:59:14.485 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:14 smithi152 ceph-mon[36876]: osdmap e33: 5 total, 4 up, 5 in 2024-03-04T15:59:14.485 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:14 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:14.485 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:14 smithi152 ceph-mon[36876]: from='osd.4 [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:59:14.485 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:14 smithi152 ceph-mon[36876]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:59:14.486 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 15:59:14 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[49800]: 2024-03-04T15:59:14.151+0000 7f648ce13700 -1 osd.4 0 waiting for initial osdmap 2024-03-04T15:59:14.486 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 15:59:14 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[49800]: 2024-03-04T15:59:14.158+0000 7f6488407700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:59:14.799 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi152/config 2024-03-04T15:59:15.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:15 smithi152 ceph-mon[36876]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T15:59:15.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:15 smithi152 ceph-mon[36876]: osdmap e34: 5 total, 4 up, 5 in 2024-03-04T15:59:15.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:15 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:15.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:15 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:15 smithi116 ceph-mon[29120]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T15:59:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:15 smithi116 ceph-mon[29120]: osdmap e34: 5 total, 4 up, 5 in 2024-03-04T15:59:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: purged_snaps scrub starts 2024-03-04T15:59:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: purged_snaps scrub ok 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: osd.4 [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947] boot 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: osdmap e35: 5 total, 5 up, 5 in 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:16 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: purged_snaps scrub starts 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: purged_snaps scrub ok 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: osd.4 [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947] boot 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: osdmap e35: 5 total, 5 up, 5 in 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:16 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:17.334 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T15:59:17.334 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T15:59:17.334 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-04T15:59:17.335 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-04T15:59:17.335 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114962 s, 912 MB/s 2024-03-04T15:59:17.335 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-04T15:59:17.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:17 smithi116 ceph-mon[29120]: Detected new or changed devices on smithi152 2024-03-04T15:59:17.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:17 smithi116 ceph-mon[29120]: Adjusting osd_memory_target on smithi152 to 7456M 2024-03-04T15:59:17.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:17 smithi116 ceph-mon[29120]: osdmap e36: 5 total, 5 up, 5 in 2024-03-04T15:59:17.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:17.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:17.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:17.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:17.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:17 smithi152 ceph-mon[36876]: Detected new or changed devices on smithi152 2024-03-04T15:59:17.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:17 smithi152 ceph-mon[36876]: Adjusting osd_memory_target on smithi152 to 7456M 2024-03-04T15:59:17.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:17 smithi152 ceph-mon[36876]: osdmap e36: 5 total, 5 up, 5 in 2024-03-04T15:59:17.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:17.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:17.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:17.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:17.902 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_2 2024-03-04T15:59:18.107 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi152/config 2024-03-04T15:59:18.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:18 smithi116 ceph-mon[29120]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:18.533 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:18 smithi152 ceph-mon[36876]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:19.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.610+0000 7f7c752bc700 1 -- 172.21.15.152:0/2415177930 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c70101dc0 msgr2=0x7f7c70102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:19.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.610+0000 7f7c752bc700 1 --2- 172.21.15.152:0/2415177930 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c70101dc0 0x7f7c70102220 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009910 tx=0x7f7c64009c20 comp rx=0 tx=0).stop 2024-03-04T15:59:19.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 -- 172.21.15.152:0/2415177930 shutdown_connections 2024-03-04T15:59:19.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 --2- 172.21.15.152:0/2415177930 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c70101dc0 0x7f7c70102220 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:19.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 --2- 172.21.15.152:0/2415177930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c70100bc0 0x7f7c70100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:19.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 -- 172.21.15.152:0/2415177930 >> 172.21.15.152:0/2415177930 conn(0x7f7c700fc140 msgr2=0x7f7c700fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:19.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 -- 172.21.15.152:0/2415177930 shutdown_connections 2024-03-04T15:59:19.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 -- 172.21.15.152:0/2415177930 wait complete. 2024-03-04T15:59:19.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 Processor -- start 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 -- start start 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c70100bc0 0x7f7c70074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c70101dc0 0x7f7c70073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c700736c0 con 0x7f7c70100bc0 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.611+0000 7f7c752bc700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70073800 con 0x7f7c70101dc0 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c6e7fc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c70101dc0 0x7f7c70073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c6e7fc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c70101dc0 0x7f7c70073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:36070/0 (socket says 172.21.15.152:36070) 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c6e7fc700 1 -- 172.21.15.152:0/1431826466 learned_addr learned my addr 172.21.15.152:0/1431826466 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c6effd700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c70100bc0 0x7f7c70074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c6e7fc700 1 -- 172.21.15.152:0/1431826466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c70100bc0 msgr2=0x7f7c70074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c6e7fc700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c70100bc0 0x7f7c70074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c6e7fc700 1 -- 172.21.15.152:0/1431826466 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c640095f0 con 0x7f7c70101dc0 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c6e7fc700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c70101dc0 0x7f7c70073180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7c64006010 tx=0x7f7c6400bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:19.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c57fff700 1 -- 172.21.15.152:0/1431826466 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c6401d070 con 0x7f7c70101dc0 2024-03-04T15:59:19.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c752bc700 1 -- 172.21.15.152:0/1431826466 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c70073a80 con 0x7f7c70101dc0 2024-03-04T15:59:19.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c57fff700 1 -- 172.21.15.152:0/1431826466 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7c6400f460 con 0x7f7c70101dc0 2024-03-04T15:59:19.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c57fff700 1 -- 172.21.15.152:0/1431826466 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c64021640 con 0x7f7c70101dc0 2024-03-04T15:59:19.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.612+0000 7f7c752bc700 1 -- 172.21.15.152:0/1431826466 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c70073f70 con 0x7f7c70101dc0 2024-03-04T15:59:19.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.613+0000 7f7c752bc700 1 -- 172.21.15.152:0/1431826466 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c701097c0 con 0x7f7c70101dc0 2024-03-04T15:59:19.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.614+0000 7f7c57fff700 1 -- 172.21.15.152:0/1431826466 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7c6400fab0 con 0x7f7c70101dc0 2024-03-04T15:59:19.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.614+0000 7f7c57fff700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7c5806c4e0 0x7f7c5806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:19.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.614+0000 7f7c57fff700 1 -- 172.21.15.152:0/1431826466 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f7c6408c700 con 0x7f7c70101dc0 2024-03-04T15:59:19.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.614+0000 7f7c6effd700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7c5806c4e0 0x7f7c5806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:19.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.615+0000 7f7c6effd700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7c5806c4e0 0x7f7c5806e9a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c70101c20 tx=0x7f7c60009190 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:19.620 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.619+0000 7f7c57fff700 1 -- 172.21.15.152:0/1431826466 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7c6405ae60 con 0x7f7c70101dc0 2024-03-04T15:59:19.816 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:19.815+0000 7f7c752bc700 1 -- 172.21.15.152:0/1431826466 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c70106710 con 0x7f7c5806c4e0 2024-03-04T15:59:20.518 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:20 smithi152 ceph-mon[36876]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:20.518 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:20.518 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:20.518 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:20 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:20.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:20 smithi116 ceph-mon[29120]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:20.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:20.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:20.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:20 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:21.528 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:21 smithi152 ceph-mon[36876]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:21.656 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:21 smithi116 ceph-mon[29120]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:22.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:22 smithi152 ceph-mon[36876]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:22.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:59:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:22 smithi116 ceph-mon[29120]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:59:23.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:23 smithi152 ceph-mon[36876]: from='client.? 172.21.15.152:0/3857137684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c06ef0f5-6894-4e01-a693-bb129d248fec"}]: dispatch 2024-03-04T15:59:23.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:23 smithi152 ceph-mon[36876]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c06ef0f5-6894-4e01-a693-bb129d248fec"}]: dispatch 2024-03-04T15:59:23.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:23 smithi152 ceph-mon[36876]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c06ef0f5-6894-4e01-a693-bb129d248fec"}]': finished 2024-03-04T15:59:23.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:23 smithi152 ceph-mon[36876]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T15:59:23.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:23 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:23.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:23 smithi152 ceph-mon[36876]: from='client.? 172.21.15.152:0/4210436745' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:23 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/3857137684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c06ef0f5-6894-4e01-a693-bb129d248fec"}]: dispatch 2024-03-04T15:59:23.768 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:23 smithi116 ceph-mon[29120]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c06ef0f5-6894-4e01-a693-bb129d248fec"}]: dispatch 2024-03-04T15:59:23.768 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:23 smithi116 ceph-mon[29120]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c06ef0f5-6894-4e01-a693-bb129d248fec"}]': finished 2024-03-04T15:59:23.768 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:23 smithi116 ceph-mon[29120]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T15:59:23.768 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:23 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:23.768 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:23 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/4210436745' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:24.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:24 smithi152 ceph-mon[36876]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:24.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:24 smithi116 ceph-mon[29120]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:26 smithi152 ceph-mon[36876]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:26.694 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:26 smithi116 ceph-mon[29120]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:28.553 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:28 smithi152 ceph-mon[36876]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:28.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:28 smithi116 ceph-mon[29120]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:30.452 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:30 smithi152 ceph-mon[36876]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:30.453 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T15:59:30.453 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:30.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:30 smithi116 ceph-mon[29120]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:30.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T15:59:30.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:31.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:31 smithi152 ceph-mon[36876]: Deploying daemon osd.5 on smithi152 2024-03-04T15:59:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:31 smithi116 ceph-mon[29120]: Deploying daemon osd.5 on smithi152 2024-03-04T15:59:32.593 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:32 smithi152 ceph-mon[36876]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:32 smithi116 ceph-mon[29120]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:34 smithi152 ceph-mon[36876]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:34.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:34 smithi116 ceph-mon[29120]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:36.273 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:36 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:36.273 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:36 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:36.273 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:36 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:36.274 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:36 smithi152 ceph-mon[36876]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:36.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:36 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:36.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:36 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:36.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:36 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:36.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:36 smithi116 ceph-mon[29120]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:37.429 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:37 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:59:37.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:37 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:59:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:38 smithi116 ceph-mon[29120]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:38.521 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:38 smithi152 ceph-mon[36876]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:38.886 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.884+0000 7f7c57fff700 1 -- 172.21.15.152:0/1431826466 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7c70106710 con 0x7f7c5806c4e0 2024-03-04T15:59:38.886 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 5 on host 'smithi152' 2024-03-04T15:59:38.889 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 -- 172.21.15.152:0/1431826466 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7c5806c4e0 msgr2=0x7f7c5806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:38.889 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7c5806c4e0 0x7f7c5806e9a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c70101c20 tx=0x7f7c60009190 comp rx=0 tx=0).stop 2024-03-04T15:59:38.889 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 -- 172.21.15.152:0/1431826466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c70101dc0 msgr2=0x7f7c70073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:38.889 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c70101dc0 0x7f7c70073180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7c64006010 tx=0x7f7c6400bd70 comp rx=0 tx=0).stop 2024-03-04T15:59:38.889 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 -- 172.21.15.152:0/1431826466 shutdown_connections 2024-03-04T15:59:38.889 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7c5806c4e0 0x7f7c5806e9a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:38.889 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c70100bc0 0x7f7c70074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:38.889 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 --2- 172.21.15.152:0/1431826466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c70101dc0 0x7f7c70073180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:38.889 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 -- 172.21.15.152:0/1431826466 >> 172.21.15.152:0/1431826466 conn(0x7f7c700fc140 msgr2=0x7f7c70104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:38.889 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 -- 172.21.15.152:0/1431826466 shutdown_connections 2024-03-04T15:59:38.890 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:38.887+0000 7f7c752bc700 1 -- 172.21.15.152:0/1431826466 wait complete. 2024-03-04T15:59:39.489 DEBUG:teuthology.orchestra.run.smithi152:osd.5> sudo journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.5.service 2024-03-04T15:59:39.491 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-04T15:59:39.492 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-04T15:59:39.720 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:59:40.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:39 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:40.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:39 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:40.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:39 smithi152 ceph-mon[36876]: from='osd.5 [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T15:59:40.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:39 smithi152 ceph-mon[36876]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T15:59:40.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:39 smithi152 ceph-mon[36876]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:40.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:39 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:40.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:39 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:40.208 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:40.208 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:40.208 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:39 smithi116 ceph-mon[29120]: from='osd.5 [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T15:59:40.208 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:39 smithi116 ceph-mon[29120]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T15:59:40.209 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:39 smithi116 ceph-mon[29120]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:40.209 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:40.209 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:39 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:41.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:40 smithi152 ceph-mon[36876]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T15:59:41.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:40 smithi152 ceph-mon[36876]: osdmap e38: 6 total, 5 up, 6 in 2024-03-04T15:59:41.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:40 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:41.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:40 smithi152 ceph-mon[36876]: from='osd.5 [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:59:41.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:40 smithi152 ceph-mon[36876]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:59:41.109 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 15:59:40 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[56761]: 2024-03-04T15:59:40.853+0000 7fbb6e817700 -1 osd.5 0 waiting for initial osdmap 2024-03-04T15:59:41.109 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 15:59:40 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[56761]: 2024-03-04T15:59:40.860+0000 7fbb66e05700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:59:41.189 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:40 smithi116 ceph-mon[29120]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T15:59:41.189 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:40 smithi116 ceph-mon[29120]: osdmap e38: 6 total, 5 up, 6 in 2024-03-04T15:59:41.189 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:40 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:41.189 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:40 smithi116 ceph-mon[29120]: from='osd.5 [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:59:41.189 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:40 smithi116 ceph-mon[29120]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T15:59:41.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.787+0000 7efc0d393700 1 -- 172.21.15.116:0/1466433240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc080ffac0 msgr2=0x7efc080ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:41.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.787+0000 7efc0d393700 1 --2- 172.21.15.116:0/1466433240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc080ffac0 0x7efc080ffee0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009910 tx=0x7efbfc009c20 comp rx=0 tx=0).stop 2024-03-04T15:59:41.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.787+0000 7efc0d393700 1 -- 172.21.15.116:0/1466433240 shutdown_connections 2024-03-04T15:59:41.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.787+0000 7efc0d393700 1 --2- 172.21.15.116:0/1466433240 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc08100cc0 0x7efc08101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:41.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.787+0000 7efc0d393700 1 --2- 172.21.15.116:0/1466433240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc080ffac0 0x7efc080ffee0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:41.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.787+0000 7efc0d393700 1 -- 172.21.15.116:0/1466433240 >> 172.21.15.116:0/1466433240 conn(0x7efc080fb060 msgr2=0x7efc080fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:41.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.787+0000 7efc0d393700 1 -- 172.21.15.116:0/1466433240 shutdown_connections 2024-03-04T15:59:41.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.787+0000 7efc0d393700 1 -- 172.21.15.116:0/1466433240 wait complete. 2024-03-04T15:59:41.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.788+0000 7efc0d393700 1 Processor -- start 2024-03-04T15:59:41.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.788+0000 7efc0d393700 1 -- start start 2024-03-04T15:59:41.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.788+0000 7efc0d393700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc080ffac0 0x7efc08194160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:41.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.788+0000 7efc0d393700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc08100cc0 0x7efc081946a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.788+0000 7efc0d393700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08194cc0 con 0x7efc080ffac0 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.788+0000 7efc0d393700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08194e00 con 0x7efc08100cc0 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.788+0000 7efc06ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc080ffac0 0x7efc08194160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.788+0000 7efc06ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc080ffac0 0x7efc08194160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56146/0 (socket says 172.21.15.116:56146) 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.788+0000 7efc06ffd700 1 -- 172.21.15.116:0/3121639484 learned_addr learned my addr 172.21.15.116:0/3121639484 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.788+0000 7efc067fc700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc08100cc0 0x7efc081946a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.789+0000 7efc06ffd700 1 -- 172.21.15.116:0/3121639484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc08100cc0 msgr2=0x7efc081946a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.789+0000 7efc06ffd700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc08100cc0 0x7efc081946a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.789+0000 7efc06ffd700 1 -- 172.21.15.116:0/3121639484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc0095f0 con 0x7efc080ffac0 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.789+0000 7efc06ffd700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc080ffac0 0x7efc08194160 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efbfc004900 tx=0x7efbfc004930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.789+0000 7efbeffff700 1 -- 172.21.15.116:0/3121639484 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbfc004020 con 0x7efc080ffac0 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.789+0000 7efc0d393700 1 -- 172.21.15.116:0/3121639484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc08199850 con 0x7efc080ffac0 2024-03-04T15:59:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.789+0000 7efbeffff700 1 -- 172.21.15.116:0/3121639484 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efbfc003680 con 0x7efc080ffac0 2024-03-04T15:59:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.789+0000 7efbeffff700 1 -- 172.21.15.116:0/3121639484 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbfc01e660 con 0x7efc080ffac0 2024-03-04T15:59:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.789+0000 7efc0d393700 1 -- 172.21.15.116:0/3121639484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc08199e20 con 0x7efc080ffac0 2024-03-04T15:59:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.790+0000 7efc0d393700 1 -- 172.21.15.116:0/3121639484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc08107510 con 0x7efc080ffac0 2024-03-04T15:59:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.791+0000 7efbeffff700 1 -- 172.21.15.116:0/3121639484 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7efbfc004180 con 0x7efc080ffac0 2024-03-04T15:59:41.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.791+0000 7efbeffff700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7efbf006c4e0 0x7efbf006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:41.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.791+0000 7efbeffff700 1 -- 172.21.15.116:0/3121639484 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4358+0+0 (secure 0 0 0) 0x7efbfc08e9a0 con 0x7efc080ffac0 2024-03-04T15:59:41.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.791+0000 7efc067fc700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7efbf006c4e0 0x7efbf006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:41.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.791+0000 7efc067fc700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7efbf006c4e0 0x7efbf006e9a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009c60 tx=0x7efbf8009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:41.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.796+0000 7efbeffff700 1 -- 172.21.15.116:0/3121639484 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7efbfc05d060 con 0x7efc080ffac0 2024-03-04T15:59:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.993+0000 7efc0d393700 1 -- 172.21.15.116:0/3121639484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7efc0819a860 con 0x7efc080ffac0 2024-03-04T15:59:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.993+0000 7efbeffff700 1 -- 172.21.15.116:0/3121639484 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7efbfc05cbf0 con 0x7efc080ffac0 2024-03-04T15:59:41.995 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:59:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.995+0000 7efc0d393700 1 -- 172.21.15.116:0/3121639484 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7efbf006c4e0 msgr2=0x7efbf006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.995+0000 7efc0d393700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7efbf006c4e0 0x7efbf006e9a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009c60 tx=0x7efbf8009260 comp rx=0 tx=0).stop 2024-03-04T15:59:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.995+0000 7efc0d393700 1 -- 172.21.15.116:0/3121639484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc080ffac0 msgr2=0x7efc08194160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.995+0000 7efc0d393700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc080ffac0 0x7efc08194160 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efbfc004900 tx=0x7efbfc004930 comp rx=0 tx=0).stop 2024-03-04T15:59:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.995+0000 7efc0d393700 1 -- 172.21.15.116:0/3121639484 shutdown_connections 2024-03-04T15:59:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.996+0000 7efc0d393700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7efbf006c4e0 0x7efbf006e9a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.996+0000 7efc0d393700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc080ffac0 0x7efc08194160 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.996+0000 7efc0d393700 1 --2- 172.21.15.116:0/3121639484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc08100cc0 0x7efc081946a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.996+0000 7efc0d393700 1 -- 172.21.15.116:0/3121639484 >> 172.21.15.116:0/3121639484 conn(0x7efc080fb060 msgr2=0x7efc08103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.996+0000 7efc0d393700 1 -- 172.21.15.116:0/3121639484 shutdown_connections 2024-03-04T15:59:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:41.996+0000 7efc0d393700 1 -- 172.21.15.116:0/3121639484 wait complete. 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:41 smithi152 ceph-mon[36876]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:41 smithi152 ceph-mon[36876]: osdmap e39: 6 total, 5 up, 6 in 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:41 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:41 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:42.110 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:41 smithi152 ceph-mon[36876]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:42.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:41 smithi116 ceph-mon[29120]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T15:59:42.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:41 smithi116 ceph-mon[29120]: osdmap e39: 6 total, 5 up, 6 in 2024-03-04T15:59:42.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:41 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:42.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:41 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:42.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:41 smithi116 ceph-mon[29120]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:42.726 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1709567981,"num_in_osds":6,"osd_in_since":1709567962,"num_remapped_pgs":0} 2024-03-04T15:59:42.727 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T15:59:42.944 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:59:42.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:42 smithi116 ceph-mon[29120]: purged_snaps scrub starts 2024-03-04T15:59:42.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:42 smithi116 ceph-mon[29120]: purged_snaps scrub ok 2024-03-04T15:59:42.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:42 smithi116 ceph-mon[29120]: osd.5 [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408] boot 2024-03-04T15:59:42.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:42 smithi116 ceph-mon[29120]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T15:59:42.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:42 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:42.977 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:42 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3121639484' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:59:43.032 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:42 smithi152 ceph-mon[36876]: purged_snaps scrub starts 2024-03-04T15:59:43.033 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:42 smithi152 ceph-mon[36876]: purged_snaps scrub ok 2024-03-04T15:59:43.033 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:42 smithi152 ceph-mon[36876]: osd.5 [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408] boot 2024-03-04T15:59:43.033 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:42 smithi152 ceph-mon[36876]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T15:59:43.033 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:42 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:43.033 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:42 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3121639484' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:59:44.196 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T15:59:44.196 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: Detected new or changed devices on smithi152 2024-03-04T15:59:44.196 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:44.196 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:44.196 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:44.196 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:44.196 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:44.196 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:44.196 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: Adjusting osd_memory_target on smithi152 to 4970M 2024-03-04T15:59:44.197 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:44.197 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:44.197 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:44.197 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:44.197 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:44.197 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:44.197 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:44.197 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:44.361 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T15:59:44.361 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: Detected new or changed devices on smithi152 2024-03-04T15:59:44.361 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:44.361 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:44.361 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:44.361 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: Adjusting osd_memory_target on smithi152 to 4970M 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:44.362 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T15:59:44.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.779+0000 7ff8e0f86700 1 -- 172.21.15.116:0/190867764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8dc105a40 msgr2=0x7ff8dc105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:44.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.779+0000 7ff8e0f86700 1 --2- 172.21.15.116:0/190867764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8dc105a40 0x7ff8dc105e20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400b0a0 tx=0x7ff8d400b3b0 comp rx=0 tx=0).stop 2024-03-04T15:59:44.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.779+0000 7ff8e0f86700 1 -- 172.21.15.116:0/190867764 shutdown_connections 2024-03-04T15:59:44.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.779+0000 7ff8e0f86700 1 --2- 172.21.15.116:0/190867764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8dc0ffa40 0x7ff8dc0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.779+0000 7ff8e0f86700 1 --2- 172.21.15.116:0/190867764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8dc105a40 0x7ff8dc105e20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.779+0000 7ff8e0f86700 1 -- 172.21.15.116:0/190867764 >> 172.21.15.116:0/190867764 conn(0x7ff8dc0fb550 msgr2=0x7ff8dc0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.779+0000 7ff8e0f86700 1 -- 172.21.15.116:0/190867764 shutdown_connections 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.779+0000 7ff8e0f86700 1 -- 172.21.15.116:0/190867764 wait complete. 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8e0f86700 1 Processor -- start 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8e0f86700 1 -- start start 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8e0f86700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8dc0ffa40 0x7ff8dc1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8e0f86700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8dc105a40 0x7ff8dc197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8e0f86700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc198140 con 0x7ff8dc105a40 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8e0f86700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc198280 con 0x7ff8dc0ffa40 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8d9d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8dc105a40 0x7ff8dc197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8d9d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8dc105a40 0x7ff8dc197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33274/0 (socket says 172.21.15.116:33274) 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8d9d9b700 1 -- 172.21.15.116:0/1150705395 learned_addr learned my addr 172.21.15.116:0/1150705395 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8da59c700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8dc0ffa40 0x7ff8dc1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8d9d9b700 1 -- 172.21.15.116:0/1150705395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8dc0ffa40 msgr2=0x7ff8dc1975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8d9d9b700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8dc0ffa40 0x7ff8dc1975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.780+0000 7ff8d9d9b700 1 -- 172.21.15.116:0/1150705395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d4009d00 con 0x7ff8dc105a40 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.781+0000 7ff8d9d9b700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8dc105a40 0x7ff8dc197b20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400c8a0 tx=0x7ff8c400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.781+0000 7ff8d37fe700 1 -- 172.21.15.116:0/1150705395 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c400cea0 con 0x7ff8dc105a40 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.781+0000 7ff8e0f86700 1 -- 172.21.15.116:0/1150705395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8dc1a0650 con 0x7ff8dc105a40 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.781+0000 7ff8d37fe700 1 -- 172.21.15.116:0/1150705395 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff8c4004830 con 0x7ff8dc105a40 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.781+0000 7ff8d37fe700 1 -- 172.21.15.116:0/1150705395 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c40055b0 con 0x7ff8dc105a40 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.781+0000 7ff8e0f86700 1 -- 172.21.15.116:0/1150705395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8dc1a0b10 con 0x7ff8dc105a40 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.782+0000 7ff8e0f86700 1 -- 172.21.15.116:0/1150705395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8dc108040 con 0x7ff8dc105a40 2024-03-04T15:59:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.782+0000 7ff8d37fe700 1 -- 172.21.15.116:0/1150705395 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff8c4007ad0 con 0x7ff8dc105a40 2024-03-04T15:59:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.783+0000 7ff8d37fe700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff8c806c4e0 0x7ff8c806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.783+0000 7ff8d37fe700 1 -- 172.21.15.116:0/1150705395 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff8c408b240 con 0x7ff8dc105a40 2024-03-04T15:59:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.783+0000 7ff8da59c700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff8c806c4e0 0x7ff8c806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.783+0000 7ff8da59c700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff8c806c4e0 0x7ff8c806e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400b830 tx=0x7ff8d40097c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.787+0000 7ff8d37fe700 1 -- 172.21.15.116:0/1150705395 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff8c4055d60 con 0x7ff8dc105a40 2024-03-04T15:59:44.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.974+0000 7ff8e0f86700 1 -- 172.21.15.116:0/1150705395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8dc04ea90 con 0x7ff8dc105a40 2024-03-04T15:59:44.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.975+0000 7ff8d37fe700 1 -- 172.21.15.116:0/1150705395 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11395 (secure 0 0 0) 0x7ff8c4059380 con 0x7ff8dc105a40 2024-03-04T15:59:44.977 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:59:44.977 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:53:48.465240+0000","modified":"2024-03-04T15:59:42.855615+0000","last_up_change":"2024-03-04T15:59:41.850845+0000","last_in_change":"2024-03-04T15:59:22.307720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:58:27.372335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d12966f-954e-4a09-bbbd-5b0f6a04d039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6803","nonce":1056902067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6805","nonce":1056902067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6809","nonce":1056902067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6807","nonce":1056902067}]},"public_addr":"172.21.15.116:6803/1056902067","cluster_addr":"172.21.15.116:6805/1056902067","heartbeat_back_addr":"172.21.15.116:6809/1056902067","heartbeat_front_addr":"172.21.15.116:6807/1056902067","state":["exists","up"]},{"osd":1,"uuid":"f990720f-a19d-48c1-9545-81e511c2690b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6811","nonce":2368425008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6813","nonce":2368425008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6817","nonce":2368425008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6815","nonce":2368425008}]},"public_addr":"172.21.15.116:6811/2368425008","cluster_addr":"172.21.15.116:6813/2368425008","heartbeat_back_addr":"172.21.15.116:6817/2368425008","heartbeat_front_addr":"172.21.15.116:6815/2368425008","state":["exists","up"]},{"osd":2,"uuid":"fe3132c8-6b15-4a11-a884-7c6a09d42955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6819","nonce":1282157941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6821","nonce":1282157941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6825","nonce":1282157941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6823","nonce":1282157941}]},"public_addr":"172.21.15.116:6819/1282157941","cluster_addr":"172.21.15.116:6821/1282157941","heartbeat_back_addr":"172.21.15.116:6825/1282157941","heartbeat_front_addr":"172.21.15.116:6823/1282157941","state":["exists","up"]},{"osd":3,"uuid":"15930a9b-f721-4c30-9c5b-bef5ef69777c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6801","nonce":2457942679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6803","nonce":2457942679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6807","nonce":2457942679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6805","nonce":2457942679}]},"public_addr":"172.21.15.152:6801/2457942679","cluster_addr":"172.21.15.152:6803/2457942679","heartbeat_back_addr":"172.21.15.152:6807/2457942679","heartbeat_front_addr":"172.21.15.152:6805/2457942679","state":["exists","up"]},{"osd":4,"uuid":"8d201b6a-6a91-4a7e-984e-e8d7f0289793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6809","nonce":3048670947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6811","nonce":3048670947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6815","nonce":3048670947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6813","nonce":3048670947}]},"public_addr":"172.21.15.152:6809/3048670947","cluster_addr":"172.21.15.152:6811/3048670947","heartbeat_back_addr":"172.21.15.152:6815/3048670947","heartbeat_front_addr":"172.21.15.152:6813/3048670947","state":["exists","up"]},{"osd":5,"uuid":"c06ef0f5-6894-4e01-a693-bb129d248fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6817","nonce":1864443408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6819","nonce":1864443408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6823","nonce":1864443408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6821","nonce":1864443408}]},"public_addr":"172.21.15.152:6817/1864443408","cluster_addr":"172.21.15.152:6819/1864443408","heartbeat_back_addr":"172.21.15.152:6823/1864443408","heartbeat_front_addr":"172.21.15.152:6821/1864443408","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:24.230215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:52.709103+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:24.072974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:47.684390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:13.571654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:39.874966+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/821201733":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/3670178561":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6800/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/831649128":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3287532122":"2024-03-05T15:54:53.931186+0000","172.21.15.116:6801/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/355050440":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/976694871":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/4142805454":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/385581892":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6801/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6800/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6801/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6800/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3323890559":"2024-03-05T15:54:21.166941+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T15:59:44.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 -- 172.21.15.116:0/1150705395 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff8c806c4e0 msgr2=0x7ff8c806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:44.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff8c806c4e0 0x7ff8c806e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400b830 tx=0x7ff8d40097c0 comp rx=0 tx=0).stop 2024-03-04T15:59:44.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 -- 172.21.15.116:0/1150705395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8dc105a40 msgr2=0x7ff8dc197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:44.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8dc105a40 0x7ff8dc197b20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400c8a0 tx=0x7ff8c400cc60 comp rx=0 tx=0).stop 2024-03-04T15:59:44.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 -- 172.21.15.116:0/1150705395 shutdown_connections 2024-03-04T15:59:44.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8dc0ffa40 0x7ff8dc1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:44.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff8c806c4e0 0x7ff8c806e9a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:44.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 --2- 172.21.15.116:0/1150705395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8dc105a40 0x7ff8dc197b20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:44.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 -- 172.21.15.116:0/1150705395 >> 172.21.15.116:0/1150705395 conn(0x7ff8dc0fb550 msgr2=0x7ff8dc0fbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:44.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 -- 172.21.15.116:0/1150705395 shutdown_connections 2024-03-04T15:59:44.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:44.977+0000 7ff8e0f86700 1 -- 172.21.15.116:0/1150705395 wait complete. 2024-03-04T15:59:45.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:45 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/1150705395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:59:45.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:45 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1150705395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:59:45.725 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T15:58:27.372335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T15:59:45.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-04T15:59:45.931 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:59:46.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:46 smithi152 ceph-mon[36876]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:46.423 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:46 smithi116 ceph-mon[29120]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:47.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.546+0000 7f0adf0b7700 1 -- 172.21.15.116:0/2286754427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8101d00 msgr2=0x7f0ad81020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:47.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.546+0000 7f0adf0b7700 1 --2- 172.21.15.116:0/2286754427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8101d00 0x7f0ad81020e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009910 tx=0x7f0acc009c20 comp rx=0 tx=0).stop 2024-03-04T15:59:47.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.546+0000 7f0adf0b7700 1 -- 172.21.15.116:0/2286754427 shutdown_connections 2024-03-04T15:59:47.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.546+0000 7f0adf0b7700 1 --2- 172.21.15.116:0/2286754427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ad8102620 0x7f0ad810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:47.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.546+0000 7f0adf0b7700 1 --2- 172.21.15.116:0/2286754427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8101d00 0x7f0ad81020e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:47.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.546+0000 7f0adf0b7700 1 -- 172.21.15.116:0/2286754427 >> 172.21.15.116:0/2286754427 conn(0x7f0ad80fb570 msgr2=0x7f0ad80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:47.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.546+0000 7f0adf0b7700 1 -- 172.21.15.116:0/2286754427 shutdown_connections 2024-03-04T15:59:47.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.546+0000 7f0adf0b7700 1 -- 172.21.15.116:0/2286754427 wait complete. 2024-03-04T15:59:47.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0adf0b7700 1 Processor -- start 2024-03-04T15:59:47.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0adf0b7700 1 -- start start 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0adf0b7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ad8101d00 0x7f0ad819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0adf0b7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8102620 0x7f0ad819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0adf0b7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad819d100 con 0x7f0ad8102620 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0adf0b7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad81965b0 con 0x7f0ad8101d00 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0ad7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8102620 0x7f0ad819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0ad7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8102620 0x7f0ad819ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33294/0 (socket says 172.21.15.116:33294) 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0ad7fff700 1 -- 172.21.15.116:0/655017592 learned_addr learned my addr 172.21.15.116:0/655017592 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0ad7fff700 1 -- 172.21.15.116:0/655017592 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ad8101d00 msgr2=0x7f0ad819c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0ad7fff700 1 --2- 172.21.15.116:0/655017592 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ad8101d00 0x7f0ad819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.547+0000 7f0ad7fff700 1 -- 172.21.15.116:0/655017592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc0095f0 con 0x7f0ad8102620 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.548+0000 7f0ad7fff700 1 --2- 172.21.15.116:0/655017592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8102620 0x7f0ad819ca70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400d6d0 tx=0x7f0ac400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.548+0000 7f0ad5ffb700 1 -- 172.21.15.116:0/655017592 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ac400f860 con 0x7f0ad8102620 2024-03-04T15:59:47.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.548+0000 7f0ad5ffb700 1 -- 172.21.15.116:0/655017592 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0ac400fe70 con 0x7f0ad8102620 2024-03-04T15:59:47.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.548+0000 7f0ad5ffb700 1 -- 172.21.15.116:0/655017592 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ac400e5e0 con 0x7f0ad8102620 2024-03-04T15:59:47.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.548+0000 7f0adf0b7700 1 -- 172.21.15.116:0/655017592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad8196890 con 0x7f0ad8102620 2024-03-04T15:59:47.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.548+0000 7f0adf0b7700 1 -- 172.21.15.116:0/655017592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad8196d90 con 0x7f0ad8102620 2024-03-04T15:59:47.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.549+0000 7f0adf0b7700 1 -- 172.21.15.116:0/655017592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad8108210 con 0x7f0ad8102620 2024-03-04T15:59:47.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.549+0000 7f0ad5ffb700 1 -- 172.21.15.116:0/655017592 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0ac4010460 con 0x7f0ad8102620 2024-03-04T15:59:47.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.550+0000 7f0ad5ffb700 1 --2- 172.21.15.116:0/655017592 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0ac806c4e0 0x7f0ac806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:47.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.550+0000 7f0ad5ffb700 1 -- 172.21.15.116:0/655017592 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0ac408b090 con 0x7f0ad8102620 2024-03-04T15:59:47.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.550+0000 7f0adce53700 1 --2- 172.21.15.116:0/655017592 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0ac806c4e0 0x7f0ac806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:47.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.550+0000 7f0adce53700 1 --2- 172.21.15.116:0/655017592 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0ac806c4e0 0x7f0ac806e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00b5c0 tx=0x7f0acc005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:47.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.554+0000 7f0ad5ffb700 1 -- 172.21.15.116:0/655017592 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0ac4059640 con 0x7f0ad8102620 2024-03-04T15:59:47.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.742+0000 7f0adf0b7700 1 -- 172.21.15.116:0/655017592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ad8197b70 con 0x7f0ad8102620 2024-03-04T15:59:47.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.743+0000 7f0ad5ffb700 1 -- 172.21.15.116:0/655017592 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ac40591d0 con 0x7f0ad8102620 2024-03-04T15:59:47.748 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T15:59:47.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 -- 172.21.15.116:0/655017592 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0ac806c4e0 msgr2=0x7f0ac806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:47.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 --2- 172.21.15.116:0/655017592 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0ac806c4e0 0x7f0ac806e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00b5c0 tx=0x7f0acc005960 comp rx=0 tx=0).stop 2024-03-04T15:59:47.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 -- 172.21.15.116:0/655017592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8102620 msgr2=0x7f0ad819ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:47.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 --2- 172.21.15.116:0/655017592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8102620 0x7f0ad819ca70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400d6d0 tx=0x7f0ac400da90 comp rx=0 tx=0).stop 2024-03-04T15:59:47.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 -- 172.21.15.116:0/655017592 shutdown_connections 2024-03-04T15:59:47.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 --2- 172.21.15.116:0/655017592 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ad8101d00 0x7f0ad819c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:47.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 --2- 172.21.15.116:0/655017592 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0ac806c4e0 0x7f0ac806e9a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:47.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 --2- 172.21.15.116:0/655017592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8102620 0x7f0ad819ca70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:47.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 -- 172.21.15.116:0/655017592 >> 172.21.15.116:0/655017592 conn(0x7f0ad80fb570 msgr2=0x7f0ad8105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:47.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 -- 172.21.15.116:0/655017592 shutdown_connections 2024-03-04T15:59:47.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:47.745+0000 7f0adf0b7700 1 -- 172.21.15.116:0/655017592 wait complete. 2024-03-04T15:59:48.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:48 smithi152 ceph-mon[36876]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:48.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:48 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/655017592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T15:59:48.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:48 smithi116 ceph-mon[29120]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:48.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:48 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/655017592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T15:59:48.509 INFO:tasks.cephadm:Setting up client nodes... 2024-03-04T15:59:48.509 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T15:59:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:59:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.334+0000 7f107aedd700 1 -- 172.21.15.116:0/3087995171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1074100cb0 msgr2=0x7f1074101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.334+0000 7f107aedd700 1 --2- 172.21.15.116:0/3087995171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1074100cb0 0x7f1074101110 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1060009910 tx=0x7f1060009c20 comp rx=0 tx=0).stop 2024-03-04T15:59:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.334+0000 7f107aedd700 1 -- 172.21.15.116:0/3087995171 shutdown_connections 2024-03-04T15:59:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.334+0000 7f107aedd700 1 --2- 172.21.15.116:0/3087995171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1074100cb0 0x7f1074101110 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.334+0000 7f107aedd700 1 --2- 172.21.15.116:0/3087995171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10740ffab0 0x7f10740ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.334+0000 7f107aedd700 1 -- 172.21.15.116:0/3087995171 >> 172.21.15.116:0/3087995171 conn(0x7f10740fb030 msgr2=0x7f10740fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.334+0000 7f107aedd700 1 -- 172.21.15.116:0/3087995171 shutdown_connections 2024-03-04T15:59:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.334+0000 7f107aedd700 1 -- 172.21.15.116:0/3087995171 wait complete. 2024-03-04T15:59:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f107aedd700 1 Processor -- start 2024-03-04T15:59:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f107aedd700 1 -- start start 2024-03-04T15:59:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f107aedd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10740ffab0 0x7f1074068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f107aedd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1074100cb0 0x7f107419b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f107aedd700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107419b710 con 0x7f1074100cb0 2024-03-04T15:59:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f107aedd700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107419b850 con 0x7f10740ffab0 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f1073fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1074100cb0 0x7f107419b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f1073fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1074100cb0 0x7f107419b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33316/0 (socket says 172.21.15.116:33316) 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f1073fff700 1 -- 172.21.15.116:0/161746648 learned_addr learned my addr 172.21.15.116:0/161746648 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f1073fff700 1 -- 172.21.15.116:0/161746648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10740ffab0 msgr2=0x7f1074068f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f1078c79700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10740ffab0 0x7f1074068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f1073fff700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10740ffab0 0x7f1074068f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f1073fff700 1 -- 172.21.15.116:0/161746648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10600095f0 con 0x7f1074100cb0 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.335+0000 7f1078c79700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10740ffab0 0x7f1074068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.336+0000 7f1073fff700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1074100cb0 0x7f107419b140 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1060006010 tx=0x7f106000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.336+0000 7f1071ffb700 1 -- 172.21.15.116:0/161746648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f106001d070 con 0x7f1074100cb0 2024-03-04T15:59:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.336+0000 7f107aedd700 1 -- 172.21.15.116:0/161746648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107419bad0 con 0x7f1074100cb0 2024-03-04T15:59:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.336+0000 7f1071ffb700 1 -- 172.21.15.116:0/161746648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f106000f460 con 0x7f1074100cb0 2024-03-04T15:59:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.336+0000 7f1071ffb700 1 -- 172.21.15.116:0/161746648 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1060021640 con 0x7f1074100cb0 2024-03-04T15:59:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.336+0000 7f107aedd700 1 -- 172.21.15.116:0/161746648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107419c040 con 0x7f1074100cb0 2024-03-04T15:59:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.337+0000 7f107aedd700 1 -- 172.21.15.116:0/161746648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10741086b0 con 0x7f1074100cb0 2024-03-04T15:59:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.337+0000 7f1071ffb700 1 -- 172.21.15.116:0/161746648 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1060004b20 con 0x7f1074100cb0 2024-03-04T15:59:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.338+0000 7f1071ffb700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106406c4e0 0x7f106406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.338+0000 7f1071ffb700 1 -- 172.21.15.116:0/161746648 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f106008c970 con 0x7f1074100cb0 2024-03-04T15:59:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.338+0000 7f1078c79700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106406c4e0 0x7f106406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.338+0000 7f1078c79700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106406c4e0 0x7f106406e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1074100b10 tx=0x7f1068008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:50.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.342+0000 7f1071ffb700 1 -- 172.21.15.116:0/161746648 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f106005af20 con 0x7f1074100cb0 2024-03-04T15:59:50.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:50 smithi116 ceph-mon[29120]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:50.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:50 smithi152 ceph-mon[36876]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.610+0000 7f107aedd700 1 -- 172.21.15.116:0/161746648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f107419c7e0 con 0x7f1074100cb0 2024-03-04T15:59:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.613+0000 7f1071ffb700 1 -- 172.21.15.116:0/161746648 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f106005aab0 con 0x7f1074100cb0 2024-03-04T15:59:50.615 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-03-04T15:59:50.615 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQD27+VlbF10JBAAOU7DQdtAHo2fGknCw7yW0g== 2024-03-04T15:59:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.615+0000 7f107aedd700 1 -- 172.21.15.116:0/161746648 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106406c4e0 msgr2=0x7f106406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.615+0000 7f107aedd700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106406c4e0 0x7f106406e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1074100b10 tx=0x7f1068008040 comp rx=0 tx=0).stop 2024-03-04T15:59:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.615+0000 7f107aedd700 1 -- 172.21.15.116:0/161746648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1074100cb0 msgr2=0x7f107419b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.615+0000 7f107aedd700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1074100cb0 0x7f107419b140 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1060006010 tx=0x7f106000be20 comp rx=0 tx=0).stop 2024-03-04T15:59:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.616+0000 7f107aedd700 1 -- 172.21.15.116:0/161746648 shutdown_connections 2024-03-04T15:59:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.616+0000 7f107aedd700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10740ffab0 0x7f1074068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.616+0000 7f107aedd700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106406c4e0 0x7f106406e9a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.616+0000 7f107aedd700 1 --2- 172.21.15.116:0/161746648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1074100cb0 0x7f107419b140 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.616+0000 7f107aedd700 1 -- 172.21.15.116:0/161746648 >> 172.21.15.116:0/161746648 conn(0x7f10740fb030 msgr2=0x7f1074103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.616+0000 7f107aedd700 1 -- 172.21.15.116:0/161746648 shutdown_connections 2024-03-04T15:59:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:50.616+0000 7f107aedd700 1 -- 172.21.15.116:0/161746648 wait complete. 2024-03-04T15:59:51.260 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T15:59:51.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-04T15:59:51.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-04T15:59:51.304 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T15:59:51.523 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi152/config 2024-03-04T15:59:51.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:51 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/161746648' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:59:51.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:51 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/161746648' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:59:51.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:51 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/161746648' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:59:51.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:51 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/161746648' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:59:52.530 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:52 smithi152 ceph-mon[36876]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:52.531 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:59:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:52 smithi116 ceph-mon[29120]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:59:53.051 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.050+0000 7f7152797700 1 -- 172.21.15.152:0/3165638785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f714c100c00 msgr2=0x7f714c101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:53.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.050+0000 7f7152797700 1 --2- 172.21.15.152:0/3165638785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f714c100c00 0x7f714c101020 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f713c009910 tx=0x7f713c009c20 comp rx=0 tx=0).stop 2024-03-04T15:59:53.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.050+0000 7f7152797700 1 -- 172.21.15.152:0/3165638785 shutdown_connections 2024-03-04T15:59:53.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.050+0000 7f7152797700 1 --2- 172.21.15.152:0/3165638785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f714c101e00 0x7f714c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:53.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.050+0000 7f7152797700 1 --2- 172.21.15.152:0/3165638785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f714c100c00 0x7f714c101020 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:53.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.050+0000 7f7152797700 1 -- 172.21.15.152:0/3165638785 >> 172.21.15.152:0/3165638785 conn(0x7f714c0fc180 msgr2=0x7f714c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:53.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.050+0000 7f7152797700 1 -- 172.21.15.152:0/3165638785 shutdown_connections 2024-03-04T15:59:53.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.050+0000 7f7152797700 1 -- 172.21.15.152:0/3165638785 wait complete. 2024-03-04T15:59:53.053 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.051+0000 7f7152797700 1 Processor -- start 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.051+0000 7f7152797700 1 -- start start 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.051+0000 7f7152797700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f714c100c00 0x7f714c196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.051+0000 7f7152797700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f714c101e00 0x7f714c1968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.051+0000 7f7152797700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c196ee0 con 0x7f714c101e00 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.051+0000 7f7152797700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c197020 con 0x7f714c100c00 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.051+0000 7f714bfff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f714c100c00 0x7f714c196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.051+0000 7f714bfff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f714c100c00 0x7f714c196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:55546/0 (socket says 172.21.15.152:55546) 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.051+0000 7f714bfff700 1 -- 172.21.15.152:0/1580674820 learned_addr learned my addr 172.21.15.152:0/1580674820 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.052+0000 7f714bfff700 1 -- 172.21.15.152:0/1580674820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f714c101e00 msgr2=0x7f714c1968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.052+0000 7f714bfff700 1 --2- 172.21.15.152:0/1580674820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f714c101e00 0x7f714c1968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.052+0000 7f714bfff700 1 -- 172.21.15.152:0/1580674820 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c0095f0 con 0x7f714c100c00 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.052+0000 7f714bfff700 1 --2- 172.21.15.152:0/1580674820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f714c100c00 0x7f714c196380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f713c00bda0 tx=0x7f713c00be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:53.054 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.052+0000 7f71497fa700 1 -- 172.21.15.152:0/1580674820 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f713c01d070 con 0x7f714c100c00 2024-03-04T15:59:53.055 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.052+0000 7f71497fa700 1 -- 172.21.15.152:0/1580674820 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f713c00f460 con 0x7f714c100c00 2024-03-04T15:59:53.055 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.052+0000 7f7152797700 1 -- 172.21.15.152:0/1580674820 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c19ba70 con 0x7f714c100c00 2024-03-04T15:59:53.055 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.052+0000 7f71497fa700 1 -- 172.21.15.152:0/1580674820 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f713c021640 con 0x7f714c100c00 2024-03-04T15:59:53.055 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.052+0000 7f7152797700 1 -- 172.21.15.152:0/1580674820 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c19bf60 con 0x7f714c100c00 2024-03-04T15:59:53.055 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.053+0000 7f7152797700 1 -- 172.21.15.152:0/1580674820 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c109760 con 0x7f714c100c00 2024-03-04T15:59:53.056 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.054+0000 7f71497fa700 1 -- 172.21.15.152:0/1580674820 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f713c00fa00 con 0x7f714c100c00 2024-03-04T15:59:53.056 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.054+0000 7f71497fa700 1 --2- 172.21.15.152:0/1580674820 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f713406c4e0 0x7f713406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:53.056 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.054+0000 7f71497fa700 1 -- 172.21.15.152:0/1580674820 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f713c08c800 con 0x7f714c100c00 2024-03-04T15:59:53.056 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.054+0000 7f714b7fe700 1 --2- 172.21.15.152:0/1580674820 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f713406c4e0 0x7f713406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:53.056 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.055+0000 7f714b7fe700 1 --2- 172.21.15.152:0/1580674820 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f713406c4e0 0x7f713406e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7140005950 tx=0x7f71400058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:53.060 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.059+0000 7f71497fa700 1 -- 172.21.15.152:0/1580674820 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f713c05adb0 con 0x7f714c100c00 2024-03-04T15:59:53.322 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.321+0000 7f7152797700 1 -- 172.21.15.152:0/1580674820 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f714c19c700 con 0x7f714c100c00 2024-03-04T15:59:53.327 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.326+0000 7f71497fa700 1 -- 172.21.15.152:0/1580674820 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f713c05a940 con 0x7f714c100c00 2024-03-04T15:59:53.327 INFO:teuthology.orchestra.run.smithi152.stdout:[client.1] 2024-03-04T15:59:53.327 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQD57+VlFuVCExAAel1pt5/mLZkz3Zrj/dZs3g== 2024-03-04T15:59:53.329 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 -- 172.21.15.152:0/1580674820 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f713406c4e0 msgr2=0x7f713406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:53.329 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 --2- 172.21.15.152:0/1580674820 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f713406c4e0 0x7f713406e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7140005950 tx=0x7f71400058e0 comp rx=0 tx=0).stop 2024-03-04T15:59:53.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 -- 172.21.15.152:0/1580674820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f714c100c00 msgr2=0x7f714c196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:53.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 --2- 172.21.15.152:0/1580674820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f714c100c00 0x7f714c196380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f713c00bda0 tx=0x7f713c00be80 comp rx=0 tx=0).stop 2024-03-04T15:59:53.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 -- 172.21.15.152:0/1580674820 shutdown_connections 2024-03-04T15:59:53.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 --2- 172.21.15.152:0/1580674820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f714c100c00 0x7f714c196380 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:53.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 --2- 172.21.15.152:0/1580674820 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f713406c4e0 0x7f713406e9a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:53.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 --2- 172.21.15.152:0/1580674820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f714c101e00 0x7f714c1968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:53.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 -- 172.21.15.152:0/1580674820 >> 172.21.15.152:0/1580674820 conn(0x7f714c0fc180 msgr2=0x7f714c105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:53.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 -- 172.21.15.152:0/1580674820 shutdown_connections 2024-03-04T15:59:53.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T15:59:53.328+0000 7f7152797700 1 -- 172.21.15.152:0/1580674820 wait complete. 2024-03-04T15:59:54.081 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T15:59:54.081 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-04T15:59:54.081 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-04T15:59:54.124 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-04T15:59:54.124 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-04T15:59:54.124 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph mgr dump --format=json 2024-03-04T15:59:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:59:54.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:54 smithi152 ceph-mon[36876]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:54.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:54 smithi152 ceph-mon[36876]: from='client.? 172.21.15.152:0/1580674820' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:59:54.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:54 smithi152 ceph-mon[36876]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:59:54.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:54 smithi152 ceph-mon[36876]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:59:54.462 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:54 smithi116 ceph-mon[29120]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:54.463 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:54 smithi116 ceph-mon[29120]: from='client.? 172.21.15.152:0/1580674820' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:59:54.463 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:54 smithi116 ceph-mon[29120]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:59:54.463 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:54 smithi116 ceph-mon[29120]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:59:56.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.131+0000 7fddfaf97700 1 -- 172.21.15.116:0/2539270011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddf4100de0 msgr2=0x7fddf41011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:56.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.131+0000 7fddfaf97700 1 --2- 172.21.15.116:0/2539270011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddf4100de0 0x7fddf41011c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdde8009910 tx=0x7fdde8009c20 comp rx=0 tx=0).stop 2024-03-04T15:59:56.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 -- 172.21.15.116:0/2539270011 shutdown_connections 2024-03-04T15:59:56.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 --2- 172.21.15.116:0/2539270011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fddf4101790 0x7fddf41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 --2- 172.21.15.116:0/2539270011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddf4100de0 0x7fddf41011c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 -- 172.21.15.116:0/2539270011 >> 172.21.15.116:0/2539270011 conn(0x7fddf40fc670 msgr2=0x7fddf40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 -- 172.21.15.116:0/2539270011 shutdown_connections 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 -- 172.21.15.116:0/2539270011 wait complete. 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 Processor -- start 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 -- start start 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fddf4100de0 0x7fddf419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddf4101790 0x7fddf419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf419d100 con 0x7fddf4101790 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.132+0000 7fddfaf97700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41965b0 con 0x7fddf4100de0 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddf4101790 0x7fddf419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddf4101790 0x7fddf419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36066/0 (socket says 172.21.15.116:36066) 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf3fff700 1 -- 172.21.15.116:0/3773124457 learned_addr learned my addr 172.21.15.116:0/3773124457 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:59:56.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf8d33700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fddf4100de0 0x7fddf419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:56.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf3fff700 1 -- 172.21.15.116:0/3773124457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fddf4100de0 msgr2=0x7fddf419c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:56.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf3fff700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fddf4100de0 0x7fddf419c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:56.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf3fff700 1 -- 172.21.15.116:0/3773124457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde80095f0 con 0x7fddf4101790 2024-03-04T15:59:56.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf3fff700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddf4101790 0x7fddf419ca70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdde000d900 tx=0x7fdde000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:56.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf1ffb700 1 -- 172.21.15.116:0/3773124457 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdde0009900 con 0x7fddf4101790 2024-03-04T15:59:56.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf1ffb700 1 -- 172.21.15.116:0/3773124457 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdde0010460 con 0x7fddf4101790 2024-03-04T15:59:56.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddfaf97700 1 -- 172.21.15.116:0/3773124457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf4196890 con 0x7fddf4101790 2024-03-04T15:59:56.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddf1ffb700 1 -- 172.21.15.116:0/3773124457 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdde000f5f0 con 0x7fddf4101790 2024-03-04T15:59:56.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.133+0000 7fddfaf97700 1 -- 172.21.15.116:0/3773124457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf4196de0 con 0x7fddf4101790 2024-03-04T15:59:56.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.134+0000 7fddfaf97700 1 -- 172.21.15.116:0/3773124457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf4109180 con 0x7fddf4101790 2024-03-04T15:59:56.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.135+0000 7fddf1ffb700 1 -- 172.21.15.116:0/3773124457 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fdde00105d0 con 0x7fddf4101790 2024-03-04T15:59:56.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.135+0000 7fddf1ffb700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fdde406c4e0 0x7fdde406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:56.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.135+0000 7fddf1ffb700 1 -- 172.21.15.116:0/3773124457 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdde008b080 con 0x7fddf4101790 2024-03-04T15:59:56.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.135+0000 7fddf8d33700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fdde406c4e0 0x7fdde406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:56.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.136+0000 7fddf8d33700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fdde406c4e0 0x7fdde406e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdde8006010 tx=0x7fdde801a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.140+0000 7fddf1ffb700 1 -- 172.21.15.116:0/3773124457 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdde0059630 con 0x7fddf4101790 2024-03-04T15:59:56.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.391+0000 7fddfaf97700 1 -- 172.21.15.116:0/3773124457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fddf4197080 con 0x7fddf4101790 2024-03-04T15:59:56.393 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:56 smithi116 ceph-mon[29120]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.395+0000 7fddf1ffb700 1 -- 172.21.15.116:0/3773124457 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173045 (secure 0 0 0) 0x7fdde00591c0 con 0x7fddf4101790 2024-03-04T15:59:56.399 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:59:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.397+0000 7fddfaf97700 1 -- 172.21.15.116:0/3773124457 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fdde406c4e0 msgr2=0x7fdde406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.397+0000 7fddfaf97700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fdde406c4e0 0x7fdde406e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdde8006010 tx=0x7fdde801a040 comp rx=0 tx=0).stop 2024-03-04T15:59:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.397+0000 7fddfaf97700 1 -- 172.21.15.116:0/3773124457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddf4101790 msgr2=0x7fddf419ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.397+0000 7fddfaf97700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddf4101790 0x7fddf419ca70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdde000d900 tx=0x7fdde000dc10 comp rx=0 tx=0).stop 2024-03-04T15:59:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.397+0000 7fddfaf97700 1 -- 172.21.15.116:0/3773124457 shutdown_connections 2024-03-04T15:59:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.397+0000 7fddfaf97700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fddf4100de0 0x7fddf419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.397+0000 7fddfaf97700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fdde406c4e0 0x7fdde406e9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.397+0000 7fddfaf97700 1 --2- 172.21.15.116:0/3773124457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddf4101790 0x7fddf419ca70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.397+0000 7fddfaf97700 1 -- 172.21.15.116:0/3773124457 >> 172.21.15.116:0/3773124457 conn(0x7fddf40fc670 msgr2=0x7fddf40fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.398+0000 7fddfaf97700 1 -- 172.21.15.116:0/3773124457 shutdown_connections 2024-03-04T15:59:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:56.398+0000 7fddfaf97700 1 -- 172.21.15.116:0/3773124457 wait complete. 2024-03-04T15:59:56.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:56 smithi152 ceph-mon[36876]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:57.144 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"active_gid":14217,"active_name":"smithi116.jwdhop","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3356602598},{"type":"v1","addr":"172.21.15.116:6801","nonce":3356602598}]},"active_addr":"172.21.15.116:6801/3356602598","active_change":"2024-03-04T15:56:07.047107+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14240,"name":"smithi152.aawieo","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.116:8443/","prometheus":"http://172.21.15.116:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":590312912}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":317318762}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":3213147898}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":232576737}]}]} 2024-03-04T15:59:57.148 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-04T15:59:57.148 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-04T15:59:57.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T15:59:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T15:59:57.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:57 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3773124457' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T15:59:57.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:57 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3773124457' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T15:59:58.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 15:59:58 smithi116 ceph-mon[29120]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 15:59:58 smithi152 ceph-mon[36876]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T15:59:59.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.135+0000 7f39a03b3700 1 -- 172.21.15.116:0/91236009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101780 msgr2=0x7f39981057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.135+0000 7f39a03b3700 1 --2- 172.21.15.116:0/91236009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101780 0x7f39981057d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3988009910 tx=0x7f3988009c20 comp rx=0 tx=0).stop 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 -- 172.21.15.116:0/91236009 shutdown_connections 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 --2- 172.21.15.116:0/91236009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101780 0x7f39981057d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 --2- 172.21.15.116:0/91236009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3998100dd0 0x7f39981011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 -- 172.21.15.116:0/91236009 >> 172.21.15.116:0/91236009 conn(0x7f39980fc640 msgr2=0x7f39980fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 -- 172.21.15.116:0/91236009 shutdown_connections 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 -- 172.21.15.116:0/91236009 wait complete. 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 Processor -- start 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 -- start start 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3998100dd0 0x7f399819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101780 0x7f399819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399819d220 con 0x7f3998101780 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f39a03b3700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39981966d0 con 0x7f3998100dd0 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f399d94e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101780 0x7f399819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f399d94e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101780 0x7f399819cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36086/0 (socket says 172.21.15.116:36086) 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.136+0000 7f399d94e700 1 -- 172.21.15.116:0/3049814903 learned_addr learned my addr 172.21.15.116:0/3049814903 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.137+0000 7f399e14f700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3998100dd0 0x7f399819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.137+0000 7f399d94e700 1 -- 172.21.15.116:0/3049814903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3998100dd0 msgr2=0x7f399819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.137+0000 7f399d94e700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3998100dd0 0x7f399819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.137+0000 7f399d94e700 1 -- 172.21.15.116:0/3049814903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39880095f0 con 0x7f3998101780 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.137+0000 7f399d94e700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101780 0x7f399819cb90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f39880092e0 tx=0x7f398800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.137+0000 7f398f7fe700 1 -- 172.21.15.116:0/3049814903 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f398801d070 con 0x7f3998101780 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.137+0000 7f398f7fe700 1 -- 172.21.15.116:0/3049814903 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3988022470 con 0x7f3998101780 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.137+0000 7f39a03b3700 1 -- 172.21.15.116:0/3049814903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3998196950 con 0x7f3998101780 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.137+0000 7f398f7fe700 1 -- 172.21.15.116:0/3049814903 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f398800f690 con 0x7f3998101780 2024-03-04T15:59:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.137+0000 7f39a03b3700 1 -- 172.21.15.116:0/3049814903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3998196e40 con 0x7f3998101780 2024-03-04T15:59:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.138+0000 7f39a03b3700 1 -- 172.21.15.116:0/3049814903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3998109120 con 0x7f3998101780 2024-03-04T15:59:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.138+0000 7f398f7fe700 1 -- 172.21.15.116:0/3049814903 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f39880225e0 con 0x7f3998101780 2024-03-04T15:59:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.139+0000 7f398f7fe700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f398406c4e0 0x7f398406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T15:59:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.139+0000 7f399e14f700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f398406c4e0 0x7f398406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T15:59:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.139+0000 7f398f7fe700 1 -- 172.21.15.116:0/3049814903 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f398808cab0 con 0x7f3998101780 2024-03-04T15:59:59.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.139+0000 7f399e14f700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f398406c4e0 0x7f398406e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3994006fd0 tx=0x7f3994008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T15:59:59.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.143+0000 7f398f7fe700 1 -- 172.21.15.116:0/3049814903 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f398805b110 con 0x7f3998101780 2024-03-04T15:59:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.334+0000 7f39a03b3700 1 -- 172.21.15.116:0/3049814903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3998197c80 con 0x7f3998101780 2024-03-04T15:59:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.335+0000 7f398f7fe700 1 -- 172.21.15.116:0/3049814903 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11395 (secure 0 0 0) 0x7f398805aca0 con 0x7f3998101780 2024-03-04T15:59:59.337 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T15:59:59.337 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:53:48.465240+0000","modified":"2024-03-04T15:59:42.855615+0000","last_up_change":"2024-03-04T15:59:41.850845+0000","last_in_change":"2024-03-04T15:59:22.307720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:58:27.372335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d12966f-954e-4a09-bbbd-5b0f6a04d039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6803","nonce":1056902067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6805","nonce":1056902067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6809","nonce":1056902067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6807","nonce":1056902067}]},"public_addr":"172.21.15.116:6803/1056902067","cluster_addr":"172.21.15.116:6805/1056902067","heartbeat_back_addr":"172.21.15.116:6809/1056902067","heartbeat_front_addr":"172.21.15.116:6807/1056902067","state":["exists","up"]},{"osd":1,"uuid":"f990720f-a19d-48c1-9545-81e511c2690b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6811","nonce":2368425008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6813","nonce":2368425008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6817","nonce":2368425008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6815","nonce":2368425008}]},"public_addr":"172.21.15.116:6811/2368425008","cluster_addr":"172.21.15.116:6813/2368425008","heartbeat_back_addr":"172.21.15.116:6817/2368425008","heartbeat_front_addr":"172.21.15.116:6815/2368425008","state":["exists","up"]},{"osd":2,"uuid":"fe3132c8-6b15-4a11-a884-7c6a09d42955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6819","nonce":1282157941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6821","nonce":1282157941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6825","nonce":1282157941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6823","nonce":1282157941}]},"public_addr":"172.21.15.116:6819/1282157941","cluster_addr":"172.21.15.116:6821/1282157941","heartbeat_back_addr":"172.21.15.116:6825/1282157941","heartbeat_front_addr":"172.21.15.116:6823/1282157941","state":["exists","up"]},{"osd":3,"uuid":"15930a9b-f721-4c30-9c5b-bef5ef69777c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6801","nonce":2457942679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6803","nonce":2457942679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6807","nonce":2457942679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6805","nonce":2457942679}]},"public_addr":"172.21.15.152:6801/2457942679","cluster_addr":"172.21.15.152:6803/2457942679","heartbeat_back_addr":"172.21.15.152:6807/2457942679","heartbeat_front_addr":"172.21.15.152:6805/2457942679","state":["exists","up"]},{"osd":4,"uuid":"8d201b6a-6a91-4a7e-984e-e8d7f0289793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6809","nonce":3048670947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6811","nonce":3048670947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6815","nonce":3048670947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6813","nonce":3048670947}]},"public_addr":"172.21.15.152:6809/3048670947","cluster_addr":"172.21.15.152:6811/3048670947","heartbeat_back_addr":"172.21.15.152:6815/3048670947","heartbeat_front_addr":"172.21.15.152:6813/3048670947","state":["exists","up"]},{"osd":5,"uuid":"c06ef0f5-6894-4e01-a693-bb129d248fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6817","nonce":1864443408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6819","nonce":1864443408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6823","nonce":1864443408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6821","nonce":1864443408}]},"public_addr":"172.21.15.152:6817/1864443408","cluster_addr":"172.21.15.152:6819/1864443408","heartbeat_back_addr":"172.21.15.152:6823/1864443408","heartbeat_front_addr":"172.21.15.152:6821/1864443408","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:24.230215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:52.709103+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:24.072974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:47.684390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:13.571654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:39.874966+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/821201733":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/3670178561":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6800/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/831649128":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3287532122":"2024-03-05T15:54:53.931186+0000","172.21.15.116:6801/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/355050440":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/976694871":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/4142805454":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/385581892":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6801/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6800/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6801/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6800/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3323890559":"2024-03-05T15:54:21.166941+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T15:59:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.337+0000 7f39a03b3700 1 -- 172.21.15.116:0/3049814903 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f398406c4e0 msgr2=0x7f398406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.337+0000 7f39a03b3700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f398406c4e0 0x7f398406e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3994006fd0 tx=0x7f3994008040 comp rx=0 tx=0).stop 2024-03-04T15:59:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.337+0000 7f39a03b3700 1 -- 172.21.15.116:0/3049814903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101780 msgr2=0x7f399819cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T15:59:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.337+0000 7f39a03b3700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101780 0x7f399819cb90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f39880092e0 tx=0x7f398800be20 comp rx=0 tx=0).stop 2024-03-04T15:59:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.337+0000 7f39a03b3700 1 -- 172.21.15.116:0/3049814903 shutdown_connections 2024-03-04T15:59:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.337+0000 7f39a03b3700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3998100dd0 0x7f399819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.337+0000 7f39a03b3700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f398406c4e0 0x7f398406e9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.337+0000 7f39a03b3700 1 --2- 172.21.15.116:0/3049814903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101780 0x7f399819cb90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T15:59:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.337+0000 7f39a03b3700 1 -- 172.21.15.116:0/3049814903 >> 172.21.15.116:0/3049814903 conn(0x7f39980fc640 msgr2=0x7f39980fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T15:59:59.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.338+0000 7f39a03b3700 1 -- 172.21.15.116:0/3049814903 shutdown_connections 2024-03-04T15:59:59.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T15:59:59.338+0000 7f39a03b3700 1 -- 172.21.15.116:0/3049814903 wait complete. 2024-03-04T16:00:00.092 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-04T16:00:00.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T16:00:00.299 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:00.423 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:00 smithi116 ceph-mon[29120]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:00.423 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:00 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3049814903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:00:00.423 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:00 smithi116 ceph-mon[29120]: overall HEALTH_OK 2024-03-04T16:00:00.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:00 smithi152 ceph-mon[36876]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:00.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:00 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3049814903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:00:00.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:00 smithi152 ceph-mon[36876]: overall HEALTH_OK 2024-03-04T16:00:02.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.100+0000 7f7d96668700 1 -- 172.21.15.116:0/21251346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d90101770 msgr2=0x7f7d901057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:02.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.101+0000 7f7d96668700 1 --2- 172.21.15.116:0/21251346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d90101770 0x7f7d901057c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009910 tx=0x7f7d84009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:02.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.101+0000 7f7d96668700 1 -- 172.21.15.116:0/21251346 shutdown_connections 2024-03-04T16:00:02.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.101+0000 7f7d96668700 1 --2- 172.21.15.116:0/21251346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d90101770 0x7f7d901057c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:02.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.101+0000 7f7d96668700 1 --2- 172.21.15.116:0/21251346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d90100dc0 0x7f7d901011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:02.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.101+0000 7f7d96668700 1 -- 172.21.15.116:0/21251346 >> 172.21.15.116:0/21251346 conn(0x7f7d900fc650 msgr2=0x7f7d900fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:02.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.101+0000 7f7d96668700 1 -- 172.21.15.116:0/21251346 shutdown_connections 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.101+0000 7f7d96668700 1 -- 172.21.15.116:0/21251346 wait complete. 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.101+0000 7f7d96668700 1 Processor -- start 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.101+0000 7f7d96668700 1 -- start start 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d96668700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d90100dc0 0x7f7d90073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d96668700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d90101770 0x7f7d900735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d96668700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d90073b10 con 0x7f7d90101770 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d96668700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d90073c50 con 0x7f7d90100dc0 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d90101770 0x7f7d900735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d90101770 0x7f7d900735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36112/0 (socket says 172.21.15.116:36112) 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8f7fe700 1 -- 172.21.15.116:0/780355878 learned_addr learned my addr 172.21.15.116:0/780355878 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8ffff700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d90100dc0 0x7f7d90073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8f7fe700 1 -- 172.21.15.116:0/780355878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d90100dc0 msgr2=0x7f7d90073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8f7fe700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d90100dc0 0x7f7d90073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8f7fe700 1 -- 172.21.15.116:0/780355878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d840095f0 con 0x7f7d90101770 2024-03-04T16:00:02.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8f7fe700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d90101770 0x7f7d900735d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d840092e0 tx=0x7f7d8400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:02.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8d7fa700 1 -- 172.21.15.116:0/780355878 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8401d070 con 0x7f7d90101770 2024-03-04T16:00:02.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8d7fa700 1 -- 172.21.15.116:0/780355878 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7d84022470 con 0x7f7d90101770 2024-03-04T16:00:02.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d96668700 1 -- 172.21.15.116:0/780355878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d90073ed0 con 0x7f7d90101770 2024-03-04T16:00:02.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d8d7fa700 1 -- 172.21.15.116:0/780355878 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8400f690 con 0x7f7d90101770 2024-03-04T16:00:02.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.102+0000 7f7d96668700 1 -- 172.21.15.116:0/780355878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d901a2c90 con 0x7f7d90101770 2024-03-04T16:00:02.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.104+0000 7f7d96668700 1 -- 172.21.15.116:0/780355878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d901091c0 con 0x7f7d90101770 2024-03-04T16:00:02.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.104+0000 7f7d8d7fa700 1 -- 172.21.15.116:0/780355878 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7d8400f8b0 con 0x7f7d90101770 2024-03-04T16:00:02.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.104+0000 7f7d8d7fa700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7d7806c4e0 0x7f7d7806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:02.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.104+0000 7f7d8d7fa700 1 -- 172.21.15.116:0/780355878 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7d8408cab0 con 0x7f7d90101770 2024-03-04T16:00:02.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.104+0000 7f7d8ffff700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7d7806c4e0 0x7f7d7806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:02.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.105+0000 7f7d8ffff700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7d7806c4e0 0x7f7d7806e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7d80005950 tx=0x7f7d80005cf0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:02.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.109+0000 7f7d8d7fa700 1 -- 172.21.15.116:0/780355878 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7d84058890 con 0x7f7d90101770 2024-03-04T16:00:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.301+0000 7f7d96668700 1 -- 172.21.15.116:0/780355878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d9004ea90 con 0x7f7d90101770 2024-03-04T16:00:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.302+0000 7f7d8d7fa700 1 -- 172.21.15.116:0/780355878 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11395 (secure 0 0 0) 0x7f7d84027020 con 0x7f7d90101770 2024-03-04T16:00:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:00:02.304 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:53:48.465240+0000","modified":"2024-03-04T15:59:42.855615+0000","last_up_change":"2024-03-04T15:59:41.850845+0000","last_in_change":"2024-03-04T15:59:22.307720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:58:27.372335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d12966f-954e-4a09-bbbd-5b0f6a04d039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6803","nonce":1056902067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6805","nonce":1056902067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6809","nonce":1056902067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6807","nonce":1056902067}]},"public_addr":"172.21.15.116:6803/1056902067","cluster_addr":"172.21.15.116:6805/1056902067","heartbeat_back_addr":"172.21.15.116:6809/1056902067","heartbeat_front_addr":"172.21.15.116:6807/1056902067","state":["exists","up"]},{"osd":1,"uuid":"f990720f-a19d-48c1-9545-81e511c2690b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6811","nonce":2368425008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6813","nonce":2368425008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6817","nonce":2368425008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6815","nonce":2368425008}]},"public_addr":"172.21.15.116:6811/2368425008","cluster_addr":"172.21.15.116:6813/2368425008","heartbeat_back_addr":"172.21.15.116:6817/2368425008","heartbeat_front_addr":"172.21.15.116:6815/2368425008","state":["exists","up"]},{"osd":2,"uuid":"fe3132c8-6b15-4a11-a884-7c6a09d42955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6819","nonce":1282157941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6821","nonce":1282157941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6825","nonce":1282157941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6823","nonce":1282157941}]},"public_addr":"172.21.15.116:6819/1282157941","cluster_addr":"172.21.15.116:6821/1282157941","heartbeat_back_addr":"172.21.15.116:6825/1282157941","heartbeat_front_addr":"172.21.15.116:6823/1282157941","state":["exists","up"]},{"osd":3,"uuid":"15930a9b-f721-4c30-9c5b-bef5ef69777c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6801","nonce":2457942679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6803","nonce":2457942679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6807","nonce":2457942679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6805","nonce":2457942679}]},"public_addr":"172.21.15.152:6801/2457942679","cluster_addr":"172.21.15.152:6803/2457942679","heartbeat_back_addr":"172.21.15.152:6807/2457942679","heartbeat_front_addr":"172.21.15.152:6805/2457942679","state":["exists","up"]},{"osd":4,"uuid":"8d201b6a-6a91-4a7e-984e-e8d7f0289793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6809","nonce":3048670947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6811","nonce":3048670947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6815","nonce":3048670947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6813","nonce":3048670947}]},"public_addr":"172.21.15.152:6809/3048670947","cluster_addr":"172.21.15.152:6811/3048670947","heartbeat_back_addr":"172.21.15.152:6815/3048670947","heartbeat_front_addr":"172.21.15.152:6813/3048670947","state":["exists","up"]},{"osd":5,"uuid":"c06ef0f5-6894-4e01-a693-bb129d248fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6817","nonce":1864443408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6819","nonce":1864443408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6823","nonce":1864443408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6821","nonce":1864443408}]},"public_addr":"172.21.15.152:6817/1864443408","cluster_addr":"172.21.15.152:6819/1864443408","heartbeat_back_addr":"172.21.15.152:6823/1864443408","heartbeat_front_addr":"172.21.15.152:6821/1864443408","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:24.230215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:52.709103+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:24.072974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:47.684390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:13.571654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:39.874966+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/821201733":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/3670178561":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6800/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/831649128":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3287532122":"2024-03-05T15:54:53.931186+0000","172.21.15.116:6801/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/355050440":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/976694871":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/4142805454":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/385581892":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6801/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6800/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6801/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6800/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3323890559":"2024-03-05T15:54:21.166941+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:00:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 -- 172.21.15.116:0/780355878 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7d7806c4e0 msgr2=0x7f7d7806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7d7806c4e0 0x7f7d7806e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7d80005950 tx=0x7f7d80005cf0 comp rx=0 tx=0).stop 2024-03-04T16:00:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 -- 172.21.15.116:0/780355878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d90101770 msgr2=0x7f7d900735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d90101770 0x7f7d900735d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d840092e0 tx=0x7f7d8400be20 comp rx=0 tx=0).stop 2024-03-04T16:00:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 -- 172.21.15.116:0/780355878 shutdown_connections 2024-03-04T16:00:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d90100dc0 0x7f7d90073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7d7806c4e0 0x7f7d7806e9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 --2- 172.21.15.116:0/780355878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d90101770 0x7f7d900735d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 -- 172.21.15.116:0/780355878 >> 172.21.15.116:0/780355878 conn(0x7f7d900fc650 msgr2=0x7f7d900fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 -- 172.21.15.116:0/780355878 shutdown_connections 2024-03-04T16:00:02.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:02.304+0000 7f7d96668700 1 -- 172.21.15.116:0/780355878 wait complete. 2024-03-04T16:00:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:02 smithi116 ceph-mon[29120]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:02.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:02 smithi152 ceph-mon[36876]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:02.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-04T16:00:02.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-04T16:00:02.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-04T16:00:02.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-04T16:00:02.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-04T16:00:02.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-04T16:00:03.141 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:03.543 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:03 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/780355878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:00:03.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:03 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/780355878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:00:03.840 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:03.840 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:04.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:04 smithi152 ceph-mon[36876]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:04.614 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:04 smithi116 ceph-mon[29120]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:06.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:06 smithi116 ceph-mon[29120]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:06.736 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:06 smithi152 ceph-mon[36876]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:00:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:00:08.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.402+0000 7f71ea63f700 1 -- 172.21.15.116:0/3245590818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71e40ffa60 msgr2=0x7f71e40ffe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:08.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.402+0000 7f71ea63f700 1 --2- 172.21.15.116:0/3245590818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71e40ffa60 0x7f71e40ffe40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f71d4009910 tx=0x7f71d4009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:08.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.403+0000 7f71ea63f700 1 -- 172.21.15.116:0/3245590818 shutdown_connections 2024-03-04T16:00:08.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.403+0000 7f71ea63f700 1 --2- 172.21.15.116:0/3245590818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71e4100410 0x7f71e41048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.403+0000 7f71ea63f700 1 --2- 172.21.15.116:0/3245590818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71e40ffa60 0x7f71e40ffe40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.403+0000 7f71ea63f700 1 -- 172.21.15.116:0/3245590818 >> 172.21.15.116:0/3245590818 conn(0x7f71e40fb550 msgr2=0x7f71e40fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.403+0000 7f71ea63f700 1 -- 172.21.15.116:0/3245590818 shutdown_connections 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.403+0000 7f71ea63f700 1 -- 172.21.15.116:0/3245590818 wait complete. 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.403+0000 7f71ea63f700 1 Processor -- start 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.403+0000 7f71ea63f700 1 -- start start 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.403+0000 7f71ea63f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71e40ffa60 0x7f71e419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71ea63f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71e4100410 0x7f71e419d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71ea63f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e419d960 con 0x7f71e4100410 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71ea63f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e41a16f0 con 0x7f71e40ffa60 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71e4100410 0x7f71e419d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71e4100410 0x7f71e419d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37476/0 (socket says 172.21.15.116:37476) 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e37fe700 1 -- 172.21.15.116:0/3561486298 learned_addr learned my addr 172.21.15.116:0/3561486298 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:08.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e3fff700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71e40ffa60 0x7f71e419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e37fe700 1 -- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71e40ffa60 msgr2=0x7f71e419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e37fe700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71e40ffa60 0x7f71e419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e37fe700 1 -- 172.21.15.116:0/3561486298 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d40095f0 con 0x7f71e4100410 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e3fff700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71e40ffa60 0x7f71e419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e37fe700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71e4100410 0x7f71e419d280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f71d800c8f0 tx=0x7f71d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e17fa700 1 -- 172.21.15.116:0/3561486298 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71d8007a50 con 0x7f71e4100410 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e41a19d0 con 0x7f71e4100410 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e17fa700 1 -- 172.21.15.116:0/3561486298 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f71d8007bb0 con 0x7f71e4100410 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71e17fa700 1 -- 172.21.15.116:0/3561486298 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71d8018630 con 0x7f71e4100410 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.404+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e41a1f20 con 0x7f71e4100410 2024-03-04T16:00:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.405+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71e4114670 con 0x7f71e4100410 2024-03-04T16:00:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.406+0000 7f71e17fa700 1 -- 172.21.15.116:0/3561486298 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f71d8007d20 con 0x7f71e4100410 2024-03-04T16:00:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.406+0000 7f71e17fa700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f71cc06c4e0 0x7f71cc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.406+0000 7f71e17fa700 1 -- 172.21.15.116:0/3561486298 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f71d808b4b0 con 0x7f71e4100410 2024-03-04T16:00:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.406+0000 7f71e3fff700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f71cc06c4e0 0x7f71cc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.406+0000 7f71e17fa700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679] conn(0x7f71cc0721c0 0x7f71cc0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.406+0000 7f71e17fa700 1 -- 172.21.15.116:0/3561486298 --> [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71cc074c90 con 0x7f71cc0721c0 2024-03-04T16:00:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.406+0000 7f71e17fa700 1 -- 172.21.15.116:0/3561486298 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71d808db80 con 0x7f71e4100410 2024-03-04T16:00:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.407+0000 7f71e3fff700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f71cc06c4e0 0x7f71cc06e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f71d400b5c0 tx=0x7f71d401a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.407+0000 7f71e8bdc700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679] conn(0x7f71cc0721c0 0x7f71cc0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.407+0000 7f71e8bdc700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679] conn(0x7f71cc0721c0 0x7f71cc0745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.408+0000 7f71e17fa700 1 -- 172.21.15.116:0/3561486298 <== osd.3 v2:172.21.15.152:6800/2457942679 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f71cc074c90 con 0x7f71cc0721c0 2024-03-04T16:00:08.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.423+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 --> [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f71e404ea90 con 0x7f71cc0721c0 2024-03-04T16:00:08.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71e17fa700 1 -- 172.21.15.116:0/3561486298 <== osd.3 v2:172.21.15.152:6800/2457942679 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f71e404ea90 con 0x7f71cc0721c0 2024-03-04T16:00:08.425 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:08 smithi116 ceph-mon[29120]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679] conn(0x7f71cc0721c0 msgr2=0x7f71cc0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679] conn(0x7f71cc0721c0 0x7f71cc0745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f71cc06c4e0 msgr2=0x7f71cc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f71cc06c4e0 0x7f71cc06e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f71d400b5c0 tx=0x7f71d401a040 comp rx=0 tx=0).stop 2024-03-04T16:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71e4100410 msgr2=0x7f71e419d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71e4100410 0x7f71e419d280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f71d800c8f0 tx=0x7f71d800ccb0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 shutdown_connections 2024-03-04T16:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:6800/2457942679,v1:172.21.15.152:6801/2457942679] conn(0x7f71cc0721c0 0x7f71cc0745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71e40ffa60 0x7f71e419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f71cc06c4e0 0x7f71cc06e9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 --2- 172.21.15.116:0/3561486298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71e4100410 0x7f71e419d280 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 >> 172.21.15.116:0/3561486298 conn(0x7f71e40fb550 msgr2=0x7f71e4110f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 shutdown_connections 2024-03-04T16:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.424+0000 7f71ea63f700 1 -- 172.21.15.116:0/3561486298 wait complete. 2024-03-04T16:00:08.561 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:08 smithi152 ceph-mon[36876]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.651+0000 7f7f36743700 1 -- 172.21.15.116:0/742616323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f300ffa40 msgr2=0x7f7f300ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.651+0000 7f7f36743700 1 --2- 172.21.15.116:0/742616323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f300ffa40 0x7f7f300ffec0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009910 tx=0x7f7f24009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.651+0000 7f7f36743700 1 -- 172.21.15.116:0/742616323 shutdown_connections 2024-03-04T16:00:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.651+0000 7f7f36743700 1 --2- 172.21.15.116:0/742616323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f300ffa40 0x7f7f300ffec0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.651+0000 7f7f36743700 1 --2- 172.21.15.116:0/742616323 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f30105a40 0x7f7f30105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.651+0000 7f7f36743700 1 -- 172.21.15.116:0/742616323 >> 172.21.15.116:0/742616323 conn(0x7f7f300fb570 msgr2=0x7f7f300fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.651+0000 7f7f36743700 1 -- 172.21.15.116:0/742616323 shutdown_connections 2024-03-04T16:00:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.651+0000 7f7f36743700 1 -- 172.21.15.116:0/742616323 wait complete. 2024-03-04T16:00:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.651+0000 7f7f36743700 1 Processor -- start 2024-03-04T16:00:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.651+0000 7f7f36743700 1 -- start start 2024-03-04T16:00:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f36743700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f300ffa40 0x7f7f300fbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f36743700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f30105a40 0x7f7f300fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f36743700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f300fdc10 con 0x7f7f300ffa40 2024-03-04T16:00:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f36743700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f300fdd80 con 0x7f7f30105a40 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f2ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f300ffa40 0x7f7f300fbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f2ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f300ffa40 0x7f7f300fbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37484/0 (socket says 172.21.15.116:37484) 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f2ffff700 1 -- 172.21.15.116:0/2592566739 learned_addr learned my addr 172.21.15.116:0/2592566739 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f2f7fe700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f30105a40 0x7f7f300fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f2ffff700 1 -- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f30105a40 msgr2=0x7f7f300fc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f2ffff700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f30105a40 0x7f7f300fc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f2ffff700 1 -- 172.21.15.116:0/2592566739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f240095f0 con 0x7f7f300ffa40 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.652+0000 7f7f2f7fe700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f30105a40 0x7f7f300fc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.653+0000 7f7f2ffff700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f300ffa40 0x7f7f300fbe60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000d900 tx=0x7f7f2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.653+0000 7f7f2d7fa700 1 -- 172.21.15.116:0/2592566739 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f20004a00 con 0x7f7f300ffa40 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.653+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f300fc9a0 con 0x7f7f300ffa40 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.653+0000 7f7f2d7fa700 1 -- 172.21.15.116:0/2592566739 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7f200054d0 con 0x7f7f300ffa40 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.653+0000 7f7f2d7fa700 1 -- 172.21.15.116:0/2592566739 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f20009d60 con 0x7f7f300ffa40 2024-03-04T16:00:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.653+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f301a7290 con 0x7f7f300ffa40 2024-03-04T16:00:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.654+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7f3010cee0 con 0x7f7f300ffa40 2024-03-04T16:00:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.654+0000 7f7f2d7fa700 1 -- 172.21.15.116:0/2592566739 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7f20005000 con 0x7f7f300ffa40 2024-03-04T16:00:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.655+0000 7f7f2d7fa700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7f1806c4e0 0x7f7f1806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.655+0000 7f7f2d7fa700 1 -- 172.21.15.116:0/2592566739 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7f2008bb90 con 0x7f7f300ffa40 2024-03-04T16:00:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.655+0000 7f7f2f7fe700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7f1806c4e0 0x7f7f1806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.655+0000 7f7f2d7fa700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947] conn(0x7f7f180721c0 0x7f7f180745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.655+0000 7f7f2d7fa700 1 -- 172.21.15.116:0/2592566739 --> [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f18074c90 con 0x7f7f180721c0 2024-03-04T16:00:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.655+0000 7f7f2d7fa700 1 -- 172.21.15.116:0/2592566739 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f2008c010 con 0x7f7f300ffa40 2024-03-04T16:00:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.655+0000 7f7f2f7fe700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7f1806c4e0 0x7f7f1806e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7f300fd650 tx=0x7f7f2401a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.655+0000 7f7f34ce0700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947] conn(0x7f7f180721c0 0x7f7f180745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.656+0000 7f7f34ce0700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947] conn(0x7f7f180721c0 0x7f7f180745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:08.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.656+0000 7f7f2d7fa700 1 -- 172.21.15.116:0/2592566739 <== osd.4 v2:172.21.15.152:6808/3048670947 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f7f18074c90 con 0x7f7f180721c0 2024-03-04T16:00:08.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.673+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 --> [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7f3004ea90 con 0x7f7f180721c0 2024-03-04T16:00:08.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.673+0000 7f7f2d7fa700 1 -- 172.21.15.116:0/2592566739 <== osd.4 v2:172.21.15.152:6808/3048670947 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7f3004ea90 con 0x7f7f180721c0 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947] conn(0x7f7f180721c0 msgr2=0x7f7f180745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947] conn(0x7f7f180721c0 0x7f7f180745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7f1806c4e0 msgr2=0x7f7f1806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7f1806c4e0 0x7f7f1806e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7f300fd650 tx=0x7f7f2401a040 comp rx=0 tx=0).stop 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f300ffa40 msgr2=0x7f7f300fbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f300ffa40 0x7f7f300fbe60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000d900 tx=0x7f7f2000dcc0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 shutdown_connections 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:6808/3048670947,v1:172.21.15.152:6809/3048670947] conn(0x7f7f180721c0 0x7f7f180745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7f1806c4e0 0x7f7f1806e9a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f300ffa40 0x7f7f300fbe60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 --2- 172.21.15.116:0/2592566739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f30105a40 0x7f7f300fc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 >> 172.21.15.116:0/2592566739 conn(0x7f7f300fb570 msgr2=0x7f7f30068f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 shutdown_connections 2024-03-04T16:00:08.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:08.674+0000 7f7f36743700 1 -- 172.21.15.116:0/2592566739 wait complete. 2024-03-04T16:00:09.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.331+0000 7feae96f4700 1 -- 172.21.15.116:0/440849115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae4100de0 msgr2=0x7feae41011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:09.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.331+0000 7feae96f4700 1 --2- 172.21.15.116:0/440849115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae4100de0 0x7feae41011c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fead4009910 tx=0x7fead4009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:09.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.331+0000 7feae96f4700 1 -- 172.21.15.116:0/440849115 shutdown_connections 2024-03-04T16:00:09.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.331+0000 7feae96f4700 1 --2- 172.21.15.116:0/440849115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feae4101790 0x7feae41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:09.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.331+0000 7feae96f4700 1 --2- 172.21.15.116:0/440849115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae4100de0 0x7feae41011c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:09.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.331+0000 7feae96f4700 1 -- 172.21.15.116:0/440849115 >> 172.21.15.116:0/440849115 conn(0x7feae40fc670 msgr2=0x7feae40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:09.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.331+0000 7feae96f4700 1 -- 172.21.15.116:0/440849115 shutdown_connections 2024-03-04T16:00:09.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.331+0000 7feae96f4700 1 -- 172.21.15.116:0/440849115 wait complete. 2024-03-04T16:00:09.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.332+0000 7feae96f4700 1 Processor -- start 2024-03-04T16:00:09.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.332+0000 7feae96f4700 1 -- start start 2024-03-04T16:00:09.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.332+0000 7feae96f4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feae4100de0 0x7feae419c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:09.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.332+0000 7feae96f4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae4101790 0x7feae419cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:09.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.332+0000 7feae96f4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae419d150 con 0x7feae4101790 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.332+0000 7feae96f4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae4196600 con 0x7feae4100de0 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.332+0000 7feae27fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae4101790 0x7feae419cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.332+0000 7feae27fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae4101790 0x7feae419cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37490/0 (socket says 172.21.15.116:37490) 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.332+0000 7feae27fc700 1 -- 172.21.15.116:0/1407478753 learned_addr learned my addr 172.21.15.116:0/1407478753 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feae2ffd700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feae4100de0 0x7feae419c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feae27fc700 1 -- 172.21.15.116:0/1407478753 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feae4100de0 msgr2=0x7feae419c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feae27fc700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feae4100de0 0x7feae419c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feae27fc700 1 -- 172.21.15.116:0/1407478753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead40095f0 con 0x7feae4101790 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feae2ffd700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feae4100de0 0x7feae419c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feae27fc700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae4101790 0x7feae419cac0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fead800b480 tx=0x7fead800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feacbfff700 1 -- 172.21.15.116:0/1407478753 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fead80118c0 con 0x7feae4101790 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feacbfff700 1 -- 172.21.15.116:0/1407478753 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fead8010380 con 0x7feae4101790 2024-03-04T16:00:09.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feacbfff700 1 -- 172.21.15.116:0/1407478753 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fead800f4f0 con 0x7feae4101790 2024-03-04T16:00:09.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae41968e0 con 0x7feae4101790 2024-03-04T16:00:09.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.333+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae4196de0 con 0x7feae4101790 2024-03-04T16:00:09.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.334+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feae410e120 con 0x7feae4101790 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.334+0000 7feacbfff700 1 -- 172.21.15.116:0/1407478753 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fead8011a20 con 0x7feae4101790 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.335+0000 7feacbfff700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7feacc06c4e0 0x7feacc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.335+0000 7feae2ffd700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7feacc06c4e0 0x7feacc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.335+0000 7feacbfff700 1 -- 172.21.15.116:0/1407478753 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fead808b1a0 con 0x7feae4101790 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.335+0000 7feacbfff700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941] conn(0x7feacc0721c0 0x7feacc0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.335+0000 7feacbfff700 1 -- 172.21.15.116:0/1407478753 --> [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feacc074c90 con 0x7feacc0721c0 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.335+0000 7feacbfff700 1 -- 172.21.15.116:0/1407478753 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fead808d870 con 0x7feae4101790 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.335+0000 7feae37fe700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941] conn(0x7feacc0721c0 0x7feacc0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.335+0000 7feae2ffd700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7feacc06c4e0 0x7feacc06e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fead400b5c0 tx=0x7fead4005480 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.335+0000 7feae37fe700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941] conn(0x7feacc0721c0 0x7feacc0745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:09.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.336+0000 7feacbfff700 1 -- 172.21.15.116:0/1407478753 <== osd.2 v2:172.21.15.116:6818/1282157941 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7feacc074c90 con 0x7feacc0721c0 2024-03-04T16:00:09.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.351+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 --> [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feae404ea90 con 0x7feacc0721c0 2024-03-04T16:00:09.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feacbfff700 1 -- 172.21.15.116:0/1407478753 <== osd.2 v2:172.21.15.116:6818/1282157941 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feae404ea90 con 0x7feacc0721c0 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941] conn(0x7feacc0721c0 msgr2=0x7feacc0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941] conn(0x7feacc0721c0 0x7feacc0745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7feacc06c4e0 msgr2=0x7feacc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7feacc06c4e0 0x7feacc06e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fead400b5c0 tx=0x7fead4005480 comp rx=0 tx=0).stop 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae4101790 msgr2=0x7feae419cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae4101790 0x7feae419cac0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fead800b480 tx=0x7fead800b840 comp rx=0 tx=0).stop 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 shutdown_connections 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6818/1282157941,v1:172.21.15.116:6819/1282157941] conn(0x7feacc0721c0 0x7feacc0745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feae4100de0 0x7feae419c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7feacc06c4e0 0x7feacc06e9a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 --2- 172.21.15.116:0/1407478753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae4101790 0x7feae419cac0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:09.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 >> 172.21.15.116:0/1407478753 conn(0x7feae40fc670 msgr2=0x7feae40fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:09.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 shutdown_connections 2024-03-04T16:00:09.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:09.352+0000 7feae96f4700 1 -- 172.21.15.116:0/1407478753 wait complete. 2024-03-04T16:00:09.829 INFO:teuthology.orchestra.run.smithi116.stdout:150323855373 2024-03-04T16:00:09.829 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-04T16:00:10.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:10 smithi116 ceph-mon[29120]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:10.487 INFO:teuthology.orchestra.run.smithi116.stdout:115964117010 2024-03-04T16:00:10.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-04T16:00:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:10 smithi152 ceph-mon[36876]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:12.092 INFO:teuthology.orchestra.run.smithi116.stdout:81604378648 2024-03-04T16:00:12.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-04T16:00:12.543 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:12 smithi116 ceph-mon[29120]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:12.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:12 smithi152 ceph-mon[36876]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:14.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.479+0000 7f69ae18e700 1 -- 172.21.15.116:0/673960712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a8102680 msgr2=0x7f69a810ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:14.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.479+0000 7f69ae18e700 1 --2- 172.21.15.116:0/673960712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a8102680 0x7f69a810ab70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6990009940 tx=0x7f6990009c50 comp rx=0 tx=0).stop 2024-03-04T16:00:14.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.479+0000 7f69ae18e700 1 -- 172.21.15.116:0/673960712 shutdown_connections 2024-03-04T16:00:14.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.479+0000 7f69ae18e700 1 --2- 172.21.15.116:0/673960712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a8102680 0x7f69a810ab70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:14.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.479+0000 7f69ae18e700 1 --2- 172.21.15.116:0/673960712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a8101d60 0x7f69a8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.479+0000 7f69ae18e700 1 -- 172.21.15.116:0/673960712 >> 172.21.15.116:0/673960712 conn(0x7f69a80fb550 msgr2=0x7f69a80fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.479+0000 7f69ae18e700 1 -- 172.21.15.116:0/673960712 shutdown_connections 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.479+0000 7f69ae18e700 1 -- 172.21.15.116:0/673960712 wait complete. 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.479+0000 7f69ae18e700 1 Processor -- start 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69ae18e700 1 -- start start 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69ae18e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a8101d60 0x7f69a819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69ae18e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a8102680 0x7f69a819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69ae18e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a819d2b0 con 0x7f69a8101d60 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69ae18e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a8196760 con 0x7f69a8102680 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69a77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a8101d60 0x7f69a819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69a77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a8101d60 0x7f69a819c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37504/0 (socket says 172.21.15.116:37504) 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69a77fe700 1 -- 172.21.15.116:0/867416028 learned_addr learned my addr 172.21.15.116:0/867416028 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69a6ffd700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a8102680 0x7f69a819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:14.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69a77fe700 1 -- 172.21.15.116:0/867416028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a8102680 msgr2=0x7f69a819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69a77fe700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a8102680 0x7f69a819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69a77fe700 1 -- 172.21.15.116:0/867416028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69900095f0 con 0x7f69a8101d60 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69a77fe700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a8101d60 0x7f69a819c6e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f699800ba50 tx=0x7f699800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.480+0000 7f69a4ff9700 1 -- 172.21.15.116:0/867416028 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f699800ca00 con 0x7f69a8101d60 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.481+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a8196a40 con 0x7f69a8101d60 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.481+0000 7f69a4ff9700 1 -- 172.21.15.116:0/867416028 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f699800cb60 con 0x7f69a8101d60 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.481+0000 7f69a4ff9700 1 -- 172.21.15.116:0/867416028 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6998007580 con 0x7f69a8101d60 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.481+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a8196f60 con 0x7f69a8101d60 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.481+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f69a810e060 con 0x7f69a8101d60 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.482+0000 7f69a4ff9700 1 -- 172.21.15.116:0/867416028 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f699800ccd0 con 0x7f69a8101d60 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.482+0000 7f69a4ff9700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f699406c4e0 0x7f699406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.482+0000 7f69a4ff9700 1 -- 172.21.15.116:0/867416028 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6998006e40 con 0x7f69a8101d60 2024-03-04T16:00:14.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.482+0000 7f69a6ffd700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f699406c4e0 0x7f699406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:14.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.483+0000 7f69a4ff9700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067] conn(0x7f69940721c0 0x7f69940745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:14.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.483+0000 7f69a4ff9700 1 -- 172.21.15.116:0/867416028 --> [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6994074c90 con 0x7f69940721c0 2024-03-04T16:00:14.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.483+0000 7f69a4ff9700 1 -- 172.21.15.116:0/867416028 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f699808b750 con 0x7f69a8101d60 2024-03-04T16:00:14.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.483+0000 7f69a7fff700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067] conn(0x7f69940721c0 0x7f69940745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:14.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.483+0000 7f69a6ffd700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f699406c4e0 0x7f699406e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6990009910 tx=0x7f6990005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:14.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.483+0000 7f69a7fff700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067] conn(0x7f69940721c0 0x7f69940745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:14.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.484+0000 7f69a4ff9700 1 -- 172.21.15.116:0/867416028 <== osd.0 v2:172.21.15.116:6802/1056902067 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6994074c90 con 0x7f69940721c0 2024-03-04T16:00:14.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.500+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 --> [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f69a804ea90 con 0x7f69940721c0 2024-03-04T16:00:14.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.500+0000 7f69a4ff9700 1 -- 172.21.15.116:0/867416028 <== osd.0 v2:172.21.15.116:6802/1056902067 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f69a804ea90 con 0x7f69940721c0 2024-03-04T16:00:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.500+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067] conn(0x7f69940721c0 msgr2=0x7f69940745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.500+0000 7f69ae18e700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067] conn(0x7f69940721c0 0x7f69940745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.500+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f699406c4e0 msgr2=0x7f699406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:14.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.500+0000 7f69ae18e700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f699406c4e0 0x7f699406e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6990009910 tx=0x7f6990005940 comp rx=0 tx=0).stop 2024-03-04T16:00:14.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.500+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a8101d60 msgr2=0x7f69a819c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:14.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.500+0000 7f69ae18e700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a8101d60 0x7f69a819c6e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f699800ba50 tx=0x7f699800be10 comp rx=0 tx=0).stop 2024-03-04T16:00:14.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.501+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 shutdown_connections 2024-03-04T16:00:14.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.501+0000 7f69ae18e700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6802/1056902067,v1:172.21.15.116:6803/1056902067] conn(0x7f69940721c0 0x7f69940745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:14.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.501+0000 7f69ae18e700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f699406c4e0 0x7f699406e9a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:14.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.501+0000 7f69ae18e700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a8101d60 0x7f69a819c6e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:14.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.501+0000 7f69ae18e700 1 --2- 172.21.15.116:0/867416028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a8102680 0x7f69a819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:14.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.501+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 >> 172.21.15.116:0/867416028 conn(0x7f69a80fb550 msgr2=0x7f69a81053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:14.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.501+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 shutdown_connections 2024-03-04T16:00:14.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:14.501+0000 7f69ae18e700 1 -- 172.21.15.116:0/867416028 wait complete. 2024-03-04T16:00:14.509 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:14.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:14 smithi152 ceph-mon[36876]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:14.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:14 smithi116 ceph-mon[29120]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:15.493 INFO:teuthology.orchestra.run.smithi116.stdout:38654705700 2024-03-04T16:00:15.493 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-04T16:00:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.873+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3727015373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4100de0 msgr2=0x7fc4e41011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.873+0000 7fc4ea4bf700 1 --2- 172.21.15.116:0/3727015373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4100de0 0x7fc4e41011c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009910 tx=0x7fc4cc009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.873+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3727015373 shutdown_connections 2024-03-04T16:00:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.873+0000 7fc4ea4bf700 1 --2- 172.21.15.116:0/3727015373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc4e4101790 0x7fc4e41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.873+0000 7fc4ea4bf700 1 --2- 172.21.15.116:0/3727015373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4100de0 0x7fc4e41011c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.873+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3727015373 >> 172.21.15.116:0/3727015373 conn(0x7fc4e40fc670 msgr2=0x7fc4e40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.873+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3727015373 shutdown_connections 2024-03-04T16:00:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.873+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3727015373 wait complete. 2024-03-04T16:00:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4ea4bf700 1 Processor -- start 2024-03-04T16:00:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4ea4bf700 1 -- start start 2024-03-04T16:00:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4ea4bf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4100de0 0x7fc4e4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4ea4bf700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc4e4101790 0x7fc4e40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4ea4bf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e4073b10 con 0x7fc4e4100de0 2024-03-04T16:00:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4ea4bf700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e4073c50 con 0x7fc4e4101790 2024-03-04T16:00:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4e3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4100de0 0x7fc4e4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4e3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4100de0 0x7fc4e4073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38688/0 (socket says 172.21.15.116:38688) 2024-03-04T16:00:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4e3fff700 1 -- 172.21.15.116:0/3166874769 learned_addr learned my addr 172.21.15.116:0/3166874769 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4e37fe700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc4e4101790 0x7fc4e40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4e3fff700 1 -- 172.21.15.116:0/3166874769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc4e4101790 msgr2=0x7fc4e40735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4e3fff700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc4e4101790 0x7fc4e40735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.874+0000 7fc4e3fff700 1 -- 172.21.15.116:0/3166874769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc0095f0 con 0x7fc4e4100de0 2024-03-04T16:00:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.875+0000 7fc4e3fff700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4100de0 0x7fc4e4073090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc006010 tx=0x7fc4cc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.875+0000 7fc4e17fa700 1 -- 172.21.15.116:0/3166874769 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4cc01d070 con 0x7fc4e4100de0 2024-03-04T16:00:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.875+0000 7fc4e17fa700 1 -- 172.21.15.116:0/3166874769 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc4cc004fa0 con 0x7fc4e4100de0 2024-03-04T16:00:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.875+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e4073ed0 con 0x7fc4e4100de0 2024-03-04T16:00:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.875+0000 7fc4e17fa700 1 -- 172.21.15.116:0/3166874769 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4cc00f690 con 0x7fc4e4100de0 2024-03-04T16:00:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.875+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e41a2cb0 con 0x7fc4e4100de0 2024-03-04T16:00:15.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.876+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc4e410e120 con 0x7fc4e4100de0 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.876+0000 7fc4e17fa700 1 -- 172.21.15.116:0/3166874769 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc4cc022470 con 0x7fc4e4100de0 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.877+0000 7fc4e17fa700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc4d006c4e0 0x7fc4d006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.877+0000 7fc4e37fe700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc4d006c4e0 0x7fc4d006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.877+0000 7fc4e17fa700 1 -- 172.21.15.116:0/3166874769 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc4cc08cba0 con 0x7fc4e4100de0 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.877+0000 7fc4e17fa700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408] conn(0x7fc4d00721c0 0x7fc4d00745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.877+0000 7fc4e17fa700 1 -- 172.21.15.116:0/3166874769 --> [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4d0074c90 con 0x7fc4d00721c0 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.877+0000 7fc4e17fa700 1 -- 172.21.15.116:0/3166874769 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4cc092280 con 0x7fc4e4100de0 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.877+0000 7fc4e37fe700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc4d006c4e0 0x7fc4d006e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4074880 tx=0x7fc4d4009190 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.877+0000 7fc4e8a5c700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408] conn(0x7fc4d00721c0 0x7fc4d00745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.878+0000 7fc4e8a5c700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408] conn(0x7fc4d00721c0 0x7fc4d00745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.878+0000 7fc4e17fa700 1 -- 172.21.15.116:0/3166874769 <== osd.5 v2:172.21.15.152:6816/1864443408 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc4d0074c90 con 0x7fc4d00721c0 2024-03-04T16:00:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.893+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 --> [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc4e404ea90 con 0x7fc4d00721c0 2024-03-04T16:00:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4e17fa700 1 -- 172.21.15.116:0/3166874769 <== osd.5 v2:172.21.15.152:6816/1864443408 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc4e404ea90 con 0x7fc4d00721c0 2024-03-04T16:00:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 >> [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408] conn(0x7fc4d00721c0 msgr2=0x7fc4d00745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408] conn(0x7fc4d00721c0 0x7fc4d00745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc4d006c4e0 msgr2=0x7fc4d006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc4d006c4e0 0x7fc4d006e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4074880 tx=0x7fc4d4009190 comp rx=0 tx=0).stop 2024-03-04T16:00:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4100de0 msgr2=0x7fc4e4073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4100de0 0x7fc4e4073090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc006010 tx=0x7fc4cc00bd70 comp rx=0 tx=0).stop 2024-03-04T16:00:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 shutdown_connections 2024-03-04T16:00:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.152:6816/1864443408,v1:172.21.15.152:6817/1864443408] conn(0x7fc4d00721c0 0x7fc4d00745e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc4d006c4e0 0x7fc4d006e9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4100de0 0x7fc4e4073090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 --2- 172.21.15.116:0/3166874769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc4e4101790 0x7fc4e40735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 >> 172.21.15.116:0/3166874769 conn(0x7fc4e40fc670 msgr2=0x7fc4e40fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 shutdown_connections 2024-03-04T16:00:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:15.894+0000 7fc4ea4bf700 1 -- 172.21.15.116:0/3166874769 wait complete. 2024-03-04T16:00:16.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:16 smithi152 ceph-mon[36876]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:16.618 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:16 smithi116 ceph-mon[29120]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:16.659 INFO:teuthology.orchestra.run.smithi116.stdout:171798691849 2024-03-04T16:00:16.659 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-04T16:00:17.027 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:17.053 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:17.070 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:18.407 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:18 smithi116 ceph-mon[29120]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:18.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:18 smithi152 ceph-mon[36876]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:19.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.540+0000 7fcd8319a700 1 -- 172.21.15.116:0/147995922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd7c1092a0 msgr2=0x7fcd7c107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:19.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.540+0000 7fcd8319a700 1 --2- 172.21.15.116:0/147995922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd7c1092a0 0x7fcd7c107d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009940 tx=0x7fcd70009c50 comp rx=0 tx=0).stop 2024-03-04T16:00:19.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.540+0000 7fcd8319a700 1 -- 172.21.15.116:0/147995922 shutdown_connections 2024-03-04T16:00:19.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.540+0000 7fcd8319a700 1 --2- 172.21.15.116:0/147995922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd7c1082d0 0x7fcd7c108730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:19.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.540+0000 7fcd8319a700 1 --2- 172.21.15.116:0/147995922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd7c1092a0 0x7fcd7c107d90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:19.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.540+0000 7fcd8319a700 1 -- 172.21.15.116:0/147995922 >> 172.21.15.116:0/147995922 conn(0x7fcd7c06d040 msgr2=0x7fcd7c06d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:19.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.541+0000 7fcd8319a700 1 -- 172.21.15.116:0/147995922 shutdown_connections 2024-03-04T16:00:19.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.541+0000 7fcd8319a700 1 -- 172.21.15.116:0/147995922 wait complete. 2024-03-04T16:00:19.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.541+0000 7fcd8319a700 1 Processor -- start 2024-03-04T16:00:19.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.541+0000 7fcd8319a700 1 -- start start 2024-03-04T16:00:19.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd8319a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd7c1082d0 0x7fcd7c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd8319a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd7c1092a0 0x7fcd7c19f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd8319a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c19fbe0 con 0x7fcd7c1082d0 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd8319a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c1a3970 con 0x7fcd7c1092a0 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd80f36700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd7c1082d0 0x7fcd7c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd80f36700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd7c1082d0 0x7fcd7c19efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38714/0 (socket says 172.21.15.116:38714) 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd80f36700 1 -- 172.21.15.116:0/2362471553 learned_addr learned my addr 172.21.15.116:0/2362471553 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd7bfff700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd7c1092a0 0x7fcd7c19f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd80f36700 1 -- 172.21.15.116:0/2362471553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd7c1092a0 msgr2=0x7fcd7c19f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd80f36700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd7c1092a0 0x7fcd7c19f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd80f36700 1 -- 172.21.15.116:0/2362471553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd700095f0 con 0x7fcd7c1082d0 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd80f36700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd7c1082d0 0x7fcd7c19efc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcd700055f0 tx=0x7fcd700056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd79ffb700 1 -- 172.21.15.116:0/2362471553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd7001d070 con 0x7fcd7c1082d0 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c1a3c50 con 0x7fcd7c1082d0 2024-03-04T16:00:19.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.542+0000 7fcd79ffb700 1 -- 172.21.15.116:0/2362471553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcd70003ba0 con 0x7fcd7c1082d0 2024-03-04T16:00:19.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.543+0000 7fcd79ffb700 1 -- 172.21.15.116:0/2362471553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd700215e0 con 0x7fcd7c1082d0 2024-03-04T16:00:19.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.543+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c1a40e0 con 0x7fcd7c1082d0 2024-03-04T16:00:19.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.544+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcd7c116890 con 0x7fcd7c1082d0 2024-03-04T16:00:19.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.544+0000 7fcd79ffb700 1 -- 172.21.15.116:0/2362471553 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fcd70003d10 con 0x7fcd7c1082d0 2024-03-04T16:00:19.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.544+0000 7fcd79ffb700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcd6c06c4e0 0x7fcd6c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:19.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.545+0000 7fcd79ffb700 1 -- 172.21.15.116:0/2362471553 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fcd7008c760 con 0x7fcd7c1082d0 2024-03-04T16:00:19.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.545+0000 7fcd7bfff700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcd6c06c4e0 0x7fcd6c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:19.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.545+0000 7fcd79ffb700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008] conn(0x7fcd6c0721c0 0x7fcd6c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:19.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.545+0000 7fcd79ffb700 1 -- 172.21.15.116:0/2362471553 --> [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd6c074c90 con 0x7fcd6c0721c0 2024-03-04T16:00:19.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.545+0000 7fcd79ffb700 1 -- 172.21.15.116:0/2362471553 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd7008ee30 con 0x7fcd7c1082d0 2024-03-04T16:00:19.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.545+0000 7fcd81737700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008] conn(0x7fcd6c0721c0 0x7fcd6c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:19.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.545+0000 7fcd7bfff700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcd6c06c4e0 0x7fcd6c06e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcd68006fd0 tx=0x7fcd68008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:19.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.545+0000 7fcd81737700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008] conn(0x7fcd6c0721c0 0x7fcd6c0745e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:19.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.546+0000 7fcd79ffb700 1 -- 172.21.15.116:0/2362471553 <== osd.1 v2:172.21.15.116:6810/2368425008 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fcd6c074c90 con 0x7fcd6c0721c0 2024-03-04T16:00:19.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.561+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 --> [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcd7c04ea90 con 0x7fcd6c0721c0 2024-03-04T16:00:19.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.561+0000 7fcd79ffb700 1 -- 172.21.15.116:0/2362471553 <== osd.1 v2:172.21.15.116:6810/2368425008 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcd7c04ea90 con 0x7fcd6c0721c0 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008] conn(0x7fcd6c0721c0 msgr2=0x7fcd6c0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008] conn(0x7fcd6c0721c0 0x7fcd6c0745e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcd6c06c4e0 msgr2=0x7fcd6c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcd6c06c4e0 0x7fcd6c06e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcd68006fd0 tx=0x7fcd68008040 comp rx=0 tx=0).stop 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd7c1082d0 msgr2=0x7fcd7c19efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd7c1082d0 0x7fcd7c19efc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcd700055f0 tx=0x7fcd700056d0 comp rx=0 tx=0).stop 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 shutdown_connections 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6810/2368425008,v1:172.21.15.116:6811/2368425008] conn(0x7fcd6c0721c0 0x7fcd6c0745e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcd6c06c4e0 0x7fcd6c06e9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd7c1082d0 0x7fcd7c19efc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 --2- 172.21.15.116:0/2362471553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd7c1092a0 0x7fcd7c19f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:19.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 >> 172.21.15.116:0/2362471553 conn(0x7fcd7c06d040 msgr2=0x7fcd7c112f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:19.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 shutdown_connections 2024-03-04T16:00:19.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:19.562+0000 7fcd8319a700 1 -- 172.21.15.116:0/2362471553 wait complete. 2024-03-04T16:00:20.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:20 smithi152 ceph-mon[36876]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:20.610 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:20 smithi116 ceph-mon[29120]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:21.630 INFO:teuthology.orchestra.run.smithi116.stdout:60129542176 2024-03-04T16:00:21.630 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-04T16:00:22.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:22 smithi152 ceph-mon[36876]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:22.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:00:22.665 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:22 smithi116 ceph-mon[29120]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:22.665 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:00:22.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.666+0000 7f5919fd9700 1 -- 172.21.15.116:0/2653117514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5914100de0 msgr2=0x7f59141011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:22.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.666+0000 7f5919fd9700 1 --2- 172.21.15.116:0/2653117514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5914100de0 0x7f59141011c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5904009910 tx=0x7f5904009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:22.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.667+0000 7f5919fd9700 1 -- 172.21.15.116:0/2653117514 shutdown_connections 2024-03-04T16:00:22.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.667+0000 7f5919fd9700 1 --2- 172.21.15.116:0/2653117514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5914101790 0x7f59141057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:22.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.667+0000 7f5919fd9700 1 --2- 172.21.15.116:0/2653117514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5914100de0 0x7f59141011c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:22.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.667+0000 7f5919fd9700 1 -- 172.21.15.116:0/2653117514 >> 172.21.15.116:0/2653117514 conn(0x7f59140fc6b0 msgr2=0x7f59140fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:22.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.667+0000 7f5919fd9700 1 -- 172.21.15.116:0/2653117514 shutdown_connections 2024-03-04T16:00:22.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.667+0000 7f5919fd9700 1 -- 172.21.15.116:0/2653117514 wait complete. 2024-03-04T16:00:22.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.667+0000 7f5919fd9700 1 Processor -- start 2024-03-04T16:00:22.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.667+0000 7f5919fd9700 1 -- start start 2024-03-04T16:00:22.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5919fd9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5914100de0 0x7f591419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:22.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5919fd9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5914101790 0x7f591419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:22.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5919fd9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591419f3a0 con 0x7f5914101790 2024-03-04T16:00:22.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5919fd9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5914198850 con 0x7f5914100de0 2024-03-04T16:00:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5912ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5914101790 0x7f591419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5912ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5914101790 0x7f591419ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38722/0 (socket says 172.21.15.116:38722) 2024-03-04T16:00:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5912ffd700 1 -- 172.21.15.116:0/926000012 learned_addr learned my addr 172.21.15.116:0/926000012 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f59137fe700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5914100de0 0x7f591419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5912ffd700 1 -- 172.21.15.116:0/926000012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5914100de0 msgr2=0x7f591419e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5912ffd700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5914100de0 0x7f591419e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5912ffd700 1 -- 172.21.15.116:0/926000012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59040095f0 con 0x7f5914101790 2024-03-04T16:00:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5912ffd700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5914101790 0x7f591419ed10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f590800b480 tx=0x7f590800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5910ff9700 1 -- 172.21.15.116:0/926000012 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5908011800 con 0x7f5914101790 2024-03-04T16:00:22.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5919fd9700 1 -- 172.21.15.116:0/926000012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5914198b30 con 0x7f5914101790 2024-03-04T16:00:22.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5910ff9700 1 -- 172.21.15.116:0/926000012 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5908011e10 con 0x7f5914101790 2024-03-04T16:00:22.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5919fd9700 1 -- 172.21.15.116:0/926000012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5914199080 con 0x7f5914101790 2024-03-04T16:00:22.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.668+0000 7f5910ff9700 1 -- 172.21.15.116:0/926000012 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5908010ea0 con 0x7f5914101790 2024-03-04T16:00:22.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.669+0000 7f5919fd9700 1 -- 172.21.15.116:0/926000012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5914104f60 con 0x7f5914101790 2024-03-04T16:00:22.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.670+0000 7f5910ff9700 1 -- 172.21.15.116:0/926000012 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5908011960 con 0x7f5914101790 2024-03-04T16:00:22.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.670+0000 7f5910ff9700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f58fc06c4e0 0x7f58fc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:22.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.670+0000 7f5910ff9700 1 -- 172.21.15.116:0/926000012 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f590808b010 con 0x7f5914101790 2024-03-04T16:00:22.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.670+0000 7f59137fe700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f58fc06c4e0 0x7f58fc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:22.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.671+0000 7f59137fe700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f58fc06c4e0 0x7f58fc06e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5904006010 tx=0x7f590401a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.675+0000 7f5910ff9700 1 -- 172.21.15.116:0/926000012 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5908059640 con 0x7f5914101790 2024-03-04T16:00:22.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.867+0000 7f5919fd9700 1 -- 172.21.15.116:0/926000012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f591404ea90 con 0x7f5914101790 2024-03-04T16:00:22.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.868+0000 7f5910ff9700 1 -- 172.21.15.116:0/926000012 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f59080591d0 con 0x7f5914101790 2024-03-04T16:00:22.870 INFO:teuthology.orchestra.run.smithi116.stdout:150323855376 2024-03-04T16:00:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 -- 172.21.15.116:0/926000012 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f58fc06c4e0 msgr2=0x7f58fc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f58fc06c4e0 0x7f58fc06e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5904006010 tx=0x7f590401a040 comp rx=0 tx=0).stop 2024-03-04T16:00:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 -- 172.21.15.116:0/926000012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5914101790 msgr2=0x7f591419ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5914101790 0x7f591419ed10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f590800b480 tx=0x7f590800b790 comp rx=0 tx=0).stop 2024-03-04T16:00:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 -- 172.21.15.116:0/926000012 shutdown_connections 2024-03-04T16:00:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5914100de0 0x7f591419e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f58fc06c4e0 0x7f58fc06e9a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 --2- 172.21.15.116:0/926000012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5914101790 0x7f591419ed10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 -- 172.21.15.116:0/926000012 >> 172.21.15.116:0/926000012 conn(0x7f59140fc6b0 msgr2=0x7f59140fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:22.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 -- 172.21.15.116:0/926000012 shutdown_connections 2024-03-04T16:00:22.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:22.870+0000 7f5919fd9700 1 -- 172.21.15.116:0/926000012 wait complete. 2024-03-04T16:00:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:23 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/926000012' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:00:23.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:23 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/926000012' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:00:24.361 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:24 smithi116 ceph-mon[29120]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.386+0000 7f4630dc8700 1 -- 172.21.15.116:0/3143879909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c105a40 msgr2=0x7f462c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.386+0000 7f4630dc8700 1 --2- 172.21.15.116:0/3143879909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c105a40 0x7f462c105e20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f461c009910 tx=0x7f461c009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.386+0000 7f4630dc8700 1 -- 172.21.15.116:0/3143879909 shutdown_connections 2024-03-04T16:00:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.386+0000 7f4630dc8700 1 --2- 172.21.15.116:0/3143879909 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f462c0ffbd0 0x7f462c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.386+0000 7f4630dc8700 1 --2- 172.21.15.116:0/3143879909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c105a40 0x7f462c105e20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.386+0000 7f4630dc8700 1 -- 172.21.15.116:0/3143879909 >> 172.21.15.116:0/3143879909 conn(0x7f462c0755a0 msgr2=0x7f462c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.386+0000 7f4630dc8700 1 -- 172.21.15.116:0/3143879909 shutdown_connections 2024-03-04T16:00:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.386+0000 7f4630dc8700 1 -- 172.21.15.116:0/3143879909 wait complete. 2024-03-04T16:00:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4630dc8700 1 Processor -- start 2024-03-04T16:00:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4630dc8700 1 -- start start 2024-03-04T16:00:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4630dc8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f462c0ffbd0 0x7f462c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4630dc8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c105a40 0x7f462c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4630dc8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c19d060 con 0x7f462c105a40 2024-03-04T16:00:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4630dc8700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c196520 con 0x7f462c0ffbd0 2024-03-04T16:00:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4629d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c105a40 0x7f462c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4629d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c105a40 0x7f462c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38734/0 (socket says 172.21.15.116:38734) 2024-03-04T16:00:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4629d9b700 1 -- 172.21.15.116:0/3253808981 learned_addr learned my addr 172.21.15.116:0/3253808981 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f462a59c700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f462c0ffbd0 0x7f462c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4629d9b700 1 -- 172.21.15.116:0/3253808981 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f462c0ffbd0 msgr2=0x7f462c19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4629d9b700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f462c0ffbd0 0x7f462c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.387+0000 7f4629d9b700 1 -- 172.21.15.116:0/3253808981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c0095f0 con 0x7f462c105a40 2024-03-04T16:00:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.388+0000 7f4629d9b700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c105a40 0x7f462c19c980 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f462000c8a0 tx=0x7f462000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.388+0000 7f461b7fe700 1 -- 172.21.15.116:0/3253808981 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f462000cea0 con 0x7f462c105a40 2024-03-04T16:00:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.388+0000 7f4630dc8700 1 -- 172.21.15.116:0/3253808981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462c196800 con 0x7f462c105a40 2024-03-04T16:00:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.388+0000 7f461b7fe700 1 -- 172.21.15.116:0/3253808981 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4620004830 con 0x7f462c105a40 2024-03-04T16:00:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.388+0000 7f461b7fe700 1 -- 172.21.15.116:0/3253808981 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46200055b0 con 0x7f462c105a40 2024-03-04T16:00:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.388+0000 7f4630dc8700 1 -- 172.21.15.116:0/3253808981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462c196d50 con 0x7f462c105a40 2024-03-04T16:00:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.389+0000 7f4630dc8700 1 -- 172.21.15.116:0/3253808981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462c109050 con 0x7f462c105a40 2024-03-04T16:00:24.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.389+0000 7f461b7fe700 1 -- 172.21.15.116:0/3253808981 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4620007ad0 con 0x7f462c105a40 2024-03-04T16:00:24.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.389+0000 7f461b7fe700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f461406c4e0 0x7f461406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:24.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.390+0000 7f461b7fe700 1 -- 172.21.15.116:0/3253808981 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f462008b240 con 0x7f462c105a40 2024-03-04T16:00:24.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.390+0000 7f462a59c700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f461406c4e0 0x7f461406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:24.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.390+0000 7f462a59c700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f461406c4e0 0x7f461406e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f461c00b5c0 tx=0x7f461c005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:24.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.394+0000 7f461b7fe700 1 -- 172.21.15.116:0/3253808981 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f46200597f0 con 0x7f462c105a40 2024-03-04T16:00:24.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.589+0000 7f4630dc8700 1 -- 172.21.15.116:0/3253808981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f462c04ea90 con 0x7f462c105a40 2024-03-04T16:00:24.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.589+0000 7f461b7fe700 1 -- 172.21.15.116:0/3253808981 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4620059380 con 0x7f462c105a40 2024-03-04T16:00:24.591 INFO:teuthology.orchestra.run.smithi116.stdout:81604378651 2024-03-04T16:00:24.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 -- 172.21.15.116:0/3253808981 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f461406c4e0 msgr2=0x7f461406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:24.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f461406c4e0 0x7f461406e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f461c00b5c0 tx=0x7f461c005960 comp rx=0 tx=0).stop 2024-03-04T16:00:24.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 -- 172.21.15.116:0/3253808981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c105a40 msgr2=0x7f462c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:24.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c105a40 0x7f462c19c980 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f462000c8a0 tx=0x7f462000cc60 comp rx=0 tx=0).stop 2024-03-04T16:00:24.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 -- 172.21.15.116:0/3253808981 shutdown_connections 2024-03-04T16:00:24.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f462c0ffbd0 0x7f462c19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:24.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f461406c4e0 0x7f461406e9a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:24.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 --2- 172.21.15.116:0/3253808981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c105a40 0x7f462c19c980 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:24.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 -- 172.21.15.116:0/3253808981 >> 172.21.15.116:0/3253808981 conn(0x7f462c0755a0 msgr2=0x7f462c0685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:24.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 -- 172.21.15.116:0/3253808981 shutdown_connections 2024-03-04T16:00:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:24.591+0000 7f4630dc8700 1 -- 172.21.15.116:0/3253808981 wait complete. 2024-03-04T16:00:24.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:24 smithi152 ceph-mon[36876]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:25.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:25 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3253808981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:00:25.611 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:25 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3253808981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:00:25.634 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378651 for osd.2 2024-03-04T16:00:25.634 DEBUG:teuthology.parallel:result is None 2024-03-04T16:00:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.132+0000 7f9a82f4c700 1 -- 172.21.15.116:0/2706024934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c1003e0 msgr2=0x7f9a7c100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.132+0000 7f9a82f4c700 1 --2- 172.21.15.116:0/2706024934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c1003e0 0x7f9a7c100860 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009910 tx=0x7f9a68009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.132+0000 7f9a82f4c700 1 -- 172.21.15.116:0/2706024934 shutdown_connections 2024-03-04T16:00:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.132+0000 7f9a82f4c700 1 --2- 172.21.15.116:0/2706024934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c1003e0 0x7f9a7c100860 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.132+0000 7f9a82f4c700 1 --2- 172.21.15.116:0/2706024934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a7c104ae0 0x7f9a7c104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.132+0000 7f9a82f4c700 1 -- 172.21.15.116:0/2706024934 >> 172.21.15.116:0/2706024934 conn(0x7f9a7c074df0 msgr2=0x7f9a7c075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.132+0000 7f9a82f4c700 1 -- 172.21.15.116:0/2706024934 shutdown_connections 2024-03-04T16:00:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.132+0000 7f9a82f4c700 1 -- 172.21.15.116:0/2706024934 wait complete. 2024-03-04T16:00:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a82f4c700 1 Processor -- start 2024-03-04T16:00:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a82f4c700 1 -- start start 2024-03-04T16:00:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a82f4c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c1003e0 0x7f9a7c1017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a82f4c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a7c104ae0 0x7f9a7c101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a82f4c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c102260 con 0x7f9a7c1003e0 2024-03-04T16:00:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a82f4c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c1023a0 con 0x7f9a7c104ae0 2024-03-04T16:00:25.999 INFO:teuthology.orchestra.run.smithi116.stdout:38654705702 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a80ce8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c1003e0 0x7f9a7c1017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a80ce8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c1003e0 0x7f9a7c1017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37500/0 (socket says 172.21.15.116:37500) 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a80ce8700 1 -- 172.21.15.116:0/274781644 learned_addr learned my addr 172.21.15.116:0/274781644 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a80ce8700 1 -- 172.21.15.116:0/274781644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a7c104ae0 msgr2=0x7f9a7c101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a80ce8700 1 --2- 172.21.15.116:0/274781644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a7c104ae0 0x7f9a7c101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.133+0000 7f9a80ce8700 1 -- 172.21.15.116:0/274781644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a680095f0 con 0x7f9a7c1003e0 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.134+0000 7f9a80ce8700 1 --2- 172.21.15.116:0/274781644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c1003e0 0x7f9a7c1017e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000c8a0 tx=0x7f9a7000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.134+0000 7f9a79ffb700 1 -- 172.21.15.116:0/274781644 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a7000cea0 con 0x7f9a7c1003e0 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.134+0000 7f9a79ffb700 1 -- 172.21.15.116:0/274781644 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9a70004830 con 0x7f9a7c1003e0 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.134+0000 7f9a82f4c700 1 -- 172.21.15.116:0/274781644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c1a29a0 con 0x7f9a7c1003e0 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.134+0000 7f9a79ffb700 1 -- 172.21.15.116:0/274781644 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a700055b0 con 0x7f9a7c1003e0 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.134+0000 7f9a82f4c700 1 -- 172.21.15.116:0/274781644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c1a2db0 con 0x7f9a7c1003e0 2024-03-04T16:00:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.135+0000 7f9a82f4c700 1 -- 172.21.15.116:0/274781644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c04ea90 con 0x7f9a7c1003e0 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.135+0000 7f9a79ffb700 1 -- 172.21.15.116:0/274781644 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f9a70007ad0 con 0x7f9a7c1003e0 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.136+0000 7f9a79ffb700 1 --2- 172.21.15.116:0/274781644 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9a6c06c4e0 0x7f9a6c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.136+0000 7f9a79ffb700 1 -- 172.21.15.116:0/274781644 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9a7008b160 con 0x7f9a7c1003e0 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.136+0000 7f9a7bfff700 1 --2- 172.21.15.116:0/274781644 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9a6c06c4e0 0x7f9a6c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.136+0000 7f9a7bfff700 1 --2- 172.21.15.116:0/274781644 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9a6c06c4e0 0x7f9a6c06e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800b5c0 tx=0x7f9a68005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.140+0000 7f9a79ffb700 1 -- 172.21.15.116:0/274781644 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9a70055d00 con 0x7f9a7c1003e0 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.337+0000 7f9a82f4c700 1 -- 172.21.15.116:0/274781644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9a7c102d10 con 0x7f9a7c1003e0 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.337+0000 7f9a79ffb700 1 -- 172.21.15.116:0/274781644 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a70059320 con 0x7f9a7c1003e0 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 -- 172.21.15.116:0/274781644 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9a6c06c4e0 msgr2=0x7f9a6c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 --2- 172.21.15.116:0/274781644 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9a6c06c4e0 0x7f9a6c06e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800b5c0 tx=0x7f9a68005960 comp rx=0 tx=0).stop 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 -- 172.21.15.116:0/274781644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c1003e0 msgr2=0x7f9a7c1017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 --2- 172.21.15.116:0/274781644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c1003e0 0x7f9a7c1017e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000c8a0 tx=0x7f9a7000cc60 comp rx=0 tx=0).stop 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 -- 172.21.15.116:0/274781644 shutdown_connections 2024-03-04T16:00:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 --2- 172.21.15.116:0/274781644 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9a6c06c4e0 0x7f9a6c06e9a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 --2- 172.21.15.116:0/274781644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c1003e0 0x7f9a7c1017e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 --2- 172.21.15.116:0/274781644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a7c104ae0 0x7f9a7c101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 -- 172.21.15.116:0/274781644 >> 172.21.15.116:0/274781644 conn(0x7f9a7c074df0 msgr2=0x7f9a7c0ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 -- 172.21.15.116:0/274781644 shutdown_connections 2024-03-04T16:00:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:25.339+0000 7f9a82f4c700 1 -- 172.21.15.116:0/274781644 wait complete. 2024-03-04T16:00:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.368+0000 7f3f164f1700 1 -- 172.21.15.116:0/1735827326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f10105a40 msgr2=0x7f3f10105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.368+0000 7f3f164f1700 1 --2- 172.21.15.116:0/1735827326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f10105a40 0x7f3f10105e20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009910 tx=0x7f3f04009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.368+0000 7f3f164f1700 1 -- 172.21.15.116:0/1735827326 shutdown_connections 2024-03-04T16:00:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.368+0000 7f3f164f1700 1 --2- 172.21.15.116:0/1735827326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f100ffbd0 0x7f3f10100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.368+0000 7f3f164f1700 1 --2- 172.21.15.116:0/1735827326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f10105a40 0x7f3f10105e20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.368+0000 7f3f164f1700 1 -- 172.21.15.116:0/1735827326 >> 172.21.15.116:0/1735827326 conn(0x7f3f100755a0 msgr2=0x7f3f100759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.368+0000 7f3f164f1700 1 -- 172.21.15.116:0/1735827326 shutdown_connections 2024-03-04T16:00:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.368+0000 7f3f164f1700 1 -- 172.21.15.116:0/1735827326 wait complete. 2024-03-04T16:00:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.368+0000 7f3f164f1700 1 Processor -- start 2024-03-04T16:00:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.368+0000 7f3f164f1700 1 -- start start 2024-03-04T16:00:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f164f1700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f100ffbd0 0x7f3f10195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f164f1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f10105a40 0x7f3f10195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f164f1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f10195ea0 con 0x7f3f10105a40 2024-03-04T16:00:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f164f1700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f10196010 con 0x7f3f100ffbd0 2024-03-04T16:00:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f0f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f10105a40 0x7f3f10195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f0f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f10105a40 0x7f3f10195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37516/0 (socket says 172.21.15.116:37516) 2024-03-04T16:00:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f0f7fe700 1 -- 172.21.15.116:0/3442616320 learned_addr learned my addr 172.21.15.116:0/3442616320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f0f7fe700 1 -- 172.21.15.116:0/3442616320 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f100ffbd0 msgr2=0x7f3f10195420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f0f7fe700 1 --2- 172.21.15.116:0/3442616320 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f100ffbd0 0x7f3f10195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f0f7fe700 1 -- 172.21.15.116:0/3442616320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f040095f0 con 0x7f3f10105a40 2024-03-04T16:00:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f0f7fe700 1 --2- 172.21.15.116:0/3442616320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f10105a40 0x7f3f10195960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000b4b0 tx=0x7f3f0000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f0d7fa700 1 -- 172.21.15.116:0/3442616320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f00011800 con 0x7f3f10105a40 2024-03-04T16:00:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f0d7fa700 1 -- 172.21.15.116:0/3442616320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3f00011e10 con 0x7f3f10105a40 2024-03-04T16:00:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f164f1700 1 -- 172.21.15.116:0/3442616320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f101a2a90 con 0x7f3f10105a40 2024-03-04T16:00:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.369+0000 7f3f0d7fa700 1 -- 172.21.15.116:0/3442616320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f0000f2e0 con 0x7f3f10105a40 2024-03-04T16:00:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.370+0000 7f3f164f1700 1 -- 172.21.15.116:0/3442616320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f101a2df0 con 0x7f3f10105a40 2024-03-04T16:00:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.370+0000 7f3f164f1700 1 -- 172.21.15.116:0/3442616320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f10108040 con 0x7f3f10105a40 2024-03-04T16:00:26.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.371+0000 7f3f0d7fa700 1 -- 172.21.15.116:0/3442616320 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3f00010380 con 0x7f3f10105a40 2024-03-04T16:00:26.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.371+0000 7f3f0d7fa700 1 --2- 172.21.15.116:0/3442616320 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3ef806c4e0 0x7f3ef806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:26.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.372+0000 7f3f0ffff700 1 --2- 172.21.15.116:0/3442616320 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3ef806c4e0 0x7f3ef806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:26.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.372+0000 7f3f0d7fa700 1 -- 172.21.15.116:0/3442616320 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3f0008b0d0 con 0x7f3f10105a40 2024-03-04T16:00:26.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.372+0000 7f3f0ffff700 1 --2- 172.21.15.116:0/3442616320 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3ef806c4e0 0x7f3ef806e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400b5c0 tx=0x7f3f04005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:26.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.376+0000 7f3f0d7fa700 1 -- 172.21.15.116:0/3442616320 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3f00059680 con 0x7f3f10105a40 2024-03-04T16:00:26.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.573+0000 7f3f164f1700 1 -- 172.21.15.116:0/3442616320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3f101961d0 con 0x7f3f10105a40 2024-03-04T16:00:26.576 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:26 smithi116 ceph-mon[29120]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:26.576 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:26 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/274781644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:00:26.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.574+0000 7f3f0d7fa700 1 -- 172.21.15.116:0/3442616320 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3f00059210 con 0x7f3f10105a40 2024-03-04T16:00:26.577 INFO:teuthology.orchestra.run.smithi116.stdout:115964117014 2024-03-04T16:00:26.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.576+0000 7f3f164f1700 1 -- 172.21.15.116:0/3442616320 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3ef806c4e0 msgr2=0x7f3ef806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:26.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.576+0000 7f3f164f1700 1 --2- 172.21.15.116:0/3442616320 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3ef806c4e0 0x7f3ef806e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400b5c0 tx=0x7f3f04005960 comp rx=0 tx=0).stop 2024-03-04T16:00:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.576+0000 7f3f164f1700 1 -- 172.21.15.116:0/3442616320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f10105a40 msgr2=0x7f3f10195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.576+0000 7f3f164f1700 1 --2- 172.21.15.116:0/3442616320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f10105a40 0x7f3f10195960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000b4b0 tx=0x7f3f0000b870 comp rx=0 tx=0).stop 2024-03-04T16:00:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.576+0000 7f3f164f1700 1 -- 172.21.15.116:0/3442616320 shutdown_connections 2024-03-04T16:00:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.576+0000 7f3f164f1700 1 --2- 172.21.15.116:0/3442616320 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f100ffbd0 0x7f3f10195420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.576+0000 7f3f164f1700 1 --2- 172.21.15.116:0/3442616320 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3ef806c4e0 0x7f3ef806e9a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.576+0000 7f3f164f1700 1 --2- 172.21.15.116:0/3442616320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f10105a40 0x7f3f10195960 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.576+0000 7f3f164f1700 1 -- 172.21.15.116:0/3442616320 >> 172.21.15.116:0/3442616320 conn(0x7f3f100755a0 msgr2=0x7f3f100fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:26.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.577+0000 7f3f164f1700 1 -- 172.21.15.116:0/3442616320 shutdown_connections 2024-03-04T16:00:26.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:26.577+0000 7f3f164f1700 1 -- 172.21.15.116:0/3442616320 wait complete. 2024-03-04T16:00:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:26 smithi152 ceph-mon[36876]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:26 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/274781644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:00:26.857 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855373 got 150323855376 for osd.4 2024-03-04T16:00:26.858 DEBUG:teuthology.parallel:result is None 2024-03-04T16:00:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.028+0000 7f6a421a8700 1 -- 172.21.15.116:0/2799768418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a3c0ff310 msgr2=0x7f6a3c0ff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.028+0000 7f6a421a8700 1 --2- 172.21.15.116:0/2799768418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a3c0ff310 0x7f6a3c0ff770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009910 tx=0x7f6a30009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.028+0000 7f6a421a8700 1 -- 172.21.15.116:0/2799768418 shutdown_connections 2024-03-04T16:00:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.028+0000 7f6a421a8700 1 --2- 172.21.15.116:0/2799768418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a3c0ff310 0x7f6a3c0ff770 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.028+0000 7f6a421a8700 1 --2- 172.21.15.116:0/2799768418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a3c106c10 0x7f6a3c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.028+0000 7f6a421a8700 1 -- 172.21.15.116:0/2799768418 >> 172.21.15.116:0/2799768418 conn(0x7f6a3c074bf0 msgr2=0x7f6a3c075000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.028+0000 7f6a421a8700 1 -- 172.21.15.116:0/2799768418 shutdown_connections 2024-03-04T16:00:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.028+0000 7f6a421a8700 1 -- 172.21.15.116:0/2799768418 wait complete. 2024-03-04T16:00:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a421a8700 1 Processor -- start 2024-03-04T16:00:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a421a8700 1 -- start start 2024-03-04T16:00:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a421a8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a3c0ff310 0x7f6a3c19c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a421a8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a3c106c10 0x7f6a3c19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a421a8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c196570 con 0x7f6a3c0ff310 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a421a8700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c1966e0 con 0x7f6a3c106c10 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a3c0ff310 0x7f6a3c19c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a3c0ff310 0x7f6a3c19c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37534/0 (socket says 172.21.15.116:37534) 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a3b7fe700 1 -- 172.21.15.116:0/3980141299 learned_addr learned my addr 172.21.15.116:0/3980141299 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a3affd700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a3c106c10 0x7f6a3c19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a3b7fe700 1 -- 172.21.15.116:0/3980141299 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a3c106c10 msgr2=0x7f6a3c19cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a3b7fe700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a3c106c10 0x7f6a3c19cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.029+0000 7f6a3b7fe700 1 -- 172.21.15.116:0/3980141299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a300095f0 con 0x7f6a3c0ff310 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.030+0000 7f6a3b7fe700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a3c0ff310 0x7f6a3c19c580 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00c8f0 tx=0x7f6a2c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.030+0000 7f6a38ff9700 1 -- 172.21.15.116:0/3980141299 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a2c007a70 con 0x7f6a3c0ff310 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.030+0000 7f6a421a8700 1 -- 172.21.15.116:0/3980141299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3c1969c0 con 0x7f6a3c0ff310 2024-03-04T16:00:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.030+0000 7f6a38ff9700 1 -- 172.21.15.116:0/3980141299 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6a2c004ce0 con 0x7f6a3c0ff310 2024-03-04T16:00:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.030+0000 7f6a38ff9700 1 -- 172.21.15.116:0/3980141299 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a2c005610 con 0x7f6a3c0ff310 2024-03-04T16:00:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.030+0000 7f6a421a8700 1 -- 172.21.15.116:0/3980141299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3c196f10 con 0x7f6a3c0ff310 2024-03-04T16:00:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.031+0000 7f6a421a8700 1 -- 172.21.15.116:0/3980141299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3c109110 con 0x7f6a3c0ff310 2024-03-04T16:00:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.031+0000 7f6a38ff9700 1 -- 172.21.15.116:0/3980141299 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6a2c004750 con 0x7f6a3c0ff310 2024-03-04T16:00:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.032+0000 7f6a38ff9700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6a2406c4e0 0x7f6a2406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.032+0000 7f6a38ff9700 1 -- 172.21.15.116:0/3980141299 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6a2c013070 con 0x7f6a3c0ff310 2024-03-04T16:00:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.032+0000 7f6a3affd700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6a2406c4e0 0x7f6a2406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.032+0000 7f6a3affd700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6a2406c4e0 0x7f6a2406e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c197e10 tx=0x7f6a3001a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:27.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.036+0000 7f6a38ff9700 1 -- 172.21.15.116:0/3980141299 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6a2c055e70 con 0x7f6a3c0ff310 2024-03-04T16:00:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.234+0000 7f6a421a8700 1 -- 172.21.15.116:0/3980141299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6a3c197190 con 0x7f6a3c0ff310 2024-03-04T16:00:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.235+0000 7f6a38ff9700 1 -- 172.21.15.116:0/3980141299 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6a2c059490 con 0x7f6a3c0ff310 2024-03-04T16:00:27.236 INFO:teuthology.orchestra.run.smithi116.stdout:171798691851 2024-03-04T16:00:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 -- 172.21.15.116:0/3980141299 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6a2406c4e0 msgr2=0x7f6a2406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6a2406c4e0 0x7f6a2406e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c197e10 tx=0x7f6a3001a040 comp rx=0 tx=0).stop 2024-03-04T16:00:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 -- 172.21.15.116:0/3980141299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a3c0ff310 msgr2=0x7f6a3c19c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a3c0ff310 0x7f6a3c19c580 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00c8f0 tx=0x7f6a2c00ccb0 comp rx=0 tx=0).stop 2024-03-04T16:00:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 -- 172.21.15.116:0/3980141299 shutdown_connections 2024-03-04T16:00:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6a2406c4e0 0x7f6a2406e9a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a3c0ff310 0x7f6a3c19c580 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 --2- 172.21.15.116:0/3980141299 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a3c106c10 0x7f6a3c19cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 -- 172.21.15.116:0/3980141299 >> 172.21.15.116:0/3980141299 conn(0x7f6a3c074bf0 msgr2=0x7f6a3c0fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 -- 172.21.15.116:0/3980141299 shutdown_connections 2024-03-04T16:00:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:27.237+0000 7f6a421a8700 1 -- 172.21.15.116:0/3980141299 wait complete. 2024-03-04T16:00:27.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:27 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3442616320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:00:27.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:27 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3980141299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:00:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:27 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3442616320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:00:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:27 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3980141299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:00:27.837 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117010 got 115964117014 for osd.3 2024-03-04T16:00:27.837 DEBUG:teuthology.parallel:result is None 2024-03-04T16:00:28.101 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:28.453 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:28 smithi116 ceph-mon[29120]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:28.473 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691849 got 171798691851 for osd.5 2024-03-04T16:00:28.473 DEBUG:teuthology.parallel:result is None 2024-03-04T16:00:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:28 smithi152 ceph-mon[36876]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:28.972 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705702 for osd.0 2024-03-04T16:00:28.972 DEBUG:teuthology.parallel:result is None 2024-03-04T16:00:30.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:30 smithi116 ceph-mon[29120]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.488+0000 7fb78fb63700 1 -- 172.21.15.116:0/2434204742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7880ff530 msgr2=0x7fb7880ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.488+0000 7fb78fb63700 1 --2- 172.21.15.116:0/2434204742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7880ff530 0x7fb7880ff910 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb778009910 tx=0x7fb778009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.488+0000 7fb78fb63700 1 -- 172.21.15.116:0/2434204742 shutdown_connections 2024-03-04T16:00:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.488+0000 7fb78fb63700 1 --2- 172.21.15.116:0/2434204742 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7880ffee0 0x7fb788105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.488+0000 7fb78fb63700 1 --2- 172.21.15.116:0/2434204742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7880ff530 0x7fb7880ff910 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.488+0000 7fb78fb63700 1 -- 172.21.15.116:0/2434204742 >> 172.21.15.116:0/2434204742 conn(0x7fb788074df0 msgr2=0x7fb788075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.488+0000 7fb78fb63700 1 -- 172.21.15.116:0/2434204742 shutdown_connections 2024-03-04T16:00:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.488+0000 7fb78fb63700 1 -- 172.21.15.116:0/2434204742 wait complete. 2024-03-04T16:00:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.489+0000 7fb78fb63700 1 Processor -- start 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.489+0000 7fb78fb63700 1 -- start start 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.489+0000 7fb78fb63700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7880ff530 0x7fb78819e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.489+0000 7fb78fb63700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7880ffee0 0x7fb78819ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.489+0000 7fb78fb63700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78819f3a0 con 0x7fb7880ff530 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.489+0000 7fb78fb63700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788198850 con 0x7fb7880ffee0 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.489+0000 7fb78d8ff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7880ff530 0x7fb78819e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.489+0000 7fb78d8ff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7880ff530 0x7fb78819e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37556/0 (socket says 172.21.15.116:37556) 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.489+0000 7fb78d8ff700 1 -- 172.21.15.116:0/2646131087 learned_addr learned my addr 172.21.15.116:0/2646131087 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.489+0000 7fb78d0fe700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7880ffee0 0x7fb78819ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.490+0000 7fb78d8ff700 1 -- 172.21.15.116:0/2646131087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7880ffee0 msgr2=0x7fb78819ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.490+0000 7fb78d8ff700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7880ffee0 0x7fb78819ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.490+0000 7fb78d8ff700 1 -- 172.21.15.116:0/2646131087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780095f0 con 0x7fb7880ff530 2024-03-04T16:00:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.490+0000 7fb78d0fe700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7880ffee0 0x7fb78819ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:00:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.490+0000 7fb78d8ff700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7880ff530 0x7fb78819e7d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb778006010 tx=0x7fb77800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.490+0000 7fb77effd700 1 -- 172.21.15.116:0/2646131087 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb77801d070 con 0x7fb7880ff530 2024-03-04T16:00:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.490+0000 7fb77effd700 1 -- 172.21.15.116:0/2646131087 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb77800f460 con 0x7fb7880ff530 2024-03-04T16:00:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.490+0000 7fb78fb63700 1 -- 172.21.15.116:0/2646131087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb788198ad0 con 0x7fb7880ff530 2024-03-04T16:00:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.490+0000 7fb77effd700 1 -- 172.21.15.116:0/2646131087 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb778021640 con 0x7fb7880ff530 2024-03-04T16:00:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.490+0000 7fb78fb63700 1 -- 172.21.15.116:0/2646131087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb788198fc0 con 0x7fb7880ff530 2024-03-04T16:00:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.491+0000 7fb78fb63700 1 -- 172.21.15.116:0/2646131087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb788104ff0 con 0x7fb7880ff530 2024-03-04T16:00:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.491+0000 7fb77effd700 1 -- 172.21.15.116:0/2646131087 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb77800faf0 con 0x7fb7880ff530 2024-03-04T16:00:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.492+0000 7fb77effd700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb77406c4e0 0x7fb77406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.492+0000 7fb78d0fe700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb77406c4e0 0x7fb77406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.492+0000 7fb77effd700 1 -- 172.21.15.116:0/2646131087 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb77808c790 con 0x7fb7880ff530 2024-03-04T16:00:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.492+0000 7fb78d0fe700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb77406c4e0 0x7fb77406e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb784005fd0 tx=0x7fb784005dc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:30.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.496+0000 7fb77effd700 1 -- 172.21.15.116:0/2646131087 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb77805ad40 con 0x7fb7880ff530 2024-03-04T16:00:30.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:30 smithi152 ceph-mon[36876]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:30.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.691+0000 7fb78fb63700 1 -- 172.21.15.116:0/2646131087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb7881049d0 con 0x7fb7880ff530 2024-03-04T16:00:30.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.691+0000 7fb77effd700 1 -- 172.21.15.116:0/2646131087 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb77805a8d0 con 0x7fb7880ff530 2024-03-04T16:00:30.693 INFO:teuthology.orchestra.run.smithi116.stdout:60129542178 2024-03-04T16:00:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.693+0000 7fb78fb63700 1 -- 172.21.15.116:0/2646131087 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb77406c4e0 msgr2=0x7fb77406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.693+0000 7fb78fb63700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb77406c4e0 0x7fb77406e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb784005fd0 tx=0x7fb784005dc0 comp rx=0 tx=0).stop 2024-03-04T16:00:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.693+0000 7fb78fb63700 1 -- 172.21.15.116:0/2646131087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7880ff530 msgr2=0x7fb78819e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.693+0000 7fb78fb63700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7880ff530 0x7fb78819e7d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb778006010 tx=0x7fb77800be20 comp rx=0 tx=0).stop 2024-03-04T16:00:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.693+0000 7fb78fb63700 1 -- 172.21.15.116:0/2646131087 shutdown_connections 2024-03-04T16:00:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.693+0000 7fb78fb63700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb77406c4e0 0x7fb77406e9a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.693+0000 7fb78fb63700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7880ff530 0x7fb78819e7d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.694+0000 7fb78fb63700 1 --2- 172.21.15.116:0/2646131087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7880ffee0 0x7fb78819ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.694+0000 7fb78fb63700 1 -- 172.21.15.116:0/2646131087 >> 172.21.15.116:0/2646131087 conn(0x7fb788074df0 msgr2=0x7fb7880fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.694+0000 7fb78fb63700 1 -- 172.21.15.116:0/2646131087 shutdown_connections 2024-03-04T16:00:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:30.694+0000 7fb78fb63700 1 -- 172.21.15.116:0/2646131087 wait complete. 2024-03-04T16:00:31.340 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542176 got 60129542178 for osd.1 2024-03-04T16:00:31.340 DEBUG:teuthology.parallel:result is None 2024-03-04T16:00:31.341 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-04T16:00:31.341 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:00:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:31.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:31 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2646131087' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:00:31.668 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:31 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2646131087' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:00:32.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:32 smithi116 ceph-mon[29120]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:32.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:32 smithi152 ceph-mon[36876]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.211+0000 7f38b212c700 1 -- 172.21.15.116:0/2861042407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac100dc0 msgr2=0x7f38ac1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.211+0000 7f38b212c700 1 --2- 172.21.15.116:0/2861042407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac100dc0 0x7f38ac1011a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f389c009910 tx=0x7f389c009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.211+0000 7f38b212c700 1 -- 172.21.15.116:0/2861042407 shutdown_connections 2024-03-04T16:00:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.211+0000 7f38b212c700 1 --2- 172.21.15.116:0/2861042407 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38ac101770 0x7f38ac1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.211+0000 7f38b212c700 1 --2- 172.21.15.116:0/2861042407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac100dc0 0x7f38ac1011a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.211+0000 7f38b212c700 1 -- 172.21.15.116:0/2861042407 >> 172.21.15.116:0/2861042407 conn(0x7f38ac0fc650 msgr2=0x7f38ac0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.211+0000 7f38b212c700 1 -- 172.21.15.116:0/2861042407 shutdown_connections 2024-03-04T16:00:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.211+0000 7f38b212c700 1 -- 172.21.15.116:0/2861042407 wait complete. 2024-03-04T16:00:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.211+0000 7f38b212c700 1 Processor -- start 2024-03-04T16:00:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.211+0000 7f38b212c700 1 -- start start 2024-03-04T16:00:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38b212c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac100dc0 0x7f38ac073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38b212c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38ac101770 0x7f38ac0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38b212c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac073b10 con 0x7f38ac100dc0 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38b212c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac073c50 con 0x7f38ac101770 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38ab7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac100dc0 0x7f38ac073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38ab7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac100dc0 0x7f38ac073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37578/0 (socket says 172.21.15.116:37578) 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38ab7fe700 1 -- 172.21.15.116:0/1048127546 learned_addr learned my addr 172.21.15.116:0/1048127546 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38aaffd700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38ac101770 0x7f38ac0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38ab7fe700 1 -- 172.21.15.116:0/1048127546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38ac101770 msgr2=0x7f38ac0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38ab7fe700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38ac101770 0x7f38ac0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38ab7fe700 1 -- 172.21.15.116:0/1048127546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c0095f0 con 0x7f38ac100dc0 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38ab7fe700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac100dc0 0x7f38ac073090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f389c006010 tx=0x7f389c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38a8ff9700 1 -- 172.21.15.116:0/1048127546 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f389c01d070 con 0x7f38ac100dc0 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38a8ff9700 1 -- 172.21.15.116:0/1048127546 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f389c00f460 con 0x7f38ac100dc0 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38b212c700 1 -- 172.21.15.116:0/1048127546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ac073ed0 con 0x7f38ac100dc0 2024-03-04T16:00:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38a8ff9700 1 -- 172.21.15.116:0/1048127546 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f389c021640 con 0x7f38ac100dc0 2024-03-04T16:00:33.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.212+0000 7f38b212c700 1 -- 172.21.15.116:0/1048127546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ac1a2d40 con 0x7f38ac100dc0 2024-03-04T16:00:33.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.213+0000 7f38b212c700 1 -- 172.21.15.116:0/1048127546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ac109160 con 0x7f38ac100dc0 2024-03-04T16:00:33.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.214+0000 7f38a8ff9700 1 -- 172.21.15.116:0/1048127546 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f389c00faf0 con 0x7f38ac100dc0 2024-03-04T16:00:33.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.214+0000 7f38a8ff9700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f389406c4e0 0x7f389406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:33.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.214+0000 7f38a8ff9700 1 -- 172.21.15.116:0/1048127546 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f389c08c820 con 0x7f38ac100dc0 2024-03-04T16:00:33.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.214+0000 7f38aaffd700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f389406c4e0 0x7f389406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:33.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.215+0000 7f38aaffd700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f389406c4e0 0x7f389406e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f38a0006fd0 tx=0x7f38a0008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:33.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.218+0000 7f38a8ff9700 1 -- 172.21.15.116:0/1048127546 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f389c057340 con 0x7f38ac100dc0 2024-03-04T16:00:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.402+0000 7f38b212c700 1 -- 172.21.15.116:0/1048127546 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38ac02ccc0 con 0x7f389406c4e0 2024-03-04T16:00:33.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.404+0000 7f38a8ff9700 1 -- 172.21.15.116:0/1048127546 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28130 (secure 0 0 0) 0x7f38ac02ccc0 con 0x7f389406c4e0 2024-03-04T16:00:33.406 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:00:33.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 -- 172.21.15.116:0/1048127546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f389406c4e0 msgr2=0x7f389406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:33.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f389406c4e0 0x7f389406e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f38a0006fd0 tx=0x7f38a0008040 comp rx=0 tx=0).stop 2024-03-04T16:00:33.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 -- 172.21.15.116:0/1048127546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac100dc0 msgr2=0x7f38ac073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:33.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac100dc0 0x7f38ac073090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f389c006010 tx=0x7f389c00be20 comp rx=0 tx=0).stop 2024-03-04T16:00:33.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 -- 172.21.15.116:0/1048127546 shutdown_connections 2024-03-04T16:00:33.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f389406c4e0 0x7f389406e9a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:33.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac100dc0 0x7f38ac073090 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:33.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 --2- 172.21.15.116:0/1048127546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38ac101770 0x7f38ac0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:33.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 -- 172.21.15.116:0/1048127546 >> 172.21.15.116:0/1048127546 conn(0x7f38ac0fc650 msgr2=0x7f38ac0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:33.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 -- 172.21.15.116:0/1048127546 shutdown_connections 2024-03-04T16:00:33.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:33.407+0000 7f38b212c700 1 -- 172.21.15.116:0/1048127546 wait complete. 2024-03-04T16:00:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-04T16:00:34.045 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-03-04T16:00:33.098656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002016"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":186,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-04T15:59:41.854086+0000","last_change":"2024-03-04T15:59:15.151143+0000","last_active":"2024-03-04T15:59:41.854086+0000","last_peered":"2024-03-04T15:59:41.854086+0000","last_clean":"2024-03-04T15:59:41.854086+0000","last_became_active":"2024-03-04T15:59:15.150733+0000","last_became_peered":"2024-03-04T15:59:15.150733+0000","last_unstale":"2024-03-04T15:59:41.854086+0000","last_undegraded":"2024-03-04T15:59:41.854086+0000","last_fullsized":"2024-03-04T15:59:41.854086+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:58:27.610108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:58:27.610108+0000","last_clean_scrub_stamp":"2024-03-04T15:58:27.610108+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:07:27.179215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Mar 4 16:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.44600000000000001}]},{"osd":2,"last update":"Mon Mar 4 16:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52600000000000002}]},{"osd":3,"last update":"Mon Mar 4 16:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.54600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 15:59:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.57999999999999996}]},{"osd":1,"last update":"Mon Mar 4 15:59:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Mon Mar 4 15:59:50 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.504},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Mar 4 16:00:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:00:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36199999999999999,"15min":0.36299999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.38200000000000001,"15min":0.38400000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Mar 4 16:00:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39700000000000002,"15min":0.39600000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.499},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39400000000000002,"15min":0.39900000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.49399999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.47999999999999998}]},{"osd":3,"last update":"Mon Mar 4 15:59:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.74299999999999999}]},{"osd":4,"last update":"Mon Mar 4 16:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.79600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 4 15:59:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.315,"15min":0.30299999999999999},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.31},"min":{"1min":0.27700000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.624}]},{"osd":2,"last update":"Mon Mar 4 15:59:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Mar 4 15:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.56899999999999995}]},{"osd":4,"last update":"Mon Mar 4 16:00:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 15:59:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.317,"15min":0.313},"min":{"1min":0.249,"5min":0.14999999999999999,"15min":0.14999999999999999},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.31900000000000001,"15min":0.314},"min":{"1min":0.25800000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.222}]},{"osd":2,"last update":"Mon Mar 4 16:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35199999999999998,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48899999999999999,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.35999999999999999,"15min":0.36499999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.505,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.25700000000000001}]},{"osd":3,"last update":"Mon Mar 4 15:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.441}]},{"osd":4,"last update":"Mon Mar 4 16:00:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.434},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:00:34.046 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:00:34.252 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:34 smithi152 ceph-mon[36876]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:34 smithi152 ceph-mon[36876]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:00:34.610 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:34 smithi116 ceph-mon[29120]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:34.610 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:34 smithi116 ceph-mon[29120]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:00:35.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.849+0000 7f0c44020700 1 -- 172.21.15.116:0/1848407663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3c101770 msgr2=0x7f0c3c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:35.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.849+0000 7f0c44020700 1 --2- 172.21.15.116:0/1848407663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3c101770 0x7f0c3c1057c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009910 tx=0x7f0c2c009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:35.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.849+0000 7f0c44020700 1 -- 172.21.15.116:0/1848407663 shutdown_connections 2024-03-04T16:00:35.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.849+0000 7f0c44020700 1 --2- 172.21.15.116:0/1848407663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3c101770 0x7f0c3c1057c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:35.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.849+0000 7f0c44020700 1 --2- 172.21.15.116:0/1848407663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c3c100dc0 0x7f0c3c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.849+0000 7f0c44020700 1 -- 172.21.15.116:0/1848407663 >> 172.21.15.116:0/1848407663 conn(0x7f0c3c0fc650 msgr2=0x7f0c3c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.849+0000 7f0c44020700 1 -- 172.21.15.116:0/1848407663 shutdown_connections 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.849+0000 7f0c44020700 1 -- 172.21.15.116:0/1848407663 wait complete. 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.849+0000 7f0c44020700 1 Processor -- start 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.849+0000 7f0c44020700 1 -- start start 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c44020700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3c100dc0 0x7f0c3c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c44020700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c3c101770 0x7f0c3c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c44020700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c073b10 con 0x7f0c3c100dc0 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c44020700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c073c50 con 0x7f0c3c101770 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c41dbc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3c100dc0 0x7f0c3c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c41dbc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3c100dc0 0x7f0c3c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45502/0 (socket says 172.21.15.116:45502) 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c41dbc700 1 -- 172.21.15.116:0/1651957284 learned_addr learned my addr 172.21.15.116:0/1651957284 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c415bb700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c3c101770 0x7f0c3c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:35.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c41dbc700 1 -- 172.21.15.116:0/1651957284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c3c101770 msgr2=0x7f0c3c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:35.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c41dbc700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c3c101770 0x7f0c3c0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:35.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c41dbc700 1 -- 172.21.15.116:0/1651957284 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c0095f0 con 0x7f0c3c100dc0 2024-03-04T16:00:35.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c41dbc700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3c100dc0 0x7f0c3c073090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800d8d0 tx=0x7f0c3800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:35.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c32ffd700 1 -- 172.21.15.116:0/1651957284 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c38009940 con 0x7f0c3c100dc0 2024-03-04T16:00:35.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c32ffd700 1 -- 172.21.15.116:0/1651957284 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0c38010460 con 0x7f0c3c100dc0 2024-03-04T16:00:35.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c44020700 1 -- 172.21.15.116:0/1651957284 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3c1a2960 con 0x7f0c3c100dc0 2024-03-04T16:00:35.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c32ffd700 1 -- 172.21.15.116:0/1651957284 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c3800f5f0 con 0x7f0c3c100dc0 2024-03-04T16:00:35.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.850+0000 7f0c44020700 1 -- 172.21.15.116:0/1651957284 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3c1a2d70 con 0x7f0c3c100dc0 2024-03-04T16:00:35.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.851+0000 7f0c44020700 1 -- 172.21.15.116:0/1651957284 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3c1091c0 con 0x7f0c3c100dc0 2024-03-04T16:00:35.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.852+0000 7f0c32ffd700 1 -- 172.21.15.116:0/1651957284 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0c380105d0 con 0x7f0c3c100dc0 2024-03-04T16:00:35.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.852+0000 7f0c32ffd700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c2806c4e0 0x7f0c2806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:35.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.852+0000 7f0c32ffd700 1 -- 172.21.15.116:0/1651957284 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0c3808b4f0 con 0x7f0c3c100dc0 2024-03-04T16:00:35.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.852+0000 7f0c415bb700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c2806c4e0 0x7f0c2806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:35.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.853+0000 7f0c415bb700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c2806c4e0 0x7f0c2806e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c074880 tx=0x7f0c2c01a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:35.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:35.857+0000 7f0c32ffd700 1 -- 172.21.15.116:0/1651957284 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0c38056010 con 0x7f0c3c100dc0 2024-03-04T16:00:36.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.040+0000 7f0c44020700 1 -- 172.21.15.116:0/1651957284 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c3c02cdd0 con 0x7f0c2806c4e0 2024-03-04T16:00:36.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.042+0000 7f0c32ffd700 1 -- 172.21.15.116:0/1651957284 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28130 (secure 0 0 0) 0x7f0c3c02cdd0 con 0x7f0c2806c4e0 2024-03-04T16:00:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:00:36.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 -- 172.21.15.116:0/1651957284 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c2806c4e0 msgr2=0x7f0c2806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:36.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c2806c4e0 0x7f0c2806e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c074880 tx=0x7f0c2c01a040 comp rx=0 tx=0).stop 2024-03-04T16:00:36.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 -- 172.21.15.116:0/1651957284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3c100dc0 msgr2=0x7f0c3c073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:36.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3c100dc0 0x7f0c3c073090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800d8d0 tx=0x7f0c3800dc90 comp rx=0 tx=0).stop 2024-03-04T16:00:36.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 -- 172.21.15.116:0/1651957284 shutdown_connections 2024-03-04T16:00:36.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c2806c4e0 0x7f0c2806e9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:36.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3c100dc0 0x7f0c3c073090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:36.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 --2- 172.21.15.116:0/1651957284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c3c101770 0x7f0c3c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:36.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 -- 172.21.15.116:0/1651957284 >> 172.21.15.116:0/1651957284 conn(0x7f0c3c0fc650 msgr2=0x7f0c3c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:36.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 -- 172.21.15.116:0/1651957284 shutdown_connections 2024-03-04T16:00:36.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:36.044+0000 7f0c44020700 1 -- 172.21.15.116:0/1651957284 wait complete. 2024-03-04T16:00:36.046 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-04T16:00:36.629 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:36 smithi116 ceph-mon[29120]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:36.651 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-03-04T16:00:35.099121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002084"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":186,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-04T15:59:41.854086+0000","last_change":"2024-03-04T15:59:15.151143+0000","last_active":"2024-03-04T15:59:41.854086+0000","last_peered":"2024-03-04T15:59:41.854086+0000","last_clean":"2024-03-04T15:59:41.854086+0000","last_became_active":"2024-03-04T15:59:15.150733+0000","last_became_peered":"2024-03-04T15:59:15.150733+0000","last_unstale":"2024-03-04T15:59:41.854086+0000","last_undegraded":"2024-03-04T15:59:41.854086+0000","last_fullsized":"2024-03-04T15:59:41.854086+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:58:27.610108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:58:27.610108+0000","last_clean_scrub_stamp":"2024-03-04T15:58:27.610108+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:07:27.179215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Mar 4 16:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.44600000000000001}]},{"osd":2,"last update":"Mon Mar 4 16:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52600000000000002}]},{"osd":3,"last update":"Mon Mar 4 16:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.54600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 15:59:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.629},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Mar 4 15:59:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Mar 4 15:59:50 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.61499999999999999}]},{"osd":4,"last update":"Mon Mar 4 16:00:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:00:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36199999999999999,"15min":0.36299999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.38200000000000001,"15min":0.38400000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Mar 4 16:00:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39700000000000002,"15min":0.39600000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.499},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39400000000000002,"15min":0.39900000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.49399999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.47999999999999998}]},{"osd":3,"last update":"Mon Mar 4 15:59:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.74299999999999999}]},{"osd":4,"last update":"Mon Mar 4 16:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.79600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 4 15:59:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.315,"15min":0.30299999999999999},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.31},"min":{"1min":0.27700000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.624}]},{"osd":2,"last update":"Mon Mar 4 15:59:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Mar 4 15:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.56899999999999995}]},{"osd":4,"last update":"Mon Mar 4 16:00:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 15:59:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.317,"15min":0.313},"min":{"1min":0.249,"5min":0.14999999999999999,"15min":0.14999999999999999},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.31900000000000001,"15min":0.314},"min":{"1min":0.25800000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.222}]},{"osd":2,"last update":"Mon Mar 4 16:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35199999999999998,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48899999999999999,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.35999999999999999,"15min":0.36499999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.505,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.25700000000000001}]},{"osd":3,"last update":"Mon Mar 4 15:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.441}]},{"osd":4,"last update":"Mon Mar 4 16:00:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.434},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:00:36.652 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-04T16:00:36.652 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-04T16:00:36.652 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-04T16:00:36.652 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph health --format=json 2024-03-04T16:00:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:36.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:36 smithi152 ceph-mon[36876]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:37 smithi116 ceph-mon[29120]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:00:37.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:37 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:00:37.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:37 smithi152 ceph-mon[36876]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:00:37.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:37 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:00:38.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.520+0000 7f2825233700 1 -- 172.21.15.116:0/3623477938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28200ffa40 msgr2=0x7f28200ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:38.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.520+0000 7f2825233700 1 --2- 172.21.15.116:0/3623477938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28200ffa40 0x7f28200ffec0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2814009910 tx=0x7f2814009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:38.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.520+0000 7f2825233700 1 -- 172.21.15.116:0/3623477938 shutdown_connections 2024-03-04T16:00:38.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.520+0000 7f2825233700 1 --2- 172.21.15.116:0/3623477938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28200ffa40 0x7f28200ffec0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:38.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.520+0000 7f2825233700 1 --2- 172.21.15.116:0/3623477938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2820105a40 0x7f2820105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:38.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.520+0000 7f2825233700 1 -- 172.21.15.116:0/3623477938 >> 172.21.15.116:0/3623477938 conn(0x7f28200fb550 msgr2=0x7f28200fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:38.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.520+0000 7f2825233700 1 -- 172.21.15.116:0/3623477938 shutdown_connections 2024-03-04T16:00:38.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.520+0000 7f2825233700 1 -- 172.21.15.116:0/3623477938 wait complete. 2024-03-04T16:00:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.521+0000 7f2825233700 1 Processor -- start 2024-03-04T16:00:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.521+0000 7f2825233700 1 -- start start 2024-03-04T16:00:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.521+0000 7f2825233700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28200ffa40 0x7f2820197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.521+0000 7f2825233700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2820105a40 0x7f2820197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.521+0000 7f2825233700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28201980f0 con 0x7f28200ffa40 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.521+0000 7f2825233700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2820198260 con 0x7f2820105a40 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f281ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28200ffa40 0x7f2820197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f281ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28200ffa40 0x7f2820197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45518/0 (socket says 172.21.15.116:45518) 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f281ed9d700 1 -- 172.21.15.116:0/2851702777 learned_addr learned my addr 172.21.15.116:0/2851702777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f281e59c700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2820105a40 0x7f2820197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f281ed9d700 1 -- 172.21.15.116:0/2851702777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2820105a40 msgr2=0x7f2820197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f281ed9d700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2820105a40 0x7f2820197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f281ed9d700 1 -- 172.21.15.116:0/2851702777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28140095f0 con 0x7f28200ffa40 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f281ed9d700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28200ffa40 0x7f2820197590 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f281000d8d0 tx=0x7f281000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f2807fff700 1 -- 172.21.15.116:0/2851702777 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28100098a0 con 0x7f28200ffa40 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f2807fff700 1 -- 172.21.15.116:0/2851702777 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2810010460 con 0x7f28200ffa40 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f2807fff700 1 -- 172.21.15.116:0/2851702777 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f281000f5f0 con 0x7f28200ffa40 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f2825233700 1 -- 172.21.15.116:0/2851702777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2820195380 con 0x7f28200ffa40 2024-03-04T16:00:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.522+0000 7f2825233700 1 -- 172.21.15.116:0/2851702777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2820195870 con 0x7f28200ffa40 2024-03-04T16:00:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.523+0000 7f2825233700 1 -- 172.21.15.116:0/2851702777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2820107fd0 con 0x7f28200ffa40 2024-03-04T16:00:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.524+0000 7f2807fff700 1 -- 172.21.15.116:0/2851702777 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f281000f750 con 0x7f28200ffa40 2024-03-04T16:00:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.524+0000 7f2807fff700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f280806c4e0 0x7f280806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.524+0000 7f281e59c700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f280806c4e0 0x7f280806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.524+0000 7f2807fff700 1 -- 172.21.15.116:0/2851702777 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f281008b310 con 0x7f28200ffa40 2024-03-04T16:00:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.524+0000 7f281e59c700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f280806c4e0 0x7f280806e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2820069680 tx=0x7f281401a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.528+0000 7f2807fff700 1 -- 172.21.15.116:0/2851702777 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2810055f20 con 0x7f28200ffa40 2024-03-04T16:00:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:38 smithi116 ceph-mon[29120]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.762+0000 7f2825233700 1 -- 172.21.15.116:0/2851702777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f282004ea90 con 0x7f28200ffa40 2024-03-04T16:00:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.762+0000 7f2807fff700 1 -- 172.21.15.116:0/2851702777 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2810059540 con 0x7f28200ffa40 2024-03-04T16:00:38.764 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:00:38.764 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T16:00:38.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 -- 172.21.15.116:0/2851702777 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f280806c4e0 msgr2=0x7f280806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f280806c4e0 0x7f280806e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2820069680 tx=0x7f281401a040 comp rx=0 tx=0).stop 2024-03-04T16:00:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 -- 172.21.15.116:0/2851702777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28200ffa40 msgr2=0x7f2820197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28200ffa40 0x7f2820197590 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f281000d8d0 tx=0x7f281000dbe0 comp rx=0 tx=0).stop 2024-03-04T16:00:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 -- 172.21.15.116:0/2851702777 shutdown_connections 2024-03-04T16:00:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f280806c4e0 0x7f280806e9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28200ffa40 0x7f2820197590 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 --2- 172.21.15.116:0/2851702777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2820105a40 0x7f2820197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 -- 172.21.15.116:0/2851702777 >> 172.21.15.116:0/2851702777 conn(0x7f28200fb550 msgr2=0x7f28200fc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 -- 172.21.15.116:0/2851702777 shutdown_connections 2024-03-04T16:00:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:38.764+0000 7f2825233700 1 -- 172.21.15.116:0/2851702777 wait complete. 2024-03-04T16:00:38.790 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:38 smithi152 ceph-mon[36876]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:39.360 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-04T16:00:39.360 INFO:tasks.cephadm:Setup complete, yielding 2024-03-04T16:00:39.360 INFO:teuthology.run_tasks:Running task print... 2024-03-04T16:00:39.371 INFO:teuthology.task.print:**** done end installing v18.2.1 cephadm ... 2024-03-04T16:00:39.371 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:00:39.402 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:00:39.402 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-04T16:00:39.617 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:39 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2851702777' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T16:00:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:39 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2851702777' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T16:00:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:40 smithi116 ceph-mon[29120]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:40.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:40 smithi152 ceph-mon[36876]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.298+0000 7f626f39b700 1 -- 172.21.15.116:0/3113801469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6268100be0 msgr2=0x7f6268101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.298+0000 7f626f39b700 1 --2- 172.21.15.116:0/3113801469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6268100be0 0x7f6268101000 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6264009910 tx=0x7f6264009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.298+0000 7f626f39b700 1 -- 172.21.15.116:0/3113801469 shutdown_connections 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.298+0000 7f626f39b700 1 --2- 172.21.15.116:0/3113801469 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6268101de0 0x7f6268102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.298+0000 7f626f39b700 1 --2- 172.21.15.116:0/3113801469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6268100be0 0x7f6268101000 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.298+0000 7f626f39b700 1 -- 172.21.15.116:0/3113801469 >> 172.21.15.116:0/3113801469 conn(0x7f62680fc140 msgr2=0x7f62680fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.298+0000 7f626f39b700 1 -- 172.21.15.116:0/3113801469 shutdown_connections 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.299+0000 7f626f39b700 1 -- 172.21.15.116:0/3113801469 wait complete. 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.299+0000 7f626f39b700 1 Processor -- start 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.299+0000 7f626f39b700 1 -- start start 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.299+0000 7f626f39b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6268100be0 0x7f6268196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.299+0000 7f626f39b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6268101de0 0x7f6268196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.299+0000 7f626f39b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6268196f70 con 0x7f6268101de0 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.299+0000 7f626f39b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62681970b0 con 0x7f6268100be0 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f626c936700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6268101de0 0x7f6268196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f626c936700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6268101de0 0x7f6268196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45530/0 (socket says 172.21.15.116:45530) 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f626c936700 1 -- 172.21.15.116:0/4178735844 learned_addr learned my addr 172.21.15.116:0/4178735844 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f626d137700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6268100be0 0x7f6268196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f626c936700 1 -- 172.21.15.116:0/4178735844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6268100be0 msgr2=0x7f6268196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f626c936700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6268100be0 0x7f6268196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f626c936700 1 -- 172.21.15.116:0/4178735844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640095f0 con 0x7f6268101de0 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f626c936700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6268101de0 0x7f6268196950 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f625800b4b0 tx=0x7f625800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f625e7fc700 1 -- 172.21.15.116:0/4178735844 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6258011800 con 0x7f6268101de0 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f626f39b700 1 -- 172.21.15.116:0/4178735844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626819bb60 con 0x7f6268101de0 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f625e7fc700 1 -- 172.21.15.116:0/4178735844 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6258011e10 con 0x7f6268101de0 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f625e7fc700 1 -- 172.21.15.116:0/4178735844 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f625800f2e0 con 0x7f6268101de0 2024-03-04T16:00:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.300+0000 7f626f39b700 1 -- 172.21.15.116:0/4178735844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626819c130 con 0x7f6268101de0 2024-03-04T16:00:41.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.302+0000 7f626f39b700 1 -- 172.21.15.116:0/4178735844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62681097c0 con 0x7f6268101de0 2024-03-04T16:00:41.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.302+0000 7f625e7fc700 1 -- 172.21.15.116:0/4178735844 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6258010380 con 0x7f6268101de0 2024-03-04T16:00:41.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.302+0000 7f625e7fc700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f625406c4e0 0x7f625406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:41.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.302+0000 7f625e7fc700 1 -- 172.21.15.116:0/4178735844 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f625808b150 con 0x7f6268101de0 2024-03-04T16:00:41.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.302+0000 7f626d137700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f625406c4e0 0x7f625406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:41.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.303+0000 7f626d137700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f625406c4e0 0x7f625406e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6264006010 tx=0x7f6264005920 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:41.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.307+0000 7f625e7fc700 1 -- 172.21.15.116:0/4178735844 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6258059700 con 0x7f6268101de0 2024-03-04T16:00:41.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.495+0000 7f626f39b700 1 -- 172.21.15.116:0/4178735844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f62681099b0 con 0x7f6268101de0 2024-03-04T16:00:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.499+0000 7f625e7fc700 1 -- 172.21.15.116:0/4178735844 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f6258059290 con 0x7f6268101de0 2024-03-04T16:00:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.501+0000 7f626f39b700 1 -- 172.21.15.116:0/4178735844 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f625406c4e0 msgr2=0x7f625406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.501+0000 7f626f39b700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f625406c4e0 0x7f625406e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6264006010 tx=0x7f6264005920 comp rx=0 tx=0).stop 2024-03-04T16:00:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.501+0000 7f626f39b700 1 -- 172.21.15.116:0/4178735844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6268101de0 msgr2=0x7f6268196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.501+0000 7f626f39b700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6268101de0 0x7f6268196950 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f625800b4b0 tx=0x7f625800b870 comp rx=0 tx=0).stop 2024-03-04T16:00:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.501+0000 7f626f39b700 1 -- 172.21.15.116:0/4178735844 shutdown_connections 2024-03-04T16:00:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.501+0000 7f626f39b700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6268100be0 0x7f6268196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.501+0000 7f626f39b700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f625406c4e0 0x7f625406e9a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.501+0000 7f626f39b700 1 --2- 172.21.15.116:0/4178735844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6268101de0 0x7f6268196950 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.501+0000 7f626f39b700 1 -- 172.21.15.116:0/4178735844 >> 172.21.15.116:0/4178735844 conn(0x7f62680fc140 msgr2=0x7f6268105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.502+0000 7f626f39b700 1 -- 172.21.15.116:0/4178735844 shutdown_connections 2024-03-04T16:00:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:41.502+0000 7f626f39b700 1 -- 172.21.15.116:0/4178735844 wait complete. 2024-03-04T16:00:42.145 INFO:teuthology.run_tasks:Running task print... 2024-03-04T16:00:42.154 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-04T16:00:42.154 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:00:42.176 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:00:42.176 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph orch status' 2024-03-04T16:00:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:42.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:42 smithi116 ceph-mon[29120]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:42.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:42 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/4178735844' entity='client.admin' 2024-03-04T16:00:42.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:42 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:42 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:42 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:42 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:00:42.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:42 smithi152 ceph-mon[36876]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:42.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:42 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/4178735844' entity='client.admin' 2024-03-04T16:00:42.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:42 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:42.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:42 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:42.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:42 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:42.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:42 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:00:44.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.141+0000 7fbc89895700 1 -- 172.21.15.116:0/296990948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc840ffab0 msgr2=0x7fbc840ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:44.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.141+0000 7fbc89895700 1 --2- 172.21.15.116:0/296990948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc840ffab0 0x7fbc840ffed0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009910 tx=0x7fbc78009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:44.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.141+0000 7fbc89895700 1 -- 172.21.15.116:0/296990948 shutdown_connections 2024-03-04T16:00:44.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.141+0000 7fbc89895700 1 --2- 172.21.15.116:0/296990948 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84100cb0 0x7fbc84101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:44.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.141+0000 7fbc89895700 1 --2- 172.21.15.116:0/296990948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc840ffab0 0x7fbc840ffed0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:44.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.141+0000 7fbc89895700 1 -- 172.21.15.116:0/296990948 >> 172.21.15.116:0/296990948 conn(0x7fbc840fb030 msgr2=0x7fbc840fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:44.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.141+0000 7fbc89895700 1 -- 172.21.15.116:0/296990948 shutdown_connections 2024-03-04T16:00:44.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.141+0000 7fbc89895700 1 -- 172.21.15.116:0/296990948 wait complete. 2024-03-04T16:00:44.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc89895700 1 Processor -- start 2024-03-04T16:00:44.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc89895700 1 -- start start 2024-03-04T16:00:44.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc89895700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc840ffab0 0x7fbc841952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:44.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc89895700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84100cb0 0x7fbc84195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:44.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc89895700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc84195e40 con 0x7fbc840ffab0 2024-03-04T16:00:44.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc89895700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc84195f80 con 0x7fbc84100cb0 2024-03-04T16:00:44.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc82ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc840ffab0 0x7fbc841952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:44.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc82ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc840ffab0 0x7fbc841952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45560/0 (socket says 172.21.15.116:45560) 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc82ffd700 1 -- 172.21.15.116:0/899616180 learned_addr learned my addr 172.21.15.116:0/899616180 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc827fc700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84100cb0 0x7fbc84195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.142+0000 7fbc82ffd700 1 -- 172.21.15.116:0/899616180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84100cb0 msgr2=0x7fbc84195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.143+0000 7fbc82ffd700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84100cb0 0x7fbc84195820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.143+0000 7fbc82ffd700 1 -- 172.21.15.116:0/899616180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc780095f0 con 0x7fbc840ffab0 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.143+0000 7fbc827fc700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84100cb0 0x7fbc84195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.143+0000 7fbc82ffd700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc840ffab0 0x7fbc841952e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800bda0 tx=0x7fbc7800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.143+0000 7fbc88893700 1 -- 172.21.15.116:0/899616180 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc7801d070 con 0x7fbc840ffab0 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.143+0000 7fbc89895700 1 -- 172.21.15.116:0/899616180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8419a9d0 con 0x7fbc840ffab0 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.143+0000 7fbc88893700 1 -- 172.21.15.116:0/899616180 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbc78004c50 con 0x7fbc840ffab0 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.143+0000 7fbc88893700 1 -- 172.21.15.116:0/899616180 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc7800f690 con 0x7fbc840ffab0 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.143+0000 7fbc89895700 1 -- 172.21.15.116:0/899616180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8419af40 con 0x7fbc840ffab0 2024-03-04T16:00:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.144+0000 7fbc89895700 1 -- 172.21.15.116:0/899616180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc84108690 con 0x7fbc840ffab0 2024-03-04T16:00:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.144+0000 7fbc88893700 1 -- 172.21.15.116:0/899616180 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fbc780229b0 con 0x7fbc840ffab0 2024-03-04T16:00:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.145+0000 7fbc88893700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbc6c06c4e0 0x7fbc6c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.145+0000 7fbc88893700 1 -- 172.21.15.116:0/899616180 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbc7808cbd0 con 0x7fbc840ffab0 2024-03-04T16:00:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.145+0000 7fbc827fc700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbc6c06c4e0 0x7fbc6c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.145+0000 7fbc827fc700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbc6c06c4e0 0x7fbc6c06e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbc74007900 tx=0x7fbc74008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:44.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.150+0000 7fbc88893700 1 -- 172.21.15.116:0/899616180 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbc7805b230 con 0x7fbc840ffab0 2024-03-04T16:00:44.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.352+0000 7fbc89895700 1 -- 172.21.15.116:0/899616180 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc84105600 con 0x7fbc6c06c4e0 2024-03-04T16:00:44.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.353+0000 7fbc88893700 1 -- 172.21.15.116:0/899616180 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fbc84105600 con 0x7fbc6c06c4e0 2024-03-04T16:00:44.355 INFO:teuthology.orchestra.run.smithi116.stdout:Backend: cephadm 2024-03-04T16:00:44.356 INFO:teuthology.orchestra.run.smithi116.stdout:Available: Yes 2024-03-04T16:00:44.356 INFO:teuthology.orchestra.run.smithi116.stdout:Paused: No 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 -- 172.21.15.116:0/899616180 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbc6c06c4e0 msgr2=0x7fbc6c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbc6c06c4e0 0x7fbc6c06e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbc74007900 tx=0x7fbc74008040 comp rx=0 tx=0).stop 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 -- 172.21.15.116:0/899616180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc840ffab0 msgr2=0x7fbc841952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc840ffab0 0x7fbc841952e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800bda0 tx=0x7fbc7800be80 comp rx=0 tx=0).stop 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 -- 172.21.15.116:0/899616180 shutdown_connections 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbc6c06c4e0 0x7fbc6c06e9a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc840ffab0 0x7fbc841952e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 --2- 172.21.15.116:0/899616180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84100cb0 0x7fbc84195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 -- 172.21.15.116:0/899616180 >> 172.21.15.116:0/899616180 conn(0x7fbc840fb030 msgr2=0x7fbc84103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 -- 172.21.15.116:0/899616180 shutdown_connections 2024-03-04T16:00:44.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:44.355+0000 7fbc89895700 1 -- 172.21.15.116:0/899616180 wait complete. 2024-03-04T16:00:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:44 smithi116 ceph-mon[29120]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:44.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:44 smithi152 ceph-mon[36876]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:45.141 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-04T16:00:45.353 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:45 smithi116 ceph-mon[29120]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:45.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:45 smithi152 ceph-mon[36876]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:46.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:46 smithi116 ceph-mon[29120]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:46.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:46 smithi152 ceph-mon[36876]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.950+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/2055269412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100bc0 msgr2=0x7f3dc4100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.950+0000 7f3dc8e3b700 1 --2- 172.21.15.116:0/2055269412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100bc0 0x7f3dc4100fe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009910 tx=0x7f3db4009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.951+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/2055269412 shutdown_connections 2024-03-04T16:00:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.951+0000 7f3dc8e3b700 1 --2- 172.21.15.116:0/2055269412 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dc4101dc0 0x7f3dc4102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.951+0000 7f3dc8e3b700 1 --2- 172.21.15.116:0/2055269412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100bc0 0x7f3dc4100fe0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.951+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/2055269412 >> 172.21.15.116:0/2055269412 conn(0x7f3dc40fc140 msgr2=0x7f3dc40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.951+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/2055269412 shutdown_connections 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.951+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/2055269412 wait complete. 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.951+0000 7f3dc8e3b700 1 Processor -- start 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.951+0000 7f3dc8e3b700 1 -- start start 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc8e3b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100bc0 0x7f3dc41963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc8e3b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dc4101dc0 0x7f3dc4196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc8e3b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4196f50 con 0x7f3dc4100bc0 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc8e3b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4197090 con 0x7f3dc4101dc0 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100bc0 0x7f3dc41963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100bc0 0x7f3dc41963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46658/0 (socket says 172.21.15.116:46658) 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc259c700 1 -- 172.21.15.116:0/3593560450 learned_addr learned my addr 172.21.15.116:0/3593560450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc1d9b700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dc4101dc0 0x7f3dc4196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc259c700 1 -- 172.21.15.116:0/3593560450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dc4101dc0 msgr2=0x7f3dc4196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc259c700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dc4101dc0 0x7f3dc4196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc259c700 1 -- 172.21.15.116:0/3593560450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db40095f0 con 0x7f3dc4100bc0 2024-03-04T16:00:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc259c700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100bc0 0x7f3dc41963f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3db400bda0 tx=0x7f3db400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3db37fe700 1 -- 172.21.15.116:0/3593560450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3db401d070 con 0x7f3dc4100bc0 2024-03-04T16:00:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/3593560450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc419bae0 con 0x7f3dc4100bc0 2024-03-04T16:00:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3db37fe700 1 -- 172.21.15.116:0/3593560450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3db4004c50 con 0x7f3dc4100bc0 2024-03-04T16:00:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3db37fe700 1 -- 172.21.15.116:0/3593560450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3db400f690 con 0x7f3dc4100bc0 2024-03-04T16:00:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.952+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/3593560450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc419c050 con 0x7f3dc4100bc0 2024-03-04T16:00:46.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.953+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/3593560450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc41097a0 con 0x7f3dc4100bc0 2024-03-04T16:00:46.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.954+0000 7f3db37fe700 1 -- 172.21.15.116:0/3593560450 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3db40229b0 con 0x7f3dc4100bc0 2024-03-04T16:00:46.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.954+0000 7f3db37fe700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3dac06c4e0 0x7f3dac06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:46.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.954+0000 7f3db37fe700 1 -- 172.21.15.116:0/3593560450 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3db408cc40 con 0x7f3dc4100bc0 2024-03-04T16:00:46.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.954+0000 7f3dc1d9b700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3dac06c4e0 0x7f3dac06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:46.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.955+0000 7f3dc1d9b700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3dac06c4e0 0x7f3dac06e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3db8006fd0 tx=0x7f3db8008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:46.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:46.959+0000 7f3db37fe700 1 -- 172.21.15.116:0/3593560450 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3db405b2a0 con 0x7f3dc4100bc0 2024-03-04T16:00:47.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.157+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/3593560450 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3dc4106710 con 0x7f3dac06c4e0 2024-03-04T16:00:47.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.167+0000 7f3db37fe700 1 -- 172.21.15.116:0/3593560450 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7f3dc4106710 con 0x7f3dac06c4e0 2024-03-04T16:00:47.169 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:00:47.169 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (3m) 2m ago 5m 26.2M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:00:47.169 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (5m) 2m ago 5m 7046k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:00:47.169 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (4m) 67s ago 4m 6903k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:00:47.169 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (5m) 2m ago 5m 6665k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:00:47.169 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (4m) 67s ago 4m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:00:47.169 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (3m) 2m ago 5m 80.5M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:00:47.169 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:9283,8765,8443 running (6m) 2m ago 6m 481M - 18.2.1 5be31c24972a 083879ca4bf4 2024-03-04T16:00:47.169 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (4m) 67s ago 4m 442M - 18.2.1 5be31c24972a c68d3e4a1dbc 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (6m) 2m ago 6m 44.0M 2048M 18.2.1 5be31c24972a 60e0bf118c4d 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (4m) 67s ago 4m 42.9M 2048M 18.2.1 5be31c24972a 52450d235516 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (5m) 2m ago 5m 15.5M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (4m) 67s ago 4m 17.3M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (3m) 2m ago 3m 42.8M 3946M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (2m) 2m ago 2m 42.2M 3946M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (2m) 2m ago 2m 40.6M 3946M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (2m) 67s ago 2m 41.1M 4970M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (98s) 67s ago 98s 38.7M 4970M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (72s) 67s ago 72s 11.4M 4970M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:00:47.170 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (3m) 2m ago 4m 40.6M - 2.43.0 a07b618ecd1d 31240d908f24 2024-03-04T16:00:47.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.169+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/3593560450 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3dac06c4e0 msgr2=0x7f3dac06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:47.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.169+0000 7f3dc8e3b700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3dac06c4e0 0x7f3dac06e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3db8006fd0 tx=0x7f3db8008040 comp rx=0 tx=0).stop 2024-03-04T16:00:47.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.169+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/3593560450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100bc0 msgr2=0x7f3dc41963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:47.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.169+0000 7f3dc8e3b700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100bc0 0x7f3dc41963f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3db400bda0 tx=0x7f3db400be80 comp rx=0 tx=0).stop 2024-03-04T16:00:47.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.169+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/3593560450 shutdown_connections 2024-03-04T16:00:47.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.169+0000 7f3dc8e3b700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f3dac06c4e0 0x7f3dac06e9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:47.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.169+0000 7f3dc8e3b700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100bc0 0x7f3dc41963f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:47.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.169+0000 7f3dc8e3b700 1 --2- 172.21.15.116:0/3593560450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dc4101dc0 0x7f3dc4196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.169+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/3593560450 >> 172.21.15.116:0/3593560450 conn(0x7f3dc40fc140 msgr2=0x7f3dc4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.169+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/3593560450 shutdown_connections 2024-03-04T16:00:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:47.170+0000 7f3dc8e3b700 1 -- 172.21.15.116:0/3593560450 wait complete. 2024-03-04T16:00:47.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:47 smithi152 ceph-mon[36876]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:47.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:47 smithi152 ceph-mon[36876]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:47.929 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:47 smithi116 ceph-mon[29120]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:47.929 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:47 smithi116 ceph-mon[29120]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:47.952 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-04T16:00:48.160 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.770+0000 7fb145a1a700 1 -- 172.21.15.116:0/1188562423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140101660 msgr2=0x7fb140103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.770+0000 7fb145a1a700 1 --2- 172.21.15.116:0/1188562423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140101660 0x7fb140103a50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb130009910 tx=0x7fb130009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.770+0000 7fb145a1a700 1 -- 172.21.15.116:0/1188562423 shutdown_connections 2024-03-04T16:00:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.770+0000 7fb145a1a700 1 --2- 172.21.15.116:0/1188562423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb140103f90 0x7fb140106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.770+0000 7fb145a1a700 1 --2- 172.21.15.116:0/1188562423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140101660 0x7fb140103a50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.770+0000 7fb145a1a700 1 -- 172.21.15.116:0/1188562423 >> 172.21.15.116:0/1188562423 conn(0x7fb1400fb050 msgr2=0x7fb1400fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.770+0000 7fb145a1a700 1 -- 172.21.15.116:0/1188562423 shutdown_connections 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.770+0000 7fb145a1a700 1 -- 172.21.15.116:0/1188562423 wait complete. 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.771+0000 7fb145a1a700 1 Processor -- start 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.771+0000 7fb145a1a700 1 -- start start 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.771+0000 7fb145a1a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140101660 0x7fb140196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.771+0000 7fb145a1a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb140103f90 0x7fb1401968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.771+0000 7fb145a1a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb140196e40 con 0x7fb140101660 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.771+0000 7fb145a1a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb140196f80 con 0x7fb140103f90 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.771+0000 7fb13effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140101660 0x7fb140196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.771+0000 7fb13effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140101660 0x7fb140196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46682/0 (socket says 172.21.15.116:46682) 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.771+0000 7fb13effd700 1 -- 172.21.15.116:0/2294830510 learned_addr learned my addr 172.21.15.116:0/2294830510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.772+0000 7fb13e7fc700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb140103f90 0x7fb1401968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.772+0000 7fb13effd700 1 -- 172.21.15.116:0/2294830510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb140103f90 msgr2=0x7fb1401968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.772+0000 7fb13effd700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb140103f90 0x7fb1401968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.772+0000 7fb13effd700 1 -- 172.21.15.116:0/2294830510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1300095f0 con 0x7fb140101660 2024-03-04T16:00:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.772+0000 7fb13effd700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140101660 0x7fb140196370 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb13000bd70 tx=0x7fb13000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.772+0000 7fb144a18700 1 -- 172.21.15.116:0/2294830510 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb13001d070 con 0x7fb140101660 2024-03-04T16:00:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.772+0000 7fb144a18700 1 -- 172.21.15.116:0/2294830510 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb130005940 con 0x7fb140101660 2024-03-04T16:00:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.772+0000 7fb144a18700 1 -- 172.21.15.116:0/2294830510 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb130021430 con 0x7fb140101660 2024-03-04T16:00:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.772+0000 7fb145a1a700 1 -- 172.21.15.116:0/2294830510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14019b9e0 con 0x7fb140101660 2024-03-04T16:00:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.772+0000 7fb145a1a700 1 -- 172.21.15.116:0/2294830510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14019bea0 con 0x7fb140101660 2024-03-04T16:00:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.773+0000 7fb145a1a700 1 -- 172.21.15.116:0/2294830510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb140190540 con 0x7fb140101660 2024-03-04T16:00:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.773+0000 7fb144a18700 1 -- 172.21.15.116:0/2294830510 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb13000f860 con 0x7fb140101660 2024-03-04T16:00:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.774+0000 7fb144a18700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb12806c4e0 0x7fb12806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.774+0000 7fb144a18700 1 -- 172.21.15.116:0/2294830510 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb13008c8e0 con 0x7fb140101660 2024-03-04T16:00:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.774+0000 7fb13e7fc700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb12806c4e0 0x7fb12806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.774+0000 7fb13e7fc700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb12806c4e0 0x7fb12806e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb13400ac50 tx=0x7fb13400a380 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:49.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.778+0000 7fb144a18700 1 -- 172.21.15.116:0/2294830510 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb13005ae90 con 0x7fb140101660 2024-03-04T16:00:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.974+0000 7fb145a1a700 1 -- 172.21.15.116:0/2294830510 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1400611d0 con 0x7fb12806c4e0 2024-03-04T16:00:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.978+0000 7fb144a18700 1 -- 172.21.15.116:0/2294830510 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1310 (secure 0 0 0) 0x7fb1400611d0 con 0x7fb12806c4e0 2024-03-04T16:00:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-04T16:00:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-04T16:00:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter 2/2 2m ago 6m * 2024-03-04T16:00:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:crash 2/2 2m ago 6m * 2024-03-04T16:00:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-04T16:00:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:mgr 2/2 2m ago 6m count:2 2024-03-04T16:00:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:mon 2/2 2m ago 5m smithi116:172.21.15.116=smithi116;smithi152:172.21.15.152=smithi152;count:2 2024-03-04T16:00:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-04T16:00:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:osd 6 2m ago - 2024-03-04T16:00:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-04T16:00:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.980+0000 7fb145a1a700 1 -- 172.21.15.116:0/2294830510 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb12806c4e0 msgr2=0x7fb12806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.980+0000 7fb145a1a700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb12806c4e0 0x7fb12806e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb13400ac50 tx=0x7fb13400a380 comp rx=0 tx=0).stop 2024-03-04T16:00:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.980+0000 7fb145a1a700 1 -- 172.21.15.116:0/2294830510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140101660 msgr2=0x7fb140196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.980+0000 7fb145a1a700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140101660 0x7fb140196370 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb13000bd70 tx=0x7fb13000bda0 comp rx=0 tx=0).stop 2024-03-04T16:00:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.980+0000 7fb145a1a700 1 -- 172.21.15.116:0/2294830510 shutdown_connections 2024-03-04T16:00:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.980+0000 7fb145a1a700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb12806c4e0 0x7fb12806e9a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.980+0000 7fb145a1a700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140101660 0x7fb140196370 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.980+0000 7fb145a1a700 1 --2- 172.21.15.116:0/2294830510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb140103f90 0x7fb1401968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.980+0000 7fb145a1a700 1 -- 172.21.15.116:0/2294830510 >> 172.21.15.116:0/2294830510 conn(0x7fb1400fb050 msgr2=0x7fb1400fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.981+0000 7fb145a1a700 1 -- 172.21.15.116:0/2294830510 shutdown_connections 2024-03-04T16:00:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:49.981+0000 7fb145a1a700 1 -- 172.21.15.116:0/2294830510 wait complete. 2024-03-04T16:00:50.551 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:50 smithi116 ceph-mon[29120]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:50.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-04T16:00:50.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:50 smithi152 ceph-mon[36876]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:51.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:51 smithi116 ceph-mon[29120]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:51.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:51 smithi152 ceph-mon[36876]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:52.444 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:52 smithi116 ceph-mon[29120]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:52.445 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:00:52.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.447+0000 7f6019f47700 1 -- 172.21.15.116:0/2502279587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100c00 msgr2=0x7f6014101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:52.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.447+0000 7f6019f47700 1 --2- 172.21.15.116:0/2502279587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100c00 0x7f6014101020 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6004009910 tx=0x7f6004009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.447+0000 7f6019f47700 1 -- 172.21.15.116:0/2502279587 shutdown_connections 2024-03-04T16:00:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.447+0000 7f6019f47700 1 --2- 172.21.15.116:0/2502279587 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6014101e00 0x7f6014102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.447+0000 7f6019f47700 1 --2- 172.21.15.116:0/2502279587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100c00 0x7f6014101020 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.447+0000 7f6019f47700 1 -- 172.21.15.116:0/2502279587 >> 172.21.15.116:0/2502279587 conn(0x7f60140fc180 msgr2=0x7f60140fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.447+0000 7f6019f47700 1 -- 172.21.15.116:0/2502279587 shutdown_connections 2024-03-04T16:00:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.447+0000 7f6019f47700 1 -- 172.21.15.116:0/2502279587 wait complete. 2024-03-04T16:00:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.447+0000 7f6019f47700 1 Processor -- start 2024-03-04T16:00:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.447+0000 7f6019f47700 1 -- start start 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f6019f47700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100c00 0x7f6014196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f6019f47700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6014101e00 0x7f60141968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f6019f47700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6014196ef0 con 0x7f6014100c00 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f6019f47700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6014197030 con 0x7f6014101e00 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f60137fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100c00 0x7f6014196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f60137fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100c00 0x7f6014196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46700/0 (socket says 172.21.15.116:46700) 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f60137fe700 1 -- 172.21.15.116:0/623572805 learned_addr learned my addr 172.21.15.116:0/623572805 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f60137fe700 1 -- 172.21.15.116:0/623572805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6014101e00 msgr2=0x7f60141968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f60137fe700 1 --2- 172.21.15.116:0/623572805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6014101e00 0x7f60141968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f60137fe700 1 -- 172.21.15.116:0/623572805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60040095f0 con 0x7f6014100c00 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f60137fe700 1 --2- 172.21.15.116:0/623572805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100c00 0x7f6014196390 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f600400bda0 tx=0x7f600400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f60117fa700 1 -- 172.21.15.116:0/623572805 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f600401d070 con 0x7f6014100c00 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f60117fa700 1 -- 172.21.15.116:0/623572805 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6004004c50 con 0x7f6014100c00 2024-03-04T16:00:52.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f6019f47700 1 -- 172.21.15.116:0/623572805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601419ba80 con 0x7f6014100c00 2024-03-04T16:00:52.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f60117fa700 1 -- 172.21.15.116:0/623572805 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f600400f690 con 0x7f6014100c00 2024-03-04T16:00:52.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.448+0000 7f6019f47700 1 -- 172.21.15.116:0/623572805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601419bf70 con 0x7f6014100c00 2024-03-04T16:00:52.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.449+0000 7f6019f47700 1 -- 172.21.15.116:0/623572805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6014109760 con 0x7f6014100c00 2024-03-04T16:00:52.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.450+0000 7f60117fa700 1 -- 172.21.15.116:0/623572805 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f60040229b0 con 0x7f6014100c00 2024-03-04T16:00:52.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.450+0000 7f60117fa700 1 --2- 172.21.15.116:0/623572805 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f600006c4e0 0x7f600006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:52.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.450+0000 7f60117fa700 1 -- 172.21.15.116:0/623572805 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f600408cc40 con 0x7f6014100c00 2024-03-04T16:00:52.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.450+0000 7f600affd700 1 --2- 172.21.15.116:0/623572805 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f600006c4e0 0x7f600006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:52.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.451+0000 7f600affd700 1 --2- 172.21.15.116:0/623572805 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f600006c4e0 0x7f600006e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00a990 tx=0x7f5ffc005c80 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:52.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.455+0000 7f60117fa700 1 -- 172.21.15.116:0/623572805 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f600405b2a0 con 0x7f6014100c00 2024-03-04T16:00:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:52 smithi152 ceph-mon[36876]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:00:52.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.648+0000 7f6019f47700 1 -- 172.21.15.116:0/623572805 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6014106750 con 0x7f600006c4e0 2024-03-04T16:00:52.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.649+0000 7f60117fa700 1 -- 172.21.15.116:0/623572805 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f6014106750 con 0x7f600006c4e0 2024-03-04T16:00:52.651 INFO:teuthology.orchestra.run.smithi116.stdout:HOST ADDR LABELS STATUS 2024-03-04T16:00:52.651 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 172.21.15.116 2024-03-04T16:00:52.651 INFO:teuthology.orchestra.run.smithi116.stdout:smithi152 172.21.15.152 2024-03-04T16:00:52.651 INFO:teuthology.orchestra.run.smithi116.stdout:2 hosts in cluster 2024-03-04T16:00:52.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.651+0000 7f6019f47700 1 -- 172.21.15.116:0/623572805 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f600006c4e0 msgr2=0x7f600006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:52.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.651+0000 7f6019f47700 1 --2- 172.21.15.116:0/623572805 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f600006c4e0 0x7f600006e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00a990 tx=0x7f5ffc005c80 comp rx=0 tx=0).stop 2024-03-04T16:00:52.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.651+0000 7f6019f47700 1 -- 172.21.15.116:0/623572805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100c00 msgr2=0x7f6014196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:52.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.651+0000 7f6019f47700 1 --2- 172.21.15.116:0/623572805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100c00 0x7f6014196390 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f600400bda0 tx=0x7f600400be80 comp rx=0 tx=0).stop 2024-03-04T16:00:52.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.651+0000 7f6019f47700 1 -- 172.21.15.116:0/623572805 shutdown_connections 2024-03-04T16:00:52.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.651+0000 7f6019f47700 1 --2- 172.21.15.116:0/623572805 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f600006c4e0 0x7f600006e9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:52.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.652+0000 7f6019f47700 1 --2- 172.21.15.116:0/623572805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100c00 0x7f6014196390 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:52.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.652+0000 7f6019f47700 1 --2- 172.21.15.116:0/623572805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6014101e00 0x7f60141968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:52.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.652+0000 7f6019f47700 1 -- 172.21.15.116:0/623572805 >> 172.21.15.116:0/623572805 conn(0x7f60140fc180 msgr2=0x7f6014105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:52.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.652+0000 7f6019f47700 1 -- 172.21.15.116:0/623572805 shutdown_connections 2024-03-04T16:00:52.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:52.652+0000 7f6019f47700 1 -- 172.21.15.116:0/623572805 wait complete. 2024-03-04T16:00:53.422 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-04T16:00:53.629 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:54.421 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:54 smithi116 ceph-mon[29120]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:54.421 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:54 smithi116 ceph-mon[29120]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:54.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:54 smithi152 ceph-mon[36876]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:54.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:54 smithi152 ceph-mon[36876]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:55.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.377+0000 7f4d30d87700 1 -- 172.21.15.116:0/850043977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d2c0ff250 msgr2=0x7f4d2c0ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:55.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.377+0000 7f4d30d87700 1 --2- 172.21.15.116:0/850043977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d2c0ff250 0x7f4d2c0ff670 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c009910 tx=0x7f4d1c009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:55.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.377+0000 7f4d30d87700 1 -- 172.21.15.116:0/850043977 shutdown_connections 2024-03-04T16:00:55.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.377+0000 7f4d30d87700 1 --2- 172.21.15.116:0/850043977 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d2c100450 0x7f4d2c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.377+0000 7f4d30d87700 1 --2- 172.21.15.116:0/850043977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d2c0ff250 0x7f4d2c0ff670 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.377+0000 7f4d30d87700 1 -- 172.21.15.116:0/850043977 >> 172.21.15.116:0/850043977 conn(0x7f4d2c0fa7f0 msgr2=0x7f4d2c0fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.377+0000 7f4d30d87700 1 -- 172.21.15.116:0/850043977 shutdown_connections 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.377+0000 7f4d30d87700 1 -- 172.21.15.116:0/850043977 wait complete. 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d30d87700 1 Processor -- start 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d30d87700 1 -- start start 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d30d87700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d2c0ff250 0x7f4d2c1021d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d30d87700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d2c100450 0x7f4d2c19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d30d87700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c19b6b0 con 0x7f4d2c100450 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d30d87700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c19b7f0 con 0x7f4d2c0ff250 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d29d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d2c100450 0x7f4d2c19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:55.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d29d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d2c100450 0x7f4d2c19b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53120/0 (socket says 172.21.15.116:53120) 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d29d9b700 1 -- 172.21.15.116:0/3312617874 learned_addr learned my addr 172.21.15.116:0/3312617874 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d2a59c700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d2c0ff250 0x7f4d2c1021d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d29d9b700 1 -- 172.21.15.116:0/3312617874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d2c0ff250 msgr2=0x7f4d2c1021d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d29d9b700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d2c0ff250 0x7f4d2c1021d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.378+0000 7f4d29d9b700 1 -- 172.21.15.116:0/3312617874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c0095f0 con 0x7f4d2c100450 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.379+0000 7f4d29d9b700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d2c100450 0x7f4d2c19b0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000d8d0 tx=0x7f4d2000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.379+0000 7f4d1b7fe700 1 -- 172.21.15.116:0/3312617874 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d20009940 con 0x7f4d2c100450 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.379+0000 7f4d1b7fe700 1 -- 172.21.15.116:0/3312617874 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4d20010460 con 0x7f4d2c100450 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.379+0000 7f4d30d87700 1 -- 172.21.15.116:0/3312617874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2c19bad0 con 0x7f4d2c100450 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.379+0000 7f4d1b7fe700 1 -- 172.21.15.116:0/3312617874 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d2000f5d0 con 0x7f4d2c100450 2024-03-04T16:00:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.379+0000 7f4d30d87700 1 -- 172.21.15.116:0/3312617874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2c19c020 con 0x7f4d2c100450 2024-03-04T16:00:55.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.380+0000 7f4d30d87700 1 -- 172.21.15.116:0/3312617874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2c107db0 con 0x7f4d2c100450 2024-03-04T16:00:55.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.380+0000 7f4d1b7fe700 1 -- 172.21.15.116:0/3312617874 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4d200105d0 con 0x7f4d2c100450 2024-03-04T16:00:55.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.381+0000 7f4d1b7fe700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f4d1406c4e0 0x7f4d1406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:55.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.381+0000 7f4d1b7fe700 1 -- 172.21.15.116:0/3312617874 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4d2008b470 con 0x7f4d2c100450 2024-03-04T16:00:55.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.381+0000 7f4d2a59c700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f4d1406c4e0 0x7f4d1406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:55.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.381+0000 7f4d2a59c700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f4d1406c4e0 0x7f4d1406e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c006010 tx=0x7f4d1c005a90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:55.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.385+0000 7f4d1b7fe700 1 -- 172.21.15.116:0/3312617874 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4d20059a20 con 0x7f4d2c100450 2024-03-04T16:00:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.580+0000 7f4d30d87700 1 -- 172.21.15.116:0/3312617874 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d2c104790 con 0x7f4d1406c4e0 2024-03-04T16:00:55.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.581+0000 7f4d1b7fe700 1 -- 172.21.15.116:0/3312617874 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f4d2c104790 con 0x7f4d1406c4e0 2024-03-04T16:00:55.583 INFO:teuthology.orchestra.run.smithi116.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-04T16:00:55.583 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-04T16:00:55.583 INFO:teuthology.orchestra.run.smithi116.stdout:smithi152 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JT400BGN 372G No 72s ago Has a FileSystem, LVM detected 2024-03-04T16:00:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 -- 172.21.15.116:0/3312617874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f4d1406c4e0 msgr2=0x7f4d1406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f4d1406c4e0 0x7f4d1406e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c006010 tx=0x7f4d1c005a90 comp rx=0 tx=0).stop 2024-03-04T16:00:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 -- 172.21.15.116:0/3312617874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d2c100450 msgr2=0x7f4d2c19b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d2c100450 0x7f4d2c19b0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000d8d0 tx=0x7f4d2000dc90 comp rx=0 tx=0).stop 2024-03-04T16:00:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 -- 172.21.15.116:0/3312617874 shutdown_connections 2024-03-04T16:00:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d2c0ff250 0x7f4d2c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f4d1406c4e0 0x7f4d1406e9a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:55.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 --2- 172.21.15.116:0/3312617874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d2c100450 0x7f4d2c19b0e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:55.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 -- 172.21.15.116:0/3312617874 >> 172.21.15.116:0/3312617874 conn(0x7f4d2c0fa7f0 msgr2=0x7f4d2c0fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:55.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 -- 172.21.15.116:0/3312617874 shutdown_connections 2024-03-04T16:00:55.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:55.584+0000 7f4d30d87700 1 -- 172.21.15.116:0/3312617874 wait complete. 2024-03-04T16:00:56.257 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:00:56.267 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:00:56.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-04T16:00:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:00:56.597 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:56 smithi116 ceph-mon[29120]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:56.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:56 smithi152 ceph-mon[36876]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:57.554 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:57 smithi116 ceph-mon[29120]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:57.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:57 smithi152 ceph-mon[36876]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.084+0000 7f97177bf700 1 -- 172.21.15.116:0/374408131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97100ffae0 msgr2=0x7f97100fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.084+0000 7f97177bf700 1 --2- 172.21.15.116:0/374408131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97100ffae0 0x7f97100fff00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f970c009910 tx=0x7f970c009c20 comp rx=0 tx=0).stop 2024-03-04T16:00:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.084+0000 7f97177bf700 1 -- 172.21.15.116:0/374408131 shutdown_connections 2024-03-04T16:00:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.084+0000 7f97177bf700 1 --2- 172.21.15.116:0/374408131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9710100ce0 0x7f9710101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.084+0000 7f97177bf700 1 --2- 172.21.15.116:0/374408131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97100ffae0 0x7f97100fff00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.084+0000 7f97177bf700 1 -- 172.21.15.116:0/374408131 >> 172.21.15.116:0/374408131 conn(0x7f97100fb080 msgr2=0x7f97100fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:00:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.084+0000 7f97177bf700 1 -- 172.21.15.116:0/374408131 shutdown_connections 2024-03-04T16:00:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.084+0000 7f97177bf700 1 -- 172.21.15.116:0/374408131 wait complete. 2024-03-04T16:00:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.084+0000 7f97177bf700 1 Processor -- start 2024-03-04T16:00:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.084+0000 7f97177bf700 1 -- start start 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f97177bf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97100ffae0 0x7f9710195290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f97177bf700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9710100ce0 0x7f97101957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f97177bf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9710195df0 con 0x7f97100ffae0 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f97177bf700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9710195f30 con 0x7f9710100ce0 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f971555b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97100ffae0 0x7f9710195290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f971555b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97100ffae0 0x7f9710195290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53146/0 (socket says 172.21.15.116:53146) 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f971555b700 1 -- 172.21.15.116:0/3632574832 learned_addr learned my addr 172.21.15.116:0/3632574832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f971555b700 1 -- 172.21.15.116:0/3632574832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9710100ce0 msgr2=0x7f97101957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f9714d5a700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9710100ce0 0x7f97101957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f971555b700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9710100ce0 0x7f97101957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f971555b700 1 -- 172.21.15.116:0/3632574832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c0095f0 con 0x7f97100ffae0 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f9714d5a700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9710100ce0 0x7f97101957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:00:58.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f971555b700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97100ffae0 0x7f9710195290 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f970c00bda0 tx=0x7f970c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f97067fc700 1 -- 172.21.15.116:0/3632574832 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f970c01d070 con 0x7f97100ffae0 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f97067fc700 1 -- 172.21.15.116:0/3632574832 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f970c004c50 con 0x7f97100ffae0 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f97177bf700 1 -- 172.21.15.116:0/3632574832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971019a980 con 0x7f97100ffae0 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f97067fc700 1 -- 172.21.15.116:0/3632574832 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f970c00f690 con 0x7f97100ffae0 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.085+0000 7f97177bf700 1 -- 172.21.15.116:0/3632574832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971019ae70 con 0x7f97100ffae0 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.086+0000 7f97177bf700 1 -- 172.21.15.116:0/3632574832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9710108640 con 0x7f97100ffae0 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.087+0000 7f97067fc700 1 -- 172.21.15.116:0/3632574832 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f970c0229b0 con 0x7f97100ffae0 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.087+0000 7f97067fc700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f96fc06c4e0 0x7f96fc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.087+0000 7f9714d5a700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f96fc06c4e0 0x7f96fc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.087+0000 7f97067fc700 1 -- 172.21.15.116:0/3632574832 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f970c08cbf0 con 0x7f97100ffae0 2024-03-04T16:00:58.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.088+0000 7f9714d5a700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f96fc06c4e0 0x7f96fc06e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9700005fd0 tx=0x7f9700005dc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:00:58.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.092+0000 7f97067fc700 1 -- 172.21.15.116:0/3632574832 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f970c05b250 con 0x7f97100ffae0 2024-03-04T16:00:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:00:58.310+0000 7f97177bf700 1 -- 172.21.15.116:0/3632574832 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9710105630 con 0x7f96fc06c4e0 2024-03-04T16:00:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:58 smithi152 ceph-mon[36876]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:58.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:58 smithi116 ceph-mon[29120]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:59.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:59 smithi152 ceph-mon[36876]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:59.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:00:59 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-04T16:00:59.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:59 smithi116 ceph-mon[29120]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:59.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:00:59 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-04T16:01:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.335+0000 7f97067fc700 1 -- 172.21.15.116:0/3632574832 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9710105630 con 0x7f96fc06c4e0 2024-03-04T16:01:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 -- 172.21.15.116:0/3632574832 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f96fc06c4e0 msgr2=0x7f96fc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f96fc06c4e0 0x7f96fc06e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9700005fd0 tx=0x7f9700005dc0 comp rx=0 tx=0).stop 2024-03-04T16:01:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 -- 172.21.15.116:0/3632574832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97100ffae0 msgr2=0x7f9710195290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97100ffae0 0x7f9710195290 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f970c00bda0 tx=0x7f970c00be80 comp rx=0 tx=0).stop 2024-03-04T16:01:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 -- 172.21.15.116:0/3632574832 shutdown_connections 2024-03-04T16:01:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f96fc06c4e0 0x7f96fc06e9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97100ffae0 0x7f9710195290 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 --2- 172.21.15.116:0/3632574832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9710100ce0 0x7f97101957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 -- 172.21.15.116:0/3632574832 >> 172.21.15.116:0/3632574832 conn(0x7f97100fb080 msgr2=0x7f9710103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 -- 172.21.15.116:0/3632574832 shutdown_connections 2024-03-04T16:01:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:00.338+0000 7f97177bf700 1 -- 172.21.15.116:0/3632574832 wait complete. 2024-03-04T16:01:00.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:00 smithi152 ceph-mon[36876]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:00.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:00 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-04T16:01:00.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:00 smithi152 ceph-mon[36876]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T16:01:00.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:00 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-04T16:01:00.677 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:00 smithi116 ceph-mon[29120]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:00.677 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:00 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116[29116]: 2024-03-04T16:01:00.312+0000 7fa135b21700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T16:01:00.677 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:00 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-04T16:01:00.677 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:00 smithi116 ceph-mon[29120]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T16:01:00.677 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:00 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-04T16:01:00.956 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-04T16:01:01.463 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: osdmap e43: 6 total, 6 up, 6 in 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: osdmap e44: 6 total, 6 up, 6 in 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: fsmap cephfs:0 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: Saving service mds.cephfs spec with placement count:4 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:01.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:01.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:01.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:01.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.pjrmzv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:01:01.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.pjrmzv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T16:01:01.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:01.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:01 smithi116 ceph-mon[29120]: Deploying daemon mds.cephfs.smithi116.pjrmzv on smithi116 2024-03-04T16:01:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-04T16:01:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: osdmap e43: 6 total, 6 up, 6 in 2024-03-04T16:01:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-04T16:01:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T16:01:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-04T16:01:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: osdmap e44: 6 total, 6 up, 6 in 2024-03-04T16:01:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: fsmap cephfs:0 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: Saving service mds.cephfs spec with placement count:4 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.pjrmzv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.pjrmzv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:01 smithi152 ceph-mon[36876]: Deploying daemon mds.cephfs.smithi116.pjrmzv on smithi116 2024-03-04T16:01:02.183 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:02.430 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:02 smithi116 ceph-mon[29120]: pgmap v179: 65 pgs: 8 creating+peering, 8 active+clean, 49 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:02.430 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:02 smithi116 ceph-mon[29120]: osdmap e45: 6 total, 6 up, 6 in 2024-03-04T16:01:02.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:02 smithi152 ceph-mon[36876]: pgmap v179: 65 pgs: 8 creating+peering, 8 active+clean, 49 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:02.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:02 smithi152 ceph-mon[36876]: osdmap e45: 6 total, 6 up, 6 in 2024-03-04T16:01:03.528 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:03 smithi116 ceph-mon[29120]: osdmap e46: 6 total, 6 up, 6 in 2024-03-04T16:01:03.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:03 smithi152 ceph-mon[36876]: osdmap e46: 6 total, 6 up, 6 in 2024-03-04T16:01:04.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.375+0000 7fb1e1019700 1 -- 172.21.15.116:0/2357142405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1dc100cb0 msgr2=0x7fb1dc101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:04.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.375+0000 7fb1e1019700 1 --2- 172.21.15.116:0/2357142405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1dc100cb0 0x7fb1dc101110 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009910 tx=0x7fb1d0009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.375+0000 7fb1e1019700 1 -- 172.21.15.116:0/2357142405 shutdown_connections 2024-03-04T16:01:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.375+0000 7fb1e1019700 1 --2- 172.21.15.116:0/2357142405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1dc100cb0 0x7fb1dc101110 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.375+0000 7fb1e1019700 1 --2- 172.21.15.116:0/2357142405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1dc0ffab0 0x7fb1dc0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.375+0000 7fb1e1019700 1 -- 172.21.15.116:0/2357142405 >> 172.21.15.116:0/2357142405 conn(0x7fb1dc0fb050 msgr2=0x7fb1dc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.375+0000 7fb1e1019700 1 -- 172.21.15.116:0/2357142405 shutdown_connections 2024-03-04T16:01:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.375+0000 7fb1e1019700 1 -- 172.21.15.116:0/2357142405 wait complete. 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1e1019700 1 Processor -- start 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1e1019700 1 -- start start 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1e1019700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1dc0ffab0 0x7fb1dc069800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1e1019700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1dc100cb0 0x7fb1dc19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1e1019700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc196f30 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1e1019700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc19da00 con 0x7fb1dc100cb0 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1dad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1dc0ffab0 0x7fb1dc069800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1dad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1dc0ffab0 0x7fb1dc069800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53174/0 (socket says 172.21.15.116:53174) 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1dad9d700 1 -- 172.21.15.116:0/263677788 learned_addr learned my addr 172.21.15.116:0/263677788 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1da59c700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1dc100cb0 0x7fb1dc19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1dad9d700 1 -- 172.21.15.116:0/263677788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1dc100cb0 msgr2=0x7fb1dc19d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1dad9d700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1dc100cb0 0x7fb1dc19d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1dad9d700 1 -- 172.21.15.116:0/263677788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d00095f0 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.376+0000 7fb1dad9d700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1dc0ffab0 0x7fb1dc069800 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400d930 tx=0x7fb1d400dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.377+0000 7fb1c3fff700 1 -- 172.21.15.116:0/263677788 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1d40041f0 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.377+0000 7fb1e1019700 1 -- 172.21.15.116:0/263677788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc19dcb0 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.377+0000 7fb1c3fff700 1 -- 172.21.15.116:0/263677788 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb1d4004350 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.377+0000 7fb1c3fff700 1 -- 172.21.15.116:0/263677788 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1d4003d20 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.377+0000 7fb1e1019700 1 -- 172.21.15.116:0/263677788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc19e200 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.378+0000 7fb1e1019700 1 -- 172.21.15.116:0/263677788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc108610 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.378+0000 7fb1c3fff700 1 -- 172.21.15.116:0/263677788 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb1d4009730 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.378+0000 7fb1c3fff700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb1c406c4e0 0x7fb1c406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:04.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.379+0000 7fb1c3fff700 1 -- 172.21.15.116:0/263677788 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fb1d408b6f0 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.379+0000 7fb1da59c700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb1c406c4e0 0x7fb1c406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:04.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.379+0000 7fb1da59c700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb1c406c4e0 0x7fb1c406e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc069d80 tx=0x7fb1d000b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.383+0000 7fb1c3fff700 1 -- 172.21.15.116:0/263677788 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb1d4059960 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.534 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:04 smithi116 ceph-mon[29120]: pgmap v182: 65 pgs: 8 creating+peering, 39 active+clean, 18 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:04.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:04 smithi152 ceph-mon[36876]: pgmap v182: 65 pgs: 8 creating+peering, 39 active+clean, 18 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:04.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.620+0000 7fb1e1019700 1 -- 172.21.15.116:0/263677788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb1dc19e870 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.621+0000 7fb1c3fff700 1 -- 172.21.15.116:0/263677788 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7fb1d4003e80 con 0x7fb1dc0ffab0 2024-03-04T16:01:04.622 INFO:teuthology.orchestra.run.smithi116.stdout:e2 2024-03-04T16:01:04.622 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:01:04.622 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 2 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:00.313822+0000 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:01:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 0 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:in 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:up {} 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:01:04.624 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 0 2024-03-04T16:01:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 -- 172.21.15.116:0/263677788 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb1c406c4e0 msgr2=0x7fb1c406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb1c406c4e0 0x7fb1c406e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc069d80 tx=0x7fb1d000b540 comp rx=0 tx=0).stop 2024-03-04T16:01:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 -- 172.21.15.116:0/263677788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1dc0ffab0 msgr2=0x7fb1dc069800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1dc0ffab0 0x7fb1dc069800 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400d930 tx=0x7fb1d400dc40 comp rx=0 tx=0).stop 2024-03-04T16:01:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 -- 172.21.15.116:0/263677788 shutdown_connections 2024-03-04T16:01:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb1c406c4e0 0x7fb1c406e9a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1dc0ffab0 0x7fb1dc069800 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 --2- 172.21.15.116:0/263677788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1dc100cb0 0x7fb1dc19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 -- 172.21.15.116:0/263677788 >> 172.21.15.116:0/263677788 conn(0x7fb1dc0fb050 msgr2=0x7fb1dc103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 -- 172.21.15.116:0/263677788 shutdown_connections 2024-03-04T16:01:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:04.624+0000 7fb1e1019700 1 -- 172.21.15.116:0/263677788 wait complete. 2024-03-04T16:01:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 2 2024-03-04T16:01:05.327 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:01:05.337 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:01:05.337 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-03-04T16:01:05.536 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:05 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/263677788' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:01:05.536 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:05 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:05.536 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:05 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:05.536 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:05 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:05.536 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:05 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.tlpshy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:01:05.537 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:05 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.tlpshy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T16:01:05.537 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:05 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:05.550 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:05.673 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:05 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/263677788' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:01:05.673 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:05 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:05.673 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:05 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:05.673 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:05 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:05.673 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:05 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.tlpshy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:01:05.673 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:05 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.tlpshy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T16:01:05.673 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:05 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:06.471 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: Deploying daemon mds.cephfs.smithi152.tlpshy on smithi152 2024-03-04T16:01:06.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: pgmap v183: 65 pgs: 8 creating+peering, 54 active+clean, 3 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:06.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.116:6826/114739798,v1:172.21.15.116:6827/114739798] up:boot 2024-03-04T16:01:06.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: daemon mds.cephfs.smithi116.pjrmzv assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-04T16:01:06.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T16:01:06.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-04T16:01:06.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: Cluster is now healthy 2024-03-04T16:01:06.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: fsmap cephfs:0 1 up:standby 2024-03-04T16:01:06.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:01:06.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:creating} 2024-03-04T16:01:06.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:06 smithi152 ceph-mon[36876]: daemon mds.cephfs.smithi116.pjrmzv is now active in filesystem cephfs as rank 0 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: Deploying daemon mds.cephfs.smithi152.tlpshy on smithi152 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: pgmap v183: 65 pgs: 8 creating+peering, 54 active+clean, 3 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.116:6826/114739798,v1:172.21.15.116:6827/114739798] up:boot 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: daemon mds.cephfs.smithi116.pjrmzv assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: Cluster is now healthy 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: fsmap cephfs:0 1 up:standby 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:creating} 2024-03-04T16:01:06.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:06 smithi116 ceph-mon[29120]: daemon mds.cephfs.smithi116.pjrmzv is now active in filesystem cephfs as rank 0 2024-03-04T16:01:07.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.143+0000 7fc2987ba700 1 -- 172.21.15.116:0/3472211896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc290100bd0 msgr2=0x7fc290100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:07.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.143+0000 7fc2987ba700 1 --2- 172.21.15.116:0/3472211896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc290100bd0 0x7fc290100ff0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009910 tx=0x7fc28c009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:07.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.143+0000 7fc2987ba700 1 -- 172.21.15.116:0/3472211896 shutdown_connections 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.143+0000 7fc2987ba700 1 --2- 172.21.15.116:0/3472211896 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc290101dd0 0x7fc290102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.143+0000 7fc2987ba700 1 --2- 172.21.15.116:0/3472211896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc290100bd0 0x7fc290100ff0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.143+0000 7fc2987ba700 1 -- 172.21.15.116:0/3472211896 >> 172.21.15.116:0/3472211896 conn(0x7fc2900fc190 msgr2=0x7fc2900fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.143+0000 7fc2987ba700 1 -- 172.21.15.116:0/3472211896 shutdown_connections 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc2987ba700 1 -- 172.21.15.116:0/3472211896 wait complete. 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc2987ba700 1 Processor -- start 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc2987ba700 1 -- start start 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc2987ba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc290100bd0 0x7fc290196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc2987ba700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc290101dd0 0x7fc2901968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc2987ba700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc290196ec0 con 0x7fc290100bd0 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc2987ba700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc290197000 con 0x7fc290101dd0 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc296556700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc290100bd0 0x7fc290196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc296556700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc290100bd0 0x7fc290196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48376/0 (socket says 172.21.15.116:48376) 2024-03-04T16:01:07.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc296556700 1 -- 172.21.15.116:0/3925670321 learned_addr learned my addr 172.21.15.116:0/3925670321 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.144+0000 7fc295d55700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc290101dd0 0x7fc2901968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.145+0000 7fc296556700 1 -- 172.21.15.116:0/3925670321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc290101dd0 msgr2=0x7fc2901968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.145+0000 7fc296556700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc290101dd0 0x7fc2901968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.145+0000 7fc296556700 1 -- 172.21.15.116:0/3925670321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0095f0 con 0x7fc290100bd0 2024-03-04T16:01:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.145+0000 7fc296556700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc290100bd0 0x7fc290196360 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc28c000c00 tx=0x7fc28c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.145+0000 7fc2877fe700 1 -- 172.21.15.116:0/3925670321 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc28c01d070 con 0x7fc290100bd0 2024-03-04T16:01:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.145+0000 7fc2987ba700 1 -- 172.21.15.116:0/3925670321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29019ba50 con 0x7fc290100bd0 2024-03-04T16:01:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.145+0000 7fc2877fe700 1 -- 172.21.15.116:0/3925670321 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc28c005dc0 con 0x7fc290100bd0 2024-03-04T16:01:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.145+0000 7fc2987ba700 1 -- 172.21.15.116:0/3925670321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29019bf40 con 0x7fc290100bd0 2024-03-04T16:01:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.145+0000 7fc2877fe700 1 -- 172.21.15.116:0/3925670321 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc28c022470 con 0x7fc290100bd0 2024-03-04T16:01:07.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.146+0000 7fc2987ba700 1 -- 172.21.15.116:0/3925670321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc290109730 con 0x7fc290100bd0 2024-03-04T16:01:07.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.146+0000 7fc2877fe700 1 -- 172.21.15.116:0/3925670321 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc28c003810 con 0x7fc290100bd0 2024-03-04T16:01:07.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.147+0000 7fc2877fe700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc27c06c4e0 0x7fc27c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:07.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.147+0000 7fc2877fe700 1 -- 172.21.15.116:0/3925670321 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fc28c08cc80 con 0x7fc290100bd0 2024-03-04T16:01:07.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.147+0000 7fc295d55700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc27c06c4e0 0x7fc27c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:07.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.147+0000 7fc295d55700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc27c06c4e0 0x7fc27c06e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc280009c60 tx=0x7fc280009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:07.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.151+0000 7fc2877fe700 1 -- 172.21.15.116:0/3925670321 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc28c05af70 con 0x7fc290100bd0 2024-03-04T16:01:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:07.397+0000 7fc2987ba700 1 -- 172.21.15.116:0/3925670321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fc29019c610 con 0x7fc290100bd0 2024-03-04T16:01:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:07 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.116:6826/114739798,v1:172.21.15.116:6827/114739798] up:active 2024-03-04T16:01:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:07 smithi152 ceph-mon[36876]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 2024-03-04T16:01:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:01:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.gkenwy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:01:07.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.gkenwy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T16:01:07.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:07 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.116:6826/114739798,v1:172.21.15.116:6827/114739798] up:active 2024-03-04T16:01:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:07 smithi116 ceph-mon[29120]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 2024-03-04T16:01:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:01:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:07.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:07.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:07.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.gkenwy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:01:07.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.gkenwy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T16:01:07.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:08.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.360+0000 7fc2877fe700 1 -- 172.21.15.116:0/3925670321 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc28c05ab00 con 0x7fc290100bd0 2024-03-04T16:01:08.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.362+0000 7fc2987ba700 1 -- 172.21.15.116:0/3925670321 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc27c06c4e0 msgr2=0x7fc27c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:08.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.362+0000 7fc2987ba700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc27c06c4e0 0x7fc27c06e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc280009c60 tx=0x7fc280009260 comp rx=0 tx=0).stop 2024-03-04T16:01:08.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.362+0000 7fc2987ba700 1 -- 172.21.15.116:0/3925670321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc290100bd0 msgr2=0x7fc290196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:08.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.362+0000 7fc2987ba700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc290100bd0 0x7fc290196360 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc28c000c00 tx=0x7fc28c00f690 comp rx=0 tx=0).stop 2024-03-04T16:01:08.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.362+0000 7fc2987ba700 1 -- 172.21.15.116:0/3925670321 shutdown_connections 2024-03-04T16:01:08.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.362+0000 7fc2987ba700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc27c06c4e0 0x7fc27c06e9a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:08.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.362+0000 7fc2987ba700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc290100bd0 0x7fc290196360 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:08.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.362+0000 7fc2987ba700 1 --2- 172.21.15.116:0/3925670321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc290101dd0 0x7fc2901968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:08.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.362+0000 7fc2987ba700 1 -- 172.21.15.116:0/3925670321 >> 172.21.15.116:0/3925670321 conn(0x7fc2900fc190 msgr2=0x7fc290105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:08.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.362+0000 7fc2987ba700 1 -- 172.21.15.116:0/3925670321 shutdown_connections 2024-03-04T16:01:08.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:08.363+0000 7fc2987ba700 1 -- 172.21.15.116:0/3925670321 wait complete. 2024-03-04T16:01:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:08 smithi152 ceph-mon[36876]: pgmap v184: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 302 B/s wr, 2 op/s 2024-03-04T16:01:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:08 smithi152 ceph-mon[36876]: Deploying daemon mds.cephfs.smithi116.gkenwy on smithi116 2024-03-04T16:01:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:08 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] up:boot 2024-03-04T16:01:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:08 smithi152 ceph-mon[36876]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 1 up:standby 2024-03-04T16:01:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:08 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:01:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:08 smithi152 ceph-mon[36876]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 1 up:standby 2024-03-04T16:01:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:08 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3925670321' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-04T16:01:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:08 smithi116 ceph-mon[29120]: pgmap v184: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 302 B/s wr, 2 op/s 2024-03-04T16:01:08.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:08 smithi116 ceph-mon[29120]: Deploying daemon mds.cephfs.smithi116.gkenwy on smithi116 2024-03-04T16:01:08.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:08 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] up:boot 2024-03-04T16:01:08.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:08 smithi116 ceph-mon[29120]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 1 up:standby 2024-03-04T16:01:08.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:08 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:01:08.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:08 smithi116 ceph-mon[29120]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 1 up:standby 2024-03-04T16:01:08.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:08 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3925670321' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-04T16:01:09.003 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:01:09.012 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:01:09.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-03-04T16:01:09.573 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:09.696 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:09 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3925670321' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-04T16:01:09.696 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:09 smithi116 ceph-mon[29120]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 1 up:standby 2024-03-04T16:01:09.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:09 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3925670321' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-04T16:01:09.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:09 smithi152 ceph-mon[36876]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 1 up:standby 2024-03-04T16:01:10.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:10 smithi116 ceph-mon[29120]: pgmap v185: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-04T16:01:10.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:10 smithi152 ceph-mon[36876]: pgmap v185: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-04T16:01:11.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 -- 172.21.15.116:0/1362405455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7188101db0 msgr2=0x7f7188102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:11.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 --2- 172.21.15.116:0/1362405455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7188101db0 0x7f7188102210 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7178009910 tx=0x7f7178009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:11.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 -- 172.21.15.116:0/1362405455 shutdown_connections 2024-03-04T16:01:11.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 --2- 172.21.15.116:0/1362405455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7188101db0 0x7f7188102210 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:11.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 --2- 172.21.15.116:0/1362405455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188100bb0 0x7f7188100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:11.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 -- 172.21.15.116:0/1362405455 >> 172.21.15.116:0/1362405455 conn(0x7f71880fc110 msgr2=0x7f71880fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:11.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 -- 172.21.15.116:0/1362405455 shutdown_connections 2024-03-04T16:01:11.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 -- 172.21.15.116:0/1362405455 wait complete. 2024-03-04T16:01:11.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 Processor -- start 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 -- start start 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188100bb0 0x7f7188196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7188101db0 0x7f7188196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7188196f60 con 0x7f7188101db0 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.348+0000 7f718ff13700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71881970a0 con 0x7f7188100bb0 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718d4ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7188101db0 0x7f7188196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718d4ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7188101db0 0x7f7188196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48394/0 (socket says 172.21.15.116:48394) 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718d4ae700 1 -- 172.21.15.116:0/2688464663 learned_addr learned my addr 172.21.15.116:0/2688464663 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718dcaf700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188100bb0 0x7f7188196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718d4ae700 1 -- 172.21.15.116:0/2688464663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188100bb0 msgr2=0x7f7188196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718d4ae700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188100bb0 0x7f7188196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718d4ae700 1 -- 172.21.15.116:0/2688464663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71780095f0 con 0x7f7188101db0 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718dcaf700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188100bb0 0x7f7188196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:01:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718d4ae700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7188101db0 0x7f7188196940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f717800bd70 tx=0x7f717800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:11.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f717effd700 1 -- 172.21.15.116:0/2688464663 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f717801d070 con 0x7f7188101db0 2024-03-04T16:01:11.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f717effd700 1 -- 172.21.15.116:0/2688464663 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7178022470 con 0x7f7188101db0 2024-03-04T16:01:11.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718ff13700 1 -- 172.21.15.116:0/2688464663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718819baf0 con 0x7f7188101db0 2024-03-04T16:01:11.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f717effd700 1 -- 172.21.15.116:0/2688464663 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f717800f670 con 0x7f7188101db0 2024-03-04T16:01:11.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.349+0000 7f718ff13700 1 -- 172.21.15.116:0/2688464663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f718819c060 con 0x7f7188101db0 2024-03-04T16:01:11.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.350+0000 7f718ff13700 1 -- 172.21.15.116:0/2688464663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71881097b0 con 0x7f7188101db0 2024-03-04T16:01:11.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.351+0000 7f717effd700 1 -- 172.21.15.116:0/2688464663 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7178022a50 con 0x7f7188101db0 2024-03-04T16:01:11.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.351+0000 7f717effd700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f717406c4e0 0x7f717406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:11.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.351+0000 7f718dcaf700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f717406c4e0 0x7f717406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:11.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.351+0000 7f717effd700 1 -- 172.21.15.116:0/2688464663 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f717808cee0 con 0x7f7188101db0 2024-03-04T16:01:11.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.352+0000 7f718dcaf700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f717406c4e0 0x7f717406e9a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7188101c10 tx=0x7f7184008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:11.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.356+0000 7f717effd700 1 -- 172.21.15.116:0/2688464663 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f717805b200 con 0x7f7188101db0 2024-03-04T16:01:11.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:11.595+0000 7f718ff13700 1 -- 172.21.15.116:0/2688464663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f718819c6a0 con 0x7f7188101db0 2024-03-04T16:01:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.373+0000 7f717effd700 1 -- 172.21.15.116:0/2688464663 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f717805ad90 con 0x7f7188101db0 2024-03-04T16:01:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 -- 172.21.15.116:0/2688464663 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f717406c4e0 msgr2=0x7f717406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f717406c4e0 0x7f717406e9a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7188101c10 tx=0x7f7184008040 comp rx=0 tx=0).stop 2024-03-04T16:01:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 -- 172.21.15.116:0/2688464663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7188101db0 msgr2=0x7f7188196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7188101db0 0x7f7188196940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f717800bd70 tx=0x7f717800be50 comp rx=0 tx=0).stop 2024-03-04T16:01:12.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 -- 172.21.15.116:0/2688464663 shutdown_connections 2024-03-04T16:01:12.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188100bb0 0x7f7188196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:12.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f717406c4e0 0x7f717406e9a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:12.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 --2- 172.21.15.116:0/2688464663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7188101db0 0x7f7188196940 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:12.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 -- 172.21.15.116:0/2688464663 >> 172.21.15.116:0/2688464663 conn(0x7f71880fc110 msgr2=0x7f7188104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:12.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 -- 172.21.15.116:0/2688464663 shutdown_connections 2024-03-04T16:01:12.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:12.376+0000 7f718ff13700 1 -- 172.21.15.116:0/2688464663 wait complete. 2024-03-04T16:01:12.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:12 smithi152 ceph-mon[36876]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-04T16:01:12.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:12 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2688464663' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-03-04T16:01:12.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:12 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:12.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:12 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:12.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:12 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:12.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:12 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.kqoybf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:01:12.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:12 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.kqoybf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T16:01:12.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:12 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:12.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:12 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:12.663 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:12 smithi116 ceph-mon[29120]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-04T16:01:12.663 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:12 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2688464663' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-03-04T16:01:12.663 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:12.663 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:12.663 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:12.663 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.kqoybf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:01:12.663 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.kqoybf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T16:01:12.663 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:12.663 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:12 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:13.106 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:01:13.117 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:01:13.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data false' 2024-03-04T16:01:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:13.641 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:13 smithi116 ceph-mon[29120]: Deploying daemon mds.cephfs.smithi152.kqoybf on smithi152 2024-03-04T16:01:13.641 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:13 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2688464663' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-03-04T16:01:13.641 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:13 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] up:boot 2024-03-04T16:01:13.642 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:13 smithi116 ceph-mon[29120]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 2 up:standby 2024-03-04T16:01:13.642 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:13 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:01:13.750 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:13 smithi152 ceph-mon[36876]: Deploying daemon mds.cephfs.smithi152.kqoybf on smithi152 2024-03-04T16:01:13.750 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:13 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2688464663' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-03-04T16:01:13.750 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:13 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] up:boot 2024-03-04T16:01:13.750 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:13 smithi152 ceph-mon[36876]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 2 up:standby 2024-03-04T16:01:13.750 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:13 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:01:14.704 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:14 smithi116 ceph-mon[29120]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-04T16:01:14.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:14 smithi152 ceph-mon[36876]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-04T16:01:15.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.042+0000 7f18e03c6700 1 -- 172.21.15.116:0/733962392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18d80feeb0 msgr2=0x7f18d80ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.042+0000 7f18e03c6700 1 --2- 172.21.15.116:0/733962392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18d80feeb0 0x7f18d80ff2d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f18d4009910 tx=0x7f18d4009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.042+0000 7f18e03c6700 1 -- 172.21.15.116:0/733962392 shutdown_connections 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.042+0000 7f18e03c6700 1 --2- 172.21.15.116:0/733962392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d8101f30 0x7f18d8102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.042+0000 7f18e03c6700 1 --2- 172.21.15.116:0/733962392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18d80feeb0 0x7f18d80ff2d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.042+0000 7f18e03c6700 1 -- 172.21.15.116:0/733962392 >> 172.21.15.116:0/733962392 conn(0x7f18d80fa7f0 msgr2=0x7f18d80fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.042+0000 7f18e03c6700 1 -- 172.21.15.116:0/733962392 shutdown_connections 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.042+0000 7f18e03c6700 1 -- 172.21.15.116:0/733962392 wait complete. 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18e03c6700 1 Processor -- start 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18e03c6700 1 -- start start 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18e03c6700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d80feeb0 0x7f18d8196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18e03c6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18d8101f30 0x7f18d81968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18e03c6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8196ee0 con 0x7f18d8101f30 2024-03-04T16:01:15.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18e03c6700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8197020 con 0x7f18d80feeb0 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18dd961700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18d8101f30 0x7f18d81968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18dd961700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18d8101f30 0x7f18d81968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57366/0 (socket says 172.21.15.116:57366) 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18dd961700 1 -- 172.21.15.116:0/279482234 learned_addr learned my addr 172.21.15.116:0/279482234 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18de162700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d80feeb0 0x7f18d8196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18dd961700 1 -- 172.21.15.116:0/279482234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d80feeb0 msgr2=0x7f18d8196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18dd961700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d80feeb0 0x7f18d8196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18dd961700 1 -- 172.21.15.116:0/279482234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d40095f0 con 0x7f18d8101f30 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18de162700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d80feeb0 0x7f18d8196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.043+0000 7f18dd961700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18d8101f30 0x7f18d81968c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f18c800d8d0 tx=0x7f18c800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.044+0000 7f18cf7fe700 1 -- 172.21.15.116:0/279482234 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18c8009940 con 0x7f18d8101f30 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.044+0000 7f18cf7fe700 1 -- 172.21.15.116:0/279482234 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f18c8010460 con 0x7f18d8101f30 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.044+0000 7f18e03c6700 1 -- 172.21.15.116:0/279482234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d819bad0 con 0x7f18d8101f30 2024-03-04T16:01:15.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.044+0000 7f18cf7fe700 1 -- 172.21.15.116:0/279482234 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18c800f5f0 con 0x7f18d8101f30 2024-03-04T16:01:15.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.044+0000 7f18e03c6700 1 -- 172.21.15.116:0/279482234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d819c020 con 0x7f18d8101f30 2024-03-04T16:01:15.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.045+0000 7f18e03c6700 1 -- 172.21.15.116:0/279482234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d8109730 con 0x7f18d8101f30 2024-03-04T16:01:15.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.045+0000 7f18cf7fe700 1 -- 172.21.15.116:0/279482234 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f18c80105d0 con 0x7f18d8101f30 2024-03-04T16:01:15.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.046+0000 7f18cf7fe700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f18c406c4e0 0x7f18c406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:15.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.046+0000 7f18cf7fe700 1 -- 172.21.15.116:0/279482234 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f18c808b830 con 0x7f18d8101f30 2024-03-04T16:01:15.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.046+0000 7f18de162700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f18c406c4e0 0x7f18c406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:15.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.046+0000 7f18de162700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f18c406c4e0 0x7f18c406e9a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f18d4006010 tx=0x7f18d4005a90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:15.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.050+0000 7f18cf7fe700 1 -- 172.21.15.116:0/279482234 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f18c8059aa0 con 0x7f18d8101f30 2024-03-04T16:01:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.308+0000 7f18e03c6700 1 -- 172.21.15.116:0/279482234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7f18d8109920 con 0x7f18d8101f30 2024-03-04T16:01:15.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.386+0000 7f18cf7fe700 1 -- 172.21.15.116:0/279482234 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v10) v1 ==== 133+0+0 (secure 0 0 0) 0x7f18c8059630 con 0x7f18d8101f30 2024-03-04T16:01:15.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 -- 172.21.15.116:0/279482234 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f18c406c4e0 msgr2=0x7f18c406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:15.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f18c406c4e0 0x7f18c406e9a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f18d4006010 tx=0x7f18d4005a90 comp rx=0 tx=0).stop 2024-03-04T16:01:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 -- 172.21.15.116:0/279482234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18d8101f30 msgr2=0x7f18d81968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18d8101f30 0x7f18d81968c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f18c800d8d0 tx=0x7f18c800dc90 comp rx=0 tx=0).stop 2024-03-04T16:01:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 -- 172.21.15.116:0/279482234 shutdown_connections 2024-03-04T16:01:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d80feeb0 0x7f18d8196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f18c406c4e0 0x7f18c406e9a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 --2- 172.21.15.116:0/279482234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18d8101f30 0x7f18d81968c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 -- 172.21.15.116:0/279482234 >> 172.21.15.116:0/279482234 conn(0x7f18d80fa7f0 msgr2=0x7f18d8105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 -- 172.21.15.116:0/279482234 shutdown_connections 2024-03-04T16:01:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:15.389+0000 7f18e03c6700 1 -- 172.21.15.116:0/279482234 wait complete. 2024-03-04T16:01:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:inline data disabled 2024-03-04T16:01:15.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:15.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:15.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:15.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:15.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/279482234' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] up:boot 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] up:standby 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/279482234' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: Dropping low affinity active daemon mds.cephfs.smithi116.pjrmzv in favor of higher affinity standby. 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: Replacing daemon mds.cephfs.smithi116.pjrmzv as rank 0 with standby daemon mds.cephfs.smithi152.tlpshy 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 3 up:standby 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: osdmap e47: 6 total, 6 up, 6 in 2024-03-04T16:01:15.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:15 smithi152 ceph-mon[36876]: fsmap cephfs:1/1 {0=cephfs.smithi152.tlpshy=up:replay} 2 up:standby 2024-03-04T16:01:15.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/279482234' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] up:boot 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] up:standby 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/279482234' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: Dropping low affinity active daemon mds.cephfs.smithi116.pjrmzv in favor of higher affinity standby. 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: Replacing daemon mds.cephfs.smithi116.pjrmzv as rank 0 with standby daemon mds.cephfs.smithi152.tlpshy 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T16:01:15.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 3 up:standby 2024-03-04T16:01:15.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:01:15.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: osdmap e47: 6 total, 6 up, 6 in 2024-03-04T16:01:15.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:15 smithi116 ceph-mon[29120]: fsmap cephfs:1/1 {0=cephfs.smithi152.tlpshy=up:replay} 2 up:standby 2024-03-04T16:01:16.369 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:01:16.379 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:01:16.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-04T16:01:17.250 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:17.710 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:17 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] up:reconnect 2024-03-04T16:01:17.710 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:17 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] up:boot 2024-03-04T16:01:17.710 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:17 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] up:standby 2024-03-04T16:01:17.710 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:17 smithi116 ceph-mon[29120]: fsmap cephfs:1/1 {0=cephfs.smithi152.tlpshy=up:reconnect} 3 up:standby 2024-03-04T16:01:17.710 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:01:17.710 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:17.710 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:17.710 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:17 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:17 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] up:reconnect 2024-03-04T16:01:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:17 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] up:boot 2024-03-04T16:01:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:17 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] up:standby 2024-03-04T16:01:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:17 smithi152 ceph-mon[36876]: fsmap cephfs:1/1 {0=cephfs.smithi152.tlpshy=up:reconnect} 3 up:standby 2024-03-04T16:01:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:01:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:17 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:18.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:18 smithi152 ceph-mon[36876]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-04T16:01:18.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:18 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] up:rejoin 2024-03-04T16:01:18.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:18 smithi152 ceph-mon[36876]: fsmap cephfs:1/1 {0=cephfs.smithi152.tlpshy=up:rejoin} 3 up:standby 2024-03-04T16:01:18.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:18 smithi152 ceph-mon[36876]: daemon mds.cephfs.smithi152.tlpshy is now active in filesystem cephfs as rank 0 2024-03-04T16:01:18.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:18 smithi152 ceph-mon[36876]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T16:01:18.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:18 smithi152 ceph-mon[36876]: Cluster is now healthy 2024-03-04T16:01:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:18 smithi116 ceph-mon[29120]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-04T16:01:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:18 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] up:rejoin 2024-03-04T16:01:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:18 smithi116 ceph-mon[29120]: fsmap cephfs:1/1 {0=cephfs.smithi152.tlpshy=up:rejoin} 3 up:standby 2024-03-04T16:01:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:18 smithi116 ceph-mon[29120]: daemon mds.cephfs.smithi152.tlpshy is now active in filesystem cephfs as rank 0 2024-03-04T16:01:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:18 smithi116 ceph-mon[29120]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T16:01:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:18 smithi116 ceph-mon[29120]: Cluster is now healthy 2024-03-04T16:01:19.671 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:19 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] up:active 2024-03-04T16:01:19.671 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:19 smithi116 ceph-mon[29120]: mds.? [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] up:standby 2024-03-04T16:01:19.671 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:19 smithi116 ceph-mon[29120]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 3 up:standby 2024-03-04T16:01:19.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:19 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] up:active 2024-03-04T16:01:19.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:19 smithi152 ceph-mon[36876]: mds.? [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] up:standby 2024-03-04T16:01:19.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:19 smithi152 ceph-mon[36876]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 3 up:standby 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 -- 172.21.15.116:0/1627715710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba8100cb0 msgr2=0x7fcba8101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 --2- 172.21.15.116:0/1627715710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba8100cb0 0x7fcba8101110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009910 tx=0x7fcb98009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 -- 172.21.15.116:0/1627715710 shutdown_connections 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 --2- 172.21.15.116:0/1627715710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba8100cb0 0x7fcba8101110 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 --2- 172.21.15.116:0/1627715710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcba80ffab0 0x7fcba80ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 -- 172.21.15.116:0/1627715710 >> 172.21.15.116:0/1627715710 conn(0x7fcba80fb050 msgr2=0x7fcba80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 -- 172.21.15.116:0/1627715710 shutdown_connections 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 -- 172.21.15.116:0/1627715710 wait complete. 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 Processor -- start 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 -- start start 2024-03-04T16:01:19.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba80ffab0 0x7fcba8196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcba8100cb0 0x7fcba8196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba8196f70 con 0x7fcba80ffab0 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.947+0000 7fcbb02b8700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba81970b0 con 0x7fcba8100cb0 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcbae054700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba80ffab0 0x7fcba8196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcbae054700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba80ffab0 0x7fcba8196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57420/0 (socket says 172.21.15.116:57420) 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcbae054700 1 -- 172.21.15.116:0/4263306145 learned_addr learned my addr 172.21.15.116:0/4263306145 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcbad853700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcba8100cb0 0x7fcba8196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcbae054700 1 -- 172.21.15.116:0/4263306145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcba8100cb0 msgr2=0x7fcba8196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcbae054700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcba8100cb0 0x7fcba8196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcbae054700 1 -- 172.21.15.116:0/4263306145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb980095f0 con 0x7fcba80ffab0 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcbae054700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba80ffab0 0x7fcba8196410 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcba400b480 tx=0x7fcba400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcb9f7fe700 1 -- 172.21.15.116:0/4263306145 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcba4011800 con 0x7fcba80ffab0 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcb9f7fe700 1 -- 172.21.15.116:0/4263306145 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcba4011e10 con 0x7fcba80ffab0 2024-03-04T16:01:19.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcbb02b8700 1 -- 172.21.15.116:0/4263306145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba819bb60 con 0x7fcba80ffab0 2024-03-04T16:01:19.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcb9f7fe700 1 -- 172.21.15.116:0/4263306145 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcba4010ea0 con 0x7fcba80ffab0 2024-03-04T16:01:19.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.948+0000 7fcbb02b8700 1 -- 172.21.15.116:0/4263306145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba819c0b0 con 0x7fcba80ffab0 2024-03-04T16:01:19.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.949+0000 7fcbb02b8700 1 -- 172.21.15.116:0/4263306145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba81086b0 con 0x7fcba80ffab0 2024-03-04T16:01:19.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.950+0000 7fcb9f7fe700 1 -- 172.21.15.116:0/4263306145 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fcba4010380 con 0x7fcba80ffab0 2024-03-04T16:01:19.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.950+0000 7fcb9f7fe700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcb9406c4e0 0x7fcb9406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:19.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.950+0000 7fcbad853700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcb9406c4e0 0x7fcb9406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:19.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.950+0000 7fcb9f7fe700 1 -- 172.21.15.116:0/4263306145 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcba408b3a0 con 0x7fcba80ffab0 2024-03-04T16:01:19.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.951+0000 7fcbad853700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcb9406c4e0 0x7fcb9406e9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcb98006010 tx=0x7fcb9800b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:19.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:19.955+0000 7fcb9f7fe700 1 -- 172.21.15.116:0/4263306145 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcba4059630 con 0x7fcba80ffab0 2024-03-04T16:01:20.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.191+0000 7fcbb02b8700 1 -- 172.21.15.116:0/4263306145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcba819c750 con 0x7fcba80ffab0 2024-03-04T16:01:20.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.192+0000 7fcb9f7fe700 1 -- 172.21.15.116:0/4263306145 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1831 (secure 0 0 0) 0x7fcba4015740 con 0x7fcba80ffab0 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:01:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:01:20.195 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:01:20.196 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:01:20.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 -- 172.21.15.116:0/4263306145 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcb9406c4e0 msgr2=0x7fcb9406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:20.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcb9406c4e0 0x7fcb9406e9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcb98006010 tx=0x7fcb9800b540 comp rx=0 tx=0).stop 2024-03-04T16:01:20.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 -- 172.21.15.116:0/4263306145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba80ffab0 msgr2=0x7fcba8196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:20.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba80ffab0 0x7fcba8196410 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcba400b480 tx=0x7fcba400b790 comp rx=0 tx=0).stop 2024-03-04T16:01:20.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 -- 172.21.15.116:0/4263306145 shutdown_connections 2024-03-04T16:01:20.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fcb9406c4e0 0x7fcb9406e9a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:20.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba80ffab0 0x7fcba8196410 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:20.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 --2- 172.21.15.116:0/4263306145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcba8100cb0 0x7fcba8196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:20.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 -- 172.21.15.116:0/4263306145 >> 172.21.15.116:0/4263306145 conn(0x7fcba80fb050 msgr2=0x7fcba8068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:20.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 -- 172.21.15.116:0/4263306145 shutdown_connections 2024-03-04T16:01:20.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:20.195+0000 7fcbb02b8700 1 -- 172.21.15.116:0/4263306145 wait complete. 2024-03-04T16:01:20.198 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:01:20.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:20 smithi116 ceph-mon[29120]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 409 B/s wr, 2 op/s 2024-03-04T16:01:20.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:20 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/4263306145' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:01:20.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:20 smithi152 ceph-mon[36876]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 409 B/s wr, 2 op/s 2024-03-04T16:01:20.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:20 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/4263306145' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:01:21.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-04T16:01:21.664 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:22 smithi116 ceph-mon[29120]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.2 KiB/s rd, 4 op/s 2024-03-04T16:01:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:22.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:22.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:22.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:01:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:22 smithi152 ceph-mon[36876]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.2 KiB/s rd, 4 op/s 2024-03-04T16:01:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:01:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.383+0000 7f3881904700 1 -- 172.21.15.116:0/512544496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f387c1021a0 msgr2=0x7f387c1025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.383+0000 7f3881904700 1 --2- 172.21.15.116:0/512544496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f387c1021a0 0x7f387c1025c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f387400b0a0 tx=0x7f387400b3b0 comp rx=0 tx=0).stop 2024-03-04T16:01:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.383+0000 7f3881904700 1 -- 172.21.15.116:0/512544496 shutdown_connections 2024-03-04T16:01:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.383+0000 7f3881904700 1 --2- 172.21.15.116:0/512544496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f387c0fec20 0x7f387c0ff080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.383+0000 7f3881904700 1 --2- 172.21.15.116:0/512544496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f387c1021a0 0x7f387c1025c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.383+0000 7f3881904700 1 -- 172.21.15.116:0/512544496 >> 172.21.15.116:0/512544496 conn(0x7f387c0fa7c0 msgr2=0x7f387c0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.383+0000 7f3881904700 1 -- 172.21.15.116:0/512544496 shutdown_connections 2024-03-04T16:01:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.383+0000 7f3881904700 1 -- 172.21.15.116:0/512544496 wait complete. 2024-03-04T16:01:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.383+0000 7f3881904700 1 Processor -- start 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f3881904700 1 -- start start 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f3881904700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f387c0fec20 0x7f387c1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f3881904700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f387c1021a0 0x7f387c0ffef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f3881904700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c100430 con 0x7f387c1021a0 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f3881904700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c100570 con 0x7f387c0fec20 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f387a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f387c1021a0 0x7f387c0ffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f387a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f387c1021a0 0x7f387c0ffef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57430/0 (socket says 172.21.15.116:57430) 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f387affd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f387c0fec20 0x7f387c1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f387a7fc700 1 -- 172.21.15.116:0/1037916860 learned_addr learned my addr 172.21.15.116:0/1037916860 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f387a7fc700 1 -- 172.21.15.116:0/1037916860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f387c0fec20 msgr2=0x7f387c1018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f387a7fc700 1 --2- 172.21.15.116:0/1037916860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f387c0fec20 0x7f387c1018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f387a7fc700 1 -- 172.21.15.116:0/1037916860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3874009d00 con 0x7f387c1021a0 2024-03-04T16:01:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f387a7fc700 1 --2- 172.21.15.116:0/1037916860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f387c1021a0 0x7f387c0ffef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f387000d8d0 tx=0x7f387000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f3880902700 1 -- 172.21.15.116:0/1037916860 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38700098a0 con 0x7f387c1021a0 2024-03-04T16:01:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f3880902700 1 -- 172.21.15.116:0/1037916860 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3870010460 con 0x7f387c1021a0 2024-03-04T16:01:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f3881904700 1 -- 172.21.15.116:0/1037916860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f387c100850 con 0x7f387c1021a0 2024-03-04T16:01:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f3880902700 1 -- 172.21.15.116:0/1037916860 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f387000f5f0 con 0x7f387c1021a0 2024-03-04T16:01:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.384+0000 7f3881904700 1 -- 172.21.15.116:0/1037916860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387c19e450 con 0x7f387c1021a0 2024-03-04T16:01:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.385+0000 7f3881904700 1 -- 172.21.15.116:0/1037916860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387c109730 con 0x7f387c1021a0 2024-03-04T16:01:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.386+0000 7f3880902700 1 -- 172.21.15.116:0/1037916860 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3870009a00 con 0x7f387c1021a0 2024-03-04T16:01:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.386+0000 7f3880902700 1 --2- 172.21.15.116:0/1037916860 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f386406c4e0 0x7f386406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.386+0000 7f3880902700 1 -- 172.21.15.116:0/1037916860 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f387008b770 con 0x7f387c1021a0 2024-03-04T16:01:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.386+0000 7f387affd700 1 --2- 172.21.15.116:0/1037916860 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f386406c4e0 0x7f386406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.387+0000 7f387affd700 1 --2- 172.21.15.116:0/1037916860 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f386406c4e0 0x7f386406e9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f387400b830 tx=0x7f38740060c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.391+0000 7f3880902700 1 -- 172.21.15.116:0/1037916860 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3870059980 con 0x7f387c1021a0 2024-03-04T16:01:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.632+0000 7f3881904700 1 -- 172.21.15.116:0/1037916860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f387c1098f0 con 0x7f387c1021a0 2024-03-04T16:01:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.633+0000 7f3880902700 1 -- 172.21.15.116:0/1037916860 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4748 (secure 0 0 0) 0x7f3870059510 con 0x7f387c1021a0 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.636+0000 7f3881904700 1 -- 172.21.15.116:0/1037916860 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f386406c4e0 msgr2=0x7f386406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.636+0000 7f3881904700 1 --2- 172.21.15.116:0/1037916860 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f386406c4e0 0x7f386406e9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f387400b830 tx=0x7f38740060c0 comp rx=0 tx=0).stop 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.636+0000 7f3881904700 1 -- 172.21.15.116:0/1037916860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f387c1021a0 msgr2=0x7f387c0ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.636+0000 7f3881904700 1 --2- 172.21.15.116:0/1037916860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f387c1021a0 0x7f387c0ffef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f387000d8d0 tx=0x7f387000dbe0 comp rx=0 tx=0).stop 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.636+0000 7f3881904700 1 -- 172.21.15.116:0/1037916860 shutdown_connections 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.636+0000 7f3881904700 1 --2- 172.21.15.116:0/1037916860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f387c0fec20 0x7f387c1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.636+0000 7f3881904700 1 --2- 172.21.15.116:0/1037916860 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f386406c4e0 0x7f386406e9a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.636+0000 7f3881904700 1 --2- 172.21.15.116:0/1037916860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f387c1021a0 0x7f387c0ffef0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.636+0000 7f3881904700 1 -- 172.21.15.116:0/1037916860 >> 172.21.15.116:0/1037916860 conn(0x7f387c0fa7c0 msgr2=0x7f387c105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.637+0000 7f3881904700 1 -- 172.21.15.116:0/1037916860 shutdown_connections 2024-03-04T16:01:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:23.637+0000 7f3881904700 1 -- 172.21.15.116:0/1037916860 wait complete. 2024-03-04T16:01:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:01:23.651 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:01:24.614 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:24 smithi116 ceph-mon[29120]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 7 op/s 2024-03-04T16:01:24.615 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:24 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1037916860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T16:01:24.615 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:24 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:24.615 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:24 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:24.635 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-04T16:01:24.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:24 smithi152 ceph-mon[36876]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 7 op/s 2024-03-04T16:01:24.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:24 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/1037916860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T16:01:24.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:24 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:24.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:24 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:25.479 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:26.590 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:26 smithi116 ceph-mon[29120]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 7 op/s 2024-03-04T16:01:26.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:26 smithi152 ceph-mon[36876]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 7 op/s 2024-03-04T16:01:28.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.360+0000 7faeaed25700 1 -- 172.21.15.116:0/3390082215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea8103ff0 msgr2=0x7faea81063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.360+0000 7faeaed25700 1 --2- 172.21.15.116:0/3390082215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea8103ff0 0x7faea81063e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fae94009910 tx=0x7fae94009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.360+0000 7faeaed25700 1 -- 172.21.15.116:0/3390082215 shutdown_connections 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.360+0000 7faeaed25700 1 --2- 172.21.15.116:0/3390082215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea8103ff0 0x7faea81063e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.360+0000 7faeaed25700 1 --2- 172.21.15.116:0/3390082215 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faea81016c0 0x7faea8103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.360+0000 7faeaed25700 1 -- 172.21.15.116:0/3390082215 >> 172.21.15.116:0/3390082215 conn(0x7faea80fb030 msgr2=0x7faea80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.360+0000 7faeaed25700 1 -- 172.21.15.116:0/3390082215 shutdown_connections 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.360+0000 7faeaed25700 1 -- 172.21.15.116:0/3390082215 wait complete. 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faeaed25700 1 Processor -- start 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faeaed25700 1 -- start start 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faeaed25700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faea81016c0 0x7faea8101020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faeaed25700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea8103ff0 0x7faea80ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faeaed25700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea80ffbb0 con 0x7faea8103ff0 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faeaed25700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea80ffcf0 con 0x7faea81016c0 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faea7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea8103ff0 0x7faea80ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faea7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea8103ff0 0x7faea80ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43856/0 (socket says 172.21.15.116:43856) 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faea7fff700 1 -- 172.21.15.116:0/913191419 learned_addr learned my addr 172.21.15.116:0/913191419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faeacac1700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faea81016c0 0x7faea8101020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faea7fff700 1 -- 172.21.15.116:0/913191419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faea81016c0 msgr2=0x7faea8101020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faea7fff700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faea81016c0 0x7faea8101020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.361+0000 7faea7fff700 1 -- 172.21.15.116:0/913191419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae940095f0 con 0x7faea8103ff0 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.362+0000 7faea7fff700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea8103ff0 0x7faea80ff670 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fae94006010 tx=0x7fae9400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.362+0000 7faea5ffb700 1 -- 172.21.15.116:0/913191419 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae9401d070 con 0x7faea8103ff0 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.362+0000 7faea5ffb700 1 -- 172.21.15.116:0/913191419 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fae9400f460 con 0x7faea8103ff0 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.362+0000 7faeaed25700 1 -- 172.21.15.116:0/913191419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea80fff70 con 0x7faea8103ff0 2024-03-04T16:01:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.362+0000 7faea5ffb700 1 -- 172.21.15.116:0/913191419 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae94021640 con 0x7faea8103ff0 2024-03-04T16:01:28.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.362+0000 7faeaed25700 1 -- 172.21.15.116:0/913191419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea8100410 con 0x7faea8103ff0 2024-03-04T16:01:28.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.363+0000 7faeaed25700 1 -- 172.21.15.116:0/913191419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea81905a0 con 0x7faea8103ff0 2024-03-04T16:01:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.363+0000 7faea5ffb700 1 -- 172.21.15.116:0/913191419 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fae9400fab0 con 0x7faea8103ff0 2024-03-04T16:01:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.364+0000 7faea5ffb700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fae9806c4e0 0x7fae9806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.364+0000 7faea5ffb700 1 -- 172.21.15.116:0/913191419 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fae9408cd10 con 0x7faea8103ff0 2024-03-04T16:01:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.364+0000 7faeacac1700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fae9806c4e0 0x7fae9806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.364+0000 7faeacac1700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fae9806c4e0 0x7fae9806e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fae9c006fd0 tx=0x7fae9c008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:28.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.369+0000 7faea5ffb700 1 -- 172.21.15.116:0/913191419 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fae9405b040 con 0x7faea8103ff0 2024-03-04T16:01:28.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.648+0000 7faeaed25700 1 -- 172.21.15.116:0/913191419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7faea802cc70 con 0x7faea8103ff0 2024-03-04T16:01:28.650 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:28 smithi116 ceph-mon[29120]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-04T16:01:28.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.648+0000 7faea5ffb700 1 -- 172.21.15.116:0/913191419 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v14) v1 ==== 78+0+83 (secure 0 0 0) 0x7fae94026070 con 0x7faea8103ff0 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.650+0000 7faeaed25700 1 -- 172.21.15.116:0/913191419 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fae9806c4e0 msgr2=0x7fae9806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.650+0000 7faeaed25700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fae9806c4e0 0x7fae9806e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fae9c006fd0 tx=0x7fae9c008040 comp rx=0 tx=0).stop 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.650+0000 7faeaed25700 1 -- 172.21.15.116:0/913191419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea8103ff0 msgr2=0x7faea80ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.650+0000 7faeaed25700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea8103ff0 0x7faea80ff670 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fae94006010 tx=0x7fae9400be20 comp rx=0 tx=0).stop 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.650+0000 7faeaed25700 1 -- 172.21.15.116:0/913191419 shutdown_connections 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.650+0000 7faeaed25700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faea81016c0 0x7faea8101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.650+0000 7faeaed25700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fae9806c4e0 0x7fae9806e9a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.650+0000 7faeaed25700 1 --2- 172.21.15.116:0/913191419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea8103ff0 0x7faea80ff670 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.650+0000 7faeaed25700 1 -- 172.21.15.116:0/913191419 >> 172.21.15.116:0/913191419 conn(0x7faea80fb030 msgr2=0x7faea80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.651+0000 7faeaed25700 1 -- 172.21.15.116:0/913191419 shutdown_connections 2024-03-04T16:01:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:28.651+0000 7faeaed25700 1 -- 172.21.15.116:0/913191419 wait complete. 2024-03-04T16:01:28.664 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:01:28.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:28 smithi152 ceph-mon[36876]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-04T16:01:29.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:29 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/913191419' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-04T16:01:29.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:29 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/913191419' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-04T16:01:29.888 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-04T16:01:29.900 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T16:01:30.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.016+0000 7f02df2ca640 1 -- 172.21.15.116:0/1343244343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d81012d0 msgr2=0x7f02d8101750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.016+0000 7f02df2ca640 1 --2- 172.21.15.116:0/1343244343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d81012d0 0x7f02d8101750 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f02c8009880 tx=0x7f02c802bce0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 -- 172.21.15.116:0/1343244343 shutdown_connections 2024-03-04T16:01:30.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 --2- 172.21.15.116:0/1343244343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d81012d0 0x7f02d8101750 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 --2- 172.21.15.116:0/1343244343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02d81000e0 0x7f02d8100500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 -- 172.21.15.116:0/1343244343 >> 172.21.15.116:0/1343244343 conn(0x7f02d80fb650 msgr2=0x7f02d80fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 -- 172.21.15.116:0/1343244343 shutdown_connections 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 -- 172.21.15.116:0/1343244343 wait complete. 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 Processor -- start 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 -- start start 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d81000e0 0x7f02d8195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02d81012d0 0x7f02d8195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d8196310 con 0x7f02d81000e0 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02df2ca640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d8196480 con 0x7f02d81012d0 2024-03-04T16:01:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.017+0000 7f02dd066640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d81000e0 0x7f02d8195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02dd066640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d81000e0 0x7f02d8195800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43862/0 (socket says 172.21.15.116:43862) 2024-03-04T16:01:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02dd066640 1 -- 172.21.15.116:0/49644474 learned_addr learned my addr 172.21.15.116:0/49644474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02dc865640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02d81012d0 0x7f02d8195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02dd066640 1 -- 172.21.15.116:0/49644474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02d81012d0 msgr2=0x7f02d8195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02dd066640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02d81012d0 0x7f02d8195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02dd066640 1 -- 172.21.15.116:0/49644474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c8009560 con 0x7f02d81000e0 2024-03-04T16:01:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02dd066640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d81000e0 0x7f02d8195800 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f02d400e940 tx=0x7f02d400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02ce7fc640 1 -- 172.21.15.116:0/49644474 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02d40096e0 con 0x7f02d81000e0 2024-03-04T16:01:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02df2ca640 1 -- 172.21.15.116:0/49644474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d819af10 con 0x7f02d81000e0 2024-03-04T16:01:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02ce7fc640 1 -- 172.21.15.116:0/49644474 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f02d40044e0 con 0x7f02d81000e0 2024-03-04T16:01:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02ce7fc640 1 -- 172.21.15.116:0/49644474 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02d4010430 con 0x7f02d81000e0 2024-03-04T16:01:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.018+0000 7f02df2ca640 1 -- 172.21.15.116:0/49644474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d819b3e0 con 0x7f02d81000e0 2024-03-04T16:01:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.019+0000 7f02df2ca640 1 -- 172.21.15.116:0/49644474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d8108e70 con 0x7f02d81000e0 2024-03-04T16:01:30.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.020+0000 7f02ce7fc640 1 -- 172.21.15.116:0/49644474 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f02d400cc90 con 0x7f02d81000e0 2024-03-04T16:01:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.020+0000 7f02ce7fc640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f02c406c510 0x7f02c406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.020+0000 7f02ce7fc640 1 -- 172.21.15.116:0/49644474 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f02d4014070 con 0x7f02d81000e0 2024-03-04T16:01:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.020+0000 7f02dc865640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f02c406c510 0x7f02c406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.021+0000 7f02dc865640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f02c406c510 0x7f02c406e9d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f02c8006010 tx=0x7f02c802c700 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.028+0000 7f02ce7fc640 1 -- 172.21.15.116:0/49644474 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f02d4056450 con 0x7f02d81000e0 2024-03-04T16:01:30.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.214+0000 7f02df2ca640 1 -- 172.21.15.116:0/49644474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02d819b720 con 0x7f02d81000e0 2024-03-04T16:01:30.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.215+0000 7f02ce7fc640 1 -- 172.21.15.116:0/49644474 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+15198 (secure 0 0 0) 0x7f02d4059a70 con 0x7f02d81000e0 2024-03-04T16:01:30.216 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:30.220 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":47,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:53:48.465240+0000","modified":"2024-03-04T16:01:15.387504+0000","last_up_change":"2024-03-04T15:59:41.850845+0000","last_in_change":"2024-03-04T15:59:22.307720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:58:27.372335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-04T16:00:58.546762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-04T16:00:59.546717+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d12966f-954e-4a09-bbbd-5b0f6a04d039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6803","nonce":1056902067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6805","nonce":1056902067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6809","nonce":1056902067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1056902067},{"type":"v1","addr":"172.21.15.116:6807","nonce":1056902067}]},"public_addr":"172.21.15.116:6803/1056902067","cluster_addr":"172.21.15.116:6805/1056902067","heartbeat_back_addr":"172.21.15.116:6809/1056902067","heartbeat_front_addr":"172.21.15.116:6807/1056902067","state":["exists","up"]},{"osd":1,"uuid":"f990720f-a19d-48c1-9545-81e511c2690b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6811","nonce":2368425008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6813","nonce":2368425008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6817","nonce":2368425008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2368425008},{"type":"v1","addr":"172.21.15.116:6815","nonce":2368425008}]},"public_addr":"172.21.15.116:6811/2368425008","cluster_addr":"172.21.15.116:6813/2368425008","heartbeat_back_addr":"172.21.15.116:6817/2368425008","heartbeat_front_addr":"172.21.15.116:6815/2368425008","state":["exists","up"]},{"osd":2,"uuid":"fe3132c8-6b15-4a11-a884-7c6a09d42955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6819","nonce":1282157941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6821","nonce":1282157941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6825","nonce":1282157941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1282157941},{"type":"v1","addr":"172.21.15.116:6823","nonce":1282157941}]},"public_addr":"172.21.15.116:6819/1282157941","cluster_addr":"172.21.15.116:6821/1282157941","heartbeat_back_addr":"172.21.15.116:6825/1282157941","heartbeat_front_addr":"172.21.15.116:6823/1282157941","state":["exists","up"]},{"osd":3,"uuid":"15930a9b-f721-4c30-9c5b-bef5ef69777c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6801","nonce":2457942679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6803","nonce":2457942679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6807","nonce":2457942679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2457942679},{"type":"v1","addr":"172.21.15.152:6805","nonce":2457942679}]},"public_addr":"172.21.15.152:6801/2457942679","cluster_addr":"172.21.15.152:6803/2457942679","heartbeat_back_addr":"172.21.15.152:6807/2457942679","heartbeat_front_addr":"172.21.15.152:6805/2457942679","state":["exists","up"]},{"osd":4,"uuid":"8d201b6a-6a91-4a7e-984e-e8d7f0289793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6809","nonce":3048670947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6811","nonce":3048670947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6815","nonce":3048670947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3048670947},{"type":"v1","addr":"172.21.15.152:6813","nonce":3048670947}]},"public_addr":"172.21.15.152:6809/3048670947","cluster_addr":"172.21.15.152:6811/3048670947","heartbeat_back_addr":"172.21.15.152:6815/3048670947","heartbeat_front_addr":"172.21.15.152:6813/3048670947","state":["exists","up"]},{"osd":5,"uuid":"c06ef0f5-6894-4e01-a693-bb129d248fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6817","nonce":1864443408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6819","nonce":1864443408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6823","nonce":1864443408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1864443408},{"type":"v1","addr":"172.21.15.152:6821","nonce":1864443408}]},"public_addr":"172.21.15.152:6817/1864443408","cluster_addr":"172.21.15.152:6819/1864443408","heartbeat_back_addr":"172.21.15.152:6823/1864443408","heartbeat_front_addr":"172.21.15.152:6821/1864443408","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:24.230215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:52.709103+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:24.072974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:47.684390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:13.571654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:39.874966+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6827/114739798":"2024-03-05T16:01:15.387483+0000","172.21.15.116:0/3323890559":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6800/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:6801/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6800/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6801/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/385581892":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/4142805454":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/976694871":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/355050440":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6801/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3287532122":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/831649128":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3670178561":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6800/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:0/821201733":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6826/114739798":"2024-03-05T16:01:15.387483+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:01:30.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 -- 172.21.15.116:0/49644474 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f02c406c510 msgr2=0x7f02c406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f02c406c510 0x7f02c406e9d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f02c8006010 tx=0x7f02c802c700 comp rx=0 tx=0).stop 2024-03-04T16:01:30.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 -- 172.21.15.116:0/49644474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d81000e0 msgr2=0x7f02d8195800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d81000e0 0x7f02d8195800 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f02d400e940 tx=0x7f02d400ee00 comp rx=0 tx=0).stop 2024-03-04T16:01:30.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 -- 172.21.15.116:0/49644474 shutdown_connections 2024-03-04T16:01:30.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f02c406c510 0x7f02c406e9d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02d81012d0 0x7f02d8195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 --2- 172.21.15.116:0/49644474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d81000e0 0x7f02d8195800 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 -- 172.21.15.116:0/49644474 >> 172.21.15.116:0/49644474 conn(0x7f02d80fb650 msgr2=0x7f02d8104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:30.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 -- 172.21.15.116:0/49644474 shutdown_connections 2024-03-04T16:01:30.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.217+0000 7f02df2ca640 1 -- 172.21.15.116:0/49644474 wait complete. 2024-03-04T16:01:30.229 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T15:58:27.372335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-04T16:00:58.546762+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-04T16:00:59.546717+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T16:01:30.230 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T16:01:30.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 -- 172.21.15.116:0/4218863779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee8100190 msgr2=0x7f7ee8100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 --2- 172.21.15.116:0/4218863779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee8100190 0x7f7ee8100610 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009880 tx=0x7f7ed402bce0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 -- 172.21.15.116:0/4218863779 shutdown_connections 2024-03-04T16:01:30.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 --2- 172.21.15.116:0/4218863779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee8100190 0x7f7ee8100610 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 --2- 172.21.15.116:0/4218863779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ee80fefa0 0x7f7ee80ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 -- 172.21.15.116:0/4218863779 >> 172.21.15.116:0/4218863779 conn(0x7f7ee80fa530 msgr2=0x7f7ee80fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 -- 172.21.15.116:0/4218863779 shutdown_connections 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 -- 172.21.15.116:0/4218863779 wait complete. 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 Processor -- start 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 -- start start 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee80fefa0 0x7f7ee8194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.332+0000 7f7eeea87640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ee8100190 0x7f7ee8194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eeea87640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee81952e0 con 0x7f7ee80fefa0 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eeea87640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee8195450 con 0x7f7ee8100190 2024-03-04T16:01:30.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eec823640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee80fefa0 0x7f7ee8194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eec823640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee80fefa0 0x7f7ee8194860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43878/0 (socket says 172.21.15.116:43878) 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eec823640 1 -- 172.21.15.116:0/1490720067 learned_addr learned my addr 172.21.15.116:0/1490720067 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7ee7fff640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ee8100190 0x7f7ee8194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eec823640 1 -- 172.21.15.116:0/1490720067 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ee8100190 msgr2=0x7f7ee8194da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eec823640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ee8100190 0x7f7ee8194da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eec823640 1 -- 172.21.15.116:0/1490720067 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed4009560 con 0x7f7ee80fefa0 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eec823640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee80fefa0 0x7f7ee8194860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00c8f0 tx=0x7f7edc00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7ee5ffb640 1 -- 172.21.15.116:0/1490720067 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7edc0041f0 con 0x7f7ee80fefa0 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7ee5ffb640 1 -- 172.21.15.116:0/1490720067 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7edc007e00 con 0x7f7ee80fefa0 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eeea87640 1 -- 172.21.15.116:0/1490720067 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee8199f20 con 0x7f7ee80fefa0 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7ee5ffb640 1 -- 172.21.15.116:0/1490720067 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7edc00f450 con 0x7f7ee80fefa0 2024-03-04T16:01:30.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.333+0000 7f7eeea87640 1 -- 172.21.15.116:0/1490720067 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee819a3f0 con 0x7f7ee80fefa0 2024-03-04T16:01:30.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.334+0000 7f7eeea87640 1 -- 172.21.15.116:0/1490720067 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee8107b80 con 0x7f7ee80fefa0 2024-03-04T16:01:30.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.335+0000 7f7ee5ffb640 1 -- 172.21.15.116:0/1490720067 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7edc004350 con 0x7f7ee80fefa0 2024-03-04T16:01:30.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.335+0000 7f7ee5ffb640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7ed806c450 0x7f7ed806e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.335+0000 7f7ee7fff640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7ed806c450 0x7f7ed806e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.335+0000 7f7ee5ffb640 1 -- 172.21.15.116:0/1490720067 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7edc013070 con 0x7f7ee80fefa0 2024-03-04T16:01:30.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.335+0000 7f7ee7fff640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7ed806c450 0x7f7ed806e910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8195d70 tx=0x7f7ed4000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:30.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.339+0000 7f7ee5ffb640 1 -- 172.21.15.116:0/1490720067 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7edc056070 con 0x7f7ee80fefa0 2024-03-04T16:01:30.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.509+0000 7f7eeea87640 1 -- 172.21.15.116:0/1490720067 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ee819a920 con 0x7f7ee80fefa0 2024-03-04T16:01:30.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.510+0000 7f7ee5ffb640 1 -- 172.21.15.116:0/1490720067 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7edc059690 con 0x7f7ee80fefa0 2024-03-04T16:01:30.511 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T16:01:30.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.511+0000 7f7eeea87640 1 -- 172.21.15.116:0/1490720067 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7ed806c450 msgr2=0x7f7ed806e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.511+0000 7f7eeea87640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7ed806c450 0x7f7ed806e910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8195d70 tx=0x7f7ed4000bc0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.511+0000 7f7eeea87640 1 -- 172.21.15.116:0/1490720067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee80fefa0 msgr2=0x7f7ee8194860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.511+0000 7f7eeea87640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee80fefa0 0x7f7ee8194860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00c8f0 tx=0x7f7edc00cdb0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.511+0000 7f7eeea87640 1 -- 172.21.15.116:0/1490720067 shutdown_connections 2024-03-04T16:01:30.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.511+0000 7f7eeea87640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f7ed806c450 0x7f7ed806e910 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.511+0000 7f7eeea87640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ee8100190 0x7f7ee8194da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.511+0000 7f7eeea87640 1 --2- 172.21.15.116:0/1490720067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee80fefa0 0x7f7ee8194860 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.511+0000 7f7eeea87640 1 -- 172.21.15.116:0/1490720067 >> 172.21.15.116:0/1490720067 conn(0x7f7ee80fa530 msgr2=0x7f7ee81033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.512+0000 7f7eeea87640 1 -- 172.21.15.116:0/1490720067 shutdown_connections 2024-03-04T16:01:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.512+0000 7f7eeea87640 1 -- 172.21.15.116:0/1490720067 wait complete. 2024-03-04T16:01:30.523 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:30 smithi116 ceph-mon[29120]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-04T16:01:30.523 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:30.523 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:30.523 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:30.523 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:30.523 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:30 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:30.523 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:30 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/49644474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:01:30.524 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-04T16:01:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.627+0000 7f54c1a51640 1 -- 172.21.15.116:0/694632498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc0fefa0 msgr2=0x7f54bc0ff3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.627+0000 7f54c1a51640 1 --2- 172.21.15.116:0/694632498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc0fefa0 0x7f54bc0ff3c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009880 tx=0x7f54ac02bce0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.627+0000 7f54c1a51640 1 -- 172.21.15.116:0/694632498 shutdown_connections 2024-03-04T16:01:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.627+0000 7f54c1a51640 1 --2- 172.21.15.116:0/694632498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc100190 0x7f54bc100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.627+0000 7f54c1a51640 1 --2- 172.21.15.116:0/694632498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc0fefa0 0x7f54bc0ff3c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.627+0000 7f54c1a51640 1 -- 172.21.15.116:0/694632498 >> 172.21.15.116:0/694632498 conn(0x7f54bc0fa530 msgr2=0x7f54bc0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:30.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.627+0000 7f54c1a51640 1 -- 172.21.15.116:0/694632498 shutdown_connections 2024-03-04T16:01:30.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.627+0000 7f54c1a51640 1 -- 172.21.15.116:0/694632498 wait complete. 2024-03-04T16:01:30.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54c1a51640 1 Processor -- start 2024-03-04T16:01:30.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54c1a51640 1 -- start start 2024-03-04T16:01:30.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54c1a51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc0fefa0 0x7f54bc068f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54c1a51640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc100190 0x7f54bc19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54c1a51640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc19ace0 con 0x7f54bc0fefa0 2024-03-04T16:01:30.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54c1a51640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc19ae20 con 0x7f54bc100190 2024-03-04T16:01:30.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54baffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc0fefa0 0x7f54bc068f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54baffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc0fefa0 0x7f54bc068f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43886/0 (socket says 172.21.15.116:43886) 2024-03-04T16:01:30.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54baffd640 1 -- 172.21.15.116:0/4133474049 learned_addr learned my addr 172.21.15.116:0/4133474049 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:30.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54b3fff640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc100190 0x7f54bc19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54baffd640 1 -- 172.21.15.116:0/4133474049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc100190 msgr2=0x7f54bc19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54baffd640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc100190 0x7f54bc19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54baffd640 1 -- 172.21.15.116:0/4133474049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac009560 con 0x7f54bc0fefa0 2024-03-04T16:01:30.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.628+0000 7f54baffd640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc0fefa0 0x7f54bc068f30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009230 tx=0x7f54ac004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:30.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.629+0000 7f54c0a4f640 1 -- 172.21.15.116:0/4133474049 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54ac0044e0 con 0x7f54bc0fefa0 2024-03-04T16:01:30.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.629+0000 7f54c1a51640 1 -- 172.21.15.116:0/4133474049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54bc19b070 con 0x7f54bc0fefa0 2024-03-04T16:01:30.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.629+0000 7f54c0a4f640 1 -- 172.21.15.116:0/4133474049 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f54ac0054b0 con 0x7f54bc0fefa0 2024-03-04T16:01:30.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.629+0000 7f54c0a4f640 1 -- 172.21.15.116:0/4133474049 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54ac034440 con 0x7f54bc0fefa0 2024-03-04T16:01:30.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.629+0000 7f54c1a51640 1 -- 172.21.15.116:0/4133474049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc19b570 con 0x7f54bc0fefa0 2024-03-04T16:01:30.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.629+0000 7f54c1a51640 1 -- 172.21.15.116:0/4133474049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54bc107d30 con 0x7f54bc0fefa0 2024-03-04T16:01:30.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.630+0000 7f54c0a4f640 1 -- 172.21.15.116:0/4133474049 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f54ac003810 con 0x7f54bc0fefa0 2024-03-04T16:01:30.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.630+0000 7f54c0a4f640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54a806c450 0x7f54a806e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.630+0000 7f54b3fff640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54a806c450 0x7f54a806e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.630+0000 7f54c0a4f640 1 -- 172.21.15.116:0/4133474049 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f54ac0adcd0 con 0x7f54bc0fefa0 2024-03-04T16:01:30.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.631+0000 7f54b3fff640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54a806c450 0x7f54a806e910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f54bc068ca0 tx=0x7f54a4005f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:30.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.635+0000 7f54c0a4f640 1 -- 172.21.15.116:0/4133474049 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f54ac078500 con 0x7f54bc0fefa0 2024-03-04T16:01:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.802+0000 7f54c1a51640 1 -- 172.21.15.116:0/4133474049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f54bc19b850 con 0x7f54bc0fefa0 2024-03-04T16:01:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.802+0000 7f54c0a4f640 1 -- 172.21.15.116:0/4133474049 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f54ac07bb20 con 0x7f54bc0fefa0 2024-03-04T16:01:30.803 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T16:01:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 -- 172.21.15.116:0/4133474049 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54a806c450 msgr2=0x7f54a806e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54a806c450 0x7f54a806e910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f54bc068ca0 tx=0x7f54a4005f90 comp rx=0 tx=0).stop 2024-03-04T16:01:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 -- 172.21.15.116:0/4133474049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc0fefa0 msgr2=0x7f54bc068f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc0fefa0 0x7f54bc068f30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009230 tx=0x7f54ac004dc0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 -- 172.21.15.116:0/4133474049 shutdown_connections 2024-03-04T16:01:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f54a806c450 0x7f54a806e910 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc100190 0x7f54bc19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 --2- 172.21.15.116:0/4133474049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc0fefa0 0x7f54bc068f30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 -- 172.21.15.116:0/4133474049 >> 172.21.15.116:0/4133474049 conn(0x7f54bc0fa530 msgr2=0x7f54bc1033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 -- 172.21.15.116:0/4133474049 shutdown_connections 2024-03-04T16:01:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.804+0000 7f54c1a51640 1 -- 172.21.15.116:0/4133474049 wait complete. 2024-03-04T16:01:30.817 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-04T16:01:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:30 smithi152 ceph-mon[36876]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-04T16:01:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:30 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:30 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/49644474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:01:30.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.919+0000 7f61e2716640 1 -- 172.21.15.116:0/1696602256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61dc0ff2d0 msgr2=0x7f61dc0ff730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.919+0000 7f61e2716640 1 --2- 172.21.15.116:0/1696602256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61dc0ff2d0 0x7f61dc0ff730 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009880 tx=0x7f61d002bce0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.919+0000 7f61e2716640 1 -- 172.21.15.116:0/1696602256 shutdown_connections 2024-03-04T16:01:30.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.919+0000 7f61e2716640 1 --2- 172.21.15.116:0/1696602256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61dc0ff2d0 0x7f61dc0ff730 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.919+0000 7f61e2716640 1 --2- 172.21.15.116:0/1696602256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61dc0fe970 0x7f61dc0fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.919+0000 7f61e2716640 1 -- 172.21.15.116:0/1696602256 >> 172.21.15.116:0/1696602256 conn(0x7f61dc0fa510 msgr2=0x7f61dc0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:30.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.919+0000 7f61e2716640 1 -- 172.21.15.116:0/1696602256 shutdown_connections 2024-03-04T16:01:30.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.919+0000 7f61e2716640 1 -- 172.21.15.116:0/1696602256 wait complete. 2024-03-04T16:01:30.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61e2716640 1 Processor -- start 2024-03-04T16:01:30.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61e2716640 1 -- start start 2024-03-04T16:01:30.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61e2716640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61dc0fe970 0x7f61dc197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61e2716640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61dc0ff2d0 0x7f61dc198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61e2716640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc198640 con 0x7f61dc0ff2d0 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61e2716640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc1987b0 con 0x7f61dc0fe970 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61dc0ff2d0 0x7f61dc198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61dc0ff2d0 0x7f61dc198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43896/0 (socket says 172.21.15.116:43896) 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61db7fe640 1 -- 172.21.15.116:0/3753948117 learned_addr learned my addr 172.21.15.116:0/3753948117 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61dbfff640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61dc0fe970 0x7f61dc197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61db7fe640 1 -- 172.21.15.116:0/3753948117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61dc0fe970 msgr2=0x7f61dc197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.920+0000 7f61db7fe640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61dc0fe970 0x7f61dc197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.921+0000 7f61db7fe640 1 -- 172.21.15.116:0/3753948117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d0009560 con 0x7f61dc0ff2d0 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.921+0000 7f61dbfff640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61dc0fe970 0x7f61dc197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.921+0000 7f61db7fe640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61dc0ff2d0 0x7f61dc198070 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009430 tx=0x7f61d002cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.921+0000 7f61d97fa640 1 -- 172.21.15.116:0/3753948117 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61d0004520 con 0x7f61dc0ff2d0 2024-03-04T16:01:30.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.921+0000 7f61e2716640 1 -- 172.21.15.116:0/3753948117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61dc19d1e0 con 0x7f61dc0ff2d0 2024-03-04T16:01:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.921+0000 7f61d97fa640 1 -- 172.21.15.116:0/3753948117 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f61d00054b0 con 0x7f61dc0ff2d0 2024-03-04T16:01:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.921+0000 7f61d97fa640 1 -- 172.21.15.116:0/3753948117 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61d0034440 con 0x7f61dc0ff2d0 2024-03-04T16:01:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.921+0000 7f61e2716640 1 -- 172.21.15.116:0/3753948117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61dc19d6e0 con 0x7f61dc0ff2d0 2024-03-04T16:01:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.922+0000 7f61e2716640 1 -- 172.21.15.116:0/3753948117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61dc10afc0 con 0x7f61dc0ff2d0 2024-03-04T16:01:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.923+0000 7f61d97fa640 1 -- 172.21.15.116:0/3753948117 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f61d0005000 con 0x7f61dc0ff2d0 2024-03-04T16:01:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.923+0000 7f61d97fa640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f61c406c450 0x7f61c406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.923+0000 7f61d97fa640 1 -- 172.21.15.116:0/3753948117 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f61d00adfa0 con 0x7f61dc0ff2d0 2024-03-04T16:01:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.923+0000 7f61dbfff640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f61c406c450 0x7f61c406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.923+0000 7f61dbfff640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f61c406c450 0x7f61c406e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0058c0 tx=0x7f61cc005d30 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:30.927+0000 7f61d97fa640 1 -- 172.21.15.116:0/3753948117 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f61d0078720 con 0x7f61dc0ff2d0 2024-03-04T16:01:31.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.094+0000 7f61e2716640 1 -- 172.21.15.116:0/3753948117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f61dc19de20 con 0x7f61dc0ff2d0 2024-03-04T16:01:31.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.094+0000 7f61d97fa640 1 -- 172.21.15.116:0/3753948117 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f61d007bd40 con 0x7f61dc0ff2d0 2024-03-04T16:01:31.096 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T16:01:31.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 -- 172.21.15.116:0/3753948117 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f61c406c450 msgr2=0x7f61c406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:31.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f61c406c450 0x7f61c406e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0058c0 tx=0x7f61cc005d30 comp rx=0 tx=0).stop 2024-03-04T16:01:31.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 -- 172.21.15.116:0/3753948117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61dc0ff2d0 msgr2=0x7f61dc198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:31.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61dc0ff2d0 0x7f61dc198070 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009430 tx=0x7f61d002cec0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 -- 172.21.15.116:0/3753948117 shutdown_connections 2024-03-04T16:01:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f61c406c450 0x7f61c406e910 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61dc0ff2d0 0x7f61dc198070 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 --2- 172.21.15.116:0/3753948117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61dc0fe970 0x7f61dc197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 -- 172.21.15.116:0/3753948117 >> 172.21.15.116:0/3753948117 conn(0x7f61dc0fa510 msgr2=0x7f61dc106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 -- 172.21.15.116:0/3753948117 shutdown_connections 2024-03-04T16:01:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.096+0000 7f61e2716640 1 -- 172.21.15.116:0/3753948117 wait complete. 2024-03-04T16:01:31.109 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T16:01:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.209+0000 7fb45aba4640 1 -- 172.21.15.116:0/1311420721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4540fefc0 msgr2=0x7fb4540ff3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.209+0000 7fb45aba4640 1 --2- 172.21.15.116:0/1311420721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4540fefc0 0x7fb4540ff3e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb448009880 tx=0x7fb44802bce0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.209+0000 7fb45aba4640 1 -- 172.21.15.116:0/1311420721 shutdown_connections 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.209+0000 7fb45aba4640 1 --2- 172.21.15.116:0/1311420721 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4541001b0 0x7fb454100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.209+0000 7fb45aba4640 1 --2- 172.21.15.116:0/1311420721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4540fefc0 0x7fb4540ff3e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.209+0000 7fb45aba4640 1 -- 172.21.15.116:0/1311420721 >> 172.21.15.116:0/1311420721 conn(0x7fb4540fa530 msgr2=0x7fb4540fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.209+0000 7fb45aba4640 1 -- 172.21.15.116:0/1311420721 shutdown_connections 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.209+0000 7fb45aba4640 1 -- 172.21.15.116:0/1311420721 wait complete. 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.209+0000 7fb45aba4640 1 Processor -- start 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.209+0000 7fb45aba4640 1 -- start start 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb45aba4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4540fefc0 0x7fb4541958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb45aba4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4541001b0 0x7fb454195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb45aba4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4541963b0 con 0x7fb4540fefc0 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb45aba4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb454196520 con 0x7fb4541001b0 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb458940640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4540fefc0 0x7fb4541958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb458940640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4540fefc0 0x7fb4541958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43910/0 (socket says 172.21.15.116:43910) 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb458940640 1 -- 172.21.15.116:0/290362391 learned_addr learned my addr 172.21.15.116:0/290362391 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb458940640 1 -- 172.21.15.116:0/290362391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4541001b0 msgr2=0x7fb454195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb453fff640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4541001b0 0x7fb454195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb458940640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4541001b0 0x7fb454195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb458940640 1 -- 172.21.15.116:0/290362391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb448009560 con 0x7fb4540fefc0 2024-03-04T16:01:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb453fff640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4541001b0 0x7fb454195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:01:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb458940640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4540fefc0 0x7fb4541958a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb448009230 tx=0x7fb448004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb451ffb640 1 -- 172.21.15.116:0/290362391 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb448005000 con 0x7fb4540fefc0 2024-03-04T16:01:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb451ffb640 1 -- 172.21.15.116:0/290362391 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb448005610 con 0x7fb4540fefc0 2024-03-04T16:01:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb451ffb640 1 -- 172.21.15.116:0/290362391 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb448034440 con 0x7fb4540fefc0 2024-03-04T16:01:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb45aba4640 1 -- 172.21.15.116:0/290362391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45419af50 con 0x7fb4540fefc0 2024-03-04T16:01:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.210+0000 7fb45aba4640 1 -- 172.21.15.116:0/290362391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45419b3d0 con 0x7fb4540fefc0 2024-03-04T16:01:31.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.211+0000 7fb45aba4640 1 -- 172.21.15.116:0/290362391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb454107d50 con 0x7fb4540fefc0 2024-03-04T16:01:31.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.212+0000 7fb451ffb640 1 -- 172.21.15.116:0/290362391 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb448003880 con 0x7fb4540fefc0 2024-03-04T16:01:31.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.212+0000 7fb451ffb640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb44406c510 0x7fb44406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:31.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.212+0000 7fb451ffb640 1 -- 172.21.15.116:0/290362391 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb4480add30 con 0x7fb4540fefc0 2024-03-04T16:01:31.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.212+0000 7fb453fff640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb44406c510 0x7fb44406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:31.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.213+0000 7fb453fff640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb44406c510 0x7fb44406e9d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb454196db0 tx=0x7fb440005e50 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.217+0000 7fb451ffb640 1 -- 172.21.15.116:0/290362391 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb448078530 con 0x7fb4540fefc0 2024-03-04T16:01:31.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.420+0000 7fb45aba4640 1 -- 172.21.15.116:0/290362391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb45419b730 con 0x7fb4540fefc0 2024-03-04T16:01:31.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.420+0000 7fb451ffb640 1 -- 172.21.15.116:0/290362391 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4748 (secure 0 0 0) 0x7fb44807bb50 con 0x7fb4540fefc0 2024-03-04T16:01:31.422 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:31.422 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.116:6829/3470200572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3470200572},{"type":"v1","addr":"172.21.15.116:6829","nonce":3470200572}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":14498,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/4270616896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":4270616896},{"type":"v1","addr":"172.21.15.116:6827","nonce":4270616896}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:01:18.395823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24247":{"gid":24247,"name":"cephfs.smithi152.tlpshy","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.152:6825/287206196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":287206196},{"type":"v1","addr":"172.21.15.152:6825","nonce":287206196}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:01:31.422 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 -- 172.21.15.116:0/290362391 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb44406c510 msgr2=0x7fb44406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb44406c510 0x7fb44406e9d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb454196db0 tx=0x7fb440005e50 comp rx=0 tx=0).stop 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 -- 172.21.15.116:0/290362391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4540fefc0 msgr2=0x7fb4541958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4540fefc0 0x7fb4541958a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb448009230 tx=0x7fb448004e30 comp rx=0 tx=0).stop 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 -- 172.21.15.116:0/290362391 shutdown_connections 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb44406c510 0x7fb44406e9d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4541001b0 0x7fb454195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 --2- 172.21.15.116:0/290362391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4540fefc0 0x7fb4541958a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 -- 172.21.15.116:0/290362391 >> 172.21.15.116:0/290362391 conn(0x7fb4540fa530 msgr2=0x7fb4540683b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 -- 172.21.15.116:0/290362391 shutdown_connections 2024-03-04T16:01:31.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:31.422+0000 7fb45aba4640 1 -- 172.21.15.116:0/290362391 wait complete. 2024-03-04T16:01:31.435 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 14, 'max_mds': 1, 'flags': 18} 2024-03-04T16:01:31.435 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-04T16:01:31.465 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-04T16:01:31.465 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-03-04T16:01:31.465 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-04T16:01:31.465 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T16:01:31.465 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T16:01:31.466 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-04T16:01:31.466 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T16:01:31.466 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-04T16:01:31.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:31.466 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-04T16:01:31.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:31.482 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-04T16:01:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:Cannot find device "ceph-brx" 2024-03-04T16:01:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:01:31.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:31.551 DEBUG:teuthology.orchestra.run.smithi152:> ip netns list 2024-03-04T16:01:31.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:31.571 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link delete ceph-brx 2024-03-04T16:01:31.648 INFO:teuthology.orchestra.run.smithi152.stderr:Cannot find device "ceph-brx" 2024-03-04T16:01:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:01:31.650 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-04T16:01:31.650 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph fs ls 2024-03-04T16:01:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:31 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1490720067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:01:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:31 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/4133474049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-04T16:01:31.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:31 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3753948117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-04T16:01:31.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:31 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/290362391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T16:01:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:31.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:31 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/1490720067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:01:31.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:31 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/4133474049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-04T16:01:31.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:31 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3753948117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-04T16:01:31.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:31 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/290362391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T16:01:32.711 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:32 smithi116 ceph-mon[29120]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-04T16:01:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:32 smithi152 ceph-mon[36876]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-04T16:01:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.434+0000 7f2ace079700 1 -- 172.21.15.116:0/1072151806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ac80ff050 msgr2=0x7f2ac80ff4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.434+0000 7f2ace079700 1 --2- 172.21.15.116:0/1072151806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ac80ff050 0x7f2ac80ff4d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009910 tx=0x7f2abc009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.435+0000 7f2ace079700 1 -- 172.21.15.116:0/1072151806 shutdown_connections 2024-03-04T16:01:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.435+0000 7f2ace079700 1 --2- 172.21.15.116:0/1072151806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ac80ff050 0x7f2ac80ff4d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.435+0000 7f2ace079700 1 --2- 172.21.15.116:0/1072151806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac8105b20 0x7f2ac8105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.435+0000 7f2ace079700 1 -- 172.21.15.116:0/1072151806 >> 172.21.15.116:0/1072151806 conn(0x7f2ac8075e30 msgr2=0x7f2ac80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.435+0000 7f2ace079700 1 -- 172.21.15.116:0/1072151806 shutdown_connections 2024-03-04T16:01:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.435+0000 7f2ace079700 1 -- 172.21.15.116:0/1072151806 wait complete. 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.435+0000 7f2ace079700 1 Processor -- start 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.435+0000 7f2ace079700 1 -- start start 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.435+0000 7f2ace079700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac80ff050 0x7f2ac819c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ace079700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ac8105b20 0x7f2ac819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ace079700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac819d140 con 0x7f2ac8105b20 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ace079700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81965f0 con 0x7f2ac80ff050 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ac8105b20 0x7f2ac819cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ac8105b20 0x7f2ac819cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43926/0 (socket says 172.21.15.116:43926) 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac6ffd700 1 -- 172.21.15.116:0/507111623 learned_addr learned my addr 172.21.15.116:0/507111623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac6ffd700 1 -- 172.21.15.116:0/507111623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac80ff050 msgr2=0x7f2ac819c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac77fe700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac80ff050 0x7f2ac819c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac6ffd700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac80ff050 0x7f2ac819c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac6ffd700 1 -- 172.21.15.116:0/507111623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc0095f0 con 0x7f2ac8105b20 2024-03-04T16:01:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac6ffd700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ac8105b20 0x7f2ac819cab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00bd70 tx=0x7f2abc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac4ff9700 1 -- 172.21.15.116:0/507111623 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2abc01d070 con 0x7f2ac8105b20 2024-03-04T16:01:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ace079700 1 -- 172.21.15.116:0/507111623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac8196870 con 0x7f2ac8105b20 2024-03-04T16:01:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac4ff9700 1 -- 172.21.15.116:0/507111623 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2abc00f460 con 0x7f2ac8105b20 2024-03-04T16:01:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ac4ff9700 1 -- 172.21.15.116:0/507111623 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2abc017630 con 0x7f2ac8105b20 2024-03-04T16:01:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.436+0000 7f2ace079700 1 -- 172.21.15.116:0/507111623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac8196d60 con 0x7f2ac8105b20 2024-03-04T16:01:33.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.437+0000 7f2ace079700 1 -- 172.21.15.116:0/507111623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac81090c0 con 0x7f2ac8105b20 2024-03-04T16:01:33.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.438+0000 7f2ac4ff9700 1 -- 172.21.15.116:0/507111623 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2abc003a60 con 0x7f2ac8105b20 2024-03-04T16:01:33.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.438+0000 7f2ac4ff9700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2ab006c4e0 0x7f2ab006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.438+0000 7f2ac4ff9700 1 -- 172.21.15.116:0/507111623 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2abc08cdf0 con 0x7f2ac8105b20 2024-03-04T16:01:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.438+0000 7f2ac77fe700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2ab006c4e0 0x7f2ab006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.439+0000 7f2ac77fe700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2ab006c4e0 0x7f2ab006e9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8005950 tx=0x7f2ab80058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:33.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.443+0000 7f2ac4ff9700 1 -- 172.21.15.116:0/507111623 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2abc05b000 con 0x7f2ac8105b20 2024-03-04T16:01:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.682+0000 7f2ace079700 1 -- 172.21.15.116:0/507111623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2ac8108aa0 con 0x7f2ac8105b20 2024-03-04T16:01:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.682+0000 7f2ac4ff9700 1 -- 172.21.15.116:0/507111623 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f2abc05ab90 con 0x7f2ac8105b20 2024-03-04T16:01:33.684 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 -- 172.21.15.116:0/507111623 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2ab006c4e0 msgr2=0x7f2ab006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2ab006c4e0 0x7f2ab006e9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8005950 tx=0x7f2ab80058e0 comp rx=0 tx=0).stop 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 -- 172.21.15.116:0/507111623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ac8105b20 msgr2=0x7f2ac819cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ac8105b20 0x7f2ac819cab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00bd70 tx=0x7f2abc00be50 comp rx=0 tx=0).stop 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 -- 172.21.15.116:0/507111623 shutdown_connections 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac80ff050 0x7f2ac819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2ab006c4e0 0x7f2ab006e9a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 --2- 172.21.15.116:0/507111623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ac8105b20 0x7f2ac819cab0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 -- 172.21.15.116:0/507111623 >> 172.21.15.116:0/507111623 conn(0x7f2ac8075e30 msgr2=0x7f2ac81044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 -- 172.21.15.116:0/507111623 shutdown_connections 2024-03-04T16:01:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:33.684+0000 7f2ace079700 1 -- 172.21.15.116:0/507111623 wait complete. 2024-03-04T16:01:34.378 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T16:01:34.378 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T16:01:34.378 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-03-04T16:01:34.378 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T16:01:34.378 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T16:01:34.378 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T16:01:34.378 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T16:01:34.379 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T16:01:34.379 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T16:01:34.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:34.380 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-03-04T16:01:34.397 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T16:01:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T16:01:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T16:01:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-04T16:01:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-03-04T16:01:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-04T16:01:34.397 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42236sec preferred_lft 42236sec 2024-03-04T16:01:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-04T16:01:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-04T16:01:34.399 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-04T16:01:34.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:34.399 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-04T16:01:34.399 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-04T16:01:34.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-03-04T16:01:34.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-03-04T16:01:34.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-03-04T16:01:34.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-04T16:01:34.400 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-04T16:01:34.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:34.578 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T16:01:34.604 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-04T16:01:34.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:34.606 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-03-04T16:01:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:34 smithi152 ceph-mon[36876]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-04T16:01:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:34 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/507111623' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T16:01:34.681 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-03-04T16:01:34.681 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T16:01:34.681 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T16:01:34.681 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T16:01:34.681 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T16:01:34.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:34.682 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-04T16:01:34.682 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-04T16:01:34.682 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T16:01:34.682 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T16:01:34.682 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T16:01:34.682 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-04T16:01:34.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:34 smithi116 ceph-mon[29120]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-04T16:01:34.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:34 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/507111623' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T16:01:34.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:34.954 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-04T16:01:34.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:34.971 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-03-04T16:01:35.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:35.025 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-04T16:01:35.025 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-04T16:01:35.025 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T16:01:35.025 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-04T16:01:35.025 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-04T16:01:35.107 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-04T16:01:35.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:35.108 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-04T16:01:35.108 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-04T16:01:35.108 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-04T16:01:35.108 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-04T16:01:35.108 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-04T16:01:35.108 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-04T16:01:35.108 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-04T16:01:35.108 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-04T16:01:35.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:35.248 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-04T16:01:35.248 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-04T16:01:35.248 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-03-04T16:01:35.248 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-03-04T16:01:35.248 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-04T16:01:35.289 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-04T16:01:35.289 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T16:01:35.289 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T16:01:35.344 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T16:01:35.344 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T16:01:35.345 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T16:01:35.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-04T16:01:35.469 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-04T16:01:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-04T16:01:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/19c3e96585741634d7e5774ecf484a641c02ecd7a9286174e2d6beceedfac7e6/merged 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/b0949977f6d3d08c3e02a91bd534f6b24628db8bafd0d9e7cedcbcaab1510075/merged 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/ca99e636a7c26a9472ccf1c5c15b446ad910c5cc5787fb5987dc0e6e6fe16c88/merged 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/2329b46f229bbccdb341e5abb16380f637ee3a691e3d1703c3f314ea49db9514/merged 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/36b109b9baaba06950fde59a8848b058d761dd29b97211e85c9c975404e946b6/merged 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/68093970c654bd2e1f69ce72ac965cf9eebf06c61cbe76a13c0180a99d57c748/merged 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/3884c1f348a68550eaa50789af34f7b0ea787a144ee079b58bc14a62da531da4/merged 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/76a19baf34e5bb29670a0577adb3aba9d7e35b82e7d6ee9329cf25649e6636e4/merged 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6cad30c81ad1850e4e90299dabb188b2755284d23b4072b507a1df852a5033f8/merged 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/42a441728750f332dd63e73d758ff4162f2fa50f7aedaddfd37321a329828941/merged 2024-03-04T16:01:35.527 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6185cc2b9592c57c82fc1cfdaf5a195284247bf2c9b2aeddf6fd7e56d30aebff/merged 2024-03-04T16:01:35.528 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/ba5f1b4b78594a7cb088e57b1460c6ce0d3d5aa63cd3801a6d94b05767c9ed60/merged 2024-03-04T16:01:35.528 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/7393c4d388eb07cc5fd1467dcd5d3d68078275cc4211335d5a365228d5e752fd/merged 2024-03-04T16:01:35.528 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-04T16:01:35.528 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T16:01:35.528 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T16:01:35.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:35.528 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-04T16:01:35.580 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-04T16:01:35.580 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-04T16:01:35.622 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-04T16:01:35.649 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-04T16:01:35.706 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-04T16:01:35.707 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-04T16:01:35.707 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-04T16:01:35.707 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-04T16:01:35.707 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-04T16:01:35.707 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-04T16:01:35.707 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-04T16:01:35.707 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-04T16:01:35.707 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-04T16:01:35.707 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-04T16:01:35.708 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-04T16:01:35.708 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-04T16:01:35.708 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T16:01:35.708 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-04T16:01:35.708 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-04T16:01:35.708 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-04T16:01:35.708 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-04T16:01:35.708 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-04T16:01:35.708 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-04T16:01:35.708 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-04T16:01:35.709 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-04T16:01:35.709 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-04T16:01:35.709 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-04T16:01:35.709 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-04T16:01:35.709 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-04T16:01:35.709 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-04T16:01:35.709 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/19c3e96585741634d7e5774ecf484a641c02ecd7a9286174e2d6beceedfac7e6/merged 2024-03-04T16:01:35.709 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/b0949977f6d3d08c3e02a91bd534f6b24628db8bafd0d9e7cedcbcaab1510075/merged 2024-03-04T16:01:35.709 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-04T16:01:35.709 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/ca99e636a7c26a9472ccf1c5c15b446ad910c5cc5787fb5987dc0e6e6fe16c88/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/2329b46f229bbccdb341e5abb16380f637ee3a691e3d1703c3f314ea49db9514/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/36b109b9baaba06950fde59a8848b058d761dd29b97211e85c9c975404e946b6/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/68093970c654bd2e1f69ce72ac965cf9eebf06c61cbe76a13c0180a99d57c748/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/3884c1f348a68550eaa50789af34f7b0ea787a144ee079b58bc14a62da531da4/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/76a19baf34e5bb29670a0577adb3aba9d7e35b82e7d6ee9329cf25649e6636e4/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6cad30c81ad1850e4e90299dabb188b2755284d23b4072b507a1df852a5033f8/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/42a441728750f332dd63e73d758ff4162f2fa50f7aedaddfd37321a329828941/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6185cc2b9592c57c82fc1cfdaf5a195284247bf2c9b2aeddf6fd7e56d30aebff/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/ba5f1b4b78594a7cb088e57b1460c6ce0d3d5aa63cd3801a6d94b05767c9ed60/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/7393c4d388eb07cc5fd1467dcd5d3d68078275cc4211335d5a365228d5e752fd/merged 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T16:01:35.710 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T16:01:35.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:35.711 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-04T16:01:35.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2024-03-04T16:01:35.744+0000 7f677ea4c480 -1 init, newargv = 0x5626f2cc4d10 newargc=15 2024-03-04T16:01:35.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[78279]: starting ceph client 2024-03-04T16:01:35.752 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[78279]: starting fuse 2024-03-04T16:01:35.763 INFO:teuthology.orchestra.run.smithi116.stdout:129 2024-03-04T16:01:35.764 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [129] 2024-03-04T16:01:35.764 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2024-03-04T16:01:35.764 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2024-03-04T16:01:35.764 DEBUG:teuthology.orchestra.run.smithi116:> import re 2024-03-04T16:01:35.764 DEBUG:teuthology.orchestra.run.smithi116:> import os 2024-03-04T16:01:35.764 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2024-03-04T16:01:35.765 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2024-03-04T16:01:35.766 DEBUG:teuthology.orchestra.run.smithi116:> return f 2024-03-04T16:01:35.766 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-04T16:01:35.766 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-04T16:01:35.766 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2024-03-04T16:01:35.766 DEBUG:teuthology.orchestra.run.smithi116:> ' 2024-03-04T16:01:35.883 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.78279.asok 2024-03-04T16:01:35.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T16:01:35.894 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.78279.asok 2024-03-04T16:01:35.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:35.894 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.78279.asok status 2024-03-04T16:01:36.029 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:01:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2024-03-04T16:01:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "7fe91d5d5842e04be3b4f514d6dd990c54b29c76", 2024-03-04T16:01:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-03-04T16:01:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2024-03-04T16:01:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2024-03-04T16:01:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-04T16:01:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "78279", 2024-03-04T16:01:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2024-03-04T16:01:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:01:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2024-03-04T16:01:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2024-03-04T16:01:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 14538, 2024-03-04T16:01:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2024-03-04T16:01:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2024-03-04T16:01:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2024-03-04T16:01:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 14538 2024-03-04T16:01:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:01:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-04T16:01:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-04T16:01:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-03-04T16:01:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1893604702 2024-03-04T16:01:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:01:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:01:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-04T16:01:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-04T16:01:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-03-04T16:01:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1893604702 2024-03-04T16:01:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:01:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.14538 192.168.0.1:0/1893604702", 2024-03-04T16:01:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/1893604702", 2024-03-04T16:01:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2024-03-04T16:01:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 14, 2024-03-04T16:01:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 47, 2024-03-04T16:01:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2024-03-04T16:01:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2024-03-04T16:01:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2024-03-04T16:01:36.033 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:01:36.038 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- ceph fs ls 2024-03-04T16:01:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:36.669 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:36 smithi116 ceph-mon[29120]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-04T16:01:36.805 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:36 smithi152 ceph-mon[36876]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-04T16:01:37.683 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:37 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:01:37.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:37 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:01:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.924+0000 7ff33534a700 1 -- 172.21.15.116:0/3269647766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330101790 msgr2=0x7ff3301057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.924+0000 7ff33534a700 1 --2- 172.21.15.116:0/3269647766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330101790 0x7ff3301057e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff318009910 tx=0x7ff318009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.924+0000 7ff33534a700 1 -- 172.21.15.116:0/3269647766 shutdown_connections 2024-03-04T16:01:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.924+0000 7ff33534a700 1 --2- 172.21.15.116:0/3269647766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330101790 0x7ff3301057e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.924+0000 7ff33534a700 1 --2- 172.21.15.116:0/3269647766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100de0 0x7ff3301011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.924+0000 7ff33534a700 1 -- 172.21.15.116:0/3269647766 >> 172.21.15.116:0/3269647766 conn(0x7ff3300fc670 msgr2=0x7ff3300fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.924+0000 7ff33534a700 1 -- 172.21.15.116:0/3269647766 shutdown_connections 2024-03-04T16:01:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.924+0000 7ff33534a700 1 -- 172.21.15.116:0/3269647766 wait complete. 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.925+0000 7ff33534a700 1 Processor -- start 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.925+0000 7ff33534a700 1 -- start start 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.925+0000 7ff33534a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100de0 0x7ff33019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.925+0000 7ff33534a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330101790 0x7ff33019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.925+0000 7ff33534a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33019d0a0 con 0x7ff330101790 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.925+0000 7ff33534a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3301965e0 con 0x7ff330100de0 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.925+0000 7ff327fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330101790 0x7ff33019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.925+0000 7ff327fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330101790 0x7ff33019caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34368/0 (socket says 172.21.15.116:34368) 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.925+0000 7ff327fff700 1 -- 172.21.15.116:0/3490598837 learned_addr learned my addr 172.21.15.116:0/3490598837 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.925+0000 7ff32effd700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100de0 0x7ff33019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.926+0000 7ff327fff700 1 -- 172.21.15.116:0/3490598837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100de0 msgr2=0x7ff33019c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.926+0000 7ff327fff700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100de0 0x7ff33019c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.926+0000 7ff327fff700 1 -- 172.21.15.116:0/3490598837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3180095f0 con 0x7ff330101790 2024-03-04T16:01:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.926+0000 7ff32effd700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100de0 0x7ff33019c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:01:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.926+0000 7ff327fff700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330101790 0x7ff33019caa0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff31800bda0 tx=0x7ff31800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.926+0000 7ff32cff9700 1 -- 172.21.15.116:0/3490598837 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff31801d070 con 0x7ff330101790 2024-03-04T16:01:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.926+0000 7ff32cff9700 1 -- 172.21.15.116:0/3490598837 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff318022470 con 0x7ff330101790 2024-03-04T16:01:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.926+0000 7ff33534a700 1 -- 172.21.15.116:0/3490598837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff330196860 con 0x7ff330101790 2024-03-04T16:01:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.926+0000 7ff32cff9700 1 -- 172.21.15.116:0/3490598837 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff31800f690 con 0x7ff330101790 2024-03-04T16:01:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.926+0000 7ff33534a700 1 -- 172.21.15.116:0/3490598837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff330196d50 con 0x7ff330101790 2024-03-04T16:01:37.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.927+0000 7ff33534a700 1 -- 172.21.15.116:0/3490598837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3301091e0 con 0x7ff330101790 2024-03-04T16:01:37.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.927+0000 7ff32cff9700 1 -- 172.21.15.116:0/3490598837 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff318022b00 con 0x7ff330101790 2024-03-04T16:01:37.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.928+0000 7ff32cff9700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff31c06c4e0 0x7ff31c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:37.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.928+0000 7ff32cff9700 1 -- 172.21.15.116:0/3490598837 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff31808ccf0 con 0x7ff330101790 2024-03-04T16:01:37.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.928+0000 7ff32effd700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff31c06c4e0 0x7ff31c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:37.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.928+0000 7ff32effd700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff31c06c4e0 0x7ff31c06e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff320005fd0 tx=0x7ff320005e30 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:37.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:37.932+0000 7ff32cff9700 1 -- 172.21.15.116:0/3490598837 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff31805b030 con 0x7ff330101790 2024-03-04T16:01:38.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.163+0000 7ff33534a700 1 -- 172.21.15.116:0/3490598837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff33004ea90 con 0x7ff330101790 2024-03-04T16:01:38.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.163+0000 7ff32cff9700 1 -- 172.21.15.116:0/3490598837 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7ff318027070 con 0x7ff330101790 2024-03-04T16:01:38.165 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-04T16:01:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 -- 172.21.15.116:0/3490598837 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff31c06c4e0 msgr2=0x7ff31c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff31c06c4e0 0x7ff31c06e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff320005fd0 tx=0x7ff320005e30 comp rx=0 tx=0).stop 2024-03-04T16:01:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 -- 172.21.15.116:0/3490598837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330101790 msgr2=0x7ff33019caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330101790 0x7ff33019caa0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff31800bda0 tx=0x7ff31800be80 comp rx=0 tx=0).stop 2024-03-04T16:01:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 -- 172.21.15.116:0/3490598837 shutdown_connections 2024-03-04T16:01:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100de0 0x7ff33019c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ff31c06c4e0 0x7ff31c06e9a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 --2- 172.21.15.116:0/3490598837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330101790 0x7ff33019caa0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 -- 172.21.15.116:0/3490598837 >> 172.21.15.116:0/3490598837 conn(0x7ff3300fc670 msgr2=0x7ff3300fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 -- 172.21.15.116:0/3490598837 shutdown_connections 2024-03-04T16:01:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:38.165+0000 7ff33534a700 1 -- 172.21.15.116:0/3490598837 wait complete. 2024-03-04T16:01:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:38 smithi116 ceph-mon[29120]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-04T16:01:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:38 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3490598837' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T16:01:38.809 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T16:01:38.809 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T16:01:38.809 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi152.front.sepia.ceph.com 2024-03-04T16:01:38.809 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-04T16:01:38.809 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-04T16:01:38.809 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T16:01:38.810 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T16:01:38.810 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T16:01:38.810 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-04T16:01:38.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:38.810 DEBUG:teuthology.orchestra.run.smithi152:> ip addr 2024-03-04T16:01:38.831 INFO:teuthology.orchestra.run.smithi152.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: valid_lft forever preferred_lft forever 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: inet6 ::1/128 scope host 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: valid_lft forever preferred_lft forever 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: link/ether 0c:c4:7a:da:6c:44 brd ff:ff:ff:ff:ff:ff 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: altname enp7s0f0 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: link/ether 0c:c4:7a:da:6c:45 brd ff:ff:ff:ff:ff:ff 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: altname enp7s0f1 2024-03-04T16:01:38.832 INFO:teuthology.orchestra.run.smithi152.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T16:01:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: link/ether 0c:c4:7a:88:6d:e2 brd ff:ff:ff:ff:ff:ff 2024-03-04T16:01:38.833 INFO:teuthology.orchestra.run.smithi152.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T16:01:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: link/ether 0c:c4:7a:88:6d:e3 brd ff:ff:ff:ff:ff:ff 2024-03-04T16:01:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: inet 172.21.15.152/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T16:01:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: valid_lft 42256sec preferred_lft 42256sec 2024-03-04T16:01:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: inet6 fe80::ec4:7aff:fe88:6de3/64 scope link 2024-03-04T16:01:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: valid_lft forever preferred_lft forever 2024-03-04T16:01:38.833 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-04T16:01:38.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:38.834 DEBUG:teuthology.orchestra.run.smithi152:> (cd / && exec bash -c ' 2024-03-04T16:01:38.834 DEBUG:teuthology.orchestra.run.smithi152:> set -e 2024-03-04T16:01:38.834 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link add name ceph-brx type bridge 2024-03-04T16:01:38.834 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip addr flush dev ceph-brx 2024-03-04T16:01:38.834 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link set ceph-brx up 2024-03-04T16:01:38.834 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-04T16:01:38.834 DEBUG:teuthology.orchestra.run.smithi152:> ') 2024-03-04T16:01:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:38 smithi152 ceph-mon[36876]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-04T16:01:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:38 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3490598837' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T16:01:38.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:38.983 DEBUG:teuthology.orchestra.run.smithi152:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T16:01:39.010 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-03-04T16:01:39.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:39.011 DEBUG:teuthology.orchestra.run.smithi152:> route 2024-03-04T16:01:39.089 INFO:teuthology.orchestra.run.smithi152.stdout:Kernel IP routing table 2024-03-04T16:01:39.089 INFO:teuthology.orchestra.run.smithi152.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T16:01:39.089 INFO:teuthology.orchestra.run.smithi152.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T16:01:39.089 INFO:teuthology.orchestra.run.smithi152.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T16:01:39.089 INFO:teuthology.orchestra.run.smithi152.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T16:01:39.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:39.090 DEBUG:teuthology.orchestra.run.smithi152:> (cd / && exec bash -c ' 2024-03-04T16:01:39.090 DEBUG:teuthology.orchestra.run.smithi152:> set -e 2024-03-04T16:01:39.090 DEBUG:teuthology.orchestra.run.smithi152:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T16:01:39.090 DEBUG:teuthology.orchestra.run.smithi152:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T16:01:39.090 DEBUG:teuthology.orchestra.run.smithi152:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T16:01:39.090 DEBUG:teuthology.orchestra.run.smithi152:> ') 2024-03-04T16:01:39.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:39.342 DEBUG:teuthology.orchestra.run.smithi152:> ip netns list 2024-03-04T16:01:39.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:39.357 DEBUG:teuthology.orchestra.run.smithi152:> ip netns list-id 2024-03-04T16:01:39.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:39.412 DEBUG:teuthology.orchestra.run.smithi152:> (cd / && exec bash -c ' 2024-03-04T16:01:39.412 DEBUG:teuthology.orchestra.run.smithi152:> set -e 2024-03-04T16:01:39.412 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-04T16:01:39.412 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-04T16:01:39.413 DEBUG:teuthology.orchestra.run.smithi152:> ') 2024-03-04T16:01:39.495 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-04T16:01:39.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:39.495 DEBUG:teuthology.orchestra.run.smithi152:> (cd / && exec bash -c ' 2024-03-04T16:01:39.495 DEBUG:teuthology.orchestra.run.smithi152:> set -e 2024-03-04T16:01:39.495 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-04T16:01:39.495 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-04T16:01:39.496 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-04T16:01:39.496 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-04T16:01:39.496 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-04T16:01:39.496 DEBUG:teuthology.orchestra.run.smithi152:> ') 2024-03-04T16:01:39.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:39.634 DEBUG:teuthology.orchestra.run.smithi152:> (cd / && exec bash -c ' 2024-03-04T16:01:39.634 DEBUG:teuthology.orchestra.run.smithi152:> set -e 2024-03-04T16:01:39.634 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link set brx.0 up 2024-03-04T16:01:39.634 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link set dev brx.0 master ceph-brx 2024-03-04T16:01:39.635 DEBUG:teuthology.orchestra.run.smithi152:> ') 2024-03-04T16:01:39.677 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-04T16:01:39.677 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T16:01:39.677 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-04T16:01:39.733 INFO:teuthology.orchestra.run.smithi152.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-04T16:01:39.733 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T16:01:39.733 DEBUG:teuthology.orchestra.run.smithi152:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-04T16:01:39.788 DEBUG:teuthology.orchestra.run.smithi152:> sudo modprobe fuse 2024-03-04T16:01:39.856 DEBUG:teuthology.orchestra.run.smithi152:> cat /proc/self/mounts | awk '{print $2}' 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/proc 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/sys 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/dev 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/kernel/security 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/shm 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/pts 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/run 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/fs/cgroup 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/fs/pstore 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/fs/bpf 2024-03-04T16:01:39.913 INFO:teuthology.orchestra.run.smithi152.stdout:/ 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/fs/selinux 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/mqueue 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/hugepages 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/kernel/debug 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/kernel/tracing 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/fs/fuse/connections 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/kernel/config 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/run/credentials/systemd-sysctl.service 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/nfs/rpc_pipefs 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/run/user/1000 2024-03-04T16:01:39.914 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/ceph 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/run/user/0 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/74e7ec41b8b6f99c552de20c71fb5d07a152d6622eac081e6ec48c26109cc2d6/merged 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/856e919d1bb18811dd08504b090386b7e65f3bf1cd19f44d48e48a0718879cce/merged 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/744ac579aaa996fb0ec11e1317e98c64ae4fe5c75724d116df17247ee76456f4/merged 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/5b6dde6816305f9d63f11183afe1cb2fca05cb905b12e115905dac08b2b46405/merged 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/aab64095df7d0b50096d30c548cffce40dcdea05e5280f764706e414e194413a/merged 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/777b9513ccdb75c56cfe6a14b09a4953f6e9f0725e2bdc700752d07c8e5cd344/merged 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/c091da15bbf07bc9811c84f0e2157638c6025d7315cb9029badd4604e9751a3f/merged 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/bf169934e825919c1d869b4720a251fc4258961d3cde8e35b9007828db91869d/merged 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/e80a34dc8c3ab921a2d7c6c19a50355e047aa3a214a91b8ebb86b571899fc214/merged 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/fa2e4a7ff5c127a2d548daca1ce4ce945b94836b80d3905dcff37ec5b042a84f/merged 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/run/netns 2024-03-04T16:01:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-04T16:01:39.916 INFO:teuthology.orchestra.run.smithi152.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-04T16:01:39.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:39.916 DEBUG:teuthology.orchestra.run.smithi152:> ls /sys/fs/fuse/connections 2024-03-04T16:01:39.967 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-04T16:01:39.968 DEBUG:teuthology.orchestra.run.smithi152:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-04T16:01:40.010 DEBUG:teuthology.orchestra.run.smithi152:> sudo modprobe fuse 2024-03-04T16:01:40.036 DEBUG:teuthology.orchestra.run.smithi152:> cat /proc/self/mounts | awk '{print $2}' 2024-03-04T16:01:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:/proc 2024-03-04T16:01:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:/sys 2024-03-04T16:01:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:/dev 2024-03-04T16:01:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/kernel/security 2024-03-04T16:01:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/shm 2024-03-04T16:01:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/pts 2024-03-04T16:01:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:/run 2024-03-04T16:01:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/fs/cgroup 2024-03-04T16:01:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/fs/pstore 2024-03-04T16:01:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/fs/bpf 2024-03-04T16:01:40.094 INFO:teuthology.orchestra.run.smithi152.stdout:/ 2024-03-04T16:01:40.094 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/fs/selinux 2024-03-04T16:01:40.094 INFO:teuthology.orchestra.run.smithi152.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T16:01:40.094 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/mqueue 2024-03-04T16:01:40.094 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/hugepages 2024-03-04T16:01:40.094 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/kernel/debug 2024-03-04T16:01:40.094 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/kernel/tracing 2024-03-04T16:01:40.094 INFO:teuthology.orchestra.run.smithi152.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-04T16:01:40.094 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/fs/fuse/connections 2024-03-04T16:01:40.094 INFO:teuthology.orchestra.run.smithi152.stdout:/sys/kernel/config 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/run/credentials/systemd-sysctl.service 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/nfs/rpc_pipefs 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/run/user/1000 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/ceph 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/run/user/0 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/74e7ec41b8b6f99c552de20c71fb5d07a152d6622eac081e6ec48c26109cc2d6/merged 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/856e919d1bb18811dd08504b090386b7e65f3bf1cd19f44d48e48a0718879cce/merged 2024-03-04T16:01:40.095 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/744ac579aaa996fb0ec11e1317e98c64ae4fe5c75724d116df17247ee76456f4/merged 2024-03-04T16:01:40.096 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/5b6dde6816305f9d63f11183afe1cb2fca05cb905b12e115905dac08b2b46405/merged 2024-03-04T16:01:40.096 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/aab64095df7d0b50096d30c548cffce40dcdea05e5280f764706e414e194413a/merged 2024-03-04T16:01:40.096 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/777b9513ccdb75c56cfe6a14b09a4953f6e9f0725e2bdc700752d07c8e5cd344/merged 2024-03-04T16:01:40.096 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/c091da15bbf07bc9811c84f0e2157638c6025d7315cb9029badd4604e9751a3f/merged 2024-03-04T16:01:40.096 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/bf169934e825919c1d869b4720a251fc4258961d3cde8e35b9007828db91869d/merged 2024-03-04T16:01:40.096 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/e80a34dc8c3ab921a2d7c6c19a50355e047aa3a214a91b8ebb86b571899fc214/merged 2024-03-04T16:01:40.096 INFO:teuthology.orchestra.run.smithi152.stdout:/var/lib/containers/storage/overlay/fa2e4a7ff5c127a2d548daca1ce4ce945b94836b80d3905dcff37ec5b042a84f/merged 2024-03-04T16:01:40.096 INFO:teuthology.orchestra.run.smithi152.stdout:/run/netns 2024-03-04T16:01:40.096 INFO:teuthology.orchestra.run.smithi152.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-04T16:01:40.096 INFO:teuthology.orchestra.run.smithi152.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-04T16:01:40.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:40.097 DEBUG:teuthology.orchestra.run.smithi152:> ls /sys/fs/fuse/connections 2024-03-04T16:01:40.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi152.stderr:2024-03-04T16:01:40.120+0000 7f29ba533480 -1 init, newargv = 0x565382b35410 newargc=15 2024-03-04T16:01:40.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi152.stderr:ceph-fuse[63079]: starting ceph client 2024-03-04T16:01:40.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi152.stderr:ceph-fuse[63079]: starting fuse 2024-03-04T16:01:40.136 INFO:teuthology.orchestra.run.smithi152.stdout:126 2024-03-04T16:01:40.137 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [126] 2024-03-04T16:01:40.137 DEBUG:teuthology.orchestra.run.smithi152:> sudo stdin-killer -- python3 -c ' 2024-03-04T16:01:40.137 DEBUG:teuthology.orchestra.run.smithi152:> import glob 2024-03-04T16:01:40.137 DEBUG:teuthology.orchestra.run.smithi152:> import re 2024-03-04T16:01:40.137 DEBUG:teuthology.orchestra.run.smithi152:> import os 2024-03-04T16:01:40.137 DEBUG:teuthology.orchestra.run.smithi152:> import subprocess 2024-03-04T16:01:40.137 DEBUG:teuthology.orchestra.run.smithi152:> 2024-03-04T16:01:40.137 DEBUG:teuthology.orchestra.run.smithi152:> def _find_admin_socket(client_name): 2024-03-04T16:01:40.137 DEBUG:teuthology.orchestra.run.smithi152:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-04T16:01:40.137 DEBUG:teuthology.orchestra.run.smithi152:> files = glob.glob(asok_path) 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> # Given a non-glob path, it better be there 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> if "*" not in asok_path: 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> assert(len(files) == 1) 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> return files[0] 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> for f in files: 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> contents = proc_f.read() 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> if mountpoint in contents: 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> return f 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-04T16:01:40.138 DEBUG:teuthology.orchestra.run.smithi152:> 2024-03-04T16:01:40.139 DEBUG:teuthology.orchestra.run.smithi152:> print(_find_admin_socket("client.1")) 2024-03-04T16:01:40.139 DEBUG:teuthology.orchestra.run.smithi152:> ' 2024-03-04T16:01:40.258 INFO:teuthology.orchestra.run.smithi152.stdout:/var/run/ceph/ceph-client.1.63079.asok 2024-03-04T16:01:40.261 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T16:01:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T16:01:40.268 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.63079.asok 2024-03-04T16:01:40.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:40.268 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.63079.asok status 2024-03-04T16:01:40.406 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-04T16:01:40.406 INFO:teuthology.orchestra.run.smithi152.stdout: "metadata": { 2024-03-04T16:01:40.406 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_sha1": "7fe91d5d5842e04be3b4f514d6dd990c54b29c76", 2024-03-04T16:01:40.406 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-03-04T16:01:40.406 INFO:teuthology.orchestra.run.smithi152.stdout: "entity_id": "1", 2024-03-04T16:01:40.406 INFO:teuthology.orchestra.run.smithi152.stdout: "hostname": "smithi152", 2024-03-04T16:01:40.406 INFO:teuthology.orchestra.run.smithi152.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-04T16:01:40.406 INFO:teuthology.orchestra.run.smithi152.stdout: "pid": "63079", 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "root": "/" 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "dentry_count": 0, 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "dentry_pinned_count": 0, 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "id": 24297, 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "inst": { 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "name": { 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "type": "client", 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "num": 24297 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "addr": { 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "type": "v1", 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "addr": "192.168.0.1:0", 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: "nonce": 2088783451 2024-03-04T16:01:40.407 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "addr": { 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "type": "v1", 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "addr": "192.168.0.1:0", 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "nonce": 2088783451 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "inst_str": "client.24297 192.168.0.1:0/2088783451", 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "addr_str": "192.168.0.1:0/2088783451", 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "inode_count": 1, 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_epoch": 14, 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_epoch": 47, 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_epoch_barrier": 0, 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "blocklisted": false, 2024-03-04T16:01:40.408 INFO:teuthology.orchestra.run.smithi152.stdout: "fs_name": "cephfs" 2024-03-04T16:01:40.409 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-04T16:01:40.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:40.415 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2024-03-04T16:01:40.415 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-04T16:01:40.434 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2024-03-04T16:01:40.435 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-04T16:01:40.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:40.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T16:01:40.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:40.514 DEBUG:teuthology.orchestra.run.smithi152:> stat --file-system '--printf=%T 2024-03-04T16:01:40.514 DEBUG:teuthology.orchestra.run.smithi152:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-04T16:01:40.529 INFO:teuthology.orchestra.run.smithi152.stdout:fuseblk 2024-03-04T16:01:40.529 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-04T16:01:40.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:01:40.530 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-04T16:01:40.599 INFO:teuthology.run_tasks:Running task print... 2024-03-04T16:01:40.609 INFO:teuthology.task.print:**** done client 2024-03-04T16:01:40.609 INFO:teuthology.run_tasks:Running task parallel... 2024-03-04T16:01:40.626 INFO:teuthology.task.parallel:starting parallel... 2024-03-04T16:01:40.626 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-04T16:01:40.627 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-04T16:01:40.627 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:01:40.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-03-04T16:01:40.628 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-04T16:01:40.628 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-04T16:01:40.631 INFO:tasks.workunit:Pulling workunits from ref 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:01:40.631 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-04T16:01:40.631 INFO:tasks.workunit:timeout=3h 2024-03-04T16:01:40.632 INFO:tasks.workunit:cleanup=True 2024-03-04T16:01:40.632 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-04T16:01:40.650 INFO:teuthology.orchestra.run.smithi116.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-04T16:01:40.651 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-04T16:01:40.651 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 81h/129d Inode: 1 Links: 2 2024-03-04T16:01:40.651 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T16:01:40.651 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-04T16:01:40.651 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-04T16:01:40.651 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-04 16:01:05.343440951 +0000 2024-03-04T16:01:40.651 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-04 16:01:40.508548421 +0000 2024-03-04T16:01:40.651 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-03-04T16:01:40.651 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-04T16:01:40.651 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-04T16:01:40.716 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:40 smithi116 ceph-mon[29120]: pgmap v201: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:40.724 DEBUG:teuthology.orchestra.run.smithi152:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-04T16:01:40.739 INFO:teuthology.orchestra.run.smithi152.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-04T16:01:40.740 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-04T16:01:40.740 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-03-04T16:01:40.740 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T16:01:40.740 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-04T16:01:40.740 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-04T16:01:40.740 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-04 16:01:40.718664439 +0000 2024-03-04T16:01:40.740 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-04 16:01:40.718664439 +0000 2024-03-04T16:01:40.740 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: - 2024-03-04T16:01:40.740 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-04T16:01:40.740 DEBUG:teuthology.orchestra.run.smithi152:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-04T16:01:40.804 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:40 smithi152 ceph-mon[36876]: pgmap v201: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:40.812 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:01:40.812 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:01:40.827 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-04T16:01:40.849 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:40.867 INFO:tasks.workunit.client.1.smithi152.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-04T16:01:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.571+0000 7fb6516c7700 1 -- 172.21.15.116:0/2277508515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb64c101e10 msgr2=0x7fb64c102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.571+0000 7fb6516c7700 1 --2- 172.21.15.116:0/2277508515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb64c101e10 0x7fb64c102270 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb634009910 tx=0x7fb634009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.571+0000 7fb6516c7700 1 -- 172.21.15.116:0/2277508515 shutdown_connections 2024-03-04T16:01:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.571+0000 7fb6516c7700 1 --2- 172.21.15.116:0/2277508515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb64c101e10 0x7fb64c102270 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.571+0000 7fb6516c7700 1 --2- 172.21.15.116:0/2277508515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb64c100c10 0x7fb64c101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.571+0000 7fb6516c7700 1 -- 172.21.15.116:0/2277508515 >> 172.21.15.116:0/2277508515 conn(0x7fb64c0fc1b0 msgr2=0x7fb64c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.571+0000 7fb6516c7700 1 -- 172.21.15.116:0/2277508515 shutdown_connections 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.571+0000 7fb6516c7700 1 -- 172.21.15.116:0/2277508515 wait complete. 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.571+0000 7fb6516c7700 1 Processor -- start 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6516c7700 1 -- start start 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6516c7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb64c100c10 0x7fb64c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6516c7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb64c101e10 0x7fb64c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6516c7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c0736c0 con 0x7fb64c101e10 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6516c7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c073800 con 0x7fb64c100c10 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6427fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb64c101e10 0x7fb64c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6427fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb64c101e10 0x7fb64c073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34376/0 (socket says 172.21.15.116:34376) 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6427fc700 1 -- 172.21.15.116:0/2706904527 learned_addr learned my addr 172.21.15.116:0/2706904527 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb64affd700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb64c100c10 0x7fb64c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6427fc700 1 -- 172.21.15.116:0/2706904527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb64c100c10 msgr2=0x7fb64c074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6427fc700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb64c100c10 0x7fb64c074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6427fc700 1 -- 172.21.15.116:0/2706904527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6340095f0 con 0x7fb64c101e10 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6427fc700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb64c101e10 0x7fb64c073180 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb634006010 tx=0x7fb63400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb648ff9700 1 -- 172.21.15.116:0/2706904527 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63401d070 con 0x7fb64c101e10 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb6516c7700 1 -- 172.21.15.116:0/2706904527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64c073a80 con 0x7fb64c101e10 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb648ff9700 1 -- 172.21.15.116:0/2706904527 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb63400f460 con 0x7fb64c101e10 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.572+0000 7fb648ff9700 1 -- 172.21.15.116:0/2706904527 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb634021640 con 0x7fb64c101e10 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.573+0000 7fb6516c7700 1 -- 172.21.15.116:0/2706904527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64c073f70 con 0x7fb64c101e10 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.573+0000 7fb6516c7700 1 -- 172.21.15.116:0/2706904527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64c109770 con 0x7fb64c101e10 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.574+0000 7fb648ff9700 1 -- 172.21.15.116:0/2706904527 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb63400fab0 con 0x7fb64c101e10 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.574+0000 7fb648ff9700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb63806c4e0 0x7fb63806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.574+0000 7fb648ff9700 1 -- 172.21.15.116:0/2706904527 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb63408cb20 con 0x7fb64c101e10 2024-03-04T16:01:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.574+0000 7fb64affd700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb63806c4e0 0x7fb63806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.575+0000 7fb64affd700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb63806c4e0 0x7fb63806e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb64c101c70 tx=0x7fb63c00b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.579+0000 7fb648ff9700 1 -- 172.21.15.116:0/2706904527 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb63405ad30 con 0x7fb64c101e10 2024-03-04T16:01:42.595 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:42 smithi116 ceph-mon[29120]: pgmap v202: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-04T16:01:42.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.765+0000 7fb6516c7700 1 -- 172.21.15.116:0/2706904527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7fb64c109960 con 0x7fb64c101e10 2024-03-04T16:01:42.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.769+0000 7fb648ff9700 1 -- 172.21.15.116:0/2706904527 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v22)=0 v22) v1 ==== 126+0+0 (secure 0 0 0) 0x7fb63405a8c0 con 0x7fb64c101e10 2024-03-04T16:01:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 -- 172.21.15.116:0/2706904527 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb63806c4e0 msgr2=0x7fb63806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb63806c4e0 0x7fb63806e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb64c101c70 tx=0x7fb63c00b410 comp rx=0 tx=0).stop 2024-03-04T16:01:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 -- 172.21.15.116:0/2706904527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb64c101e10 msgr2=0x7fb64c073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb64c101e10 0x7fb64c073180 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb634006010 tx=0x7fb63400bd70 comp rx=0 tx=0).stop 2024-03-04T16:01:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 -- 172.21.15.116:0/2706904527 shutdown_connections 2024-03-04T16:01:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb64c100c10 0x7fb64c074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fb63806c4e0 0x7fb63806e9a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 --2- 172.21.15.116:0/2706904527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb64c101e10 0x7fb64c073180 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 -- 172.21.15.116:0/2706904527 >> 172.21.15.116:0/2706904527 conn(0x7fb64c0fc1b0 msgr2=0x7fb64c105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 -- 172.21.15.116:0/2706904527 shutdown_connections 2024-03-04T16:01:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:42.772+0000 7fb6516c7700 1 -- 172.21.15.116:0/2706904527 wait complete. 2024-03-04T16:01:42.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:42 smithi152 ceph-mon[36876]: pgmap v202: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-04T16:01:43.517 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-04T16:01:43.518 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:01:43.518 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-04T16:01:43.787 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:44.025 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:43 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2706904527' entity='client.admin' 2024-03-04T16:01:44.025 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:44.025 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:43 smithi116 ceph-mon[29120]: pgmap v203: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-04T16:01:44.025 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:44.025 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:44.025 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:43 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:44.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:43 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2706904527' entity='client.admin' 2024-03-04T16:01:44.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:44.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:43 smithi152 ceph-mon[36876]: pgmap v203: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-04T16:01:44.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:44.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:44.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:43 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:45.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.516+0000 7f47b2454700 1 -- 172.21.15.116:0/3948892543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47ac100bc0 msgr2=0x7f47ac100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:45.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.516+0000 7f47b2454700 1 --2- 172.21.15.116:0/3948892543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47ac100bc0 0x7f47ac100fe0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f479c009910 tx=0x7f479c009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:45.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.517+0000 7f47b2454700 1 -- 172.21.15.116:0/3948892543 shutdown_connections 2024-03-04T16:01:45.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.517+0000 7f47b2454700 1 --2- 172.21.15.116:0/3948892543 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47ac101dc0 0x7f47ac102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:45.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.517+0000 7f47b2454700 1 --2- 172.21.15.116:0/3948892543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47ac100bc0 0x7f47ac100fe0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:45.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.517+0000 7f47b2454700 1 -- 172.21.15.116:0/3948892543 >> 172.21.15.116:0/3948892543 conn(0x7f47ac0fc140 msgr2=0x7f47ac0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:45.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.517+0000 7f47b2454700 1 -- 172.21.15.116:0/3948892543 shutdown_connections 2024-03-04T16:01:45.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.517+0000 7f47b2454700 1 -- 172.21.15.116:0/3948892543 wait complete. 2024-03-04T16:01:45.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.517+0000 7f47b2454700 1 Processor -- start 2024-03-04T16:01:45.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.517+0000 7f47b2454700 1 -- start start 2024-03-04T16:01:45.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.517+0000 7f47b2454700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47ac100bc0 0x7f47ac1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:45.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47b2454700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47ac101dc0 0x7f47ac196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:45.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47b2454700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac196f50 con 0x7f47ac101dc0 2024-03-04T16:01:45.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47b2454700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac197090 con 0x7f47ac100bc0 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47ab7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47ac101dc0 0x7f47ac196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47ab7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47ac101dc0 0x7f47ac196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45874/0 (socket says 172.21.15.116:45874) 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47ab7fe700 1 -- 172.21.15.116:0/380950611 learned_addr learned my addr 172.21.15.116:0/380950611 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47abfff700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47ac100bc0 0x7f47ac1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47ab7fe700 1 -- 172.21.15.116:0/380950611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47ac100bc0 msgr2=0x7f47ac1963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47ab7fe700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47ac100bc0 0x7f47ac1963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47ab7fe700 1 -- 172.21.15.116:0/380950611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c0095f0 con 0x7f47ac101dc0 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47abfff700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47ac100bc0 0x7f47ac1963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47ab7fe700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47ac101dc0 0x7f47ac196930 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f47a000b4b0 tx=0x7f47a000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47a97fa700 1 -- 172.21.15.116:0/380950611 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a0011800 con 0x7f47ac101dc0 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.518+0000 7f47b2454700 1 -- 172.21.15.116:0/380950611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ac19bb40 con 0x7f47ac101dc0 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.519+0000 7f47a97fa700 1 -- 172.21.15.116:0/380950611 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f47a0011e10 con 0x7f47ac101dc0 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.519+0000 7f47a97fa700 1 -- 172.21.15.116:0/380950611 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a000f2e0 con 0x7f47ac101dc0 2024-03-04T16:01:45.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.519+0000 7f47b2454700 1 -- 172.21.15.116:0/380950611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ac19c110 con 0x7f47ac101dc0 2024-03-04T16:01:45.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.520+0000 7f47b2454700 1 -- 172.21.15.116:0/380950611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ac1097a0 con 0x7f47ac101dc0 2024-03-04T16:01:45.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.520+0000 7f47a97fa700 1 -- 172.21.15.116:0/380950611 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f47a0010380 con 0x7f47ac101dc0 2024-03-04T16:01:45.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.521+0000 7f47a97fa700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f479406c4e0 0x7f479406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:45.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.521+0000 7f47a97fa700 1 -- 172.21.15.116:0/380950611 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f47a008b3f0 con 0x7f47ac101dc0 2024-03-04T16:01:45.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.521+0000 7f47abfff700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f479406c4e0 0x7f479406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:45.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.521+0000 7f47abfff700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f479406c4e0 0x7f479406e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f479c00b5c0 tx=0x7f479c005e20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:45.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.525+0000 7f47a97fa700 1 -- 172.21.15.116:0/380950611 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f47a0059680 con 0x7f47ac101dc0 2024-03-04T16:01:45.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.717+0000 7f47b2454700 1 -- 172.21.15.116:0/380950611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f47ac109990 con 0x7f47ac101dc0 2024-03-04T16:01:45.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.718+0000 7f47a97fa700 1 -- 172.21.15.116:0/380950611 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v22)=0 v22) v1 ==== 155+0+0 (secure 0 0 0) 0x7f47a0059210 con 0x7f47ac101dc0 2024-03-04T16:01:45.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 -- 172.21.15.116:0/380950611 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f479406c4e0 msgr2=0x7f479406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:45.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f479406c4e0 0x7f479406e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f479c00b5c0 tx=0x7f479c005e20 comp rx=0 tx=0).stop 2024-03-04T16:01:45.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 -- 172.21.15.116:0/380950611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47ac101dc0 msgr2=0x7f47ac196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:45.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47ac101dc0 0x7f47ac196930 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f47a000b4b0 tx=0x7f47a000b870 comp rx=0 tx=0).stop 2024-03-04T16:01:45.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 -- 172.21.15.116:0/380950611 shutdown_connections 2024-03-04T16:01:45.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47ac100bc0 0x7f47ac1963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:45.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f479406c4e0 0x7f479406e9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:45.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 --2- 172.21.15.116:0/380950611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47ac101dc0 0x7f47ac196930 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:45.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 -- 172.21.15.116:0/380950611 >> 172.21.15.116:0/380950611 conn(0x7f47ac0fc140 msgr2=0x7f47ac104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:45.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 -- 172.21.15.116:0/380950611 shutdown_connections 2024-03-04T16:01:45.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:45.720+0000 7f47b2454700 1 -- 172.21.15.116:0/380950611 wait complete. 2024-03-04T16:01:46.393 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:46 smithi116 ceph-mon[29120]: pgmap v204: 65 pgs: 65 active+clean; 585 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-03-04T16:01:46.414 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-04T16:01:46.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:46 smithi152 ceph-mon[36876]: pgmap v204: 65 pgs: 65 active+clean; 585 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-03-04T16:01:46.624 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.325+0000 7f005e4e2700 1 -- 172.21.15.116:0/1258566085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0058100cb0 msgr2=0x7f0058101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.325+0000 7f005e4e2700 1 --2- 172.21.15.116:0/1258566085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0058100cb0 0x7f0058101110 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f004c009910 tx=0x7f004c009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.325+0000 7f005e4e2700 1 -- 172.21.15.116:0/1258566085 shutdown_connections 2024-03-04T16:01:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.325+0000 7f005e4e2700 1 --2- 172.21.15.116:0/1258566085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0058100cb0 0x7f0058101110 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.325+0000 7f005e4e2700 1 --2- 172.21.15.116:0/1258566085 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00580ffab0 0x7f00580ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.325+0000 7f005e4e2700 1 -- 172.21.15.116:0/1258566085 >> 172.21.15.116:0/1258566085 conn(0x7f00580fb030 msgr2=0x7f00580fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.325+0000 7f005e4e2700 1 -- 172.21.15.116:0/1258566085 shutdown_connections 2024-03-04T16:01:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.325+0000 7f005e4e2700 1 -- 172.21.15.116:0/1258566085 wait complete. 2024-03-04T16:01:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.325+0000 7f005e4e2700 1 Processor -- start 2024-03-04T16:01:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.325+0000 7f005e4e2700 1 -- start start 2024-03-04T16:01:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f005e4e2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00580ffab0 0x7f0058068fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f005e4e2700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0058100cb0 0x7f005819b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f005e4e2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005819b680 con 0x7f00580ffab0 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f005e4e2700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005819b7c0 con 0x7f0058100cb0 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f0057fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00580ffab0 0x7f0058068fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f0057fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00580ffab0 0x7f0058068fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45884/0 (socket says 172.21.15.116:45884) 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f0057fff700 1 -- 172.21.15.116:0/358899552 learned_addr learned my addr 172.21.15.116:0/358899552 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f00577fe700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0058100cb0 0x7f005819b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f0057fff700 1 -- 172.21.15.116:0/358899552 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0058100cb0 msgr2=0x7f005819b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f0057fff700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0058100cb0 0x7f005819b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f0057fff700 1 -- 172.21.15.116:0/358899552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c0095f0 con 0x7f00580ffab0 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.326+0000 7f0057fff700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00580ffab0 0x7f0058068fd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f004800d900 tx=0x7f004800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.327+0000 7f00557fa700 1 -- 172.21.15.116:0/358899552 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00480041f0 con 0x7f00580ffab0 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.327+0000 7f00557fa700 1 -- 172.21.15.116:0/358899552 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0048004350 con 0x7f00580ffab0 2024-03-04T16:01:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.327+0000 7f005e4e2700 1 -- 172.21.15.116:0/358899552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005819baa0 con 0x7f00580ffab0 2024-03-04T16:01:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.327+0000 7f00557fa700 1 -- 172.21.15.116:0/358899552 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0048003d20 con 0x7f00580ffab0 2024-03-04T16:01:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.327+0000 7f005e4e2700 1 -- 172.21.15.116:0/358899552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005819bff0 con 0x7f00580ffab0 2024-03-04T16:01:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.328+0000 7f005e4e2700 1 -- 172.21.15.116:0/358899552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0058108610 con 0x7f00580ffab0 2024-03-04T16:01:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.328+0000 7f00557fa700 1 -- 172.21.15.116:0/358899552 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0048009730 con 0x7f00580ffab0 2024-03-04T16:01:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.329+0000 7f00557fa700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f004006c4e0 0x7f004006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.329+0000 7f00557fa700 1 -- 172.21.15.116:0/358899552 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f004808b690 con 0x7f00580ffab0 2024-03-04T16:01:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.329+0000 7f00577fe700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f004006c4e0 0x7f004006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.330+0000 7f00577fe700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f004006c4e0 0x7f004006e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f004c006010 tx=0x7f004c00b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.334+0000 7f00557fa700 1 -- 172.21.15.116:0/358899552 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0048059920 con 0x7f00580ffab0 2024-03-04T16:01:48.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:48 smithi116 ceph-mon[29120]: pgmap v205: 65 pgs: 65 active+clean; 585 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-03-04T16:01:48.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.525+0000 7f005e4e2700 1 -- 172.21.15.116:0/358899552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f005819c6f0 con 0x7f00580ffab0 2024-03-04T16:01:48.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.525+0000 7f00557fa700 1 -- 172.21.15.116:0/358899552 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v22)=0 v22) v1 ==== 163+0+0 (secure 0 0 0) 0x7f0048003e80 con 0x7f00580ffab0 2024-03-04T16:01:48.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 -- 172.21.15.116:0/358899552 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f004006c4e0 msgr2=0x7f004006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:48.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f004006c4e0 0x7f004006e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f004c006010 tx=0x7f004c00b540 comp rx=0 tx=0).stop 2024-03-04T16:01:48.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 -- 172.21.15.116:0/358899552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00580ffab0 msgr2=0x7f0058068fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00580ffab0 0x7f0058068fd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f004800d900 tx=0x7f004800dc10 comp rx=0 tx=0).stop 2024-03-04T16:01:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 -- 172.21.15.116:0/358899552 shutdown_connections 2024-03-04T16:01:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f004006c4e0 0x7f004006e9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00580ffab0 0x7f0058068fd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 --2- 172.21.15.116:0/358899552 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0058100cb0 0x7f005819b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 -- 172.21.15.116:0/358899552 >> 172.21.15.116:0/358899552 conn(0x7f00580fb030 msgr2=0x7f0058103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 -- 172.21.15.116:0/358899552 shutdown_connections 2024-03-04T16:01:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:48.527+0000 7f005e4e2700 1 -- 172.21.15.116:0/358899552 wait complete. 2024-03-04T16:01:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:48 smithi152 ceph-mon[36876]: pgmap v205: 65 pgs: 65 active+clean; 585 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-03-04T16:01:49.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-04T16:01:49.545 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:50.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:50 smithi152 ceph-mon[36876]: pgmap v206: 65 pgs: 65 active+clean; 585 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-03-04T16:01:50.684 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:50 smithi116 ceph-mon[29120]: pgmap v206: 65 pgs: 65 active+clean; 585 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-03-04T16:01:51.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.180+0000 7fe723349700 1 -- 172.21.15.116:0/1884005295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe71c100be0 msgr2=0x7fe71c101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:51.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.180+0000 7fe723349700 1 --2- 172.21.15.116:0/1884005295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe71c100be0 0x7fe71c101000 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe718009910 tx=0x7fe718009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:51.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.180+0000 7fe723349700 1 -- 172.21.15.116:0/1884005295 shutdown_connections 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.180+0000 7fe723349700 1 --2- 172.21.15.116:0/1884005295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe71c101de0 0x7fe71c102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.180+0000 7fe723349700 1 --2- 172.21.15.116:0/1884005295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe71c100be0 0x7fe71c101000 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.180+0000 7fe723349700 1 -- 172.21.15.116:0/1884005295 >> 172.21.15.116:0/1884005295 conn(0x7fe71c0fc140 msgr2=0x7fe71c0fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.181+0000 7fe723349700 1 -- 172.21.15.116:0/1884005295 shutdown_connections 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.181+0000 7fe723349700 1 -- 172.21.15.116:0/1884005295 wait complete. 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.181+0000 7fe723349700 1 Processor -- start 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.181+0000 7fe723349700 1 -- start start 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.181+0000 7fe723349700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe71c100be0 0x7fe71c196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.181+0000 7fe723349700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe71c101de0 0x7fe71c196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.181+0000 7fe723349700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c196f70 con 0x7fe71c100be0 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.181+0000 7fe723349700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c1970b0 con 0x7fe71c101de0 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.181+0000 7fe7210e5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe71c100be0 0x7fe71c196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe7210e5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe71c100be0 0x7fe71c196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45896/0 (socket says 172.21.15.116:45896) 2024-03-04T16:01:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe7210e5700 1 -- 172.21.15.116:0/1361429645 learned_addr learned my addr 172.21.15.116:0/1361429645 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe7208e4700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe71c101de0 0x7fe71c196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe7210e5700 1 -- 172.21.15.116:0/1361429645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe71c101de0 msgr2=0x7fe71c196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe7210e5700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe71c101de0 0x7fe71c196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe7210e5700 1 -- 172.21.15.116:0/1361429645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7180095f0 con 0x7fe71c100be0 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe7210e5700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe71c100be0 0x7fe71c196410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe71800bda0 tx=0x7fe71800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe7127fc700 1 -- 172.21.15.116:0/1361429645 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe71801d070 con 0x7fe71c100be0 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe723349700 1 -- 172.21.15.116:0/1361429645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71c19bb00 con 0x7fe71c100be0 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe7127fc700 1 -- 172.21.15.116:0/1361429645 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe718004c50 con 0x7fe71c100be0 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe7127fc700 1 -- 172.21.15.116:0/1361429645 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe71800f690 con 0x7fe71c100be0 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.182+0000 7fe723349700 1 -- 172.21.15.116:0/1361429645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71c19c070 con 0x7fe71c100be0 2024-03-04T16:01:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.183+0000 7fe723349700 1 -- 172.21.15.116:0/1361429645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe71c1097c0 con 0x7fe71c100be0 2024-03-04T16:01:51.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.183+0000 7fe7127fc700 1 -- 172.21.15.116:0/1361429645 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe71800f900 con 0x7fe71c100be0 2024-03-04T16:01:51.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.184+0000 7fe7127fc700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fe70806c4e0 0x7fe70806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:51.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.184+0000 7fe7127fc700 1 -- 172.21.15.116:0/1361429645 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe71808d040 con 0x7fe71c100be0 2024-03-04T16:01:51.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.184+0000 7fe7208e4700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fe70806c4e0 0x7fe70806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:51.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.184+0000 7fe7208e4700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fe70806c4e0 0x7fe70806e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe70c006fd0 tx=0x7fe70c008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:51.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.188+0000 7fe7127fc700 1 -- 172.21.15.116:0/1361429645 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe71805b250 con 0x7fe71c100be0 2024-03-04T16:01:51.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.377+0000 7fe723349700 1 -- 172.21.15.116:0/1361429645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fe71c19c6b0 con 0x7fe71c100be0 2024-03-04T16:01:51.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.377+0000 7fe7127fc700 1 -- 172.21.15.116:0/1361429645 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7fe71805ade0 con 0x7fe71c100be0 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.379+0000 7fe723349700 1 -- 172.21.15.116:0/1361429645 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fe70806c4e0 msgr2=0x7fe70806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.379+0000 7fe723349700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fe70806c4e0 0x7fe70806e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe70c006fd0 tx=0x7fe70c008040 comp rx=0 tx=0).stop 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.379+0000 7fe723349700 1 -- 172.21.15.116:0/1361429645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe71c100be0 msgr2=0x7fe71c196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.379+0000 7fe723349700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe71c100be0 0x7fe71c196410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe71800bda0 tx=0x7fe71800be80 comp rx=0 tx=0).stop 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.379+0000 7fe723349700 1 -- 172.21.15.116:0/1361429645 shutdown_connections 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.379+0000 7fe723349700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fe70806c4e0 0x7fe70806e9a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.379+0000 7fe723349700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe71c100be0 0x7fe71c196410 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.379+0000 7fe723349700 1 --2- 172.21.15.116:0/1361429645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe71c101de0 0x7fe71c196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.379+0000 7fe723349700 1 -- 172.21.15.116:0/1361429645 >> 172.21.15.116:0/1361429645 conn(0x7fe71c0fc140 msgr2=0x7fe71c105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.380+0000 7fe723349700 1 -- 172.21.15.116:0/1361429645 shutdown_connections 2024-03-04T16:01:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:51.380+0000 7fe723349700 1 -- 172.21.15.116:0/1361429645 wait complete. 2024-03-04T16:01:52.060 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-04T16:01:52.278 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:52.591 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:52 smithi116 ceph-mon[29120]: pgmap v207: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-03-04T16:01:52.591 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:01:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:52 smithi152 ceph-mon[36876]: pgmap v207: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-03-04T16:01:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:01:54.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.100+0000 7f9322f55700 1 -- 172.21.15.116:0/657586521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f931c0ff290 msgr2=0x7f931c0ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:54.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.100+0000 7f9322f55700 1 --2- 172.21.15.116:0/657586521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f931c0ff290 0x7f931c0ff6b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f931000aa20 tx=0x7f931000ad30 comp rx=0 tx=0).stop 2024-03-04T16:01:54.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.100+0000 7f9322f55700 1 -- 172.21.15.116:0/657586521 shutdown_connections 2024-03-04T16:01:54.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.100+0000 7f9322f55700 1 --2- 172.21.15.116:0/657586521 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f931c100490 0x7f931c1008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:54.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.100+0000 7f9322f55700 1 --2- 172.21.15.116:0/657586521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f931c0ff290 0x7f931c0ff6b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:54.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.100+0000 7f9322f55700 1 -- 172.21.15.116:0/657586521 >> 172.21.15.116:0/657586521 conn(0x7f931c0fa810 msgr2=0x7f931c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:54.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.100+0000 7f9322f55700 1 -- 172.21.15.116:0/657586521 shutdown_connections 2024-03-04T16:01:54.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.100+0000 7f9322f55700 1 -- 172.21.15.116:0/657586521 wait complete. 2024-03-04T16:01:54.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9322f55700 1 Processor -- start 2024-03-04T16:01:54.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9322f55700 1 -- start start 2024-03-04T16:01:54.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9322f55700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f931c0ff290 0x7f931c196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:54.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9322f55700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f931c100490 0x7f931c196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:54.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9322f55700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c197060 con 0x7f931c0ff290 2024-03-04T16:01:54.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9322f55700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c1971a0 con 0x7f931c100490 2024-03-04T16:01:54.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9320cf1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f931c0ff290 0x7f931c196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:54.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9320cf1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f931c0ff290 0x7f931c196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45900/0 (socket says 172.21.15.116:45900) 2024-03-04T16:01:54.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9320cf1700 1 -- 172.21.15.116:0/1106518360 learned_addr learned my addr 172.21.15.116:0/1106518360 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:54.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f931bfff700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f931c100490 0x7f931c196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:54.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9320cf1700 1 -- 172.21.15.116:0/1106518360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f931c100490 msgr2=0x7f931c196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:54.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.101+0000 7f9320cf1700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f931c100490 0x7f931c196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:54.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.102+0000 7f9320cf1700 1 -- 172.21.15.116:0/1106518360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931000a700 con 0x7f931c0ff290 2024-03-04T16:01:54.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.102+0000 7f931bfff700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f931c100490 0x7f931c196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:01:54.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.102+0000 7f9320cf1700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f931c0ff290 0x7f931c196500 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f931000d010 tx=0x7f931000fe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:54.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.102+0000 7f9319ffb700 1 -- 172.21.15.116:0/1106518360 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9310005070 con 0x7f931c0ff290 2024-03-04T16:01:54.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.102+0000 7f9319ffb700 1 -- 172.21.15.116:0/1106518360 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f93100051d0 con 0x7f931c0ff290 2024-03-04T16:01:54.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.102+0000 7f9319ffb700 1 -- 172.21.15.116:0/1106518360 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9310016990 con 0x7f931c0ff290 2024-03-04T16:01:54.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.102+0000 7f9322f55700 1 -- 172.21.15.116:0/1106518360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c19bbf0 con 0x7f931c0ff290 2024-03-04T16:01:54.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.102+0000 7f9322f55700 1 -- 172.21.15.116:0/1106518360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c19c0e0 con 0x7f931c0ff290 2024-03-04T16:01:54.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.103+0000 7f9322f55700 1 -- 172.21.15.116:0/1106518360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c1075e0 con 0x7f931c0ff290 2024-03-04T16:01:54.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.104+0000 7f9319ffb700 1 -- 172.21.15.116:0/1106518360 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f9310016bc0 con 0x7f931c0ff290 2024-03-04T16:01:54.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.104+0000 7f9319ffb700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f930c06c4e0 0x7f930c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:54.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.104+0000 7f931bfff700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f930c06c4e0 0x7f930c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:54.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.104+0000 7f9319ffb700 1 -- 172.21.15.116:0/1106518360 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9310012070 con 0x7f931c0ff290 2024-03-04T16:01:54.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.105+0000 7f931bfff700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f930c06c4e0 0x7f930c06e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f93080099d0 tx=0x7f9308008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.109+0000 7f9319ffb700 1 -- 172.21.15.116:0/1106518360 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f931005b890 con 0x7f931c0ff290 2024-03-04T16:01:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.314+0000 7f9322f55700 1 -- 172.21.15.116:0/1106518360 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", "target": ["mon-mgr", ""]}) v1 -- 0x7f931c104de0 con 0x7f930c06c4e0 2024-03-04T16:01:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.320+0000 7f9319ffb700 1 -- 172.21.15.116:0/1106518360 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f931c104de0 con 0x7f930c06c4e0 2024-03-04T16:01:54.321 INFO:teuthology.orchestra.run.smithi116.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:01:54.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 -- 172.21.15.116:0/1106518360 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f930c06c4e0 msgr2=0x7f930c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:54.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f930c06c4e0 0x7f930c06e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f93080099d0 tx=0x7f9308008040 comp rx=0 tx=0).stop 2024-03-04T16:01:54.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 -- 172.21.15.116:0/1106518360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f931c0ff290 msgr2=0x7f931c196500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:54.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f931c0ff290 0x7f931c196500 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f931000d010 tx=0x7f931000fe40 comp rx=0 tx=0).stop 2024-03-04T16:01:54.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 -- 172.21.15.116:0/1106518360 shutdown_connections 2024-03-04T16:01:54.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f930c06c4e0 0x7f930c06e9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:54.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f931c0ff290 0x7f931c196500 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:54.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 --2- 172.21.15.116:0/1106518360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f931c100490 0x7f931c196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:54.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 -- 172.21.15.116:0/1106518360 >> 172.21.15.116:0/1106518360 conn(0x7f931c0fa810 msgr2=0x7f931c1036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:54.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 -- 172.21.15.116:0/1106518360 shutdown_connections 2024-03-04T16:01:54.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:54.322+0000 7f9322f55700 1 -- 172.21.15.116:0/1106518360 wait complete. 2024-03-04T16:01:54.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:54 smithi152 ceph-mon[36876]: pgmap v208: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-04T16:01:54.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:54 smithi116 ceph-mon[29120]: pgmap v208: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-04T16:01:55.049 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-04T16:01:55.049 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:01:55.050 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-04T16:01:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:01:55.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:55 smithi152 ceph-mon[36876]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:55.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:55 smithi152 ceph-mon[36876]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:01:55.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:55 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:55.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:55 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:55.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:55 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:55.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:55 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:55.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:55 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:55.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:55 smithi152 ceph-mon[36876]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:01:55.639 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:55 smithi116 ceph-mon[29120]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:55.640 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:55 smithi116 ceph-mon[29120]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:01:55.640 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:55 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:55.640 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:55 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:55.640 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:55 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:55.640 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:55 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:55.640 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:55 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:01:55.640 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:55 smithi116 ceph-mon[29120]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:01:56.457 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:56 smithi116 ceph-mon[29120]: pgmap v209: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-04T16:01:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:56 smithi152 ceph-mon[36876]: pgmap v209: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-04T16:01:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.376+0000 7fbdc3183700 1 -- 172.21.15.116:0/734472627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdbc100c00 msgr2=0x7fbdbc101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.376+0000 7fbdc3183700 1 --2- 172.21.15.116:0/734472627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdbc100c00 0x7fbdbc101020 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009910 tx=0x7fbdb0009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.376+0000 7fbdc3183700 1 -- 172.21.15.116:0/734472627 shutdown_connections 2024-03-04T16:01:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.376+0000 7fbdc3183700 1 --2- 172.21.15.116:0/734472627 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdbc101e00 0x7fbdbc102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.376+0000 7fbdc3183700 1 --2- 172.21.15.116:0/734472627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdbc100c00 0x7fbdbc101020 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.376+0000 7fbdc3183700 1 -- 172.21.15.116:0/734472627 >> 172.21.15.116:0/734472627 conn(0x7fbdbc0fc180 msgr2=0x7fbdbc0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.376+0000 7fbdc3183700 1 -- 172.21.15.116:0/734472627 shutdown_connections 2024-03-04T16:01:57.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.376+0000 7fbdc3183700 1 -- 172.21.15.116:0/734472627 wait complete. 2024-03-04T16:01:57.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.377+0000 7fbdc3183700 1 Processor -- start 2024-03-04T16:01:57.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.377+0000 7fbdc3183700 1 -- start start 2024-03-04T16:01:57.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.377+0000 7fbdc3183700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdbc100c00 0x7fbdbc196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:57.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.377+0000 7fbdc3183700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdbc101e00 0x7fbdbc1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:57.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.377+0000 7fbdc3183700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc196ef0 con 0x7fbdbc100c00 2024-03-04T16:01:57.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.377+0000 7fbdc3183700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc197030 con 0x7fbdbc101e00 2024-03-04T16:01:57.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.377+0000 7fbdc0f1f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdbc100c00 0x7fbdbc196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:57.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.377+0000 7fbdc0f1f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdbc100c00 0x7fbdbc196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49694/0 (socket says 172.21.15.116:49694) 2024-03-04T16:01:57.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.377+0000 7fbdc0f1f700 1 -- 172.21.15.116:0/407465374 learned_addr learned my addr 172.21.15.116:0/407465374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:57.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.377+0000 7fbdbbfff700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdbc101e00 0x7fbdbc1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:57.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.378+0000 7fbdbbfff700 1 -- 172.21.15.116:0/407465374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdbc100c00 msgr2=0x7fbdbc196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:57.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.378+0000 7fbdbbfff700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdbc100c00 0x7fbdbc196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:57.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.378+0000 7fbdbbfff700 1 -- 172.21.15.116:0/407465374 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb00095f0 con 0x7fbdbc101e00 2024-03-04T16:01:57.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.378+0000 7fbdc0f1f700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdbc100c00 0x7fbdbc196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T16:01:57.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.378+0000 7fbdbbfff700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdbc101e00 0x7fbdbc1968d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbda800d8d0 tx=0x7fbda800dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.378+0000 7fbdb9ffb700 1 -- 172.21.15.116:0/407465374 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda800f860 con 0x7fbdbc101e00 2024-03-04T16:01:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.378+0000 7fbdc3183700 1 -- 172.21.15.116:0/407465374 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdbc19bae0 con 0x7fbdbc101e00 2024-03-04T16:01:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.378+0000 7fbdb9ffb700 1 -- 172.21.15.116:0/407465374 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbda800fe70 con 0x7fbdbc101e00 2024-03-04T16:01:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.378+0000 7fbdb9ffb700 1 -- 172.21.15.116:0/407465374 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda800e5e0 con 0x7fbdbc101e00 2024-03-04T16:01:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.379+0000 7fbdc3183700 1 -- 172.21.15.116:0/407465374 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdbc19c030 con 0x7fbdbc101e00 2024-03-04T16:01:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.380+0000 7fbdc3183700 1 -- 172.21.15.116:0/407465374 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdbc109760 con 0x7fbdbc101e00 2024-03-04T16:01:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.380+0000 7fbdb9ffb700 1 -- 172.21.15.116:0/407465374 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fbda8010460 con 0x7fbdbc101e00 2024-03-04T16:01:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.380+0000 7fbdb9ffb700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbdac06c4e0 0x7fbdac06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.380+0000 7fbdb9ffb700 1 -- 172.21.15.116:0/407465374 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fbda808b450 con 0x7fbdbc101e00 2024-03-04T16:01:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.380+0000 7fbdc0f1f700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbdac06c4e0 0x7fbdac06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.381+0000 7fbdc0f1f700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbdac06c4e0 0x7fbdac06e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0006010 tx=0x7fbdb000b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.385+0000 7fbdb9ffb700 1 -- 172.21.15.116:0/407465374 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbda8059640 con 0x7fbdbc101e00 2024-03-04T16:01:57.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.629+0000 7fbdc3183700 1 -- 172.21.15.116:0/407465374 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdbc106750 con 0x7fbdac06c4e0 2024-03-04T16:01:57.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.631+0000 7fbdb9ffb700 1 -- 172.21.15.116:0/407465374 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fbdbc106750 con 0x7fbdac06c4e0 2024-03-04T16:01:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.633+0000 7fbdc3183700 1 -- 172.21.15.116:0/407465374 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbdac06c4e0 msgr2=0x7fbdac06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.633+0000 7fbdc3183700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbdac06c4e0 0x7fbdac06e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0006010 tx=0x7fbdb000b540 comp rx=0 tx=0).stop 2024-03-04T16:01:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.633+0000 7fbdc3183700 1 -- 172.21.15.116:0/407465374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdbc101e00 msgr2=0x7fbdbc1968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.633+0000 7fbdc3183700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdbc101e00 0x7fbdbc1968d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbda800d8d0 tx=0x7fbda800dbe0 comp rx=0 tx=0).stop 2024-03-04T16:01:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.634+0000 7fbdc3183700 1 -- 172.21.15.116:0/407465374 shutdown_connections 2024-03-04T16:01:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.634+0000 7fbdc3183700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fbdac06c4e0 0x7fbdac06e9a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.634+0000 7fbdc3183700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdbc100c00 0x7fbdbc196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.634+0000 7fbdc3183700 1 --2- 172.21.15.116:0/407465374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdbc101e00 0x7fbdbc1968d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.634+0000 7fbdc3183700 1 -- 172.21.15.116:0/407465374 >> 172.21.15.116:0/407465374 conn(0x7fbdbc0fc180 msgr2=0x7fbdbc105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.634+0000 7fbdc3183700 1 -- 172.21.15.116:0/407465374 shutdown_connections 2024-03-04T16:01:57.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.634+0000 7fbdc3183700 1 -- 172.21.15.116:0/407465374 wait complete. 2024-03-04T16:01:57.646 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:01:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.749+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/587080025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c58100cb0 msgr2=0x7f5c58101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.749+0000 7f5c5f6b0700 1 --2- 172.21.15.116:0/587080025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c58100cb0 0x7f5c58101110 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009910 tx=0x7f5c48009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.749+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/587080025 shutdown_connections 2024-03-04T16:01:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.749+0000 7f5c5f6b0700 1 --2- 172.21.15.116:0/587080025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c58100cb0 0x7f5c58101110 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.749+0000 7f5c5f6b0700 1 --2- 172.21.15.116:0/587080025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c580ffab0 0x7f5c580ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.749+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/587080025 >> 172.21.15.116:0/587080025 conn(0x7f5c580fb030 msgr2=0x7f5c580fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.749+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/587080025 shutdown_connections 2024-03-04T16:01:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.749+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/587080025 wait complete. 2024-03-04T16:01:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.750+0000 7f5c5f6b0700 1 Processor -- start 2024-03-04T16:01:57.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.750+0000 7f5c5f6b0700 1 -- start start 2024-03-04T16:01:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.750+0000 7f5c5f6b0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c580ffab0 0x7f5c58196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.750+0000 7f5c5f6b0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c58100cb0 0x7f5c58196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.750+0000 7f5c5f6b0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58196ea0 con 0x7f5c58100cb0 2024-03-04T16:01:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.750+0000 7f5c5f6b0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58196fe0 con 0x7f5c580ffab0 2024-03-04T16:01:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.750+0000 7f5c5cc4b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c58100cb0 0x7f5c58196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.750+0000 7f5c5cc4b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c58100cb0 0x7f5c58196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49706/0 (socket says 172.21.15.116:49706) 2024-03-04T16:01:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.750+0000 7f5c5cc4b700 1 -- 172.21.15.116:0/2433406405 learned_addr learned my addr 172.21.15.116:0/2433406405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.750+0000 7f5c5d44c700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c580ffab0 0x7f5c58196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.751+0000 7f5c5cc4b700 1 -- 172.21.15.116:0/2433406405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c580ffab0 msgr2=0x7f5c58196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.751+0000 7f5c5cc4b700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c580ffab0 0x7f5c58196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.751+0000 7f5c5cc4b700 1 -- 172.21.15.116:0/2433406405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c480095f0 con 0x7f5c58100cb0 2024-03-04T16:01:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.751+0000 7f5c5cc4b700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c58100cb0 0x7f5c58196880 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5c48006010 tx=0x7f5c4800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.751+0000 7f5c4e7fc700 1 -- 172.21.15.116:0/2433406405 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c4801d070 con 0x7f5c58100cb0 2024-03-04T16:01:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.751+0000 7f5c4e7fc700 1 -- 172.21.15.116:0/2433406405 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5c4800f460 con 0x7f5c58100cb0 2024-03-04T16:01:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.751+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/2433406405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5819ba30 con 0x7f5c58100cb0 2024-03-04T16:01:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.751+0000 7f5c4e7fc700 1 -- 172.21.15.116:0/2433406405 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c48021640 con 0x7f5c58100cb0 2024-03-04T16:01:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.751+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/2433406405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5819bf20 con 0x7f5c58100cb0 2024-03-04T16:01:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.752+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/2433406405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c58108610 con 0x7f5c58100cb0 2024-03-04T16:01:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.753+0000 7f5c4e7fc700 1 -- 172.21.15.116:0/2433406405 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5c4800fab0 con 0x7f5c58100cb0 2024-03-04T16:01:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.753+0000 7f5c4e7fc700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f5c4406c4e0 0x7f5c4406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.753+0000 7f5c4e7fc700 1 -- 172.21.15.116:0/2433406405 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5c4808cb20 con 0x7f5c58100cb0 2024-03-04T16:01:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.753+0000 7f5c5d44c700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f5c4406c4e0 0x7f5c4406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.754+0000 7f5c5d44c700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f5c4406c4e0 0x7f5c4406e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5c58100b10 tx=0x7f5c5400b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:57.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.758+0000 7f5c4e7fc700 1 -- 172.21.15.116:0/2433406405 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5c4805ad30 con 0x7f5c58100cb0 2024-03-04T16:01:57.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.996+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/2433406405 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c58068b20 con 0x7f5c4406c4e0 2024-03-04T16:01:57.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:57.998+0000 7f5c4e7fc700 1 -- 172.21.15.116:0/2433406405 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f5c58068b20 con 0x7f5c4406c4e0 2024-03-04T16:01:58.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/2433406405 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f5c4406c4e0 msgr2=0x7f5c4406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f5c4406c4e0 0x7f5c4406e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5c58100b10 tx=0x7f5c5400b410 comp rx=0 tx=0).stop 2024-03-04T16:01:58.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/2433406405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c58100cb0 msgr2=0x7f5c58196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c58100cb0 0x7f5c58196880 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5c48006010 tx=0x7f5c4800bd70 comp rx=0 tx=0).stop 2024-03-04T16:01:58.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/2433406405 shutdown_connections 2024-03-04T16:01:58.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c580ffab0 0x7f5c58196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f5c4406c4e0 0x7f5c4406e9a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 --2- 172.21.15.116:0/2433406405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c58100cb0 0x7f5c58196880 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/2433406405 >> 172.21.15.116:0/2433406405 conn(0x7f5c580fb030 msgr2=0x7f5c58068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:58.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/2433406405 shutdown_connections 2024-03-04T16:01:58.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.000+0000 7f5c5f6b0700 1 -- 172.21.15.116:0/2433406405 wait complete. 2024-03-04T16:01:58.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.109+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/3709111828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf4101640 msgr2=0x7f0cf4103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.109+0000 7f0cfa4d4700 1 --2- 172.21.15.116:0/3709111828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf4101640 0x7f0cf4103a30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009910 tx=0x7f0ce4009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:58.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.109+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/3709111828 shutdown_connections 2024-03-04T16:01:58.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.109+0000 7f0cfa4d4700 1 --2- 172.21.15.116:0/3709111828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0cf4103f70 0x7f0cf4106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.109+0000 7f0cfa4d4700 1 --2- 172.21.15.116:0/3709111828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf4101640 0x7f0cf4103a30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.109+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/3709111828 >> 172.21.15.116:0/3709111828 conn(0x7f0cf40fb030 msgr2=0x7f0cf40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:58.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.109+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/3709111828 shutdown_connections 2024-03-04T16:01:58.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.109+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/3709111828 wait complete. 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.110+0000 7f0cfa4d4700 1 Processor -- start 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.110+0000 7f0cfa4d4700 1 -- start start 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.110+0000 7f0cfa4d4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf4101640 0x7f0cf4196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.110+0000 7f0cfa4d4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0cf4103f70 0x7f0cf4196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.110+0000 7f0cfa4d4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4196e60 con 0x7f0cf4101640 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.110+0000 7f0cfa4d4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4196fa0 con 0x7f0cf4103f70 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf4101640 0x7f0cf4196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf4101640 0x7f0cf4196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49714/0 (socket says 172.21.15.116:49714) 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf3fff700 1 -- 172.21.15.116:0/652270740 learned_addr learned my addr 172.21.15.116:0/652270740 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf37fe700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0cf4103f70 0x7f0cf4196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf3fff700 1 -- 172.21.15.116:0/652270740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0cf4103f70 msgr2=0x7f0cf4196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf3fff700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0cf4103f70 0x7f0cf4196840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf3fff700 1 -- 172.21.15.116:0/652270740 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40095f0 con 0x7f0cf4101640 2024-03-04T16:01:58.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf3fff700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf4101640 0x7f0cf4196300 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4006010 tx=0x7f0ce400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:58.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf17fa700 1 -- 172.21.15.116:0/652270740 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ce401d070 con 0x7f0cf4101640 2024-03-04T16:01:58.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf17fa700 1 -- 172.21.15.116:0/652270740 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0ce400f460 con 0x7f0cf4101640 2024-03-04T16:01:58.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/652270740 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf419b9f0 con 0x7f0cf4101640 2024-03-04T16:01:58.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cf17fa700 1 -- 172.21.15.116:0/652270740 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ce4017630 con 0x7f0cf4101640 2024-03-04T16:01:58.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.111+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/652270740 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf419bee0 con 0x7f0cf4101640 2024-03-04T16:01:58.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.112+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/652270740 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf4190520 con 0x7f0cf4101640 2024-03-04T16:01:58.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.113+0000 7f0cf17fa700 1 -- 172.21.15.116:0/652270740 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0ce4003a60 con 0x7f0cf4101640 2024-03-04T16:01:58.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.113+0000 7f0cf17fa700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0cdc06c4e0 0x7f0cdc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:58.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.113+0000 7f0cf17fa700 1 -- 172.21.15.116:0/652270740 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f0ce408cdf0 con 0x7f0cf4101640 2024-03-04T16:01:58.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.113+0000 7f0cf37fe700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0cdc06c4e0 0x7f0cdc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:58.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.114+0000 7f0cf37fe700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0cdc06c4e0 0x7f0cdc06e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8005950 tx=0x7f0ce80058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.118+0000 7f0cf17fa700 1 -- 172.21.15.116:0/652270740 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0ce405b000 con 0x7f0cf4101640 2024-03-04T16:01:58.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.344+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/652270740 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cf40611d0 con 0x7f0cdc06c4e0 2024-03-04T16:01:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.355+0000 7f0cf17fa700 1 -- 172.21.15.116:0/652270740 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7f0cf40611d0 con 0x7f0cdc06c4e0 2024-03-04T16:01:58.356 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:01:58.356 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (5m) 28s ago 6m 25.1M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:01:58.356 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (6m) 28s ago 6m 7390k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:01:58.356 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (5m) 34s ago 5m 7285k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:01:58.356 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (6m) 28s ago 6m 6665k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:01:58.356 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (5m) 34s ago 5m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (4m) 28s ago 6m 87.2M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (46s) 28s ago 46s 15.3M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (54s) 28s ago 53s 11.3M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (44s) 34s ago 43s 12.9M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (51s) 34s ago 51s 16.2M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:9283,8765,8443 running (8m) 28s ago 8m 489M - 18.2.1 5be31c24972a 083879ca4bf4 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (5m) 34s ago 5m 443M - 18.2.1 5be31c24972a c68d3e4a1dbc 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (8m) 28s ago 8m 45.4M 2048M 18.2.1 5be31c24972a 60e0bf118c4d 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (5m) 34s ago 5m 37.6M 2048M 18.2.1 5be31c24972a 52450d235516 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (6m) 28s ago 6m 15.4M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (5m) 34s ago 5m 16.9M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (4m) 28s ago 4m 50.5M 3946M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (4m) 28s ago 4m 46.8M 3946M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:01:58.357 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (3m) 28s ago 3m 46.4M 3946M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:01:58.358 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (3m) 34s ago 3m 46.2M 4970M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:01:58.358 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (2m) 34s ago 2m 43.8M 4970M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:01:58.358 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (2m) 34s ago 2m 44.0M 4970M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:01:58.358 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (4m) 28s ago 5m 55.8M - 2.43.0 a07b618ecd1d 31240d908f24 2024-03-04T16:01:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/652270740 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0cdc06c4e0 msgr2=0x7f0cdc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0cdc06c4e0 0x7f0cdc06e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8005950 tx=0x7f0ce80058e0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/652270740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf4101640 msgr2=0x7f0cf4196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf4101640 0x7f0cf4196300 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4006010 tx=0x7f0ce400be20 comp rx=0 tx=0).stop 2024-03-04T16:01:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/652270740 shutdown_connections 2024-03-04T16:01:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0cdc06c4e0 0x7f0cdc06e9a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf4101640 0x7f0cf4196300 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 --2- 172.21.15.116:0/652270740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0cf4103f70 0x7f0cf4196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/652270740 >> 172.21.15.116:0/652270740 conn(0x7f0cf40fb030 msgr2=0x7f0cf40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/652270740 shutdown_connections 2024-03-04T16:01:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.357+0000 7f0cfa4d4700 1 -- 172.21.15.116:0/652270740 wait complete. 2024-03-04T16:01:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.465+0000 7f251160b700 1 -- 172.21.15.116:0/4270842080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250c101f20 msgr2=0x7f250c102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.465+0000 7f251160b700 1 --2- 172.21.15.116:0/4270842080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250c101f20 0x7f250c102380 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2500009910 tx=0x7f2500009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.465+0000 7f251160b700 1 -- 172.21.15.116:0/4270842080 shutdown_connections 2024-03-04T16:01:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.465+0000 7f251160b700 1 --2- 172.21.15.116:0/4270842080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250c101f20 0x7f250c102380 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.465+0000 7f251160b700 1 --2- 172.21.15.116:0/4270842080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f250c0feea0 0x7f250c0ff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.465+0000 7f251160b700 1 -- 172.21.15.116:0/4270842080 >> 172.21.15.116:0/4270842080 conn(0x7f250c0fa7c0 msgr2=0x7f250c0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.465+0000 7f251160b700 1 -- 172.21.15.116:0/4270842080 shutdown_connections 2024-03-04T16:01:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.465+0000 7f251160b700 1 -- 172.21.15.116:0/4270842080 wait complete. 2024-03-04T16:01:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.466+0000 7f251160b700 1 Processor -- start 2024-03-04T16:01:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.466+0000 7f251160b700 1 -- start start 2024-03-04T16:01:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.466+0000 7f251160b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f250c0feea0 0x7f250c196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.466+0000 7f251160b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250c101f20 0x7f250c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.466+0000 7f251160b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c196e60 con 0x7f250c101f20 2024-03-04T16:01:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.466+0000 7f251160b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c196fa0 con 0x7f250c0feea0 2024-03-04T16:01:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f250a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250c101f20 0x7f250c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f250a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250c101f20 0x7f250c1968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49724/0 (socket says 172.21.15.116:49724) 2024-03-04T16:01:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f250a7fc700 1 -- 172.21.15.116:0/3941152202 learned_addr learned my addr 172.21.15.116:0/3941152202 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f250affd700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f250c0feea0 0x7f250c196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f250a7fc700 1 -- 172.21.15.116:0/3941152202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f250c0feea0 msgr2=0x7f250c196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f250a7fc700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f250c0feea0 0x7f250c196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f250a7fc700 1 -- 172.21.15.116:0/3941152202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25000095f0 con 0x7f250c101f20 2024-03-04T16:01:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f250a7fc700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250c101f20 0x7f250c1968d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f250000b5c0 tx=0x7f250000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f24f3fff700 1 -- 172.21.15.116:0/3941152202 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250001d070 con 0x7f250c101f20 2024-03-04T16:01:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f24f3fff700 1 -- 172.21.15.116:0/3941152202 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2500022470 con 0x7f250c101f20 2024-03-04T16:01:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f251160b700 1 -- 172.21.15.116:0/3941152202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250c19ba00 con 0x7f250c101f20 2024-03-04T16:01:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f24f3fff700 1 -- 172.21.15.116:0/3941152202 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250000f690 con 0x7f250c101f20 2024-03-04T16:01:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.467+0000 7f251160b700 1 -- 172.21.15.116:0/3941152202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250c19bec0 con 0x7f250c101f20 2024-03-04T16:01:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.468+0000 7f251160b700 1 -- 172.21.15.116:0/3941152202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250c109720 con 0x7f250c101f20 2024-03-04T16:01:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.469+0000 7f24f3fff700 1 -- 172.21.15.116:0/3941152202 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2500004b70 con 0x7f250c101f20 2024-03-04T16:01:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.470+0000 7f24f3fff700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f24f406c4e0 0x7f24f406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.470+0000 7f24f3fff700 1 -- 172.21.15.116:0/3941152202 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f250008ce80 con 0x7f250c101f20 2024-03-04T16:01:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.470+0000 7f250affd700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f24f406c4e0 0x7f24f406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.470+0000 7f250affd700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f24f406c4e0 0x7f24f406e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005fd0 tx=0x7f24fc005e50 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:58.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.474+0000 7f24f3fff700 1 -- 172.21.15.116:0/3941152202 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f250005b1c0 con 0x7f250c101f20 2024-03-04T16:01:58.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:58 smithi116 ceph-mon[29120]: pgmap v210: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-04T16:01:58.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.786+0000 7f251160b700 1 -- 172.21.15.116:0/3941152202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f250c19c980 con 0x7f250c101f20 2024-03-04T16:01:58.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.787+0000 7f24f3fff700 1 -- 172.21.15.116:0/3941152202 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f250005ad50 con 0x7f250c101f20 2024-03-04T16:01:58.788 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:01:58.788 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-03-04T16:01:58.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:01:58.790 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.789+0000 7f251160b700 1 -- 172.21.15.116:0/3941152202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f24f406c4e0 msgr2=0x7f24f406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.789+0000 7f251160b700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f24f406c4e0 0x7f24f406e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005fd0 tx=0x7f24fc005e50 comp rx=0 tx=0).stop 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.789+0000 7f251160b700 1 -- 172.21.15.116:0/3941152202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250c101f20 msgr2=0x7f250c1968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.789+0000 7f251160b700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250c101f20 0x7f250c1968d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f250000b5c0 tx=0x7f250000be20 comp rx=0 tx=0).stop 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.789+0000 7f251160b700 1 -- 172.21.15.116:0/3941152202 shutdown_connections 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.789+0000 7f251160b700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f250c0feea0 0x7f250c196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.789+0000 7f251160b700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f24f406c4e0 0x7f24f406e9a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.789+0000 7f251160b700 1 --2- 172.21.15.116:0/3941152202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250c101f20 0x7f250c1968d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.789+0000 7f251160b700 1 -- 172.21.15.116:0/3941152202 >> 172.21.15.116:0/3941152202 conn(0x7f250c0fa7c0 msgr2=0x7f250c105150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.790+0000 7f251160b700 1 -- 172.21.15.116:0/3941152202 shutdown_connections 2024-03-04T16:01:58.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.790+0000 7f251160b700 1 -- 172.21.15.116:0/3941152202 wait complete. 2024-03-04T16:01:58.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:58 smithi152 ceph-mon[36876]: pgmap v210: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-04T16:01:58.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.902+0000 7f10865ce700 1 -- 172.21.15.116:0/1001961950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1080101f20 msgr2=0x7f1080102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.902+0000 7f10865ce700 1 --2- 172.21.15.116:0/1001961950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1080101f20 0x7f1080102380 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1074009910 tx=0x7f1074009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:58.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.902+0000 7f10865ce700 1 -- 172.21.15.116:0/1001961950 shutdown_connections 2024-03-04T16:01:58.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.902+0000 7f10865ce700 1 --2- 172.21.15.116:0/1001961950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1080101f20 0x7f1080102380 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.902+0000 7f10865ce700 1 --2- 172.21.15.116:0/1001961950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10800feea0 0x7f10800ff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.902+0000 7f10865ce700 1 -- 172.21.15.116:0/1001961950 >> 172.21.15.116:0/1001961950 conn(0x7f10800fa7c0 msgr2=0x7f10800fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:58.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.902+0000 7f10865ce700 1 -- 172.21.15.116:0/1001961950 shutdown_connections 2024-03-04T16:01:58.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.902+0000 7f10865ce700 1 -- 172.21.15.116:0/1001961950 wait complete. 2024-03-04T16:01:58.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.902+0000 7f10865ce700 1 Processor -- start 2024-03-04T16:01:58.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.902+0000 7f10865ce700 1 -- start start 2024-03-04T16:01:58.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f10865ce700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10800feea0 0x7f1080196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:58.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f10865ce700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1080101f20 0x7f1080196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f10865ce700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1080196f80 con 0x7f1080101f20 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f10865ce700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10801970c0 con 0x7f10800feea0 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f107f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1080101f20 0x7f1080196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f107f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1080101f20 0x7f1080196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49736/0 (socket says 172.21.15.116:49736) 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f107f7fe700 1 -- 172.21.15.116:0/641463867 learned_addr learned my addr 172.21.15.116:0/641463867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f107ffff700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10800feea0 0x7f1080196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f107f7fe700 1 -- 172.21.15.116:0/641463867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10800feea0 msgr2=0x7f1080196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f107f7fe700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10800feea0 0x7f1080196420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f107f7fe700 1 -- 172.21.15.116:0/641463867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10740095f0 con 0x7f1080101f20 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f107ffff700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10800feea0 0x7f1080196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.903+0000 7f107f7fe700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1080101f20 0x7f1080196960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f107400b5c0 tx=0x7f107400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.904+0000 7f107d7fa700 1 -- 172.21.15.116:0/641463867 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f107401d070 con 0x7f1080101f20 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.904+0000 7f107d7fa700 1 -- 172.21.15.116:0/641463867 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1074022470 con 0x7f1080101f20 2024-03-04T16:01:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.904+0000 7f10865ce700 1 -- 172.21.15.116:0/641463867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108019bb10 con 0x7f1080101f20 2024-03-04T16:01:58.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.904+0000 7f107d7fa700 1 -- 172.21.15.116:0/641463867 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f107400f690 con 0x7f1080101f20 2024-03-04T16:01:58.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.904+0000 7f10865ce700 1 -- 172.21.15.116:0/641463867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108019c080 con 0x7f1080101f20 2024-03-04T16:01:58.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.905+0000 7f10865ce700 1 -- 172.21.15.116:0/641463867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10801097c0 con 0x7f1080101f20 2024-03-04T16:01:58.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.905+0000 7f107d7fa700 1 -- 172.21.15.116:0/641463867 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1074004b70 con 0x7f1080101f20 2024-03-04T16:01:58.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.906+0000 7f107d7fa700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106806c4e0 0x7f106806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:58.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.906+0000 7f107d7fa700 1 -- 172.21.15.116:0/641463867 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f107408d410 con 0x7f1080101f20 2024-03-04T16:01:58.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.906+0000 7f107ffff700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106806c4e0 0x7f106806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:58.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.907+0000 7f107ffff700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106806c4e0 0x7f106806e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1080068150 tx=0x7f1070008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:58.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:58.910+0000 7f107d7fa700 1 -- 172.21.15.116:0/641463867 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f107405b750 con 0x7f1080101f20 2024-03-04T16:01:59.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.171+0000 7f10865ce700 1 -- 172.21.15.116:0/641463867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f10801099b0 con 0x7f1080101f20 2024-03-04T16:01:59.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.172+0000 7f107d7fa700 1 -- 172.21.15.116:0/641463867 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1831 (secure 0 0 0) 0x7f1074027740 con 0x7f1080101f20 2024-03-04T16:01:59.173 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:01:59.173 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:01:59.173 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:01:59.173 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:01:59.173 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:01:59.174 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:59.176 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:01:59.176 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:01:59.176 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:01:59.176 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:01:59.176 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.174+0000 7f10865ce700 1 -- 172.21.15.116:0/641463867 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106806c4e0 msgr2=0x7f106806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.174+0000 7f10865ce700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106806c4e0 0x7f106806e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1080068150 tx=0x7f1070008040 comp rx=0 tx=0).stop 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.174+0000 7f10865ce700 1 -- 172.21.15.116:0/641463867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1080101f20 msgr2=0x7f1080196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.174+0000 7f10865ce700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1080101f20 0x7f1080196960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f107400b5c0 tx=0x7f107400be20 comp rx=0 tx=0).stop 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.175+0000 7f10865ce700 1 -- 172.21.15.116:0/641463867 shutdown_connections 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.175+0000 7f10865ce700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10800feea0 0x7f1080196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.175+0000 7f10865ce700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f106806c4e0 0x7f106806e9a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.175+0000 7f10865ce700 1 --2- 172.21.15.116:0/641463867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1080101f20 0x7f1080196960 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.175+0000 7f10865ce700 1 -- 172.21.15.116:0/641463867 >> 172.21.15.116:0/641463867 conn(0x7f10800fa7c0 msgr2=0x7f1080105150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.175+0000 7f10865ce700 1 -- 172.21.15.116:0/641463867 shutdown_connections 2024-03-04T16:01:59.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.175+0000 7f10865ce700 1 -- 172.21.15.116:0/641463867 wait complete. 2024-03-04T16:01:59.178 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:01:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.287+0000 7ffae16f3700 1 -- 172.21.15.116:0/700839250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffadc100cb0 msgr2=0x7ffadc101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.287+0000 7ffae16f3700 1 --2- 172.21.15.116:0/700839250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffadc100cb0 0x7ffadc101110 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ffad0009910 tx=0x7ffad0009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.287+0000 7ffae16f3700 1 -- 172.21.15.116:0/700839250 shutdown_connections 2024-03-04T16:01:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.287+0000 7ffae16f3700 1 --2- 172.21.15.116:0/700839250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffadc100cb0 0x7ffadc101110 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.287+0000 7ffae16f3700 1 --2- 172.21.15.116:0/700839250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffadc0ffab0 0x7ffadc0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.287+0000 7ffae16f3700 1 -- 172.21.15.116:0/700839250 >> 172.21.15.116:0/700839250 conn(0x7ffadc0fb030 msgr2=0x7ffadc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.287+0000 7ffae16f3700 1 -- 172.21.15.116:0/700839250 shutdown_connections 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.287+0000 7ffae16f3700 1 -- 172.21.15.116:0/700839250 wait complete. 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.288+0000 7ffae16f3700 1 Processor -- start 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.288+0000 7ffae16f3700 1 -- start start 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.288+0000 7ffae16f3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffadc0ffab0 0x7ffadc103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.288+0000 7ffae16f3700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffadc100cb0 0x7ffadc102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.288+0000 7ffae16f3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc1025d0 con 0x7ffadc0ffab0 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.288+0000 7ffae16f3700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc102710 con 0x7ffadc100cb0 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.288+0000 7ffadaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffadc0ffab0 0x7ffadc103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.288+0000 7ffadaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffadc0ffab0 0x7ffadc103a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49760/0 (socket says 172.21.15.116:49760) 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.288+0000 7ffadaffd700 1 -- 172.21.15.116:0/1079599178 learned_addr learned my addr 172.21.15.116:0/1079599178 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.289+0000 7ffada7fc700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffadc100cb0 0x7ffadc102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.289+0000 7ffadaffd700 1 -- 172.21.15.116:0/1079599178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffadc100cb0 msgr2=0x7ffadc102090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.289+0000 7ffadaffd700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffadc100cb0 0x7ffadc102090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.289+0000 7ffadaffd700 1 -- 172.21.15.116:0/1079599178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00095f0 con 0x7ffadc0ffab0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.289+0000 7ffadaffd700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffadc0ffab0 0x7ffadc103a40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00b480 tx=0x7ffacc00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.289+0000 7ffac3fff700 1 -- 172.21.15.116:0/1079599178 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffacc011800 con 0x7ffadc0ffab0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.289+0000 7ffac3fff700 1 -- 172.21.15.116:0/1079599178 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffacc011e10 con 0x7ffadc0ffab0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.289+0000 7ffac3fff700 1 -- 172.21.15.116:0/1079599178 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffacc010ea0 con 0x7ffadc0ffab0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.289+0000 7ffae16f3700 1 -- 172.21.15.116:0/1079599178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffadc1029f0 con 0x7ffadc0ffab0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.289+0000 7ffae16f3700 1 -- 172.21.15.116:0/1079599178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffadc102ef0 con 0x7ffadc0ffab0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.290+0000 7ffae16f3700 1 -- 172.21.15.116:0/1079599178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffadc1086b0 con 0x7ffadc0ffab0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.291+0000 7ffac3fff700 1 -- 172.21.15.116:0/1079599178 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ffacc010380 con 0x7ffadc0ffab0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.291+0000 7ffac3fff700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffac406c4e0 0x7ffac406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.292+0000 7ffac3fff700 1 -- 172.21.15.116:0/1079599178 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ffacc08b440 con 0x7ffadc0ffab0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.292+0000 7ffada7fc700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffac406c4e0 0x7ffac406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.292+0000 7ffada7fc700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffac406c4e0 0x7ffac406e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffad0006010 tx=0x7ffad000b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:59.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.296+0000 7ffac3fff700 1 -- 172.21.15.116:0/1079599178 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffacc059650 con 0x7ffadc0ffab0 2024-03-04T16:01:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:59 smithi116 ceph-mon[29120]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:59 smithi116 ceph-mon[29120]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:59 smithi116 ceph-mon[29120]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:59.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:59 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3941152202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:01:59.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:01:59 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/641463867' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:01:59.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.559+0000 7ffae16f3700 1 -- 172.21.15.116:0/1079599178 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffadc068b20 con 0x7ffac406c4e0 2024-03-04T16:01:59.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.560+0000 7ffac3fff700 1 -- 172.21.15.116:0/1079599178 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7ffadc068b20 con 0x7ffac406c4e0 2024-03-04T16:01:59.561 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:01:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:01:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:01:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:01:59.562 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2024-03-04T16:01:59.562 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "", 2024-03-04T16:01:59.562 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 image", 2024-03-04T16:01:59.562 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:01:59.562 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:01:59.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 -- 172.21.15.116:0/1079599178 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffac406c4e0 msgr2=0x7ffac406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:59.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffac406c4e0 0x7ffac406e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffad0006010 tx=0x7ffad000b540 comp rx=0 tx=0).stop 2024-03-04T16:01:59.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 -- 172.21.15.116:0/1079599178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffadc0ffab0 msgr2=0x7ffadc103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:59.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffadc0ffab0 0x7ffadc103a40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00b480 tx=0x7ffacc00b790 comp rx=0 tx=0).stop 2024-03-04T16:01:59.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 -- 172.21.15.116:0/1079599178 shutdown_connections 2024-03-04T16:01:59.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7ffac406c4e0 0x7ffac406e9a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffadc0ffab0 0x7ffadc103a40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 --2- 172.21.15.116:0/1079599178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffadc100cb0 0x7ffadc102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 -- 172.21.15.116:0/1079599178 >> 172.21.15.116:0/1079599178 conn(0x7ffadc0fb030 msgr2=0x7ffadc068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:59.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 -- 172.21.15.116:0/1079599178 shutdown_connections 2024-03-04T16:01:59.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.562+0000 7ffae16f3700 1 -- 172.21.15.116:0/1079599178 wait complete. 2024-03-04T16:01:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.673+0000 7fa626ccd700 1 -- 172.21.15.116:0/142801390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa620101640 msgr2=0x7fa620103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.673+0000 7fa626ccd700 1 --2- 172.21.15.116:0/142801390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa620101640 0x7fa620103a30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa614009910 tx=0x7fa614009c20 comp rx=0 tx=0).stop 2024-03-04T16:01:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.673+0000 7fa626ccd700 1 -- 172.21.15.116:0/142801390 shutdown_connections 2024-03-04T16:01:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.673+0000 7fa626ccd700 1 --2- 172.21.15.116:0/142801390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa620103f70 0x7fa620106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.673+0000 7fa626ccd700 1 --2- 172.21.15.116:0/142801390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa620101640 0x7fa620103a30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.673+0000 7fa626ccd700 1 -- 172.21.15.116:0/142801390 >> 172.21.15.116:0/142801390 conn(0x7fa6200fb030 msgr2=0x7fa6200fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.673+0000 7fa626ccd700 1 -- 172.21.15.116:0/142801390 shutdown_connections 2024-03-04T16:01:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.673+0000 7fa626ccd700 1 -- 172.21.15.116:0/142801390 wait complete. 2024-03-04T16:01:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.674+0000 7fa626ccd700 1 Processor -- start 2024-03-04T16:01:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.674+0000 7fa626ccd700 1 -- start start 2024-03-04T16:01:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.674+0000 7fa626ccd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa620101640 0x7fa620100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.674+0000 7fa626ccd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa620103f70 0x7fa6200ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.674+0000 7fa626ccd700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200ffb30 con 0x7fa620101640 2024-03-04T16:01:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.674+0000 7fa626ccd700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200ffc70 con 0x7fa620103f70 2024-03-04T16:01:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.674+0000 7fa624a69700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa620101640 0x7fa620100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.674+0000 7fa624a69700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa620101640 0x7fa620100fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49778/0 (socket says 172.21.15.116:49778) 2024-03-04T16:01:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.674+0000 7fa624a69700 1 -- 172.21.15.116:0/1551286546 learned_addr learned my addr 172.21.15.116:0/1551286546 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:01:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.674+0000 7fa61ffff700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa620103f70 0x7fa6200ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.675+0000 7fa624a69700 1 -- 172.21.15.116:0/1551286546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa620103f70 msgr2=0x7fa6200ff5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.675+0000 7fa624a69700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa620103f70 0x7fa6200ff5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.675+0000 7fa624a69700 1 -- 172.21.15.116:0/1551286546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6140095f0 con 0x7fa620101640 2024-03-04T16:01:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.675+0000 7fa624a69700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa620101640 0x7fa620100fa0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa614006010 tx=0x7fa61400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.675+0000 7fa61dffb700 1 -- 172.21.15.116:0/1551286546 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa61401d070 con 0x7fa620101640 2024-03-04T16:01:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.675+0000 7fa61dffb700 1 -- 172.21.15.116:0/1551286546 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa61400f460 con 0x7fa620101640 2024-03-04T16:01:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.675+0000 7fa61dffb700 1 -- 172.21.15.116:0/1551286546 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa614021640 con 0x7fa620101640 2024-03-04T16:01:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.675+0000 7fa626ccd700 1 -- 172.21.15.116:0/1551286546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6200ffef0 con 0x7fa620101640 2024-03-04T16:01:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.675+0000 7fa626ccd700 1 -- 172.21.15.116:0/1551286546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa620100390 con 0x7fa620101640 2024-03-04T16:01:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.676+0000 7fa626ccd700 1 -- 172.21.15.116:0/1551286546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6201905a0 con 0x7fa620101640 2024-03-04T16:01:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.677+0000 7fa61dffb700 1 -- 172.21.15.116:0/1551286546 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fa61400fab0 con 0x7fa620101640 2024-03-04T16:01:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.677+0000 7fa61dffb700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa61006c4e0 0x7fa61006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:01:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.677+0000 7fa61dffb700 1 -- 172.21.15.116:0/1551286546 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa61408cd30 con 0x7fa620101640 2024-03-04T16:01:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.677+0000 7fa61ffff700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa61006c4e0 0x7fa61006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:01:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.677+0000 7fa61ffff700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa61006c4e0 0x7fa61006e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa60c006fd0 tx=0x7fa60c008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:01:59.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.682+0000 7fa61dffb700 1 -- 172.21.15.116:0/1551286546 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa61405b040 con 0x7fa620101640 2024-03-04T16:01:59.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:59 smithi152 ceph-mon[36876]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:59.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:59 smithi152 ceph-mon[36876]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:59.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:59 smithi152 ceph-mon[36876]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:59.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:59 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3941152202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:01:59.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:01:59 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/641463867' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:01:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.975+0000 7fa626ccd700 1 -- 172.21.15.116:0/1551286546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa62002ced0 con 0x7fa620101640 2024-03-04T16:01:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.975+0000 7fa61dffb700 1 -- 172.21.15.116:0/1551286546 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fa614026020 con 0x7fa620101640 2024-03-04T16:01:59.977 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:01:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.978+0000 7fa626ccd700 1 -- 172.21.15.116:0/1551286546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa61006c4e0 msgr2=0x7fa61006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.978+0000 7fa626ccd700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa61006c4e0 0x7fa61006e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa60c006fd0 tx=0x7fa60c008040 comp rx=0 tx=0).stop 2024-03-04T16:01:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.978+0000 7fa626ccd700 1 -- 172.21.15.116:0/1551286546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa620101640 msgr2=0x7fa620100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:01:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.978+0000 7fa626ccd700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa620101640 0x7fa620100fa0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa614006010 tx=0x7fa61400be20 comp rx=0 tx=0).stop 2024-03-04T16:01:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.978+0000 7fa626ccd700 1 -- 172.21.15.116:0/1551286546 shutdown_connections 2024-03-04T16:01:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.978+0000 7fa626ccd700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fa61006c4e0 0x7fa61006e9a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.978+0000 7fa626ccd700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa620101640 0x7fa620100fa0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.978+0000 7fa626ccd700 1 --2- 172.21.15.116:0/1551286546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa620103f70 0x7fa6200ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:01:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.978+0000 7fa626ccd700 1 -- 172.21.15.116:0/1551286546 >> 172.21.15.116:0/1551286546 conn(0x7fa6200fb030 msgr2=0x7fa6200fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:01:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.979+0000 7fa626ccd700 1 -- 172.21.15.116:0/1551286546 shutdown_connections 2024-03-04T16:01:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:01:59.979+0000 7fa626ccd700 1 -- 172.21.15.116:0/1551286546 wait complete. 2024-03-04T16:02:00.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:00 smithi116 ceph-mon[29120]: pgmap v211: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-03-04T16:02:00.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:00 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1551286546' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:02:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:00 smithi152 ceph-mon[36876]: pgmap v211: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-03-04T16:02:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:00 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/1551286546' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:02:01.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:01 smithi116 ceph-mon[29120]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:01.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:01 smithi152 ceph-mon[36876]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:02.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:02 smithi116 ceph-mon[29120]: pgmap v212: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-03-04T16:02:02.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:02 smithi152 ceph-mon[36876]: pgmap v212: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-03-04T16:02:04.660 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:04 smithi152 ceph-mon[36876]: pgmap v213: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:04.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:04 smithi116 ceph-mon[29120]: pgmap v213: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:06.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:06 smithi116 ceph-mon[29120]: pgmap v214: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:06.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:06 smithi152 ceph-mon[36876]: pgmap v214: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:02:07.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:02:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:08 smithi116 ceph-mon[29120]: pgmap v215: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:08.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:08 smithi152 ceph-mon[36876]: pgmap v215: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:10.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:10 smithi116 ceph-mon[29120]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:10.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:10 smithi152 ceph-mon[36876]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:12.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:12 smithi116 ceph-mon[29120]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:12.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:12 smithi152 ceph-mon[36876]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:14.652 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:14 smithi152 ceph-mon[36876]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:14.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:14 smithi116 ceph-mon[29120]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:16.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:16 smithi116 ceph-mon[29120]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:16.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:16 smithi152 ceph-mon[36876]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:18 smithi116 ceph-mon[29120]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:18.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:18 smithi152 ceph-mon[36876]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:20.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:20 smithi116 ceph-mon[29120]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:20.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:20 smithi152 ceph-mon[36876]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:22 smithi116 ceph-mon[29120]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:02:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:22 smithi152 ceph-mon[36876]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:02:24.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:24 smithi116 ceph-mon[29120]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:24.750 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:24 smithi152 ceph-mon[36876]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:26 smithi116 ceph-mon[29120]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:26.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:26 smithi152 ceph-mon[36876]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:28.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:28 smithi116 ceph-mon[29120]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:28.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:28 smithi152 ceph-mon[36876]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:30.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.085+0000 7f493ff16700 1 -- 172.21.15.116:0/3386695045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4938101700 msgr2=0x7f4938103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:30.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.085+0000 7f493ff16700 1 --2- 172.21.15.116:0/3386695045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4938101700 0x7f4938103af0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4934009910 tx=0x7f4934009c20 comp rx=0 tx=0).stop 2024-03-04T16:02:30.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.085+0000 7f493ff16700 1 -- 172.21.15.116:0/3386695045 shutdown_connections 2024-03-04T16:02:30.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.085+0000 7f493ff16700 1 --2- 172.21.15.116:0/3386695045 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4938104030 0x7f4938106420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.085+0000 7f493ff16700 1 --2- 172.21.15.116:0/3386695045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4938101700 0x7f4938103af0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.085+0000 7f493ff16700 1 -- 172.21.15.116:0/3386695045 >> 172.21.15.116:0/3386695045 conn(0x7f49380fb070 msgr2=0x7f49380fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.085+0000 7f493ff16700 1 -- 172.21.15.116:0/3386695045 shutdown_connections 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.085+0000 7f493ff16700 1 -- 172.21.15.116:0/3386695045 wait complete. 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493ff16700 1 Processor -- start 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493ff16700 1 -- start start 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493ff16700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4938101700 0x7f4938196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493ff16700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4938104030 0x7f4938196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493ff16700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4938196ea0 con 0x7f4938104030 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493ff16700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4938196fe0 con 0x7f4938101700 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493d4b1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4938104030 0x7f4938196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:30.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493d4b1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4938104030 0x7f4938196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54704/0 (socket says 172.21.15.116:54704) 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493d4b1700 1 -- 172.21.15.116:0/830499774 learned_addr learned my addr 172.21.15.116:0/830499774 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493dcb2700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4938101700 0x7f4938196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493d4b1700 1 -- 172.21.15.116:0/830499774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4938101700 msgr2=0x7f4938196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493d4b1700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4938101700 0x7f4938196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493d4b1700 1 -- 172.21.15.116:0/830499774 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49340095f0 con 0x7f4938104030 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.086+0000 7f493d4b1700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4938104030 0x7f4938196880 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f492800d8d0 tx=0x7f492800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.087+0000 7f492effd700 1 -- 172.21.15.116:0/830499774 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49280098a0 con 0x7f4938104030 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.087+0000 7f492effd700 1 -- 172.21.15.116:0/830499774 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4928010460 con 0x7f4938104030 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.087+0000 7f493ff16700 1 -- 172.21.15.116:0/830499774 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493819ba90 con 0x7f4938104030 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.087+0000 7f492effd700 1 -- 172.21.15.116:0/830499774 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f492800f5f0 con 0x7f4938104030 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.087+0000 7f493ff16700 1 -- 172.21.15.116:0/830499774 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493819bfe0 con 0x7f4938104030 2024-03-04T16:02:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.088+0000 7f493ff16700 1 -- 172.21.15.116:0/830499774 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4938190560 con 0x7f4938104030 2024-03-04T16:02:30.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.088+0000 7f492effd700 1 -- 172.21.15.116:0/830499774 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4928009a00 con 0x7f4938104030 2024-03-04T16:02:30.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.088+0000 7f492effd700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f492406c4e0 0x7f492406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:30.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.089+0000 7f492effd700 1 -- 172.21.15.116:0/830499774 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f492808b770 con 0x7f4938104030 2024-03-04T16:02:30.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.089+0000 7f493dcb2700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f492406c4e0 0x7f492406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:30.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.089+0000 7f493dcb2700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f492406c4e0 0x7f492406e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f493400b5c0 tx=0x7f4934005ee0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:30.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.093+0000 7f492effd700 1 -- 172.21.15.116:0/830499774 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4928059980 con 0x7f4938104030 2024-03-04T16:02:30.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.326+0000 7f493ff16700 1 -- 172.21.15.116:0/830499774 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f49380611d0 con 0x7f492406c4e0 2024-03-04T16:02:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.327+0000 7f492effd700 1 -- 172.21.15.116:0/830499774 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f49380611d0 con 0x7f492406c4e0 2024-03-04T16:02:30.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 -- 172.21.15.116:0/830499774 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f492406c4e0 msgr2=0x7f492406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:30.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f492406c4e0 0x7f492406e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f493400b5c0 tx=0x7f4934005ee0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 -- 172.21.15.116:0/830499774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4938104030 msgr2=0x7f4938196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:30.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4938104030 0x7f4938196880 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f492800d8d0 tx=0x7f492800dbe0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 -- 172.21.15.116:0/830499774 shutdown_connections 2024-03-04T16:02:30.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4938101700 0x7f4938196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f492406c4e0 0x7f492406e9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 --2- 172.21.15.116:0/830499774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4938104030 0x7f4938196880 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 -- 172.21.15.116:0/830499774 >> 172.21.15.116:0/830499774 conn(0x7f49380fb070 msgr2=0x7f49380fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:30.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 -- 172.21.15.116:0/830499774 shutdown_connections 2024-03-04T16:02:30.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.329+0000 7f493ff16700 1 -- 172.21.15.116:0/830499774 wait complete. 2024-03-04T16:02:30.341 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:02:30.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.440+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/838174021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d680ffab0 msgr2=0x7f1d680ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.440+0000 7f1d6f8ee700 1 --2- 172.21.15.116:0/838174021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d680ffab0 0x7f1d680ffed0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1d64009910 tx=0x7f1d64009c20 comp rx=0 tx=0).stop 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.440+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/838174021 shutdown_connections 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.440+0000 7f1d6f8ee700 1 --2- 172.21.15.116:0/838174021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d68100cb0 0x7f1d68101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.440+0000 7f1d6f8ee700 1 --2- 172.21.15.116:0/838174021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d680ffab0 0x7f1d680ffed0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.440+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/838174021 >> 172.21.15.116:0/838174021 conn(0x7f1d680fb030 msgr2=0x7f1d680fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.440+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/838174021 shutdown_connections 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.440+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/838174021 wait complete. 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6f8ee700 1 Processor -- start 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6f8ee700 1 -- start start 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6f8ee700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d680ffab0 0x7f1d68195240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6f8ee700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d68100cb0 0x7f1d68195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6f8ee700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68195da0 con 0x7f1d68100cb0 2024-03-04T16:02:30.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6f8ee700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68195ee0 con 0x7f1d680ffab0 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6ce89700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d68100cb0 0x7f1d68195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6d68a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d680ffab0 0x7f1d68195240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6ce89700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d68100cb0 0x7f1d68195780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54712/0 (socket says 172.21.15.116:54712) 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6ce89700 1 -- 172.21.15.116:0/1320137737 learned_addr learned my addr 172.21.15.116:0/1320137737 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.441+0000 7f1d6d68a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d680ffab0 0x7f1d68195240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:58812/0 (socket says 172.21.15.116:58812) 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.442+0000 7f1d6ce89700 1 -- 172.21.15.116:0/1320137737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d680ffab0 msgr2=0x7f1d68195240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.442+0000 7f1d6ce89700 1 --2- 172.21.15.116:0/1320137737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d680ffab0 0x7f1d68195240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.442+0000 7f1d6ce89700 1 -- 172.21.15.116:0/1320137737 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d640095f0 con 0x7f1d68100cb0 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.442+0000 7f1d6ce89700 1 --2- 172.21.15.116:0/1320137737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d68100cb0 0x7f1d68195780 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800d8d0 tx=0x7f1d5800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.442+0000 7f1d5e7fc700 1 -- 172.21.15.116:0/1320137737 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d580098a0 con 0x7f1d68100cb0 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.442+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/1320137737 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d68068470 con 0x7f1d68100cb0 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.442+0000 7f1d5e7fc700 1 -- 172.21.15.116:0/1320137737 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1d58010460 con 0x7f1d68100cb0 2024-03-04T16:02:30.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.442+0000 7f1d5e7fc700 1 -- 172.21.15.116:0/1320137737 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d5800f5f0 con 0x7f1d68100cb0 2024-03-04T16:02:30.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.442+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/1320137737 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d680689c0 con 0x7f1d68100cb0 2024-03-04T16:02:30.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.444+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/1320137737 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d68108610 con 0x7f1d68100cb0 2024-03-04T16:02:30.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.444+0000 7f1d5e7fc700 1 -- 172.21.15.116:0/1320137737 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1d5800f750 con 0x7f1d68100cb0 2024-03-04T16:02:30.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.444+0000 7f1d5e7fc700 1 --2- 172.21.15.116:0/1320137737 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1d5406c4e0 0x7f1d5406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:30.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.444+0000 7f1d5e7fc700 1 -- 172.21.15.116:0/1320137737 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1d5808b6c0 con 0x7f1d68100cb0 2024-03-04T16:02:30.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.444+0000 7f1d6d68a700 1 --2- 172.21.15.116:0/1320137737 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1d5406c4e0 0x7f1d5406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:30.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.445+0000 7f1d6d68a700 1 --2- 172.21.15.116:0/1320137737 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1d5406c4e0 0x7f1d5406e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d64006010 tx=0x7f1d64005ec0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:30.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.449+0000 7f1d5e7fc700 1 -- 172.21.15.116:0/1320137737 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1d58009a00 con 0x7f1d68100cb0 2024-03-04T16:02:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.669+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/1320137737 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d68105600 con 0x7f1d5406c4e0 2024-03-04T16:02:30.672 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:30 smithi116 ceph-mon[29120]: pgmap v226: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.671+0000 7f1d5e7fc700 1 -- 172.21.15.116:0/1320137737 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f1d68105600 con 0x7f1d5406c4e0 2024-03-04T16:02:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/1320137737 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1d5406c4e0 msgr2=0x7f1d5406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 --2- 172.21.15.116:0/1320137737 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1d5406c4e0 0x7f1d5406e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d64006010 tx=0x7f1d64005ec0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/1320137737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d68100cb0 msgr2=0x7f1d68195780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 --2- 172.21.15.116:0/1320137737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d68100cb0 0x7f1d68195780 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800d8d0 tx=0x7f1d5800dbe0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/1320137737 shutdown_connections 2024-03-04T16:02:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 --2- 172.21.15.116:0/1320137737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d680ffab0 0x7f1d68195240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 --2- 172.21.15.116:0/1320137737 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f1d5406c4e0 0x7f1d5406e9a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 --2- 172.21.15.116:0/1320137737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d68100cb0 0x7f1d68195780 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/1320137737 >> 172.21.15.116:0/1320137737 conn(0x7f1d680fb030 msgr2=0x7f1d68103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/1320137737 shutdown_connections 2024-03-04T16:02:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.673+0000 7f1d6f8ee700 1 -- 172.21.15.116:0/1320137737 wait complete. 2024-03-04T16:02:30.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.780+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2097326652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ccc0ffab0 msgr2=0x7f6ccc0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:30.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.780+0000 7f6cd2e10700 1 --2- 172.21.15.116:0/2097326652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ccc0ffab0 0x7f6ccc0ffed0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009910 tx=0x7f6cc0009c20 comp rx=0 tx=0).stop 2024-03-04T16:02:30.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.780+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2097326652 shutdown_connections 2024-03-04T16:02:30.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.780+0000 7f6cd2e10700 1 --2- 172.21.15.116:0/2097326652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ccc100cb0 0x7f6ccc101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.780+0000 7f6cd2e10700 1 --2- 172.21.15.116:0/2097326652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ccc0ffab0 0x7f6ccc0ffed0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.780+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2097326652 >> 172.21.15.116:0/2097326652 conn(0x7f6ccc0fb030 msgr2=0x7f6ccc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:30.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.780+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2097326652 shutdown_connections 2024-03-04T16:02:30.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.780+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2097326652 wait complete. 2024-03-04T16:02:30.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.781+0000 7f6cd2e10700 1 Processor -- start 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.781+0000 7f6cd2e10700 1 -- start start 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.781+0000 7f6cd2e10700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ccc0ffab0 0x7f6ccc068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.781+0000 7f6cd2e10700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ccc100cb0 0x7f6ccc19b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.781+0000 7f6cd2e10700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc19b600 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.781+0000 7f6cd2e10700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc19b740 con 0x7f6ccc100cb0 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.781+0000 7f6cd0bac700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ccc0ffab0 0x7f6ccc068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.781+0000 7f6cd0bac700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ccc0ffab0 0x7f6ccc068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54728/0 (socket says 172.21.15.116:54728) 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.781+0000 7f6cd0bac700 1 -- 172.21.15.116:0/2933192874 learned_addr learned my addr 172.21.15.116:0/2933192874 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.781+0000 7f6ccbfff700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ccc100cb0 0x7f6ccc19b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.782+0000 7f6cd0bac700 1 -- 172.21.15.116:0/2933192874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ccc100cb0 msgr2=0x7f6ccc19b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.782+0000 7f6cd0bac700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ccc100cb0 0x7f6ccc19b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.782+0000 7f6cd0bac700 1 -- 172.21.15.116:0/2933192874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc00095f0 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.782+0000 7f6cd0bac700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ccc0ffab0 0x7f6ccc068f70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0000c00 tx=0x7f6cc000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:30.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.782+0000 7f6cc9ffb700 1 -- 172.21.15.116:0/2933192874 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cc001d070 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.782+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2933192874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ccc19b9c0 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.782+0000 7f6cc9ffb700 1 -- 172.21.15.116:0/2933192874 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6cc0005dc0 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.782+0000 7f6cc9ffb700 1 -- 172.21.15.116:0/2933192874 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cc0022470 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.782+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2933192874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ccc19beb0 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.783+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2933192874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ccc108610 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.784+0000 7f6cc9ffb700 1 -- 172.21.15.116:0/2933192874 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6cc000fa70 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.784+0000 7f6cc9ffb700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6cbc06c4e0 0x7f6cbc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:30.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.784+0000 7f6cc9ffb700 1 -- 172.21.15.116:0/2933192874 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6cc008cde0 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.784+0000 7f6ccbfff700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6cbc06c4e0 0x7f6cbc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:30.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.785+0000 7f6ccbfff700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6cbc06c4e0 0x7f6cbc06e9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0694f0 tx=0x7f6cb8009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:30.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:30.789+0000 7f6cc9ffb700 1 -- 172.21.15.116:0/2933192874 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6cc005aff0 con 0x7f6ccc0ffab0 2024-03-04T16:02:30.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:30 smithi152 ceph-mon[36876]: pgmap v226: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.025+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2933192874 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ccc105600 con 0x7f6cbc06c4e0 2024-03-04T16:02:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.044+0000 7f6cc9ffb700 1 -- 172.21.15.116:0/2933192874 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7f6ccc105600 con 0x7f6cbc06c4e0 2024-03-04T16:02:31.045 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (5m) 60s ago 7m 25.1M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (7m) 60s ago 7m 7390k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (6m) 66s ago 6m 7285k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (7m) 60s ago 7m 6665k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (6m) 66s ago 6m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (5m) 60s ago 6m 87.2M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (79s) 60s ago 79s 15.3M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (87s) 60s ago 86s 11.3M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (76s) 66s ago 76s 12.9M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (84s) 66s ago 83s 16.2M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:9283,8765,8443 running (8m) 60s ago 8m 489M - 18.2.1 5be31c24972a 083879ca4bf4 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (6m) 66s ago 6m 443M - 18.2.1 5be31c24972a c68d3e4a1dbc 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (8m) 60s ago 8m 45.4M 2048M 18.2.1 5be31c24972a 60e0bf118c4d 2024-03-04T16:02:31.046 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (5m) 66s ago 5m 37.6M 2048M 18.2.1 5be31c24972a 52450d235516 2024-03-04T16:02:31.047 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (7m) 60s ago 7m 15.4M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:02:31.047 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (6m) 66s ago 6m 16.9M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:02:31.047 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (5m) 60s ago 5m 50.5M 3946M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:02:31.047 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (4m) 60s ago 4m 46.8M 3946M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:02:31.047 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (4m) 60s ago 4m 46.4M 3946M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:02:31.047 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (3m) 66s ago 3m 46.2M 4970M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:02:31.047 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (3m) 66s ago 3m 43.8M 4970M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:02:31.047 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (2m) 66s ago 2m 44.0M 4970M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:02:31.047 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (5m) 60s ago 6m 55.8M - 2.43.0 a07b618ecd1d 31240d908f24 2024-03-04T16:02:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2933192874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6cbc06c4e0 msgr2=0x7f6cbc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6cbc06c4e0 0x7f6cbc06e9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0694f0 tx=0x7f6cb8009260 comp rx=0 tx=0).stop 2024-03-04T16:02:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2933192874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ccc0ffab0 msgr2=0x7f6ccc068f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ccc0ffab0 0x7f6ccc068f70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0000c00 tx=0x7f6cc000f690 comp rx=0 tx=0).stop 2024-03-04T16:02:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2933192874 shutdown_connections 2024-03-04T16:02:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f6cbc06c4e0 0x7f6cbc06e9a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ccc0ffab0 0x7f6ccc068f70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 --2- 172.21.15.116:0/2933192874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ccc100cb0 0x7f6ccc19b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2933192874 >> 172.21.15.116:0/2933192874 conn(0x7f6ccc0fb030 msgr2=0x7f6ccc103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2933192874 shutdown_connections 2024-03-04T16:02:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.046+0000 7f6cd2e10700 1 -- 172.21.15.116:0/2933192874 wait complete. 2024-03-04T16:02:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.155+0000 7f32ca98c700 1 -- 172.21.15.116:0/4267792276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32c4101dc0 msgr2=0x7f32c4102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.155+0000 7f32ca98c700 1 --2- 172.21.15.116:0/4267792276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32c4101dc0 0x7f32c4102220 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009910 tx=0x7f32b8009c20 comp rx=0 tx=0).stop 2024-03-04T16:02:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.155+0000 7f32ca98c700 1 -- 172.21.15.116:0/4267792276 shutdown_connections 2024-03-04T16:02:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.155+0000 7f32ca98c700 1 --2- 172.21.15.116:0/4267792276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32c4101dc0 0x7f32c4102220 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.155+0000 7f32ca98c700 1 --2- 172.21.15.116:0/4267792276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32c4100bc0 0x7f32c4100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.155+0000 7f32ca98c700 1 -- 172.21.15.116:0/4267792276 >> 172.21.15.116:0/4267792276 conn(0x7f32c40fc140 msgr2=0x7f32c40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.155+0000 7f32ca98c700 1 -- 172.21.15.116:0/4267792276 shutdown_connections 2024-03-04T16:02:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.155+0000 7f32ca98c700 1 -- 172.21.15.116:0/4267792276 wait complete. 2024-03-04T16:02:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.155+0000 7f32ca98c700 1 Processor -- start 2024-03-04T16:02:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.155+0000 7f32ca98c700 1 -- start start 2024-03-04T16:02:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32ca98c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32c4100bc0 0x7f32c4196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32ca98c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32c4101dc0 0x7f32c41968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32ca98c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c4196ed0 con 0x7f32c4100bc0 2024-03-04T16:02:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32ca98c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c4197010 con 0x7f32c4101dc0 2024-03-04T16:02:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32c3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32c4100bc0 0x7f32c4196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32c3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32c4100bc0 0x7f32c4196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54756/0 (socket says 172.21.15.116:54756) 2024-03-04T16:02:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32c3fff700 1 -- 172.21.15.116:0/3176594151 learned_addr learned my addr 172.21.15.116:0/3176594151 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:02:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32c37fe700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32c4101dc0 0x7f32c41968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32c3fff700 1 -- 172.21.15.116:0/3176594151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32c4101dc0 msgr2=0x7f32c41968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32c3fff700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32c4101dc0 0x7f32c41968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32c3fff700 1 -- 172.21.15.116:0/3176594151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b80095f0 con 0x7f32c4100bc0 2024-03-04T16:02:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.156+0000 7f32c3fff700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32c4100bc0 0x7f32c4196370 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f32b400e8f0 tx=0x7f32b400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.157+0000 7f32c17fa700 1 -- 172.21.15.116:0/3176594151 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32b4009790 con 0x7f32c4100bc0 2024-03-04T16:02:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.157+0000 7f32ca98c700 1 -- 172.21.15.116:0/3176594151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c419bac0 con 0x7f32c4100bc0 2024-03-04T16:02:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.157+0000 7f32c17fa700 1 -- 172.21.15.116:0/3176594151 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f32b400c810 con 0x7f32c4100bc0 2024-03-04T16:02:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.157+0000 7f32c17fa700 1 -- 172.21.15.116:0/3176594151 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32b4003d20 con 0x7f32c4100bc0 2024-03-04T16:02:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.157+0000 7f32ca98c700 1 -- 172.21.15.116:0/3176594151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c419c010 con 0x7f32c4100bc0 2024-03-04T16:02:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.158+0000 7f32ca98c700 1 -- 172.21.15.116:0/3176594151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c4109720 con 0x7f32c4100bc0 2024-03-04T16:02:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.158+0000 7f32c17fa700 1 -- 172.21.15.116:0/3176594151 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f32b40041f0 con 0x7f32c4100bc0 2024-03-04T16:02:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.159+0000 7f32c17fa700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f32ac06c4e0 0x7f32ac06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.159+0000 7f32c17fa700 1 -- 172.21.15.116:0/3176594151 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f32b4014070 con 0x7f32c4100bc0 2024-03-04T16:02:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.159+0000 7f32c37fe700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f32ac06c4e0 0x7f32ac06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.159+0000 7f32c37fe700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f32ac06c4e0 0x7f32ac06e9a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009fd0 tx=0x7f32b8005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.163+0000 7f32c17fa700 1 -- 172.21.15.116:0/3176594151 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f32b4059d90 con 0x7f32c4100bc0 2024-03-04T16:02:31.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.453+0000 7f32ca98c700 1 -- 172.21.15.116:0/3176594151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f32c419c620 con 0x7f32c4100bc0 2024-03-04T16:02:31.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.454+0000 7f32c17fa700 1 -- 172.21.15.116:0/3176594151 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f32b4059920 con 0x7f32c4100bc0 2024-03-04T16:02:31.456 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:02:31.456 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:02:31.456 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-03-04T16:02:31.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:02:31.456 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:02:31.456 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-03-04T16:02:31.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:02:31.456 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:02:31.457 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-03-04T16:02:31.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:02:31.457 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:02:31.457 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:02:31.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:02:31.457 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:02:31.457 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-03-04T16:02:31.457 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:02:31.457 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:02:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 -- 172.21.15.116:0/3176594151 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f32ac06c4e0 msgr2=0x7f32ac06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f32ac06c4e0 0x7f32ac06e9a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009fd0 tx=0x7f32b8005940 comp rx=0 tx=0).stop 2024-03-04T16:02:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 -- 172.21.15.116:0/3176594151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32c4100bc0 msgr2=0x7f32c4196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32c4100bc0 0x7f32c4196370 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f32b400e8f0 tx=0x7f32b400ec00 comp rx=0 tx=0).stop 2024-03-04T16:02:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 -- 172.21.15.116:0/3176594151 shutdown_connections 2024-03-04T16:02:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f32ac06c4e0 0x7f32ac06e9a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32c4100bc0 0x7f32c4196370 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 --2- 172.21.15.116:0/3176594151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32c4101dc0 0x7f32c41968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 -- 172.21.15.116:0/3176594151 >> 172.21.15.116:0/3176594151 conn(0x7f32c40fc140 msgr2=0x7f32c4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:31.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 -- 172.21.15.116:0/3176594151 shutdown_connections 2024-03-04T16:02:31.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.456+0000 7f32ca98c700 1 -- 172.21.15.116:0/3176594151 wait complete. 2024-03-04T16:02:31.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.563+0000 7f2b75387700 1 -- 172.21.15.116:0/4060643279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b70101e00 msgr2=0x7f2b70102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.563+0000 7f2b75387700 1 --2- 172.21.15.116:0/4060643279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b70101e00 0x7f2b70102260 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009910 tx=0x7f2b64009c20 comp rx=0 tx=0).stop 2024-03-04T16:02:31.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.563+0000 7f2b75387700 1 -- 172.21.15.116:0/4060643279 shutdown_connections 2024-03-04T16:02:31.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.563+0000 7f2b75387700 1 --2- 172.21.15.116:0/4060643279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b70101e00 0x7f2b70102260 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.563+0000 7f2b75387700 1 --2- 172.21.15.116:0/4060643279 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b70100c00 0x7f2b70101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.563+0000 7f2b75387700 1 -- 172.21.15.116:0/4060643279 >> 172.21.15.116:0/4060643279 conn(0x7f2b700fc180 msgr2=0x7f2b700fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:31.565 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:31 smithi116 ceph-mon[29120]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:31.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.563+0000 7f2b75387700 1 -- 172.21.15.116:0/4060643279 shutdown_connections 2024-03-04T16:02:31.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.563+0000 7f2b75387700 1 -- 172.21.15.116:0/4060643279 wait complete. 2024-03-04T16:02:31.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.564+0000 7f2b75387700 1 Processor -- start 2024-03-04T16:02:31.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.564+0000 7f2b75387700 1 -- start start 2024-03-04T16:02:31.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.564+0000 7f2b75387700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b70100c00 0x7f2b70196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.564+0000 7f2b75387700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b70101e00 0x7f2b701968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.564+0000 7f2b75387700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b70196ee0 con 0x7f2b70100c00 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.564+0000 7f2b75387700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b70197020 con 0x7f2b70101e00 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.564+0000 7f2b6effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b70100c00 0x7f2b70196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.564+0000 7f2b6effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b70100c00 0x7f2b70196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54778/0 (socket says 172.21.15.116:54778) 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.564+0000 7f2b6effd700 1 -- 172.21.15.116:0/173145885 learned_addr learned my addr 172.21.15.116:0/173145885 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.564+0000 7f2b6e7fc700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b70101e00 0x7f2b701968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.565+0000 7f2b6effd700 1 -- 172.21.15.116:0/173145885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b70101e00 msgr2=0x7f2b701968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.565+0000 7f2b6effd700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b70101e00 0x7f2b701968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.565+0000 7f2b6effd700 1 -- 172.21.15.116:0/173145885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b640095f0 con 0x7f2b70100c00 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.565+0000 7f2b6effd700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b70100c00 0x7f2b70196380 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000d900 tx=0x7f2b6000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.565+0000 7f2b57fff700 1 -- 172.21.15.116:0/173145885 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b6000b920 con 0x7f2b70100c00 2024-03-04T16:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.565+0000 7f2b57fff700 1 -- 172.21.15.116:0/173145885 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2b60010460 con 0x7f2b70100c00 2024-03-04T16:02:31.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.565+0000 7f2b75387700 1 -- 172.21.15.116:0/173145885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7019bad0 con 0x7f2b70100c00 2024-03-04T16:02:31.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.565+0000 7f2b57fff700 1 -- 172.21.15.116:0/173145885 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b6000f590 con 0x7f2b70100c00 2024-03-04T16:02:31.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.565+0000 7f2b75387700 1 -- 172.21.15.116:0/173145885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7019c020 con 0x7f2b70100c00 2024-03-04T16:02:31.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.566+0000 7f2b75387700 1 -- 172.21.15.116:0/173145885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b70109760 con 0x7f2b70100c00 2024-03-04T16:02:31.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.567+0000 7f2b57fff700 1 -- 172.21.15.116:0/173145885 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2b600105d0 con 0x7f2b70100c00 2024-03-04T16:02:31.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.567+0000 7f2b57fff700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2b5806c4e0 0x7f2b5806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:31.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.567+0000 7f2b6e7fc700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2b5806c4e0 0x7f2b5806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:31.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.567+0000 7f2b57fff700 1 -- 172.21.15.116:0/173145885 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2b6008b860 con 0x7f2b70100c00 2024-03-04T16:02:31.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.568+0000 7f2b6e7fc700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2b5806c4e0 0x7f2b5806e9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400b5c0 tx=0x7f2b64005ee0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:31.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.572+0000 7f2b57fff700 1 -- 172.21.15.116:0/173145885 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2b60059a70 con 0x7f2b70100c00 2024-03-04T16:02:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.825+0000 7f2b75387700 1 -- 172.21.15.116:0/173145885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2b7019cb70 con 0x7f2b70100c00 2024-03-04T16:02:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.825+0000 7f2b57fff700 1 -- 172.21.15.116:0/173145885 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1831 (secure 0 0 0) 0x7f2b60016740 con 0x7f2b70100c00 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:02:31.827 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:02:31.828 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:02:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:02:31.830 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:02:31.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 -- 172.21.15.116:0/173145885 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2b5806c4e0 msgr2=0x7f2b5806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2b5806c4e0 0x7f2b5806e9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400b5c0 tx=0x7f2b64005ee0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 -- 172.21.15.116:0/173145885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b70100c00 msgr2=0x7f2b70196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b70100c00 0x7f2b70196380 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000d900 tx=0x7f2b6000dc10 comp rx=0 tx=0).stop 2024-03-04T16:02:31.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 -- 172.21.15.116:0/173145885 shutdown_connections 2024-03-04T16:02:31.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f2b5806c4e0 0x7f2b5806e9a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b70100c00 0x7f2b70196380 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 --2- 172.21.15.116:0/173145885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b70101e00 0x7f2b701968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 -- 172.21.15.116:0/173145885 >> 172.21.15.116:0/173145885 conn(0x7f2b700fc180 msgr2=0x7f2b70105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:31.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 -- 172.21.15.116:0/173145885 shutdown_connections 2024-03-04T16:02:31.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.827+0000 7f2b75387700 1 -- 172.21.15.116:0/173145885 wait complete. 2024-03-04T16:02:31.831 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:02:31.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:31 smithi152 ceph-mon[36876]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:31.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 -- 172.21.15.116:0/3163283617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a8103fd0 msgr2=0x7fc8a81063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 --2- 172.21.15.116:0/3163283617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a8103fd0 0x7fc8a81063c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc898009910 tx=0x7fc898009c20 comp rx=0 tx=0).stop 2024-03-04T16:02:31.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 -- 172.21.15.116:0/3163283617 shutdown_connections 2024-03-04T16:02:31.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 --2- 172.21.15.116:0/3163283617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a8103fd0 0x7fc8a81063c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 --2- 172.21.15.116:0/3163283617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8a81016a0 0x7fc8a8103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 -- 172.21.15.116:0/3163283617 >> 172.21.15.116:0/3163283617 conn(0x7fc8a80fb030 msgr2=0x7fc8a80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:31.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 -- 172.21.15.116:0/3163283617 shutdown_connections 2024-03-04T16:02:31.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 -- 172.21.15.116:0/3163283617 wait complete. 2024-03-04T16:02:31.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 Processor -- start 2024-03-04T16:02:31.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 -- start start 2024-03-04T16:02:31.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a81016a0 0x7fc8a8196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.941+0000 7fc8afa9d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8a8103fd0 0x7fc8a8196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8afa9d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a8196e90 con 0x7fc8a81016a0 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8afa9d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a8196fd0 con 0x7fc8a8103fd0 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8ad839700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a81016a0 0x7fc8a8196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8ad839700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a81016a0 0x7fc8a8196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54792/0 (socket says 172.21.15.116:54792) 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8ad839700 1 -- 172.21.15.116:0/2538245497 learned_addr learned my addr 172.21.15.116:0/2538245497 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8ad038700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8a8103fd0 0x7fc8a8196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8ad839700 1 -- 172.21.15.116:0/2538245497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8a8103fd0 msgr2=0x7fc8a8196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8ad839700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8a8103fd0 0x7fc8a8196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8ad839700 1 -- 172.21.15.116:0/2538245497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8980095f0 con 0x7fc8a81016a0 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8ad839700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a81016a0 0x7fc8a8196330 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400e920 tx=0x7fc8a400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc89effd700 1 -- 172.21.15.116:0/2538245497 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a400c9d0 con 0x7fc8a81016a0 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc89effd700 1 -- 172.21.15.116:0/2538245497 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc8a4010430 con 0x7fc8a81016a0 2024-03-04T16:02:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8afa9d700 1 -- 172.21.15.116:0/2538245497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a819ba80 con 0x7fc8a81016a0 2024-03-04T16:02:31.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc89effd700 1 -- 172.21.15.116:0/2538245497 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a4018650 con 0x7fc8a81016a0 2024-03-04T16:02:31.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.942+0000 7fc8afa9d700 1 -- 172.21.15.116:0/2538245497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a819bfd0 con 0x7fc8a81016a0 2024-03-04T16:02:31.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.943+0000 7fc8afa9d700 1 -- 172.21.15.116:0/2538245497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a8190520 con 0x7fc8a81016a0 2024-03-04T16:02:31.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.944+0000 7fc89effd700 1 -- 172.21.15.116:0/2538245497 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc8a40105a0 con 0x7fc8a81016a0 2024-03-04T16:02:31.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.944+0000 7fc89effd700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc89406c4e0 0x7fc89406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:31.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.945+0000 7fc89effd700 1 -- 172.21.15.116:0/2538245497 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc8a4014070 con 0x7fc8a81016a0 2024-03-04T16:02:31.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.945+0000 7fc8ad038700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc89406c4e0 0x7fc89406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:31.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.945+0000 7fc8ad038700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc89406c4e0 0x7fc89406e9a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc89800b6f0 tx=0x7fc898005a90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:31.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:31.949+0000 7fc89effd700 1 -- 172.21.15.116:0/2538245497 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc8a40520e0 con 0x7fc8a81016a0 2024-03-04T16:02:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.182+0000 7fc8afa9d700 1 -- 172.21.15.116:0/2538245497 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8a80611d0 con 0x7fc89406c4e0 2024-03-04T16:02:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.183+0000 7fc89effd700 1 -- 172.21.15.116:0/2538245497 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fc8a80611d0 con 0x7fc89406c4e0 2024-03-04T16:02:32.184 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:02:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:02:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:02:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:02:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2024-03-04T16:02:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "", 2024-03-04T16:02:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 image", 2024-03-04T16:02:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:02:32.184 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:02:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 -- 172.21.15.116:0/2538245497 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc89406c4e0 msgr2=0x7fc89406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc89406c4e0 0x7fc89406e9a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc89800b6f0 tx=0x7fc898005a90 comp rx=0 tx=0).stop 2024-03-04T16:02:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 -- 172.21.15.116:0/2538245497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a81016a0 msgr2=0x7fc8a8196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a81016a0 0x7fc8a8196330 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400e920 tx=0x7fc8a400ec30 comp rx=0 tx=0).stop 2024-03-04T16:02:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 -- 172.21.15.116:0/2538245497 shutdown_connections 2024-03-04T16:02:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc89406c4e0 0x7fc89406e9a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a81016a0 0x7fc8a8196330 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 --2- 172.21.15.116:0/2538245497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8a8103fd0 0x7fc8a8196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 -- 172.21.15.116:0/2538245497 >> 172.21.15.116:0/2538245497 conn(0x7fc8a80fb030 msgr2=0x7fc8a80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 -- 172.21.15.116:0/2538245497 shutdown_connections 2024-03-04T16:02:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.185+0000 7fc8afa9d700 1 -- 172.21.15.116:0/2538245497 wait complete. 2024-03-04T16:02:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.293+0000 7f880cf9b700 1 -- 172.21.15.116:0/2591475350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8808101e00 msgr2=0x7f8808102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.293+0000 7f880cf9b700 1 --2- 172.21.15.116:0/2591475350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8808101e00 0x7f8808102260 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f87fc009910 tx=0x7f87fc009c20 comp rx=0 tx=0).stop 2024-03-04T16:02:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.293+0000 7f880cf9b700 1 -- 172.21.15.116:0/2591475350 shutdown_connections 2024-03-04T16:02:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.293+0000 7f880cf9b700 1 --2- 172.21.15.116:0/2591475350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8808101e00 0x7f8808102260 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.293+0000 7f880cf9b700 1 --2- 172.21.15.116:0/2591475350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8808100c00 0x7f8808101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.293+0000 7f880cf9b700 1 -- 172.21.15.116:0/2591475350 >> 172.21.15.116:0/2591475350 conn(0x7f88080fc180 msgr2=0x7f88080fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.293+0000 7f880cf9b700 1 -- 172.21.15.116:0/2591475350 shutdown_connections 2024-03-04T16:02:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.293+0000 7f880cf9b700 1 -- 172.21.15.116:0/2591475350 wait complete. 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880cf9b700 1 Processor -- start 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880cf9b700 1 -- start start 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880cf9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8808100c00 0x7f8808074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880cf9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8808101e00 0x7f8808073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880cf9b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88080736c0 con 0x7f8808100c00 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880cf9b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8808073800 con 0x7f8808101e00 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8808100c00 0x7f8808074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f8805d9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8808101e00 0x7f8808073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8808100c00 0x7f8808074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54814/0 (socket says 172.21.15.116:54814) 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f8805d9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8808101e00 0x7f8808073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:58914/0 (socket says 172.21.15.116:58914) 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880659c700 1 -- 172.21.15.116:0/1139069425 learned_addr learned my addr 172.21.15.116:0/1139069425 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:02:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880659c700 1 -- 172.21.15.116:0/1139069425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8808101e00 msgr2=0x7f8808073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880659c700 1 --2- 172.21.15.116:0/1139069425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8808101e00 0x7f8808073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880659c700 1 -- 172.21.15.116:0/1139069425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc0095f0 con 0x7f8808100c00 2024-03-04T16:02:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.294+0000 7f880659c700 1 --2- 172.21.15.116:0/1139069425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8808100c00 0x7f8808074b30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f87f800e8f0 tx=0x7f87f800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.295+0000 7f87f77fe700 1 -- 172.21.15.116:0/1139069425 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87f8009790 con 0x7f8808100c00 2024-03-04T16:02:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.295+0000 7f880cf9b700 1 -- 172.21.15.116:0/1139069425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8808073ae0 con 0x7f8808100c00 2024-03-04T16:02:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.295+0000 7f87f77fe700 1 -- 172.21.15.116:0/1139069425 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f87f800c810 con 0x7f8808100c00 2024-03-04T16:02:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.295+0000 7f87f77fe700 1 -- 172.21.15.116:0/1139069425 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87f8003d20 con 0x7f8808100c00 2024-03-04T16:02:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.295+0000 7f880cf9b700 1 -- 172.21.15.116:0/1139069425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880819e3e0 con 0x7f8808100c00 2024-03-04T16:02:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.296+0000 7f880cf9b700 1 -- 172.21.15.116:0/1139069425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8808109760 con 0x7f8808100c00 2024-03-04T16:02:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.296+0000 7f87f77fe700 1 -- 172.21.15.116:0/1139069425 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f87f80041f0 con 0x7f8808100c00 2024-03-04T16:02:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.297+0000 7f87f77fe700 1 --2- 172.21.15.116:0/1139069425 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f87f006c4e0 0x7f87f006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:02:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.297+0000 7f87f77fe700 1 -- 172.21.15.116:0/1139069425 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f87f8014070 con 0x7f8808100c00 2024-03-04T16:02:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.297+0000 7f8805d9b700 1 --2- 172.21.15.116:0/1139069425 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f87f006c4e0 0x7f87f006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:02:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.297+0000 7f8805d9b700 1 --2- 172.21.15.116:0/1139069425 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f87f006c4e0 0x7f87f006e9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f87fc009fd0 tx=0x7f87fc005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:02:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.301+0000 7f87f77fe700 1 -- 172.21.15.116:0/1139069425 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f87f8059f20 con 0x7f8808100c00 2024-03-04T16:02:32.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:32 smithi116 ceph-mon[29120]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:32.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:32 smithi116 ceph-mon[29120]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:32.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:32 smithi116 ceph-mon[29120]: pgmap v227: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:32.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:32 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3176594151' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:02:32.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:32 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/173145885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:02:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.587+0000 7f880cf9b700 1 -- 172.21.15.116:0/1139069425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8808109950 con 0x7f8808100c00 2024-03-04T16:02:32.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.587+0000 7f87f77fe700 1 -- 172.21.15.116:0/1139069425 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f87f8059ab0 con 0x7f8808100c00 2024-03-04T16:02:32.589 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 -- 172.21.15.116:0/1139069425 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f87f006c4e0 msgr2=0x7f87f006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 --2- 172.21.15.116:0/1139069425 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f87f006c4e0 0x7f87f006e9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f87fc009fd0 tx=0x7f87fc005940 comp rx=0 tx=0).stop 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 -- 172.21.15.116:0/1139069425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8808100c00 msgr2=0x7f8808074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 --2- 172.21.15.116:0/1139069425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8808100c00 0x7f8808074b30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f87f800e8f0 tx=0x7f87f800ec00 comp rx=0 tx=0).stop 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 -- 172.21.15.116:0/1139069425 shutdown_connections 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 --2- 172.21.15.116:0/1139069425 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f87f006c4e0 0x7f87f006e9a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 --2- 172.21.15.116:0/1139069425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8808100c00 0x7f8808074b30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 --2- 172.21.15.116:0/1139069425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8808101e00 0x7f8808073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 -- 172.21.15.116:0/1139069425 >> 172.21.15.116:0/1139069425 conn(0x7f88080fc180 msgr2=0x7f8808105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 -- 172.21.15.116:0/1139069425 shutdown_connections 2024-03-04T16:02:32.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:02:32.589+0000 7f880cf9b700 1 -- 172.21.15.116:0/1139069425 wait complete. 2024-03-04T16:02:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:32 smithi152 ceph-mon[36876]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:32 smithi152 ceph-mon[36876]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:32 smithi152 ceph-mon[36876]: pgmap v227: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:32 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3176594151' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:02:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:32 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/173145885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:02:33.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:33 smithi116 ceph-mon[29120]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:33.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:33 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1139069425' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:02:33.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:33 smithi152 ceph-mon[36876]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:33.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:33 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/1139069425' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:02:34.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:34 smithi116 ceph-mon[29120]: pgmap v228: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:34.758 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:34 smithi152 ceph-mon[36876]: pgmap v228: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:36.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:36 smithi116 ceph-mon[29120]: pgmap v229: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:36.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:36 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:02:36.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:36 smithi116 ceph-mon[29120]: Upgrade: Target is version 19.0.0-1826-g9ca0b5de (unknown) 2024-03-04T16:02:36.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:36 smithi116 ceph-mon[29120]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, digests ['quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977'] 2024-03-04T16:02:36.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:36 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:36.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:36 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:02:36.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:36 smithi116 ceph-mon[29120]: Upgrade: Need to upgrade myself (mgr.smithi116.jwdhop) 2024-03-04T16:02:36.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:36 smithi152 ceph-mon[36876]: pgmap v229: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:36 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:02:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:36 smithi152 ceph-mon[36876]: Upgrade: Target is version 19.0.0-1826-g9ca0b5de (unknown) 2024-03-04T16:02:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:36 smithi152 ceph-mon[36876]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, digests ['quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977'] 2024-03-04T16:02:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:36 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:36 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:02:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:36 smithi152 ceph-mon[36876]: Upgrade: Need to upgrade myself (mgr.smithi116.jwdhop) 2024-03-04T16:02:37.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:37 smithi116 ceph-mon[29120]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 on smithi152 2024-03-04T16:02:37.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:37 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:02:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:37 smithi152 ceph-mon[36876]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 on smithi152 2024-03-04T16:02:37.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:37 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:02:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:38 smithi116 ceph-mon[29120]: pgmap v230: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:38 smithi152 ceph-mon[36876]: pgmap v230: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:40.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:40 smithi152 ceph-mon[36876]: pgmap v231: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:40 smithi116 ceph-mon[29120]: pgmap v231: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:42.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:42 smithi152 ceph-mon[36876]: pgmap v232: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:42.653 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:42 smithi116 ceph-mon[29120]: pgmap v232: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:44.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:44 smithi152 ceph-mon[36876]: pgmap v233: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:44 smithi116 ceph-mon[29120]: pgmap v233: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:46.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:46 smithi152 ceph-mon[36876]: pgmap v234: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:46.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:46 smithi116 ceph-mon[29120]: pgmap v234: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:48.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:48 smithi152 ceph-mon[36876]: pgmap v235: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:48.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:48 smithi116 ceph-mon[29120]: pgmap v235: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:50.315 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:50 smithi152 ceph-mon[36876]: pgmap v236: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:50.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:50 smithi116 ceph-mon[29120]: pgmap v236: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:52 smithi152 ceph-mon[36876]: pgmap v237: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:52 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:02:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:52 smithi116 ceph-mon[29120]: pgmap v237: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:52 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:02:54.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:54 smithi152 ceph-mon[36876]: pgmap v238: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:54.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:54 smithi116 ceph-mon[29120]: pgmap v238: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:56.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:56 smithi152 ceph-mon[36876]: pgmap v239: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:56 smithi116 ceph-mon[29120]: pgmap v239: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:58.601 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:02:58 smithi116 ceph-mon[29120]: pgmap v240: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:58.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:02:58 smithi152 ceph-mon[36876]: pgmap v240: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:00.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:00 smithi152 ceph-mon[36876]: pgmap v241: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:00.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:00 smithi116 ceph-mon[29120]: pgmap v241: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:02.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:02 smithi152 ceph-mon[36876]: pgmap v242: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:02.702 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:02 smithi116 ceph-mon[29120]: pgmap v242: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.700+0000 7f0c4f108700 1 -- 172.21.15.116:0/1892395857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c48100cb0 msgr2=0x7f0c48101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.700+0000 7f0c4f108700 1 --2- 172.21.15.116:0/1892395857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c48100cb0 0x7f0c48101110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009910 tx=0x7f0c34009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.701+0000 7f0c4f108700 1 -- 172.21.15.116:0/1892395857 shutdown_connections 2024-03-04T16:03:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.701+0000 7f0c4f108700 1 --2- 172.21.15.116:0/1892395857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c48100cb0 0x7f0c48101110 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.701+0000 7f0c4f108700 1 --2- 172.21.15.116:0/1892395857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c480ffab0 0x7f0c480ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.701+0000 7f0c4f108700 1 -- 172.21.15.116:0/1892395857 >> 172.21.15.116:0/1892395857 conn(0x7f0c480fb050 msgr2=0x7f0c480fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.701+0000 7f0c4f108700 1 -- 172.21.15.116:0/1892395857 shutdown_connections 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.701+0000 7f0c4f108700 1 -- 172.21.15.116:0/1892395857 wait complete. 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.701+0000 7f0c4f108700 1 Processor -- start 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.701+0000 7f0c4f108700 1 -- start start 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c4f108700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c480ffab0 0x7f0c48198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c4f108700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c48100cb0 0x7f0c48198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c4f108700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c481990b0 con 0x7f0c48100cb0 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c4f108700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c481991f0 con 0x7f0c480ffab0 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c47fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c48100cb0 0x7f0c48198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c47fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c48100cb0 0x7f0c48198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38522/0 (socket says 172.21.15.116:38522) 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c47fff700 1 -- 172.21.15.116:0/1629059548 learned_addr learned my addr 172.21.15.116:0/1629059548 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c4cea4700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c480ffab0 0x7f0c48198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c47fff700 1 -- 172.21.15.116:0/1629059548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c480ffab0 msgr2=0x7f0c48198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c47fff700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c480ffab0 0x7f0c48198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c47fff700 1 -- 172.21.15.116:0/1629059548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c340095f0 con 0x7f0c48100cb0 2024-03-04T16:03:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c47fff700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c48100cb0 0x7f0c48198a90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0c34006010 tx=0x7f0c3400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.702+0000 7f0c45ffb700 1 -- 172.21.15.116:0/1629059548 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c3401d070 con 0x7f0c48100cb0 2024-03-04T16:03:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.703+0000 7f0c45ffb700 1 -- 172.21.15.116:0/1629059548 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0c3400f460 con 0x7f0c48100cb0 2024-03-04T16:03:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.703+0000 7f0c4f108700 1 -- 172.21.15.116:0/1629059548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4819dc40 con 0x7f0c48100cb0 2024-03-04T16:03:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.703+0000 7f0c45ffb700 1 -- 172.21.15.116:0/1629059548 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c34021640 con 0x7f0c48100cb0 2024-03-04T16:03:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.703+0000 7f0c4f108700 1 -- 172.21.15.116:0/1629059548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4819e130 con 0x7f0c48100cb0 2024-03-04T16:03:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.704+0000 7f0c4f108700 1 -- 172.21.15.116:0/1629059548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c48108610 con 0x7f0c48100cb0 2024-03-04T16:03:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.704+0000 7f0c45ffb700 1 -- 172.21.15.116:0/1629059548 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0c3400fab0 con 0x7f0c48100cb0 2024-03-04T16:03:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.705+0000 7f0c45ffb700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c3806c4e0 0x7f0c3806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.705+0000 7f0c45ffb700 1 -- 172.21.15.116:0/1629059548 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f0c3408cb00 con 0x7f0c48100cb0 2024-03-04T16:03:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.705+0000 7f0c4cea4700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c3806c4e0 0x7f0c3806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.705+0000 7f0c4cea4700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c3806c4e0 0x7f0c3806e9a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0c48100b10 tx=0x7f0c3c00b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:02.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.709+0000 7f0c45ffb700 1 -- 172.21.15.116:0/1629059548 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0c3405ad10 con 0x7f0c48100cb0 2024-03-04T16:03:02.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.925+0000 7f0c4f108700 1 -- 172.21.15.116:0/1629059548 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c480693b0 con 0x7f0c3806c4e0 2024-03-04T16:03:02.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.926+0000 7f0c45ffb700 1 -- 172.21.15.116:0/1629059548 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f0c480693b0 con 0x7f0c3806c4e0 2024-03-04T16:03:02.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 -- 172.21.15.116:0/1629059548 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c3806c4e0 msgr2=0x7f0c3806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:02.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c3806c4e0 0x7f0c3806e9a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0c48100b10 tx=0x7f0c3c00b410 comp rx=0 tx=0).stop 2024-03-04T16:03:02.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 -- 172.21.15.116:0/1629059548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c48100cb0 msgr2=0x7f0c48198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:02.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c48100cb0 0x7f0c48198a90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0c34006010 tx=0x7f0c3400bd70 comp rx=0 tx=0).stop 2024-03-04T16:03:02.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 -- 172.21.15.116:0/1629059548 shutdown_connections 2024-03-04T16:03:02.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c480ffab0 0x7f0c48198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:02.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f0c3806c4e0 0x7f0c3806e9a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:02.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 --2- 172.21.15.116:0/1629059548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c48100cb0 0x7f0c48198a90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:02.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 -- 172.21.15.116:0/1629059548 >> 172.21.15.116:0/1629059548 conn(0x7f0c480fb050 msgr2=0x7f0c48068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:02.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 -- 172.21.15.116:0/1629059548 shutdown_connections 2024-03-04T16:03:02.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:02.928+0000 7f0c4f108700 1 -- 172.21.15.116:0/1629059548 wait complete. 2024-03-04T16:03:02.940 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:03:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.039+0000 7f076d4cb700 1 -- 172.21.15.116:0/3627940121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0768101dc0 msgr2=0x7f0768102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.039+0000 7f076d4cb700 1 --2- 172.21.15.116:0/3627940121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0768101dc0 0x7f0768102220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0758009910 tx=0x7f0758009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.039+0000 7f076d4cb700 1 -- 172.21.15.116:0/3627940121 shutdown_connections 2024-03-04T16:03:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.039+0000 7f076d4cb700 1 --2- 172.21.15.116:0/3627940121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0768101dc0 0x7f0768102220 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.039+0000 7f076d4cb700 1 --2- 172.21.15.116:0/3627940121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0768100bc0 0x7f0768100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.039+0000 7f076d4cb700 1 -- 172.21.15.116:0/3627940121 >> 172.21.15.116:0/3627940121 conn(0x7f07680fc140 msgr2=0x7f07680fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.040+0000 7f076d4cb700 1 -- 172.21.15.116:0/3627940121 shutdown_connections 2024-03-04T16:03:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.040+0000 7f076d4cb700 1 -- 172.21.15.116:0/3627940121 wait complete. 2024-03-04T16:03:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.040+0000 7f076d4cb700 1 Processor -- start 2024-03-04T16:03:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.040+0000 7f076d4cb700 1 -- start start 2024-03-04T16:03:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f076d4cb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0768100bc0 0x7f0768196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f076d4cb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0768101dc0 0x7f07681968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f076d4cb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0768196ed0 con 0x7f0768101dc0 2024-03-04T16:03:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f076d4cb700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0768197010 con 0x7f0768100bc0 2024-03-04T16:03:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f07667fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0768101dc0 0x7f07681968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f07667fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0768101dc0 0x7f07681968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38540/0 (socket says 172.21.15.116:38540) 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f0766ffd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0768100bc0 0x7f0768196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f07667fc700 1 -- 172.21.15.116:0/3170629929 learned_addr learned my addr 172.21.15.116:0/3170629929 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f07667fc700 1 -- 172.21.15.116:0/3170629929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0768100bc0 msgr2=0x7f0768196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f07667fc700 1 --2- 172.21.15.116:0/3170629929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0768100bc0 0x7f0768196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f07667fc700 1 -- 172.21.15.116:0/3170629929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07580095f0 con 0x7f0768101dc0 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f07667fc700 1 --2- 172.21.15.116:0/3170629929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0768101dc0 0x7f07681968b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0758006010 tx=0x7f075800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f074ffff700 1 -- 172.21.15.116:0/3170629929 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f075801d070 con 0x7f0768101dc0 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.041+0000 7f074ffff700 1 -- 172.21.15.116:0/3170629929 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f075800f460 con 0x7f0768101dc0 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.042+0000 7f074ffff700 1 -- 172.21.15.116:0/3170629929 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0758021640 con 0x7f0768101dc0 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.042+0000 7f076d4cb700 1 -- 172.21.15.116:0/3170629929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076819ba60 con 0x7f0768101dc0 2024-03-04T16:03:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.042+0000 7f076d4cb700 1 -- 172.21.15.116:0/3170629929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076819be90 con 0x7f0768101dc0 2024-03-04T16:03:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.043+0000 7f076d4cb700 1 -- 172.21.15.116:0/3170629929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0768109720 con 0x7f0768101dc0 2024-03-04T16:03:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.043+0000 7f074ffff700 1 -- 172.21.15.116:0/3170629929 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f075800fab0 con 0x7f0768101dc0 2024-03-04T16:03:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.044+0000 7f074ffff700 1 --2- 172.21.15.116:0/3170629929 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f075006c4e0 0x7f075006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.044+0000 7f0766ffd700 1 --2- 172.21.15.116:0/3170629929 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f075006c4e0 0x7f075006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.044+0000 7f074ffff700 1 -- 172.21.15.116:0/3170629929 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f075808caa0 con 0x7f0768101dc0 2024-03-04T16:03:03.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.044+0000 7f0766ffd700 1 --2- 172.21.15.116:0/3170629929 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f075006c4e0 0x7f075006e9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0768101c20 tx=0x7f075c00b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.048+0000 7f074ffff700 1 -- 172.21.15.116:0/3170629929 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f075805ad30 con 0x7f0768101dc0 2024-03-04T16:03:03.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.261+0000 7f076d4cb700 1 -- 172.21.15.116:0/3170629929 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0768106710 con 0x7f075006c4e0 2024-03-04T16:03:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.262+0000 7f074ffff700 1 -- 172.21.15.116:0/3170629929 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f0768106710 con 0x7f075006c4e0 2024-03-04T16:03:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 -- 172.21.15.116:0/3170629929 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f075006c4e0 msgr2=0x7f075006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 --2- 172.21.15.116:0/3170629929 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f075006c4e0 0x7f075006e9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0768101c20 tx=0x7f075c00b410 comp rx=0 tx=0).stop 2024-03-04T16:03:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 -- 172.21.15.116:0/3170629929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0768101dc0 msgr2=0x7f07681968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 --2- 172.21.15.116:0/3170629929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0768101dc0 0x7f07681968b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0758006010 tx=0x7f075800bd70 comp rx=0 tx=0).stop 2024-03-04T16:03:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 -- 172.21.15.116:0/3170629929 shutdown_connections 2024-03-04T16:03:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 --2- 172.21.15.116:0/3170629929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0768100bc0 0x7f0768196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 --2- 172.21.15.116:0/3170629929 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f075006c4e0 0x7f075006e9a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 --2- 172.21.15.116:0/3170629929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0768101dc0 0x7f07681968b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 -- 172.21.15.116:0/3170629929 >> 172.21.15.116:0/3170629929 conn(0x7f07680fc140 msgr2=0x7f0768104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 -- 172.21.15.116:0/3170629929 shutdown_connections 2024-03-04T16:03:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.264+0000 7f076d4cb700 1 -- 172.21.15.116:0/3170629929 wait complete. 2024-03-04T16:03:03.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.372+0000 7fd18a6e5700 1 -- 172.21.15.116:0/3278089377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd184100cb0 msgr2=0x7fd184101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:03.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.372+0000 7fd18a6e5700 1 --2- 172.21.15.116:0/3278089377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd184100cb0 0x7fd184101110 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd178009910 tx=0x7fd178009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:03.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.372+0000 7fd18a6e5700 1 -- 172.21.15.116:0/3278089377 shutdown_connections 2024-03-04T16:03:03.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.372+0000 7fd18a6e5700 1 --2- 172.21.15.116:0/3278089377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd184100cb0 0x7fd184101110 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.372+0000 7fd18a6e5700 1 --2- 172.21.15.116:0/3278089377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1840ffab0 0x7fd1840ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.372+0000 7fd18a6e5700 1 -- 172.21.15.116:0/3278089377 >> 172.21.15.116:0/3278089377 conn(0x7fd1840fb030 msgr2=0x7fd1840fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:03.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.372+0000 7fd18a6e5700 1 -- 172.21.15.116:0/3278089377 shutdown_connections 2024-03-04T16:03:03.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.372+0000 7fd18a6e5700 1 -- 172.21.15.116:0/3278089377 wait complete. 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.373+0000 7fd18a6e5700 1 Processor -- start 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.373+0000 7fd18a6e5700 1 -- start start 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.373+0000 7fd18a6e5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1840ffab0 0x7fd184068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.373+0000 7fd18a6e5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd184100cb0 0x7fd18419b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.373+0000 7fd18a6e5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18419b690 con 0x7fd184100cb0 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.373+0000 7fd18a6e5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18419b7d0 con 0x7fd1840ffab0 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd183fff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1840ffab0 0x7fd184068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd1837fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd184100cb0 0x7fd18419b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd183fff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1840ffab0 0x7fd184068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:46384/0 (socket says 172.21.15.116:46384) 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd1837fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd184100cb0 0x7fd18419b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38556/0 (socket says 172.21.15.116:38556) 2024-03-04T16:03:03.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd183fff700 1 -- 172.21.15.116:0/1294854239 learned_addr learned my addr 172.21.15.116:0/1294854239 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:03.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd1837fe700 1 -- 172.21.15.116:0/1294854239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1840ffab0 msgr2=0x7fd184068f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:03.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd1837fe700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1840ffab0 0x7fd184068f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd1837fe700 1 -- 172.21.15.116:0/1294854239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1780095f0 con 0x7fd184100cb0 2024-03-04T16:03:03.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd183fff700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1840ffab0 0x7fd184068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T16:03:03.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd1837fe700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd184100cb0 0x7fd18419b0c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd178009910 tx=0x7fd17800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:03.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd1817fa700 1 -- 172.21.15.116:0/1294854239 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd17801d070 con 0x7fd184100cb0 2024-03-04T16:03:03.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd1817fa700 1 -- 172.21.15.116:0/1294854239 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd17800f460 con 0x7fd184100cb0 2024-03-04T16:03:03.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd18a6e5700 1 -- 172.21.15.116:0/1294854239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18419ba50 con 0x7fd184100cb0 2024-03-04T16:03:03.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.374+0000 7fd1817fa700 1 -- 172.21.15.116:0/1294854239 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd178021640 con 0x7fd184100cb0 2024-03-04T16:03:03.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.375+0000 7fd18a6e5700 1 -- 172.21.15.116:0/1294854239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18419bf40 con 0x7fd184100cb0 2024-03-04T16:03:03.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.376+0000 7fd18a6e5700 1 -- 172.21.15.116:0/1294854239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd184108610 con 0x7fd184100cb0 2024-03-04T16:03:03.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.376+0000 7fd1817fa700 1 -- 172.21.15.116:0/1294854239 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd17800fa40 con 0x7fd184100cb0 2024-03-04T16:03:03.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.377+0000 7fd1817fa700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fd16c06c4e0 0x7fd16c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:03.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.377+0000 7fd1817fa700 1 -- 172.21.15.116:0/1294854239 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd17808c940 con 0x7fd184100cb0 2024-03-04T16:03:03.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.377+0000 7fd183fff700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fd16c06c4e0 0x7fd16c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.377+0000 7fd183fff700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fd16c06c4e0 0x7fd16c06e9a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd1740105c0 tx=0x7fd17400b4a0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:03.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.382+0000 7fd1817fa700 1 -- 172.21.15.116:0/1294854239 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd17805abd0 con 0x7fd184100cb0 2024-03-04T16:03:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.595+0000 7fd18a6e5700 1 -- 172.21.15.116:0/1294854239 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd184105600 con 0x7fd16c06c4e0 2024-03-04T16:03:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.606+0000 7fd1817fa700 1 -- 172.21.15.116:0/1294854239 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7fd184105600 con 0x7fd16c06c4e0 2024-03-04T16:03:03.608 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:03:03.608 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (6m) 93s ago 7m 25.1M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:03:03.608 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (7m) 93s ago 7m 7390k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:03:03.608 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (6m) 99s ago 6m 7285k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:03:03.608 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (7m) 93s ago 7m 6665k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:03:03.609 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (6m) 99s ago 6m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:03:03.609 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (5m) 93s ago 7m 87.2M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:03:03.609 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (112s) 93s ago 111s 15.3M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:03:03.609 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (119s) 93s ago 118s 11.3M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:03:03.609 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (109s) 99s ago 109s 12.9M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:03:03.609 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (116s) 99s ago 116s 16.2M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:03:03.610 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:9283,8765,8443 running (9m) 93s ago 9m 489M - 18.2.1 5be31c24972a 083879ca4bf4 2024-03-04T16:03:03.610 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (6m) 99s ago 6m 443M - 18.2.1 5be31c24972a c68d3e4a1dbc 2024-03-04T16:03:03.610 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (9m) 93s ago 9m 45.4M 2048M 18.2.1 5be31c24972a 60e0bf118c4d 2024-03-04T16:03:03.610 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (6m) 99s ago 6m 37.6M 2048M 18.2.1 5be31c24972a 52450d235516 2024-03-04T16:03:03.610 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (7m) 93s ago 7m 15.4M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:03:03.610 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (6m) 99s ago 6m 16.9M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:03:03.610 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (5m) 93s ago 5m 50.5M 3946M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:03:03.611 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (5m) 93s ago 5m 46.8M 3946M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:03:03.611 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (4m) 93s ago 4m 46.4M 3946M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:03:03.611 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (4m) 99s ago 4m 46.2M 4970M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:03:03.611 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (3m) 99s ago 3m 43.8M 4970M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:03:03.612 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (3m) 99s ago 3m 44.0M 4970M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:03:03.612 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (5m) 93s ago 7m 55.8M - 2.43.0 a07b618ecd1d 31240d908f24 2024-03-04T16:03:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 -- 172.21.15.116:0/1294854239 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fd16c06c4e0 msgr2=0x7fd16c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fd16c06c4e0 0x7fd16c06e9a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd1740105c0 tx=0x7fd17400b4a0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 -- 172.21.15.116:0/1294854239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd184100cb0 msgr2=0x7fd18419b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd184100cb0 0x7fd18419b0c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd178009910 tx=0x7fd17800bd70 comp rx=0 tx=0).stop 2024-03-04T16:03:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 -- 172.21.15.116:0/1294854239 shutdown_connections 2024-03-04T16:03:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1840ffab0 0x7fd184068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fd16c06c4e0 0x7fd16c06e9a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 --2- 172.21.15.116:0/1294854239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd184100cb0 0x7fd18419b0c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 -- 172.21.15.116:0/1294854239 >> 172.21.15.116:0/1294854239 conn(0x7fd1840fb030 msgr2=0x7fd184103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 -- 172.21.15.116:0/1294854239 shutdown_connections 2024-03-04T16:03:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.608+0000 7fd18a6e5700 1 -- 172.21.15.116:0/1294854239 wait complete. 2024-03-04T16:03:03.621 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'. 2024-03-04T16:03:03.621 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T16:03:03.622 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-04T16:03:03.622 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-04T16:03:03.622 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2024-03-04T16:03:03.622 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T16:03:03.622 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-04T16:03:03.622 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-04T16:03:03.623 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T16:03:03.623 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2024-03-04T16:03:03.623 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T16:03:03.623 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2024-03-04T16:03:03.623 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T16:03:03.623 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2024-03-04T16:03:03.623 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T16:03:03.624 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-04T16:03:03.624 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T16:03:03.624 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at 9ca0b5de35e Merge branch 'bz2068026-nfs-idmap' of https://github.com/asm0deuz/ceph into wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T16:03:03.626 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-04T16:03:03.655 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-04T16:03:03.657 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-04T16:03:03.657 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-04T16:03:03.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.715+0000 7f1204a88700 1 -- 172.21.15.116:0/1600290207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1200100cb0 msgr2=0x7f1200101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:03.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.715+0000 7f1204a88700 1 --2- 172.21.15.116:0/1600290207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1200100cb0 0x7f1200101110 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009910 tx=0x7f11f4009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:03.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.716+0000 7f1204a88700 1 -- 172.21.15.116:0/1600290207 shutdown_connections 2024-03-04T16:03:03.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.716+0000 7f1204a88700 1 --2- 172.21.15.116:0/1600290207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1200100cb0 0x7f1200101110 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.716+0000 7f1204a88700 1 --2- 172.21.15.116:0/1600290207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12000ffab0 0x7f12000ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.716+0000 7f1204a88700 1 -- 172.21.15.116:0/1600290207 >> 172.21.15.116:0/1600290207 conn(0x7f12000fb030 msgr2=0x7f12000fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:03.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.716+0000 7f1204a88700 1 -- 172.21.15.116:0/1600290207 shutdown_connections 2024-03-04T16:03:03.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.716+0000 7f1204a88700 1 -- 172.21.15.116:0/1600290207 wait complete. 2024-03-04T16:03:03.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f1204a88700 1 Processor -- start 2024-03-04T16:03:03.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f1204a88700 1 -- start start 2024-03-04T16:03:03.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f1204a88700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12000ffab0 0x7f1200068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:03.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f1204a88700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1200100cb0 0x7f120019b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:03.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f1204a88700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120019b690 con 0x7f1200100cb0 2024-03-04T16:03:03.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f1204a88700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120019b7d0 con 0x7f12000ffab0 2024-03-04T16:03:03.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f11fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1200100cb0 0x7f120019b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:03.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f11fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1200100cb0 0x7f120019b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38574/0 (socket says 172.21.15.116:38574) 2024-03-04T16:03:03.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f11fdd9b700 1 -- 172.21.15.116:0/379147183 learned_addr learned my addr 172.21.15.116:0/379147183 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:03.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f11fe59c700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12000ffab0 0x7f1200068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:03.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f11fdd9b700 1 -- 172.21.15.116:0/379147183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12000ffab0 msgr2=0x7f1200068f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:03.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f11fdd9b700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12000ffab0 0x7f1200068f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:03.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f11fdd9b700 1 -- 172.21.15.116:0/379147183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f40095f0 con 0x7f1200100cb0 2024-03-04T16:03:03.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.717+0000 7f11fdd9b700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1200100cb0 0x7f120019b0c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f11f4000c00 tx=0x7f11f400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:03.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.718+0000 7f11ef7fe700 1 -- 172.21.15.116:0/379147183 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11f401d070 con 0x7f1200100cb0 2024-03-04T16:03:03.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.718+0000 7f11ef7fe700 1 -- 172.21.15.116:0/379147183 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f11f4005dc0 con 0x7f1200100cb0 2024-03-04T16:03:03.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.718+0000 7f1204a88700 1 -- 172.21.15.116:0/379147183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120019ba50 con 0x7f1200100cb0 2024-03-04T16:03:03.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.718+0000 7f11ef7fe700 1 -- 172.21.15.116:0/379147183 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11f40226a0 con 0x7f1200100cb0 2024-03-04T16:03:03.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.718+0000 7f1204a88700 1 -- 172.21.15.116:0/379147183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120019bf40 con 0x7f1200100cb0 2024-03-04T16:03:03.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.719+0000 7f1204a88700 1 -- 172.21.15.116:0/379147183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1200108630 con 0x7f1200100cb0 2024-03-04T16:03:03.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.719+0000 7f11ef7fe700 1 -- 172.21.15.116:0/379147183 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f11f400faf0 con 0x7f1200100cb0 2024-03-04T16:03:03.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.720+0000 7f11ef7fe700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f11e806c4e0 0x7f11e806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:03.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.720+0000 7f11fe59c700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f11e806c4e0 0x7f11e806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:03.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.720+0000 7f11ef7fe700 1 -- 172.21.15.116:0/379147183 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f11f408cdb0 con 0x7f1200100cb0 2024-03-04T16:03:03.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.720+0000 7f11fe59c700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f11e806c4e0 0x7f11e806e9a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1200100b10 tx=0x7f11f0009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:03.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:03.724+0000 7f11ef7fe700 1 -- 172.21.15.116:0/379147183 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f11f405afc0 con 0x7f1200100cb0 2024-03-04T16:03:04.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.006+0000 7f1204a88700 1 -- 172.21.15.116:0/379147183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f120019c580 con 0x7f1200100cb0 2024-03-04T16:03:04.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.006+0000 7f11ef7fe700 1 -- 172.21.15.116:0/379147183 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f11f405ab50 con 0x7f1200100cb0 2024-03-04T16:03:04.008 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:03:04.008 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:03:04.008 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-03-04T16:03:04.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:03:04.008 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:03:04.008 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-03-04T16:03:04.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:03:04.008 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:03:04.008 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-03-04T16:03:04.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:03:04.009 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:03:04.009 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:03:04.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:03:04.009 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:03:04.009 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-03-04T16:03:04.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:03:04.009 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:03:04.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 -- 172.21.15.116:0/379147183 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f11e806c4e0 msgr2=0x7f11e806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f11e806c4e0 0x7f11e806e9a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1200100b10 tx=0x7f11f0009260 comp rx=0 tx=0).stop 2024-03-04T16:03:04.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 -- 172.21.15.116:0/379147183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1200100cb0 msgr2=0x7f120019b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1200100cb0 0x7f120019b0c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f11f4000c00 tx=0x7f11f400f690 comp rx=0 tx=0).stop 2024-03-04T16:03:04.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 -- 172.21.15.116:0/379147183 shutdown_connections 2024-03-04T16:03:04.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12000ffab0 0x7f1200068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f11e806c4e0 0x7f11e806e9a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 --2- 172.21.15.116:0/379147183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1200100cb0 0x7f120019b0c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 -- 172.21.15.116:0/379147183 >> 172.21.15.116:0/379147183 conn(0x7f12000fb030 msgr2=0x7f1200103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:04.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 -- 172.21.15.116:0/379147183 shutdown_connections 2024-03-04T16:03:04.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.009+0000 7f1204a88700 1 -- 172.21.15.116:0/379147183 wait complete. 2024-03-04T16:03:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.113+0000 7f03d9c6b700 1 -- 172.21.15.116:0/812582435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03d4103ff0 msgr2=0x7f03d41063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.113+0000 7f03d9c6b700 1 --2- 172.21.15.116:0/812582435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03d4103ff0 0x7f03d41063e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f03bc009910 tx=0x7f03bc009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.113+0000 7f03d9c6b700 1 -- 172.21.15.116:0/812582435 shutdown_connections 2024-03-04T16:03:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.113+0000 7f03d9c6b700 1 --2- 172.21.15.116:0/812582435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03d4103ff0 0x7f03d41063e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.113+0000 7f03d9c6b700 1 --2- 172.21.15.116:0/812582435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03d41016c0 0x7f03d4103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.113+0000 7f03d9c6b700 1 -- 172.21.15.116:0/812582435 >> 172.21.15.116:0/812582435 conn(0x7f03d40fb030 msgr2=0x7f03d40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.113+0000 7f03d9c6b700 1 -- 172.21.15.116:0/812582435 shutdown_connections 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.113+0000 7f03d9c6b700 1 -- 172.21.15.116:0/812582435 wait complete. 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.113+0000 7f03d9c6b700 1 Processor -- start 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03d9c6b700 1 -- start start 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03d9c6b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03d41016c0 0x7f03d4196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03d9c6b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03d4103ff0 0x7f03d4196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03d9c6b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d4196e90 con 0x7f03d4103ff0 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03d9c6b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d4196fd0 con 0x7f03d41016c0 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03caffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03d4103ff0 0x7f03d4196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03caffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03d4103ff0 0x7f03d4196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38584/0 (socket says 172.21.15.116:38584) 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03caffd700 1 -- 172.21.15.116:0/2259467243 learned_addr learned my addr 172.21.15.116:0/2259467243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03d37fe700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03d41016c0 0x7f03d4196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03caffd700 1 -- 172.21.15.116:0/2259467243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03d41016c0 msgr2=0x7f03d4196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03caffd700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03d41016c0 0x7f03d4196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.114+0000 7f03caffd700 1 -- 172.21.15.116:0/2259467243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc0095f0 con 0x7f03d4103ff0 2024-03-04T16:03:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.115+0000 7f03caffd700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03d4103ff0 0x7f03d4196870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f03bc006010 tx=0x7f03bc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.115+0000 7f03d17fa700 1 -- 172.21.15.116:0/2259467243 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03bc01d070 con 0x7f03d4103ff0 2024-03-04T16:03:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.115+0000 7f03d17fa700 1 -- 172.21.15.116:0/2259467243 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f03bc00f460 con 0x7f03d4103ff0 2024-03-04T16:03:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.115+0000 7f03d17fa700 1 -- 172.21.15.116:0/2259467243 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03bc021640 con 0x7f03d4103ff0 2024-03-04T16:03:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.115+0000 7f03d9c6b700 1 -- 172.21.15.116:0/2259467243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d419ba20 con 0x7f03d4103ff0 2024-03-04T16:03:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.115+0000 7f03d9c6b700 1 -- 172.21.15.116:0/2259467243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d419bf10 con 0x7f03d4103ff0 2024-03-04T16:03:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.116+0000 7f03d9c6b700 1 -- 172.21.15.116:0/2259467243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d4190520 con 0x7f03d4103ff0 2024-03-04T16:03:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.116+0000 7f03d17fa700 1 -- 172.21.15.116:0/2259467243 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f03bc00fab0 con 0x7f03d4103ff0 2024-03-04T16:03:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.117+0000 7f03d17fa700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f03c006c4e0 0x7f03c006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.117+0000 7f03d17fa700 1 -- 172.21.15.116:0/2259467243 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f03bc08cb20 con 0x7f03d4103ff0 2024-03-04T16:03:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.117+0000 7f03d37fe700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f03c006c4e0 0x7f03c006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.117+0000 7f03d37fe700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f03c006c4e0 0x7f03c006e9a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f03c4005950 tx=0x7f03c400b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.121+0000 7f03d17fa700 1 -- 172.21.15.116:0/2259467243 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f03bc05ad30 con 0x7f03d4103ff0 2024-03-04T16:03:04.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.363+0000 7f03d9c6b700 1 -- 172.21.15.116:0/2259467243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f03d402cc70 con 0x7f03d4103ff0 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.363+0000 7f03d17fa700 1 -- 172.21.15.116:0/2259467243 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1831 (secure 0 0 0) 0x7f03bc05a8c0 con 0x7f03d4103ff0 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:03:04.365 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:03:04.366 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:03:04.367 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:03:04.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.365+0000 7f03d9c6b700 1 -- 172.21.15.116:0/2259467243 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f03c006c4e0 msgr2=0x7f03c006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.365+0000 7f03d9c6b700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f03c006c4e0 0x7f03c006e9a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f03c4005950 tx=0x7f03c400b410 comp rx=0 tx=0).stop 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.365+0000 7f03d9c6b700 1 -- 172.21.15.116:0/2259467243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03d4103ff0 msgr2=0x7f03d4196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.365+0000 7f03d9c6b700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03d4103ff0 0x7f03d4196870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f03bc006010 tx=0x7f03bc00bd70 comp rx=0 tx=0).stop 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.365+0000 7f03d9c6b700 1 -- 172.21.15.116:0/2259467243 shutdown_connections 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.365+0000 7f03d9c6b700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03d41016c0 0x7f03d4196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.365+0000 7f03d9c6b700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f03c006c4e0 0x7f03c006e9a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.365+0000 7f03d9c6b700 1 --2- 172.21.15.116:0/2259467243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03d4103ff0 0x7f03d4196870 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.365+0000 7f03d9c6b700 1 -- 172.21.15.116:0/2259467243 >> 172.21.15.116:0/2259467243 conn(0x7f03d40fb030 msgr2=0x7f03d40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.365+0000 7f03d9c6b700 1 -- 172.21.15.116:0/2259467243 shutdown_connections 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.366+0000 7f03d9c6b700 1 -- 172.21.15.116:0/2259467243 wait complete. 2024-03-04T16:03:04.369 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:03:04.402 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-04T16:03:04.409 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:04 smithi152 ceph-mon[36876]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:04.410 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:04 smithi152 ceph-mon[36876]: pgmap v243: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:04.410 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:04 smithi152 ceph-mon[36876]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:04.410 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:04 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/379147183' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:03:04.473 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-04T16:03:04.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.474+0000 7f9f57c25700 1 -- 172.21.15.116:0/1958418123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f500ffdf0 msgr2=0x7f9f50100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.474+0000 7f9f57c25700 1 --2- 172.21.15.116:0/1958418123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f500ffdf0 0x7f9f50100270 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009910 tx=0x7f9f40009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:04.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.474+0000 7f9f57c25700 1 -- 172.21.15.116:0/1958418123 shutdown_connections 2024-03-04T16:03:04.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.474+0000 7f9f57c25700 1 --2- 172.21.15.116:0/1958418123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f500ffdf0 0x7f9f50100270 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.474+0000 7f9f57c25700 1 --2- 172.21.15.116:0/1958418123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f500ff490 0x7f9f500ff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.474+0000 7f9f57c25700 1 -- 172.21.15.116:0/1958418123 >> 172.21.15.116:0/1958418123 conn(0x7f9f500fb030 msgr2=0x7f9f500fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:04.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.474+0000 7f9f57c25700 1 -- 172.21.15.116:0/1958418123 shutdown_connections 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.474+0000 7f9f57c25700 1 -- 172.21.15.116:0/1958418123 wait complete. 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f57c25700 1 Processor -- start 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f57c25700 1 -- start start 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f57c25700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f500ff490 0x7f9f50198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f57c25700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f500ffdf0 0x7f9f50198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f57c25700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f50199080 con 0x7f9f500ffdf0 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f57c25700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f501991c0 con 0x7f9f500ff490 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f551c0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f500ffdf0 0x7f9f50198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f551c0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f500ffdf0 0x7f9f50198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38588/0 (socket says 172.21.15.116:38588) 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f551c0700 1 -- 172.21.15.116:0/1052984493 learned_addr learned my addr 172.21.15.116:0/1052984493 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f559c1700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f500ff490 0x7f9f50198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f551c0700 1 -- 172.21.15.116:0/1052984493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f500ff490 msgr2=0x7f9f50198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f551c0700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f500ff490 0x7f9f50198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f551c0700 1 -- 172.21.15.116:0/1052984493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f400095f0 con 0x7f9f500ffdf0 2024-03-04T16:03:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.475+0000 7f9f551c0700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f500ffdf0 0x7f9f50198a60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000bd70 tx=0x7f9f4000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.476+0000 7f9f46ffd700 1 -- 172.21.15.116:0/1052984493 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f4001d070 con 0x7f9f500ffdf0 2024-03-04T16:03:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.476+0000 7f9f46ffd700 1 -- 172.21.15.116:0/1052984493 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9f40005dc0 con 0x7f9f500ffdf0 2024-03-04T16:03:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.476+0000 7f9f46ffd700 1 -- 172.21.15.116:0/1052984493 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f40021410 con 0x7f9f500ffdf0 2024-03-04T16:03:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.476+0000 7f9f57c25700 1 -- 172.21.15.116:0/1052984493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5019dc10 con 0x7f9f500ffdf0 2024-03-04T16:03:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.476+0000 7f9f57c25700 1 -- 172.21.15.116:0/1052984493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5019e100 con 0x7f9f500ffdf0 2024-03-04T16:03:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.477+0000 7f9f57c25700 1 -- 172.21.15.116:0/1052984493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5010b910 con 0x7f9f500ffdf0 2024-03-04T16:03:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.477+0000 7f9f46ffd700 1 -- 172.21.15.116:0/1052984493 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f9f4000f7d0 con 0x7f9f500ffdf0 2024-03-04T16:03:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.478+0000 7f9f46ffd700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9f3c06c4e0 0x7f9f3c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.478+0000 7f9f46ffd700 1 -- 172.21.15.116:0/1052984493 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9f4008ca70 con 0x7f9f500ffdf0 2024-03-04T16:03:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.478+0000 7f9f559c1700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9f3c06c4e0 0x7f9f3c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.478+0000 7f9f559c1700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9f3c06c4e0 0x7f9f3c06e9a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c007c60 tx=0x7f9f4c0073d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:04.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.486+0000 7f9f46ffd700 1 -- 172.21.15.116:0/1052984493 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9f4005ac80 con 0x7f9f500ffdf0 2024-03-04T16:03:04.517 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-04T16:03:04.519 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-04T16:03:04.519 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-04T16:03:04.558 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-04T16:03:04.561 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T16:03:04.561 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-04T16:03:04.616 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-04T16:03:04.620 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-04T16:03:04.620 DEBUG:teuthology.orchestra.run.smithi116:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-04T16:03:04.683 INFO:tasks.workunit.client.0.smithi116.stderr:+ mkdir -p fsstress 2024-03-04T16:03:04.687 INFO:tasks.workunit.client.0.smithi116.stderr:+ pushd fsstress 2024-03-04T16:03:04.689 INFO:tasks.workunit.client.0.smithi116.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-04T16:03:04.689 INFO:tasks.workunit.client.0.smithi116.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-04T16:03:04.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.701+0000 7f9f57c25700 1 -- 172.21.15.116:0/1052984493 --> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f501091e0 con 0x7f9f3c06c4e0 2024-03-04T16:03:04.703 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:04 smithi116 ceph-mon[29120]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:04.703 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:04 smithi116 ceph-mon[29120]: pgmap v243: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:04.703 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:04 smithi116 ceph-mon[29120]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:04.703 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:04 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/379147183' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:03:04.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.703+0000 7f9f46ffd700 1 -- 172.21.15.116:0/1052984493 <== mgr.14217 v2:172.21.15.116:6800/3356602598 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f9f501091e0 con 0x7f9f3c06c4e0 2024-03-04T16:03:04.704 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:03:04.704 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:03:04.704 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:03:04.704 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:03:04.704 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2024-03-04T16:03:04.704 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "0/23 daemons upgraded", 2024-03-04T16:03:04.704 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 image on host smithi152", 2024-03-04T16:03:04.704 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:03:04.704 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:03:04.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 -- 172.21.15.116:0/1052984493 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9f3c06c4e0 msgr2=0x7f9f3c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9f3c06c4e0 0x7f9f3c06e9a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c007c60 tx=0x7f9f4c0073d0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 -- 172.21.15.116:0/1052984493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f500ffdf0 msgr2=0x7f9f50198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f500ffdf0 0x7f9f50198a60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000bd70 tx=0x7f9f4000bda0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 -- 172.21.15.116:0/1052984493 shutdown_connections 2024-03-04T16:03:04.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f500ff490 0x7f9f50198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7f9f3c06c4e0 0x7f9f3c06e9a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 --2- 172.21.15.116:0/1052984493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f500ffdf0 0x7f9f50198a60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 -- 172.21.15.116:0/1052984493 >> 172.21.15.116:0/1052984493 conn(0x7f9f500fb030 msgr2=0x7f9f50107ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:04.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 -- 172.21.15.116:0/1052984493 shutdown_connections 2024-03-04T16:03:04.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.705+0000 7f9f57c25700 1 -- 172.21.15.116:0/1052984493 wait complete. 2024-03-04T16:03:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.808+0000 7fc5d5f20700 1 -- 172.21.15.116:0/114990281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0100bb0 msgr2=0x7fc5d0100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.808+0000 7fc5d5f20700 1 --2- 172.21.15.116:0/114990281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0100bb0 0x7fc5d0100fd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4009910 tx=0x7fc5c4009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.808+0000 7fc5d5f20700 1 -- 172.21.15.116:0/114990281 shutdown_connections 2024-03-04T16:03:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.808+0000 7fc5d5f20700 1 --2- 172.21.15.116:0/114990281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d0101db0 0x7fc5d0102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.808+0000 7fc5d5f20700 1 --2- 172.21.15.116:0/114990281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0100bb0 0x7fc5d0100fd0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.808+0000 7fc5d5f20700 1 -- 172.21.15.116:0/114990281 >> 172.21.15.116:0/114990281 conn(0x7fc5d00fc110 msgr2=0x7fc5d00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.808+0000 7fc5d5f20700 1 -- 172.21.15.116:0/114990281 shutdown_connections 2024-03-04T16:03:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.808+0000 7fc5d5f20700 1 -- 172.21.15.116:0/114990281 wait complete. 2024-03-04T16:03:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.808+0000 7fc5d5f20700 1 Processor -- start 2024-03-04T16:03:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5d5f20700 1 -- start start 2024-03-04T16:03:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5d5f20700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d0100bb0 0x7fc5d0074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5d5f20700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0101db0 0x7fc5d0073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5d5f20700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00736c0 con 0x7fc5d0101db0 2024-03-04T16:03:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5d5f20700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d0073800 con 0x7fc5d0100bb0 2024-03-04T16:03:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5ceffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0101db0 0x7fc5d0073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5ceffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0101db0 0x7fc5d0073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46684/0 (socket says 172.21.15.116:46684) 2024-03-04T16:03:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5ceffd700 1 -- 172.21.15.116:0/3845435564 learned_addr learned my addr 172.21.15.116:0/3845435564 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5cf7fe700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d0100bb0 0x7fc5d0074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5ceffd700 1 -- 172.21.15.116:0/3845435564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d0100bb0 msgr2=0x7fc5d0074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:04.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5ceffd700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d0100bb0 0x7fc5d0074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:04.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5ceffd700 1 -- 172.21.15.116:0/3845435564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c40095f0 con 0x7fc5d0101db0 2024-03-04T16:03:04.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.809+0000 7fc5ceffd700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0101db0 0x7fc5d0073180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc5c000d8d0 tx=0x7fc5c000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:04.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.810+0000 7fc5ccff9700 1 -- 172.21.15.116:0/3845435564 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5c00098a0 con 0x7fc5d0101db0 2024-03-04T16:03:04.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.810+0000 7fc5d5f20700 1 -- 172.21.15.116:0/3845435564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d0073ae0 con 0x7fc5d0101db0 2024-03-04T16:03:04.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.810+0000 7fc5ccff9700 1 -- 172.21.15.116:0/3845435564 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc5c0010460 con 0x7fc5d0101db0 2024-03-04T16:03:04.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.810+0000 7fc5ccff9700 1 -- 172.21.15.116:0/3845435564 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5c000f5f0 con 0x7fc5d0101db0 2024-03-04T16:03:04.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.810+0000 7fc5d5f20700 1 -- 172.21.15.116:0/3845435564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d019e3e0 con 0x7fc5d0101db0 2024-03-04T16:03:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.810+0000 7fc5d5f20700 1 -- 172.21.15.116:0/3845435564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d0109710 con 0x7fc5d0101db0 2024-03-04T16:03:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.811+0000 7fc5ccff9700 1 -- 172.21.15.116:0/3845435564 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc5c0009a00 con 0x7fc5d0101db0 2024-03-04T16:03:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.811+0000 7fc5ccff9700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc5b806c4e0 0x7fc5b806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.811+0000 7fc5cf7fe700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc5b806c4e0 0x7fc5b806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.811+0000 7fc5ccff9700 1 -- 172.21.15.116:0/3845435564 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc5c008b770 con 0x7fc5d0101db0 2024-03-04T16:03:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.812+0000 7fc5cf7fe700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc5b806c4e0 0x7fc5b806e9a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4006010 tx=0x7fc5c4005ec0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:04.816+0000 7fc5ccff9700 1 -- 172.21.15.116:0/3845435564 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc5c0059980 con 0x7fc5d0101db0 2024-03-04T16:03:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.086+0000 7fc5d5f20700 1 -- 172.21.15.116:0/3845435564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc5d019ec60 con 0x7fc5d0101db0 2024-03-04T16:03:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.086+0000 7fc5ccff9700 1 -- 172.21.15.116:0/3845435564 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fc5c0059510 con 0x7fc5d0101db0 2024-03-04T16:03:05.087 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:03:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 -- 172.21.15.116:0/3845435564 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc5b806c4e0 msgr2=0x7fc5b806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc5b806c4e0 0x7fc5b806e9a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4006010 tx=0x7fc5c4005ec0 comp rx=0 tx=0).stop 2024-03-04T16:03:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 -- 172.21.15.116:0/3845435564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0101db0 msgr2=0x7fc5d0073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0101db0 0x7fc5d0073180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc5c000d8d0 tx=0x7fc5c000dbe0 comp rx=0 tx=0).stop 2024-03-04T16:03:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 -- 172.21.15.116:0/3845435564 shutdown_connections 2024-03-04T16:03:05.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d0100bb0 0x7fc5d0074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:05.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.116:6800/3356602598,v1:172.21.15.116:6801/3356602598] conn(0x7fc5b806c4e0 0x7fc5b806e9a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:05.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 --2- 172.21.15.116:0/3845435564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0101db0 0x7fc5d0073180 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:05.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 -- 172.21.15.116:0/3845435564 >> 172.21.15.116:0/3845435564 conn(0x7fc5d00fc110 msgr2=0x7fc5d0104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:05.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 -- 172.21.15.116:0/3845435564 shutdown_connections 2024-03-04T16:03:05.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:05.088+0000 7fc5d5f20700 1 -- 172.21.15.116:0/3845435564 wait complete. 2024-03-04T16:03:05.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:05 smithi116 ceph-mon[29120]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:05.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:05 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2259467243' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:03:05.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:05 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3845435564' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:03:05.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:05 smithi152 ceph-mon[36876]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:05.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:05 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2259467243' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:03:05.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:05 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3845435564' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:03:06.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:06 smithi152 ceph-mon[36876]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:06.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:06 smithi152 ceph-mon[36876]: pgmap v244: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:06.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:06 smithi152 ceph-mon[36876]: Upgrade: Updating mgr.smithi152.aawieo 2024-03-04T16:03:06.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:06 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:06.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:06 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.aawieo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:03:06.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:06 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:03:06.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:06 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:06.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:06 smithi152 ceph-mon[36876]: Deploying daemon mgr.smithi152.aawieo on smithi152 2024-03-04T16:03:06.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:06 smithi116 ceph-mon[29120]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:06.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:06 smithi116 ceph-mon[29120]: pgmap v244: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:06.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:06 smithi116 ceph-mon[29120]: Upgrade: Updating mgr.smithi152.aawieo 2024-03-04T16:03:06.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:06 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:06.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:06 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.aawieo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:03:06.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:06 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:03:06.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:06 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:06.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:06 smithi116 ceph-mon[29120]: Deploying daemon mgr.smithi152.aawieo on smithi152 2024-03-04T16:03:07.473 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:07 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:07 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:08.720 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:08 smithi152 ceph-mon[36876]: pgmap v245: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:08.721 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:08 smithi116 ceph-mon[29120]: pgmap v245: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:09.146 INFO:tasks.workunit.client.1.smithi152.stderr:Updating files: 91% (11284/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-04T16:03:09.340 INFO:tasks.workunit.client.1.smithi152.stderr:Note: switching to '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'. 2024-03-04T16:03:09.340 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr:state without impacting any branches by switching back to a branch. 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr: git switch -c 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr:Or undo this operation with: 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr: git switch - 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-03-04T16:03:09.341 INFO:tasks.workunit.client.1.smithi152.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-04T16:03:09.342 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-03-04T16:03:09.342 INFO:tasks.workunit.client.1.smithi152.stderr:HEAD is now at 9ca0b5de35e Merge branch 'bz2068026-nfs-idmap' of https://github.com/asm0deuz/ceph into wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T16:03:09.344 DEBUG:teuthology.orchestra.run.smithi152:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-04T16:03:09.499 INFO:tasks.workunit.client.1.smithi152.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-04T16:03:09.501 INFO:tasks.workunit.client.1.smithi152.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-04T16:03:09.501 INFO:tasks.workunit.client.1.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-04T16:03:09.808 INFO:tasks.workunit.client.0.smithi116.stderr:+ tar xzf ltp-full.tgz 2024-03-04T16:03:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:10 smithi152 ceph-mon[36876]: pgmap v246: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:10 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:10 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:10 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:10.691 INFO:tasks.workunit.client.1.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-04T16:03:10.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:10 smithi116 ceph-mon[29120]: pgmap v246: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:10.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:10 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:10.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:10 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:10.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:10 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:10.811 INFO:tasks.workunit.client.1.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-04T16:03:10.857 INFO:tasks.workunit.client.1.smithi152.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-04T16:03:10.858 INFO:tasks.workunit.client.1.smithi152.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-04T16:03:10.859 INFO:tasks.workunit.client.1.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-04T16:03:10.906 INFO:tasks.workunit.client.1.smithi152.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-04T16:03:10.910 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T16:03:10.910 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-04T16:03:10.927 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-04T16:03:10.929 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-04T16:03:10.929 DEBUG:teuthology.orchestra.run.smithi152:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-04T16:03:10.995 INFO:tasks.workunit.client.1.smithi152.stderr:+ mkdir -p fsstress 2024-03-04T16:03:10.998 INFO:tasks.workunit.client.1.smithi152.stderr:+ pushd fsstress 2024-03-04T16:03:10.999 INFO:tasks.workunit.client.1.smithi152.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-04T16:03:10.999 INFO:tasks.workunit.client.1.smithi152.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-04T16:03:12.541 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:12 smithi152 ceph-mon[36876]: pgmap v247: 65 pgs: 65 active+clean; 939 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 33 KiB/s wr, 4 op/s 2024-03-04T16:03:12.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:12 smithi116 ceph-mon[29120]: pgmap v247: 65 pgs: 65 active+clean; 939 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 33 KiB/s wr, 4 op/s 2024-03-04T16:03:14.488 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:14 smithi152 ceph-mon[36876]: pgmap v248: 65 pgs: 65 active+clean; 11 MiB data, 199 MiB used, 536 GiB / 536 GiB avail; 877 KiB/s wr, 44 op/s 2024-03-04T16:03:14.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:14 smithi116 ceph-mon[29120]: pgmap v248: 65 pgs: 65 active+clean; 11 MiB data, 199 MiB used, 536 GiB / 536 GiB avail; 877 KiB/s wr, 44 op/s 2024-03-04T16:03:16.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:16 smithi116 ceph-mon[29120]: pgmap v249: 65 pgs: 65 active+clean; 16 MiB data, 212 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s wr, 66 op/s 2024-03-04T16:03:16.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:16 smithi152 ceph-mon[36876]: pgmap v249: 65 pgs: 65 active+clean; 16 MiB data, 212 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s wr, 66 op/s 2024-03-04T16:03:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:18 smithi116 ceph-mon[29120]: pgmap v250: 65 pgs: 65 active+clean; 18 MiB data, 238 MiB used, 536 GiB / 536 GiB avail; 1.5 MiB/s wr, 114 op/s 2024-03-04T16:03:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:18 smithi116 ceph-mon[29120]: Standby manager daemon smithi152.aawieo restarted 2024-03-04T16:03:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:18 smithi116 ceph-mon[29120]: Standby manager daemon smithi152.aawieo started 2024-03-04T16:03:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:18 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/crt"}]: dispatch 2024-03-04T16:03:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:18 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:03:18.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:18 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/key"}]: dispatch 2024-03-04T16:03:18.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:18 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:03:18.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:18 smithi152 ceph-mon[36876]: pgmap v250: 65 pgs: 65 active+clean; 18 MiB data, 238 MiB used, 536 GiB / 536 GiB avail; 1.5 MiB/s wr, 114 op/s 2024-03-04T16:03:18.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:18 smithi152 ceph-mon[36876]: Standby manager daemon smithi152.aawieo restarted 2024-03-04T16:03:18.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:18 smithi152 ceph-mon[36876]: Standby manager daemon smithi152.aawieo started 2024-03-04T16:03:18.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:18 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/crt"}]: dispatch 2024-03-04T16:03:18.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:18 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:03:18.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:18 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/key"}]: dispatch 2024-03-04T16:03:18.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:18 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:03:19.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:19 smithi152 ceph-mon[36876]: mgrmap e20: smithi116.jwdhop(active, since 7m), standbys: smithi152.aawieo 2024-03-04T16:03:19.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:19 smithi116 ceph-mon[29120]: mgrmap e20: smithi116.jwdhop(active, since 7m), standbys: smithi152.aawieo 2024-03-04T16:03:20.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:20 smithi152 ceph-mon[36876]: pgmap v251: 65 pgs: 65 active+clean; 21 MiB data, 258 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s wr, 165 op/s 2024-03-04T16:03:20.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:20 smithi116 ceph-mon[29120]: pgmap v251: 65 pgs: 65 active+clean; 21 MiB data, 258 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s wr, 165 op/s 2024-03-04T16:03:22.627 INFO:tasks.workunit.client.1.smithi152.stderr:+ tar xzf ltp-full.tgz 2024-03-04T16:03:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:22 smithi116 ceph-mon[29120]: pgmap v252: 65 pgs: 65 active+clean; 33 MiB data, 293 MiB used, 536 GiB / 536 GiB avail; 2.9 MiB/s wr, 221 op/s 2024-03-04T16:03:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:22 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:22 smithi152 ceph-mon[36876]: pgmap v252: 65 pgs: 65 active+clean; 33 MiB data, 293 MiB used, 536 GiB / 536 GiB avail; 2.9 MiB/s wr, 221 op/s 2024-03-04T16:03:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:22 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:24.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:24 smithi116 ceph-mon[29120]: pgmap v253: 65 pgs: 65 active+clean; 44 MiB data, 328 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s wr, 283 op/s 2024-03-04T16:03:24.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:24 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:24.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:24 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:24.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:24 smithi152 ceph-mon[36876]: pgmap v253: 65 pgs: 65 active+clean; 44 MiB data, 328 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s wr, 283 op/s 2024-03-04T16:03:24.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:24 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:24.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:24 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:25.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:25 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:25.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:25 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:25.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:25 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:25.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:25 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:26.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:26 smithi152 ceph-mon[36876]: pgmap v254: 65 pgs: 65 active+clean; 50 MiB data, 348 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s wr, 269 op/s 2024-03-04T16:03:26.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:26 smithi116 ceph-mon[29120]: pgmap v254: 65 pgs: 65 active+clean; 50 MiB data, 348 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s wr, 269 op/s 2024-03-04T16:03:28.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:28 smithi152 ceph-mon[36876]: pgmap v255: 65 pgs: 65 active+clean; 57 MiB data, 398 MiB used, 536 GiB / 536 GiB avail; 3.6 MiB/s wr, 310 op/s 2024-03-04T16:03:28.920 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:28 smithi116 ceph-mon[29120]: pgmap v255: 65 pgs: 65 active+clean; 57 MiB data, 398 MiB used, 536 GiB / 536 GiB avail; 3.6 MiB/s wr, 310 op/s 2024-03-04T16:03:30.829 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:30 smithi152 ceph-mon[36876]: pgmap v256: 65 pgs: 65 active+clean; 63 MiB data, 434 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s wr, 347 op/s 2024-03-04T16:03:30.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:30 smithi116 ceph-mon[29120]: pgmap v256: 65 pgs: 65 active+clean; 63 MiB data, 434 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s wr, 347 op/s 2024-03-04T16:03:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:32 smithi152 ceph-mon[36876]: pgmap v257: 65 pgs: 65 active+clean; 69 MiB data, 470 MiB used, 536 GiB / 536 GiB avail; 4.3 MiB/s wr, 377 op/s 2024-03-04T16:03:32.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:32 smithi116 ceph-mon[29120]: pgmap v257: 65 pgs: 65 active+clean; 69 MiB data, 470 MiB used, 536 GiB / 536 GiB avail; 4.3 MiB/s wr, 377 op/s 2024-03-04T16:03:34.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:34.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:34.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr fail", "who": "smithi116.jwdhop"}]: dispatch 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: osdmap e48: 6 total, 6 up, 6 in 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "mgr fail", "who": "smithi116.jwdhop"}]': finished 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: mgrmap e21: smithi152.aawieo(active, starting, since 0.00617855s) 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:03:34.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mgr metadata", "who": "smithi152.aawieo", "id": "smithi152.aawieo"}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: Manager daemon smithi152.aawieo is now available 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi152.aawieo/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:34.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi152.aawieo/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:34.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi152.aawieo/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:34.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:34 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi152.aawieo/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr fail", "who": "smithi116.jwdhop"}]: dispatch 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: osdmap e48: 6 total, 6 up, 6 in 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.14217 172.21.15.116:0/3661235009' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "mgr fail", "who": "smithi116.jwdhop"}]': finished 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: mgrmap e21: smithi152.aawieo(active, starting, since 0.00617855s) 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:03:34.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mgr metadata", "who": "smithi152.aawieo", "id": "smithi152.aawieo"}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: Manager daemon smithi152.aawieo is now available 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:34.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi152.aawieo/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:34.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi152.aawieo/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:34.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi152.aawieo/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:34.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:34 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi152.aawieo/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:35.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.206+0000 7f6e4ce56700 1 -- 172.21.15.116:0/2336937866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e480ffd90 msgr2=0x7f6e48100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:35.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.206+0000 7f6e4ce56700 1 --2- 172.21.15.116:0/2336937866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e480ffd90 0x7f6e48100210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009910 tx=0x7f6e30009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:35.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.206+0000 7f6e4ce56700 1 -- 172.21.15.116:0/2336937866 shutdown_connections 2024-03-04T16:03:35.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.206+0000 7f6e4ce56700 1 --2- 172.21.15.116:0/2336937866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e480ffd90 0x7f6e48100210 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:35.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.206+0000 7f6e4ce56700 1 --2- 172.21.15.116:0/2336937866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e480ff430 0x7f6e480ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:35.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.206+0000 7f6e4ce56700 1 -- 172.21.15.116:0/2336937866 >> 172.21.15.116:0/2336937866 conn(0x7f6e480fb050 msgr2=0x7f6e480fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:35.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.206+0000 7f6e4ce56700 1 -- 172.21.15.116:0/2336937866 shutdown_connections 2024-03-04T16:03:35.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.206+0000 7f6e4ce56700 1 -- 172.21.15.116:0/2336937866 wait complete. 2024-03-04T16:03:35.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e4ce56700 1 Processor -- start 2024-03-04T16:03:35.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e4ce56700 1 -- start start 2024-03-04T16:03:35.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e4ce56700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e480ff430 0x7f6e48198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e4ce56700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e480ffd90 0x7f6e48198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e4ce56700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e48199080 con 0x7f6e480ffd90 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e4ce56700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e481991c0 con 0x7f6e480ff430 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e45d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e480ffd90 0x7f6e48198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e45d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e480ffd90 0x7f6e48198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52448/0 (socket says 172.21.15.116:52448) 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e45d9b700 1 -- 172.21.15.116:0/3457940192 learned_addr learned my addr 172.21.15.116:0/3457940192 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e4659c700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e480ff430 0x7f6e48198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e45d9b700 1 -- 172.21.15.116:0/3457940192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e480ff430 msgr2=0x7f6e48198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e45d9b700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e480ff430 0x7f6e48198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.207+0000 7f6e45d9b700 1 -- 172.21.15.116:0/3457940192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e300095f0 con 0x7f6e480ffd90 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.208+0000 7f6e45d9b700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e480ffd90 0x7f6e48198a60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6e30000c00 tx=0x7f6e3000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.208+0000 7f6e3f7fe700 1 -- 172.21.15.116:0/3457940192 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e3001d070 con 0x7f6e480ffd90 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.208+0000 7f6e4ce56700 1 -- 172.21.15.116:0/3457940192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4819dc10 con 0x7f6e480ffd90 2024-03-04T16:03:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.208+0000 7f6e3f7fe700 1 -- 172.21.15.116:0/3457940192 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6e30005dc0 con 0x7f6e480ffd90 2024-03-04T16:03:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.208+0000 7f6e3f7fe700 1 -- 172.21.15.116:0/3457940192 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e300226a0 con 0x7f6e480ffd90 2024-03-04T16:03:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.208+0000 7f6e4ce56700 1 -- 172.21.15.116:0/3457940192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4819e100 con 0x7f6e480ffd90 2024-03-04T16:03:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.209+0000 7f6e4ce56700 1 -- 172.21.15.116:0/3457940192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4810b900 con 0x7f6e480ffd90 2024-03-04T16:03:35.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.209+0000 7f6e3f7fe700 1 -- 172.21.15.116:0/3457940192 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) v1 ==== 45910+0+0 (secure 0 0 0) 0x7f6e30022800 con 0x7f6e480ffd90 2024-03-04T16:03:35.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.209+0000 7f6e3f7fe700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6e34038d00 0x7f6e3403b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:35.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.209+0000 7f6e3f7fe700 1 -- 172.21.15.116:0/3457940192 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f6e3004e030 con 0x7f6e480ffd90 2024-03-04T16:03:35.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.210+0000 7f6e4659c700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6e34038d00 0x7f6e3403b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:35.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.210+0000 7f6e4659c700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6e34038d00 0x7f6e3403b1c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6e40009c60 tx=0x7f6e40009260 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.214+0000 7f6e3f7fe700 1 -- 172.21.15.116:0/3457940192 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f6e30020de0 con 0x7f6e480ffd90 2024-03-04T16:03:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:35 smithi116 ceph-mon[29120]: Deploying cephadm binary to smithi116 2024-03-04T16:03:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:35 smithi116 ceph-mon[29120]: Deploying cephadm binary to smithi152 2024-03-04T16:03:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:35 smithi116 ceph-mon[29120]: [04/Mar/2024:16:03:34] ENGINE Bus STARTING 2024-03-04T16:03:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:35 smithi116 ceph-mon[29120]: mgrmap e22: smithi152.aawieo(active, since 1.00811s) 2024-03-04T16:03:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:35 smithi116 ceph-mon[29120]: pgmap v3: 65 pgs: 65 active+clean; 87 MiB data, 516 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:35 smithi116 ceph-mon[29120]: [04/Mar/2024:16:03:34] ENGINE Serving on https://172.21.15.152:7150 2024-03-04T16:03:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:35 smithi116 ceph-mon[29120]: [04/Mar/2024:16:03:34] ENGINE Serving on http://172.21.15.152:8765 2024-03-04T16:03:35.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:35 smithi116 ceph-mon[29120]: [04/Mar/2024:16:03:34] ENGINE Bus STARTED 2024-03-04T16:03:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.484+0000 7f6e4ce56700 1 -- 172.21.15.116:0/3457940192 --> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e48109180 con 0x7f6e34038d00 2024-03-04T16:03:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.491+0000 7f6e3f7fe700 1 -- 172.21.15.116:0/3457940192 <== mgr.24387 v2:172.21.15.152:6828/2881662614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f6e48109180 con 0x7f6e34038d00 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 -- 172.21.15.116:0/3457940192 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6e34038d00 msgr2=0x7f6e3403b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6e34038d00 0x7f6e3403b1c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6e40009c60 tx=0x7f6e40009260 comp rx=0 tx=0).stop 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 -- 172.21.15.116:0/3457940192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e480ffd90 msgr2=0x7f6e48198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e480ffd90 0x7f6e48198a60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6e30000c00 tx=0x7f6e3000f690 comp rx=0 tx=0).stop 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 -- 172.21.15.116:0/3457940192 shutdown_connections 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6e34038d00 0x7f6e3403b1c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e480ff430 0x7f6e48198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 --2- 172.21.15.116:0/3457940192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e480ffd90 0x7f6e48198a60 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 -- 172.21.15.116:0/3457940192 >> 172.21.15.116:0/3457940192 conn(0x7f6e480fb050 msgr2=0x7f6e48107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 -- 172.21.15.116:0/3457940192 shutdown_connections 2024-03-04T16:03:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.494+0000 7f6e4ce56700 1 -- 172.21.15.116:0/3457940192 wait complete. 2024-03-04T16:03:35.512 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:03:35.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:35 smithi152 ceph-mon[36876]: Deploying cephadm binary to smithi116 2024-03-04T16:03:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:35 smithi152 ceph-mon[36876]: Deploying cephadm binary to smithi152 2024-03-04T16:03:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:35 smithi152 ceph-mon[36876]: [04/Mar/2024:16:03:34] ENGINE Bus STARTING 2024-03-04T16:03:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:35 smithi152 ceph-mon[36876]: mgrmap e22: smithi152.aawieo(active, since 1.00811s) 2024-03-04T16:03:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:35 smithi152 ceph-mon[36876]: pgmap v3: 65 pgs: 65 active+clean; 87 MiB data, 516 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:35 smithi152 ceph-mon[36876]: [04/Mar/2024:16:03:34] ENGINE Serving on https://172.21.15.152:7150 2024-03-04T16:03:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:35 smithi152 ceph-mon[36876]: [04/Mar/2024:16:03:34] ENGINE Serving on http://172.21.15.152:8765 2024-03-04T16:03:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:35 smithi152 ceph-mon[36876]: [04/Mar/2024:16:03:34] ENGINE Bus STARTED 2024-03-04T16:03:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.636+0000 7f8d056bf700 1 -- 172.21.15.116:0/3889844735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d000ffab0 msgr2=0x7f8d000ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.636+0000 7f8d056bf700 1 --2- 172.21.15.116:0/3889844735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d000ffab0 0x7f8d000ffed0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0009910 tx=0x7f8cf0009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.636+0000 7f8d056bf700 1 -- 172.21.15.116:0/3889844735 shutdown_connections 2024-03-04T16:03:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.636+0000 7f8d056bf700 1 --2- 172.21.15.116:0/3889844735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d00100cb0 0x7f8d00101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:35.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.636+0000 7f8d056bf700 1 --2- 172.21.15.116:0/3889844735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d000ffab0 0x7f8d000ffed0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:35.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.636+0000 7f8d056bf700 1 -- 172.21.15.116:0/3889844735 >> 172.21.15.116:0/3889844735 conn(0x7f8d000fb030 msgr2=0x7f8d000fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:35.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.636+0000 7f8d056bf700 1 -- 172.21.15.116:0/3889844735 shutdown_connections 2024-03-04T16:03:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.637+0000 7f8d056bf700 1 -- 172.21.15.116:0/3889844735 wait complete. 2024-03-04T16:03:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.637+0000 7f8d056bf700 1 Processor -- start 2024-03-04T16:03:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8d056bf700 1 -- start start 2024-03-04T16:03:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8d056bf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d000ffab0 0x7f8d00196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8d056bf700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d00100cb0 0x7f8d001968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8d056bf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d00196ee0 con 0x7f8d000ffab0 2024-03-04T16:03:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8d056bf700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d00197020 con 0x7f8d00100cb0 2024-03-04T16:03:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8cfe7fc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d00100cb0 0x7f8d001968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8cfe7fc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d00100cb0 0x7f8d001968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:35992/0 (socket says 172.21.15.116:35992) 2024-03-04T16:03:35.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8cfe7fc700 1 -- 172.21.15.116:0/501407510 learned_addr learned my addr 172.21.15.116:0/501407510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:35.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8cfeffd700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d000ffab0 0x7f8d00196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8cfe7fc700 1 -- 172.21.15.116:0/501407510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d000ffab0 msgr2=0x7f8d00196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8cfe7fc700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d000ffab0 0x7f8d00196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.638+0000 7f8cfe7fc700 1 -- 172.21.15.116:0/501407510 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf00095f0 con 0x7f8d00100cb0 2024-03-04T16:03:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.639+0000 7f8cfe7fc700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d00100cb0 0x7f8d001968c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400d900 tx=0x7f8cf400dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.639+0000 7f8ce7fff700 1 -- 172.21.15.116:0/501407510 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cf4004a00 con 0x7f8d00100cb0 2024-03-04T16:03:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.639+0000 7f8d056bf700 1 -- 172.21.15.116:0/501407510 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0019bad0 con 0x7f8d00100cb0 2024-03-04T16:03:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.639+0000 7f8ce7fff700 1 -- 172.21.15.116:0/501407510 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8cf40054d0 con 0x7f8d00100cb0 2024-03-04T16:03:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.639+0000 7f8ce7fff700 1 -- 172.21.15.116:0/501407510 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cf4009d00 con 0x7f8d00100cb0 2024-03-04T16:03:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.639+0000 7f8d056bf700 1 -- 172.21.15.116:0/501407510 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0019c020 con 0x7f8d00100cb0 2024-03-04T16:03:35.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.641+0000 7f8d056bf700 1 -- 172.21.15.116:0/501407510 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d00108630 con 0x7f8d00100cb0 2024-03-04T16:03:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.641+0000 7f8ce7fff700 1 -- 172.21.15.116:0/501407510 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 22) v1 ==== 45910+0+0 (secure 0 0 0) 0x7f8cf4005020 con 0x7f8d00100cb0 2024-03-04T16:03:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.641+0000 7f8ce7fff700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8ce8038d50 0x7f8ce803b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.642+0000 7f8ce7fff700 1 -- 172.21.15.116:0/501407510 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f8cf404cb30 con 0x7f8d00100cb0 2024-03-04T16:03:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.642+0000 7f8cfeffd700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8ce8038d50 0x7f8ce803b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.643+0000 7f8cfeffd700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8ce8038d50 0x7f8ce803b210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0006010 tx=0x7f8cf0005ec0 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:35.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:35.649+0000 7f8ce7fff700 1 -- 172.21.15.116:0/501407510 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f8cf400fab0 con 0x7f8d00100cb0 2024-03-04T16:03:36.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.003+0000 7f8d056bf700 1 -- 172.21.15.116:0/501407510 --> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d00068b20 con 0x7f8ce8038d50 2024-03-04T16:03:36.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.010+0000 7f8ce7fff700 1 -- 172.21.15.116:0/501407510 <== mgr.24387 v2:172.21.15.152:6828/2881662614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f8d00068b20 con 0x7f8ce8038d50 2024-03-04T16:03:36.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 -- 172.21.15.116:0/501407510 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8ce8038d50 msgr2=0x7f8ce803b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:36.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8ce8038d50 0x7f8ce803b210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0006010 tx=0x7f8cf0005ec0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 -- 172.21.15.116:0/501407510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d00100cb0 msgr2=0x7f8d001968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:36.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d00100cb0 0x7f8d001968c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400d900 tx=0x7f8cf400dc10 comp rx=0 tx=0).stop 2024-03-04T16:03:36.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 -- 172.21.15.116:0/501407510 shutdown_connections 2024-03-04T16:03:36.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8ce8038d50 0x7f8ce803b210 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d000ffab0 0x7f8d00196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 --2- 172.21.15.116:0/501407510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d00100cb0 0x7f8d001968c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 -- 172.21.15.116:0/501407510 >> 172.21.15.116:0/501407510 conn(0x7f8d000fb030 msgr2=0x7f8d00068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:36.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 -- 172.21.15.116:0/501407510 shutdown_connections 2024-03-04T16:03:36.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.013+0000 7f8d056bf700 1 -- 172.21.15.116:0/501407510 wait complete. 2024-03-04T16:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.140+0000 7fbdb8915700 1 -- 172.21.15.116:0/2909076587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdb4100cb0 msgr2=0x7fbdb4101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.140+0000 7fbdb8915700 1 --2- 172.21.15.116:0/2909076587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdb4100cb0 0x7fbdb4101110 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbda8009910 tx=0x7fbda8009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 -- 172.21.15.116:0/2909076587 shutdown_connections 2024-03-04T16:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 --2- 172.21.15.116:0/2909076587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdb4100cb0 0x7fbdb4101110 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 --2- 172.21.15.116:0/2909076587 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdb40ffab0 0x7fbdb40ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 -- 172.21.15.116:0/2909076587 >> 172.21.15.116:0/2909076587 conn(0x7fbdb40fb050 msgr2=0x7fbdb40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 -- 172.21.15.116:0/2909076587 shutdown_connections 2024-03-04T16:03:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 -- 172.21.15.116:0/2909076587 wait complete. 2024-03-04T16:03:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 Processor -- start 2024-03-04T16:03:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 -- start start 2024-03-04T16:03:36.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdb40ffab0 0x7fbdb4196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:36.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdb4100cb0 0x7fbdb4196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4196ea0 con 0x7fbdb4100cb0 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.141+0000 7fbdb8915700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4196fe0 con 0x7fbdb40ffab0 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbdb1d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdb4100cb0 0x7fbdb4196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbdb1d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdb4100cb0 0x7fbdb4196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52482/0 (socket says 172.21.15.116:52482) 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbdb1d9b700 1 -- 172.21.15.116:0/2315224116 learned_addr learned my addr 172.21.15.116:0/2315224116 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbdb259c700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdb40ffab0 0x7fbdb4196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbdb1d9b700 1 -- 172.21.15.116:0/2315224116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdb40ffab0 msgr2=0x7fbdb4196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbdb1d9b700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdb40ffab0 0x7fbdb4196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbdb1d9b700 1 -- 172.21.15.116:0/2315224116 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda80095f0 con 0x7fbdb4100cb0 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbdb1d9b700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdb4100cb0 0x7fbdb4196880 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbda8006010 tx=0x7fbda800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbda37fe700 1 -- 172.21.15.116:0/2315224116 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda801d070 con 0x7fbdb4100cb0 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbdb8915700 1 -- 172.21.15.116:0/2315224116 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb419ba30 con 0x7fbdb4100cb0 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbda37fe700 1 -- 172.21.15.116:0/2315224116 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbda800f460 con 0x7fbdb4100cb0 2024-03-04T16:03:36.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbda37fe700 1 -- 172.21.15.116:0/2315224116 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda8021640 con 0x7fbdb4100cb0 2024-03-04T16:03:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.142+0000 7fbdb8915700 1 -- 172.21.15.116:0/2315224116 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb419bf20 con 0x7fbdb4100cb0 2024-03-04T16:03:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.143+0000 7fbda37fe700 1 -- 172.21.15.116:0/2315224116 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) v1 ==== 45910+0+0 (secure 0 0 0) 0x7fbda802b440 con 0x7fbdb4100cb0 2024-03-04T16:03:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.143+0000 7fbdb8915700 1 -- 172.21.15.116:0/2315224116 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb4108610 con 0x7fbdb4100cb0 2024-03-04T16:03:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.143+0000 7fbda37fe700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fbd9c038d00 0x7fbd9c03b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.144+0000 7fbda37fe700 1 -- 172.21.15.116:0/2315224116 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fbda8003a60 con 0x7fbdb4100cb0 2024-03-04T16:03:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.144+0000 7fbdb259c700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fbd9c038d00 0x7fbd9c03b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.144+0000 7fbdb259c700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fbd9c038d00 0x7fbd9c03b1c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4100b10 tx=0x7fbda400b410 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:36.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.149+0000 7fbda37fe700 1 -- 172.21.15.116:0/2315224116 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fbda80298f0 con 0x7fbdb4100cb0 2024-03-04T16:03:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.267+0000 7fbda37fe700 1 -- 172.21.15.116:0/2315224116 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 23) v1 ==== 46068+0+0 (secure 0 0 0) 0x7fbda800fab0 con 0x7fbdb4100cb0 2024-03-04T16:03:36.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.392+0000 7fbdb8915700 1 -- 172.21.15.116:0/2315224116 --> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdb40693b0 con 0x7fbd9c038d00 2024-03-04T16:03:36.396 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:36 smithi116 ceph-mon[29120]: pgmap v4: 65 pgs: 65 active+clean; 87 MiB data, 516 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:36.396 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:36 smithi116 ceph-mon[29120]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:36.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.407+0000 7fbda37fe700 1 -- 172.21.15.116:0/2315224116 <== mgr.24387 v2:172.21.15.152:6828/2881662614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fbdb40693b0 con 0x7fbd9c038d00 2024-03-04T16:03:36.408 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:03:36.408 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (6m) 2m ago 8m 25.1M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:03:36.408 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (8m) 2m ago 8m 7390k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:03:36.408 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (7m) 13s ago 7m 7524k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (8m) 2m ago 8m 6665k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (7m) 13s ago 7m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (6m) 2m ago 7m 87.2M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (2m) 2m ago 2m 15.3M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (2m) 2m ago 2m 11.3M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (2m) 13s ago 2m 14.1M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (2m) 13s ago 2m 28.8M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:9283,8765,8443 running (9m) 2m ago 9m 489M - 18.2.1 5be31c24972a 083879ca4bf4 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (27s) 13s ago 7m 61.7M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6756dd312323 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (9m) 2m ago 9m 45.4M 2048M 18.2.1 5be31c24972a 60e0bf118c4d 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (7m) 13s ago 7m 37.8M 2048M 18.2.1 5be31c24972a 52450d235516 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (8m) 2m ago 8m 15.4M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (7m) 13s ago 7m 16.4M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (6m) 2m ago 6m 50.5M 3946M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:03:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (5m) 2m ago 5m 46.8M 3946M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:03:36.410 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (5m) 2m ago 5m 46.4M 3946M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:03:36.410 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (4m) 13s ago 4m 54.3M 4970M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:03:36.410 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (4m) 13s ago 4m 46.3M 4970M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:03:36.410 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (4m) 13s ago 4m 58.8M 4970M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:03:36.410 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (6m) 2m ago 7m 55.8M - 2.43.0 a07b618ecd1d 31240d908f24 2024-03-04T16:03:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 -- 172.21.15.116:0/2315224116 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fbd9c038d00 msgr2=0x7fbd9c03b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fbd9c038d00 0x7fbd9c03b1c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4100b10 tx=0x7fbda400b410 comp rx=0 tx=0).stop 2024-03-04T16:03:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 -- 172.21.15.116:0/2315224116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdb4100cb0 msgr2=0x7fbdb4196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdb4100cb0 0x7fbdb4196880 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbda8006010 tx=0x7fbda800bd70 comp rx=0 tx=0).stop 2024-03-04T16:03:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 -- 172.21.15.116:0/2315224116 shutdown_connections 2024-03-04T16:03:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fbd9c038d00 0x7fbd9c03b1c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdb40ffab0 0x7fbdb4196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 --2- 172.21.15.116:0/2315224116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdb4100cb0 0x7fbdb4196880 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 -- 172.21.15.116:0/2315224116 >> 172.21.15.116:0/2315224116 conn(0x7fbdb40fb050 msgr2=0x7fbdb4068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:36.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 -- 172.21.15.116:0/2315224116 shutdown_connections 2024-03-04T16:03:36.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.411+0000 7fbdb8915700 1 -- 172.21.15.116:0/2315224116 wait complete. 2024-03-04T16:03:36.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.529+0000 7f7eb34b0700 1 -- 172.21.15.116:0/4081566414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eac100c00 msgr2=0x7f7eac101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:36.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.529+0000 7f7eb34b0700 1 --2- 172.21.15.116:0/4081566414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eac100c00 0x7f7eac101020 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8009910 tx=0x7f7ea8009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.529+0000 7f7eb34b0700 1 -- 172.21.15.116:0/4081566414 shutdown_connections 2024-03-04T16:03:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.529+0000 7f7eb34b0700 1 --2- 172.21.15.116:0/4081566414 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7eac101e00 0x7f7eac102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.529+0000 7f7eb34b0700 1 --2- 172.21.15.116:0/4081566414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eac100c00 0x7f7eac101020 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.529+0000 7f7eb34b0700 1 -- 172.21.15.116:0/4081566414 >> 172.21.15.116:0/4081566414 conn(0x7f7eac0fc180 msgr2=0x7f7eac0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.529+0000 7f7eb34b0700 1 -- 172.21.15.116:0/4081566414 shutdown_connections 2024-03-04T16:03:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.529+0000 7f7eb34b0700 1 -- 172.21.15.116:0/4081566414 wait complete. 2024-03-04T16:03:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.530+0000 7f7eb34b0700 1 Processor -- start 2024-03-04T16:03:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.530+0000 7f7eb34b0700 1 -- start start 2024-03-04T16:03:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.530+0000 7f7eb34b0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7eac100c00 0x7f7eac196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.530+0000 7f7eb34b0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eac101e00 0x7f7eac1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.530+0000 7f7eb34b0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac196ef0 con 0x7f7eac101e00 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.530+0000 7f7eb34b0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac197030 con 0x7f7eac100c00 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.530+0000 7f7eb0a4b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eac101e00 0x7f7eac1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.530+0000 7f7eb0a4b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eac101e00 0x7f7eac1968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52504/0 (socket says 172.21.15.116:52504) 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.530+0000 7f7eb0a4b700 1 -- 172.21.15.116:0/2368185821 learned_addr learned my addr 172.21.15.116:0/2368185821 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.530+0000 7f7eb124c700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7eac100c00 0x7f7eac196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.531+0000 7f7eb0a4b700 1 -- 172.21.15.116:0/2368185821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7eac100c00 msgr2=0x7f7eac196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.531+0000 7f7eb0a4b700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7eac100c00 0x7f7eac196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.531+0000 7f7eb0a4b700 1 -- 172.21.15.116:0/2368185821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea80095f0 con 0x7f7eac101e00 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.531+0000 7f7eb0a4b700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eac101e00 0x7f7eac1968d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00d900 tx=0x7f7e9c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.531+0000 7f7ea27fc700 1 -- 172.21.15.116:0/2368185821 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e9c0041f0 con 0x7f7eac101e00 2024-03-04T16:03:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.531+0000 7f7eb34b0700 1 -- 172.21.15.116:0/2368185821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac19bae0 con 0x7f7eac101e00 2024-03-04T16:03:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.531+0000 7f7ea27fc700 1 -- 172.21.15.116:0/2368185821 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7e9c004ce0 con 0x7f7eac101e00 2024-03-04T16:03:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.531+0000 7f7ea27fc700 1 -- 172.21.15.116:0/2368185821 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e9c00b750 con 0x7f7eac101e00 2024-03-04T16:03:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.531+0000 7f7eb34b0700 1 -- 172.21.15.116:0/2368185821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac19c030 con 0x7f7eac101e00 2024-03-04T16:03:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.532+0000 7f7eb34b0700 1 -- 172.21.15.116:0/2368185821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eac109760 con 0x7f7eac101e00 2024-03-04T16:03:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.532+0000 7f7ea27fc700 1 -- 172.21.15.116:0/2368185821 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) v1 ==== 46068+0+0 (secure 0 0 0) 0x7f7e9c00b9c0 con 0x7f7eac101e00 2024-03-04T16:03:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.532+0000 7f7ea27fc700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f7e98038eb0 0x7f7e9803b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.532+0000 7f7ea27fc700 1 -- 172.21.15.116:0/2368185821 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f7e9c04d0b0 con 0x7f7eac101e00 2024-03-04T16:03:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.532+0000 7f7eb124c700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f7e98038eb0 0x7f7e9803b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.533+0000 7f7eb124c700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f7e98038eb0 0x7f7e9803b370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8006010 tx=0x7f7ea800b560 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.538+0000 7f7ea27fc700 1 -- 172.21.15.116:0/2368185821 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f7e9c022760 con 0x7f7eac101e00 2024-03-04T16:03:36.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:36 smithi152 ceph-mon[36876]: pgmap v4: 65 pgs: 65 active+clean; 87 MiB data, 516 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:36 smithi152 ceph-mon[36876]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.903+0000 7f7eb34b0700 1 -- 172.21.15.116:0/2368185821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7eac19c640 con 0x7f7eac101e00 2024-03-04T16:03:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.904+0000 7f7ea27fc700 1 -- 172.21.15.116:0/2368185821 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f7e9c00f3c0 con 0x7f7eac101e00 2024-03-04T16:03:36.905 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:03:36.905 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:03:36.905 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:03:36.906 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:03:36.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 -- 172.21.15.116:0/2368185821 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f7e98038eb0 msgr2=0x7f7e9803b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:36.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f7e98038eb0 0x7f7e9803b370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8006010 tx=0x7f7ea800b560 comp rx=0 tx=0).stop 2024-03-04T16:03:36.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 -- 172.21.15.116:0/2368185821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eac101e00 msgr2=0x7f7eac1968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:36.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eac101e00 0x7f7eac1968d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00d900 tx=0x7f7e9c00dc10 comp rx=0 tx=0).stop 2024-03-04T16:03:36.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 -- 172.21.15.116:0/2368185821 shutdown_connections 2024-03-04T16:03:36.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f7e98038eb0 0x7f7e9803b370 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7eac100c00 0x7f7eac196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 --2- 172.21.15.116:0/2368185821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eac101e00 0x7f7eac1968d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:36.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 -- 172.21.15.116:0/2368185821 >> 172.21.15.116:0/2368185821 conn(0x7f7eac0fc180 msgr2=0x7f7eac105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:36.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 -- 172.21.15.116:0/2368185821 shutdown_connections 2024-03-04T16:03:36.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:36.907+0000 7f7eb34b0700 1 -- 172.21.15.116:0/2368185821 wait complete. 2024-03-04T16:03:37.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.028+0000 7f33ad5be700 1 -- 172.21.15.116:0/3157268999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33a8101650 msgr2=0x7f33a8103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.028+0000 7f33ad5be700 1 --2- 172.21.15.116:0/3157268999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33a8101650 0x7f33a8103a40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3398009910 tx=0x7f3398009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.028+0000 7f33ad5be700 1 -- 172.21.15.116:0/3157268999 shutdown_connections 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.028+0000 7f33ad5be700 1 --2- 172.21.15.116:0/3157268999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f33a8103f80 0x7f33a8106370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.028+0000 7f33ad5be700 1 --2- 172.21.15.116:0/3157268999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33a8101650 0x7f33a8103a40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.028+0000 7f33ad5be700 1 -- 172.21.15.116:0/3157268999 >> 172.21.15.116:0/3157268999 conn(0x7f33a80fb060 msgr2=0x7f33a80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.028+0000 7f33ad5be700 1 -- 172.21.15.116:0/3157268999 shutdown_connections 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.028+0000 7f33ad5be700 1 -- 172.21.15.116:0/3157268999 wait complete. 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.029+0000 7f33ad5be700 1 Processor -- start 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.029+0000 7f33ad5be700 1 -- start start 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.029+0000 7f33ad5be700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33a8101650 0x7f33a8196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.029+0000 7f33ad5be700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f33a8103f80 0x7f33a81968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:37.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.029+0000 7f33ad5be700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a8196ee0 con 0x7f33a8101650 2024-03-04T16:03:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.029+0000 7f33ad5be700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a8197020 con 0x7f33a8103f80 2024-03-04T16:03:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.029+0000 7f33a6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33a8101650 0x7f33a8196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.029+0000 7f33a6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33a8101650 0x7f33a8196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52514/0 (socket says 172.21.15.116:52514) 2024-03-04T16:03:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.029+0000 7f33a6ffd700 1 -- 172.21.15.116:0/4249803727 learned_addr learned my addr 172.21.15.116:0/4249803727 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.030+0000 7f33a6ffd700 1 -- 172.21.15.116:0/4249803727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f33a8103f80 msgr2=0x7f33a81968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.030+0000 7f33a6ffd700 1 --2- 172.21.15.116:0/4249803727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f33a8103f80 0x7f33a81968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.030+0000 7f33a6ffd700 1 -- 172.21.15.116:0/4249803727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33980095f0 con 0x7f33a8101650 2024-03-04T16:03:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.030+0000 7f33a6ffd700 1 --2- 172.21.15.116:0/4249803727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33a8101650 0x7f33a8196380 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f339800bd70 tx=0x7f339800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.030+0000 7f338ffff700 1 -- 172.21.15.116:0/4249803727 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f339801d070 con 0x7f33a8101650 2024-03-04T16:03:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.030+0000 7f338ffff700 1 -- 172.21.15.116:0/4249803727 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3398022470 con 0x7f33a8101650 2024-03-04T16:03:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.030+0000 7f33ad5be700 1 -- 172.21.15.116:0/4249803727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a819ba20 con 0x7f33a8101650 2024-03-04T16:03:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.030+0000 7f33ad5be700 1 -- 172.21.15.116:0/4249803727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a819bf10 con 0x7f33a8101650 2024-03-04T16:03:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.030+0000 7f338ffff700 1 -- 172.21.15.116:0/4249803727 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f339800f670 con 0x7f33a8101650 2024-03-04T16:03:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.031+0000 7f33ad5be700 1 -- 172.21.15.116:0/4249803727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394005320 con 0x7f33a8101650 2024-03-04T16:03:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.031+0000 7f338ffff700 1 -- 172.21.15.116:0/4249803727 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) v1 ==== 46068+0+0 (secure 0 0 0) 0x7f339800f8e0 con 0x7f33a8101650 2024-03-04T16:03:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.031+0000 7f338ffff700 1 --2- 172.21.15.116:0/4249803727 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3390038eb0 0x7f339003b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.031+0000 7f338ffff700 1 -- 172.21.15.116:0/4249803727 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f339804e570 con 0x7f33a8101650 2024-03-04T16:03:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.032+0000 7f33a67fc700 1 --2- 172.21.15.116:0/4249803727 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3390038eb0 0x7f339003b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.032+0000 7f33a67fc700 1 --2- 172.21.15.116:0/4249803727 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3390038eb0 0x7f339003b370 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f339c005950 tx=0x7f339c0058e0 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:37.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.037+0000 7f338ffff700 1 -- 172.21.15.116:0/4249803727 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f3398020c40 con 0x7f33a8101650 2024-03-04T16:03:37.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.310+0000 7f33ad5be700 1 -- 172.21.15.116:0/4249803727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3394005cc0 con 0x7f33a8101650 2024-03-04T16:03:37.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.310+0000 7f338ffff700 1 -- 172.21.15.116:0/4249803727 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1831 (secure 0 0 0) 0x7f3398027030 con 0x7f33a8101650 2024-03-04T16:03:37.312 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:03:37.312 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:03:37.313 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:03:37.313 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:03:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:03:37.313 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:03:37.313 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:03:37.313 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:03:37.313 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:03:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:03:37.315 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:03:37.315 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:03:37.315 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:03:37.315 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:03:37.315 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:03:37.315 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:03:37.315 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:03:37.315 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:03:37.315 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:03:37.316 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:03:37.316 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:03:37.316 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:03:37.316 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:03:37.316 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:03:37.316 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:03:37.316 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:03:37.316 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:03:37.316 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:03:37.316 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:03:37.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 -- 172.21.15.116:0/4249803727 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3390038eb0 msgr2=0x7f339003b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:37.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 --2- 172.21.15.116:0/4249803727 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3390038eb0 0x7f339003b370 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f339c005950 tx=0x7f339c0058e0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 -- 172.21.15.116:0/4249803727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33a8101650 msgr2=0x7f33a8196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 --2- 172.21.15.116:0/4249803727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33a8101650 0x7f33a8196380 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f339800bd70 tx=0x7f339800be50 comp rx=0 tx=0).stop 2024-03-04T16:03:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 -- 172.21.15.116:0/4249803727 shutdown_connections 2024-03-04T16:03:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 --2- 172.21.15.116:0/4249803727 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3390038eb0 0x7f339003b370 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 --2- 172.21.15.116:0/4249803727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33a8101650 0x7f33a8196380 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 --2- 172.21.15.116:0/4249803727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f33a8103f80 0x7f33a81968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 -- 172.21.15.116:0/4249803727 >> 172.21.15.116:0/4249803727 conn(0x7f33a80fb060 msgr2=0x7f33a80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 -- 172.21.15.116:0/4249803727 shutdown_connections 2024-03-04T16:03:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.313+0000 7f33ad5be700 1 -- 172.21.15.116:0/4249803727 wait complete. 2024-03-04T16:03:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:03:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.424+0000 7f9959cb5700 1 -- 172.21.15.116:0/2625598082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99541016a0 msgr2=0x7f9954103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.424+0000 7f9959cb5700 1 --2- 172.21.15.116:0/2625598082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99541016a0 0x7f9954103a90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9948009910 tx=0x7f9948009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.424+0000 7f9959cb5700 1 -- 172.21.15.116:0/2625598082 shutdown_connections 2024-03-04T16:03:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.424+0000 7f9959cb5700 1 --2- 172.21.15.116:0/2625598082 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954103fd0 0x7f99541063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.424+0000 7f9959cb5700 1 --2- 172.21.15.116:0/2625598082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99541016a0 0x7f9954103a90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.424+0000 7f9959cb5700 1 -- 172.21.15.116:0/2625598082 >> 172.21.15.116:0/2625598082 conn(0x7f99540fb010 msgr2=0x7f99540fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.424+0000 7f9959cb5700 1 -- 172.21.15.116:0/2625598082 shutdown_connections 2024-03-04T16:03:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.424+0000 7f9959cb5700 1 -- 172.21.15.116:0/2625598082 wait complete. 2024-03-04T16:03:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.424+0000 7f9959cb5700 1 Processor -- start 2024-03-04T16:03:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f9959cb5700 1 -- start start 2024-03-04T16:03:37.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f9959cb5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99541016a0 0x7f9954101000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f9959cb5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954103fd0 0x7f99540ff650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f9959cb5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99540ffb90 con 0x7f99541016a0 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f9959cb5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99540ffcd0 con 0x7f9954103fd0 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f99537fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99541016a0 0x7f9954101000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f99537fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99541016a0 0x7f9954101000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52536/0 (socket says 172.21.15.116:52536) 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f99537fe700 1 -- 172.21.15.116:0/2369809275 learned_addr learned my addr 172.21.15.116:0/2369809275 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f9952ffd700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954103fd0 0x7f99540ff650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f99537fe700 1 -- 172.21.15.116:0/2369809275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954103fd0 msgr2=0x7f99540ff650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f99537fe700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954103fd0 0x7f99540ff650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f99537fe700 1 -- 172.21.15.116:0/2369809275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480095f0 con 0x7f99541016a0 2024-03-04T16:03:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f99537fe700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99541016a0 0x7f9954101000 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9948000c00 tx=0x7f994800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.425+0000 7f9950ff9700 1 -- 172.21.15.116:0/2369809275 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f994801d070 con 0x7f99541016a0 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.426+0000 7f9950ff9700 1 -- 172.21.15.116:0/2369809275 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9948005dc0 con 0x7f99541016a0 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.426+0000 7f9950ff9700 1 -- 172.21.15.116:0/2369809275 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99480226a0 con 0x7f99541016a0 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.426+0000 7f9959cb5700 1 -- 172.21.15.116:0/2369809275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99540fff50 con 0x7f99541016a0 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.426+0000 7f9959cb5700 1 -- 172.21.15.116:0/2369809275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9954100440 con 0x7f99541016a0 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.426+0000 7f9950ff9700 1 -- 172.21.15.116:0/2369809275 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) v1 ==== 46068+0+0 (secure 0 0 0) 0x7f9948022800 con 0x7f99541016a0 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.427+0000 7f9950ff9700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f993c038eb0 0x7f993c03b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.427+0000 7f9950ff9700 1 -- 172.21.15.116:0/2369809275 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f994804f4a0 con 0x7f99541016a0 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.427+0000 7f9959cb5700 1 -- 172.21.15.116:0/2369809275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9954190520 con 0x7f99541016a0 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.427+0000 7f9952ffd700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f993c038eb0 0x7f993c03b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.427+0000 7f9952ffd700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f993c038eb0 0x7f993c03b370 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9944009c60 tx=0x7f9944009260 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:37.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.432+0000 7f9950ff9700 1 -- 172.21.15.116:0/2369809275 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f9948022ba0 con 0x7f99541016a0 2024-03-04T16:03:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:37 smithi152 ceph-mon[36876]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:37 smithi152 ceph-mon[36876]: mgrmap e23: smithi152.aawieo(active, since 3s) 2024-03-04T16:03:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:37 smithi152 ceph-mon[36876]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:37 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2368185821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:03:37.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.672+0000 7f9959cb5700 1 -- 172.21.15.116:0/2369809275 --> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99540611d0 con 0x7f993c038eb0 2024-03-04T16:03:37.674 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:37 smithi116 ceph-mon[29120]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:37.674 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:37 smithi116 ceph-mon[29120]: mgrmap e23: smithi152.aawieo(active, since 3s) 2024-03-04T16:03:37.674 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:37 smithi116 ceph-mon[29120]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:37.674 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:37 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2368185821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:03:37.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.686+0000 7f9950ff9700 1 -- 172.21.15.116:0/2369809275 <== mgr.24387 v2:172.21.15.152:6828/2881662614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f99540611d0 con 0x7f993c038eb0 2024-03-04T16:03:37.688 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:03:37.688 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:03:37.688 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:03:37.688 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:03:37.688 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2024-03-04T16:03:37.688 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "1/23 daemons upgraded", 2024-03-04T16:03:37.688 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2024-03-04T16:03:37.688 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:03:37.688 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 -- 172.21.15.116:0/2369809275 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f993c038eb0 msgr2=0x7f993c03b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f993c038eb0 0x7f993c03b370 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9944009c60 tx=0x7f9944009260 comp rx=0 tx=0).stop 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 -- 172.21.15.116:0/2369809275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99541016a0 msgr2=0x7f9954101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99541016a0 0x7f9954101000 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9948000c00 tx=0x7f994800f690 comp rx=0 tx=0).stop 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 -- 172.21.15.116:0/2369809275 shutdown_connections 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f993c038eb0 0x7f993c03b370 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99541016a0 0x7f9954101000 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 --2- 172.21.15.116:0/2369809275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954103fd0 0x7f99540ff650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 -- 172.21.15.116:0/2369809275 >> 172.21.15.116:0/2369809275 conn(0x7f99540fb010 msgr2=0x7f99540fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 -- 172.21.15.116:0/2369809275 shutdown_connections 2024-03-04T16:03:37.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.689+0000 7f9959cb5700 1 -- 172.21.15.116:0/2369809275 wait complete. 2024-03-04T16:03:37.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.807+0000 7f2bfa1f8700 1 -- 172.21.15.116:0/299352765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bf40fee80 msgr2=0x7f2bf40ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.807+0000 7f2bfa1f8700 1 --2- 172.21.15.116:0/299352765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bf40fee80 0x7f2bf40ff2a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009910 tx=0x7f2be8009c20 comp rx=0 tx=0).stop 2024-03-04T16:03:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.807+0000 7f2bfa1f8700 1 -- 172.21.15.116:0/299352765 shutdown_connections 2024-03-04T16:03:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.807+0000 7f2bfa1f8700 1 --2- 172.21.15.116:0/299352765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bf4101e60 0x7f2bf41022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.807+0000 7f2bfa1f8700 1 --2- 172.21.15.116:0/299352765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bf40fee80 0x7f2bf40ff2a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.807+0000 7f2bfa1f8700 1 -- 172.21.15.116:0/299352765 >> 172.21.15.116:0/299352765 conn(0x7f2bf40fa7c0 msgr2=0x7f2bf40fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.807+0000 7f2bfa1f8700 1 -- 172.21.15.116:0/299352765 shutdown_connections 2024-03-04T16:03:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.807+0000 7f2bfa1f8700 1 -- 172.21.15.116:0/299352765 wait complete. 2024-03-04T16:03:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.808+0000 7f2bfa1f8700 1 Processor -- start 2024-03-04T16:03:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.808+0000 7f2bfa1f8700 1 -- start start 2024-03-04T16:03:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.808+0000 7f2bfa1f8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bf40fee80 0x7f2bf41017c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.808+0000 7f2bfa1f8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bf4101e60 0x7f2bf40ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.808+0000 7f2bfa1f8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf4100350 con 0x7f2bf4101e60 2024-03-04T16:03:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.808+0000 7f2bfa1f8700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf4100490 con 0x7f2bf40fee80 2024-03-04T16:03:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.809+0000 7f2bf2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bf4101e60 0x7f2bf40ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.809+0000 7f2bf2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bf4101e60 0x7f2bf40ffe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52554/0 (socket says 172.21.15.116:52554) 2024-03-04T16:03:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.809+0000 7f2bf2ffd700 1 -- 172.21.15.116:0/911992245 learned_addr learned my addr 172.21.15.116:0/911992245 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:03:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.809+0000 7f2bf37fe700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bf40fee80 0x7f2bf41017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.809+0000 7f2bf2ffd700 1 -- 172.21.15.116:0/911992245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bf40fee80 msgr2=0x7f2bf41017c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.809+0000 7f2bf2ffd700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bf40fee80 0x7f2bf41017c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.809+0000 7f2bf2ffd700 1 -- 172.21.15.116:0/911992245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80095f0 con 0x7f2bf4101e60 2024-03-04T16:03:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.809+0000 7f2bf2ffd700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bf4101e60 0x7f2bf40ffe10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2be400d900 tx=0x7f2be400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.810+0000 7f2bf0ff9700 1 -- 172.21.15.116:0/911992245 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2be4004a00 con 0x7f2bf4101e60 2024-03-04T16:03:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.810+0000 7f2bf0ff9700 1 -- 172.21.15.116:0/911992245 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2be40054d0 con 0x7f2bf4101e60 2024-03-04T16:03:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.810+0000 7f2bf0ff9700 1 -- 172.21.15.116:0/911992245 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2be4009d00 con 0x7f2bf4101e60 2024-03-04T16:03:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.810+0000 7f2bfa1f8700 1 -- 172.21.15.116:0/911992245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf4100770 con 0x7f2bf4101e60 2024-03-04T16:03:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.810+0000 7f2bfa1f8700 1 -- 172.21.15.116:0/911992245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf4100c00 con 0x7f2bf4101e60 2024-03-04T16:03:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.811+0000 7f2bda7fc700 1 -- 172.21.15.116:0/911992245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd40052f0 con 0x7f2bf4101e60 2024-03-04T16:03:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.811+0000 7f2bf0ff9700 1 -- 172.21.15.116:0/911992245 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) v1 ==== 46068+0+0 (secure 0 0 0) 0x7f2be4010460 con 0x7f2bf4101e60 2024-03-04T16:03:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.812+0000 7f2bf0ff9700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f2bdc038f00 0x7f2bdc03b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:03:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.812+0000 7f2bf0ff9700 1 -- 172.21.15.116:0/911992245 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f2be404cf10 con 0x7f2bf4101e60 2024-03-04T16:03:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.812+0000 7f2bf37fe700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f2bdc038f00 0x7f2bdc03b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:03:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.813+0000 7f2bf37fe700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f2bdc038f00 0x7f2bdc03b3c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2be8006010 tx=0x7f2be800b540 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:03:37.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:37.817+0000 7f2bf0ff9700 1 -- 172.21.15.116:0/911992245 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f2be4010e40 con 0x7f2bf4101e60 2024-03-04T16:03:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.165+0000 7f2bda7fc700 1 -- 172.21.15.116:0/911992245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2bd4005160 con 0x7f2bf4101e60 2024-03-04T16:03:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.166+0000 7f2bf0ff9700 1 -- 172.21.15.116:0/911992245 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f2be4017030 con 0x7f2bf4101e60 2024-03-04T16:03:38.167 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.168+0000 7f2bda7fc700 1 -- 172.21.15.116:0/911992245 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f2bdc038f00 msgr2=0x7f2bdc03b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.168+0000 7f2bda7fc700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f2bdc038f00 0x7f2bdc03b3c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2be8006010 tx=0x7f2be800b540 comp rx=0 tx=0).stop 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.168+0000 7f2bda7fc700 1 -- 172.21.15.116:0/911992245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bf4101e60 msgr2=0x7f2bf40ffe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.168+0000 7f2bda7fc700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bf4101e60 0x7f2bf40ffe10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2be400d900 tx=0x7f2be400dc10 comp rx=0 tx=0).stop 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.168+0000 7f2bda7fc700 1 -- 172.21.15.116:0/911992245 shutdown_connections 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.168+0000 7f2bda7fc700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f2bdc038f00 0x7f2bdc03b3c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.168+0000 7f2bda7fc700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bf40fee80 0x7f2bf41017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.168+0000 7f2bda7fc700 1 --2- 172.21.15.116:0/911992245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bf4101e60 0x7f2bf40ffe10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.168+0000 7f2bda7fc700 1 -- 172.21.15.116:0/911992245 >> 172.21.15.116:0/911992245 conn(0x7f2bf40fa7c0 msgr2=0x7f2bf4105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.169+0000 7f2bda7fc700 1 -- 172.21.15.116:0/911992245 shutdown_connections 2024-03-04T16:03:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:03:38.169+0000 7f2bda7fc700 1 -- 172.21.15.116:0/911992245 wait complete. 2024-03-04T16:03:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:38 smithi152 ceph-mon[36876]: pgmap v5: 65 pgs: 65 active+clean; 87 MiB data, 516 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:38 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/4249803727' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:03:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:38 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/911992245' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:03:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:38 smithi116 ceph-mon[29120]: pgmap v5: 65 pgs: 65 active+clean; 87 MiB data, 516 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:03:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:38 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/4249803727' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:03:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:38 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/911992245' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:03:39.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:39 smithi152 ceph-mon[36876]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:39 smithi116 ceph-mon[29120]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:40.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:40 smithi152 ceph-mon[36876]: pgmap v6: 65 pgs: 65 active+clean; 95 MiB data, 549 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s wr, 140 op/s 2024-03-04T16:03:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:40 smithi116 ceph-mon[29120]: pgmap v6: 65 pgs: 65 active+clean; 95 MiB data, 549 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s wr, 140 op/s 2024-03-04T16:03:42.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:42 smithi152 ceph-mon[36876]: pgmap v7: 65 pgs: 65 active+clean; 108 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 29 KiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-03-04T16:03:42.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:42 smithi152 ceph-mon[36876]: Standby manager daemon smithi116.jwdhop started 2024-03-04T16:03:42.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:42 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.116:0/1363482223' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.jwdhop/crt"}]: dispatch 2024-03-04T16:03:42.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:42 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.116:0/1363482223' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:03:42.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:42 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.116:0/1363482223' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.jwdhop/key"}]: dispatch 2024-03-04T16:03:42.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:42 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.116:0/1363482223' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:03:42.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:42 smithi116 ceph-mon[29120]: pgmap v7: 65 pgs: 65 active+clean; 108 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 29 KiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-03-04T16:03:42.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:42 smithi116 ceph-mon[29120]: Standby manager daemon smithi116.jwdhop started 2024-03-04T16:03:42.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:42 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.116:0/1363482223' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.jwdhop/crt"}]: dispatch 2024-03-04T16:03:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:42 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.116:0/1363482223' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:03:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:42 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.116:0/1363482223' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.jwdhop/key"}]: dispatch 2024-03-04T16:03:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:42 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.116:0/1363482223' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:03:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:43 smithi152 ceph-mon[36876]: mgrmap e24: smithi152.aawieo(active, since 9s), standbys: smithi116.jwdhop 2024-03-04T16:03:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:43 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mgr metadata", "who": "smithi116.jwdhop", "id": "smithi116.jwdhop"}]: dispatch 2024-03-04T16:03:43.714 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:43 smithi116 ceph-mon[29120]: mgrmap e24: smithi152.aawieo(active, since 9s), standbys: smithi116.jwdhop 2024-03-04T16:03:43.714 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:43 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mgr metadata", "who": "smithi116.jwdhop", "id": "smithi116.jwdhop"}]: dispatch 2024-03-04T16:03:44.359 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:44 smithi116 ceph-mon[29120]: pgmap v8: 65 pgs: 65 active+clean; 108 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 23 KiB/s rd, 2.5 MiB/s wr, 231 op/s 2024-03-04T16:03:44.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:44 smithi152 ceph-mon[36876]: pgmap v8: 65 pgs: 65 active+clean; 108 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 23 KiB/s rd, 2.5 MiB/s wr, 231 op/s 2024-03-04T16:03:46.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:46 smithi152 ceph-mon[36876]: pgmap v9: 65 pgs: 65 active+clean; 129 MiB data, 678 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 378 op/s 2024-03-04T16:03:46.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:46 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:46.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:46 smithi116 ceph-mon[29120]: pgmap v9: 65 pgs: 65 active+clean; 129 MiB data, 678 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 378 op/s 2024-03-04T16:03:46.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:46 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:47.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:47 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:47.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:47 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:48.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:48 smithi152 ceph-mon[36876]: pgmap v10: 65 pgs: 65 active+clean; 129 MiB data, 678 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 3.7 MiB/s wr, 346 op/s 2024-03-04T16:03:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:48 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:48 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:48 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:48 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:48 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:48 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:48 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:48 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:48 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:48.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:48 smithi116 ceph-mon[29120]: pgmap v10: 65 pgs: 65 active+clean; 129 MiB data, 678 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 3.7 MiB/s wr, 346 op/s 2024-03-04T16:03:48.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:48 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:48 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:48 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:48 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:48 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:48 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:48 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:48 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:48.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:48 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:49 smithi116 ceph-mon[29120]: Adjusting osd_memory_target on smithi116 to 1216M 2024-03-04T16:03:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:49 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:49.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:49 smithi152 ceph-mon[36876]: Adjusting osd_memory_target on smithi116 to 1216M 2024-03-04T16:03:49.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:49 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:50.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:50 smithi152 ceph-mon[36876]: pgmap v11: 65 pgs: 65 active+clean; 135 MiB data, 681 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 4.2 MiB/s wr, 394 op/s 2024-03-04T16:03:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:50 smithi116 ceph-mon[29120]: pgmap v11: 65 pgs: 65 active+clean; 135 MiB data, 681 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 4.2 MiB/s wr, 394 op/s 2024-03-04T16:03:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:52 smithi152 ceph-mon[36876]: pgmap v12: 65 pgs: 65 active+clean; 142 MiB data, 718 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 4.2 MiB/s wr, 432 op/s 2024-03-04T16:03:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:52 smithi116 ceph-mon[29120]: pgmap v12: 65 pgs: 65 active+clean; 142 MiB data, 718 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 4.2 MiB/s wr, 432 op/s 2024-03-04T16:03:53.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:53 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:53.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:53 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:53.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:53 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:53.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:53 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:54.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:54 smithi152 ceph-mon[36876]: pgmap v13: 65 pgs: 65 active+clean; 142 MiB data, 718 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s wr, 318 op/s 2024-03-04T16:03:54.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:54 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:54.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:54 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:54.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:54 smithi116 ceph-mon[29120]: pgmap v13: 65 pgs: 65 active+clean; 142 MiB data, 718 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s wr, 318 op/s 2024-03-04T16:03:54.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:54 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:54.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:54 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:56.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:56 smithi152 ceph-mon[36876]: pgmap v14: 65 pgs: 65 active+clean; 159 MiB data, 787 MiB used, 536 GiB / 536 GiB avail; 4.5 MiB/s wr, 490 op/s 2024-03-04T16:03:56.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:56 smithi116 ceph-mon[29120]: pgmap v14: 65 pgs: 65 active+clean; 159 MiB data, 787 MiB used, 536 GiB / 536 GiB avail; 4.5 MiB/s wr, 490 op/s 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: pgmap v15: 65 pgs: 65 active+clean; 159 MiB data, 787 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s wr, 317 op/s 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:58.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:58.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:58 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:58.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: pgmap v15: 65 pgs: 65 active+clean; 159 MiB data, 787 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s wr, 317 op/s 2024-03-04T16:03:58.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:58.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:58.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:58.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:03:58.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:58.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:58 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:59.755 INFO:tasks.workunit.client.0.smithi116.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-04T16:03:59.761 INFO:tasks.workunit.client.0.smithi116.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-04T16:03:59.761 INFO:tasks.workunit.client.0.smithi116.stderr:+ make 2024-03-04T16:03:59.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:59 smithi152 ceph-mon[36876]: Adjusting osd_memory_target on smithi152 to 2240M 2024-03-04T16:03:59.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:59 smithi152 ceph-mon[36876]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T16:03:59.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:03:59 smithi152 ceph-mon[36876]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T16:03:59.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:59 smithi116 ceph-mon[29120]: Adjusting osd_memory_target on smithi152 to 2240M 2024-03-04T16:03:59.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:59 smithi116 ceph-mon[29120]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T16:03:59.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:03:59 smithi116 ceph-mon[29120]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T16:04:00.236 INFO:tasks.workunit.client.0.smithi116.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-04T16:04:00.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: pgmap v16: 65 pgs: 65 active+clean; 162 MiB data, 806 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s wr, 371 op/s 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:00.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:00 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:04:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:04:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:04:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:04:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: pgmap v16: 65 pgs: 65 active+clean; 162 MiB data, 806 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s wr, 371 op/s 2024-03-04T16:04:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:04:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:04:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:00.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:00.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:00.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:00 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:01.762 INFO:tasks.workunit.client.0.smithi116.stderr:++ readlink -f fsstress 2024-03-04T16:04:01.764 INFO:tasks.workunit.client.0.smithi116.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-04T16:04:01.764 INFO:tasks.workunit.client.0.smithi116.stderr:+ popd 2024-03-04T16:04:01.765 INFO:tasks.workunit.client.0.smithi116.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-04T16:04:01.765 INFO:tasks.workunit.client.0.smithi116.stderr:+ popd 2024-03-04T16:04:01.765 INFO:tasks.workunit.client.0.smithi116.stdout:~/cephtest/mnt.0/client.0/tmp 2024-03-04T16:04:01.766 INFO:tasks.workunit.client.0.smithi116.stderr:++ mktemp -d -p . 2024-03-04T16:04:01.769 INFO:tasks.workunit.client.0.smithi116.stderr:+ T=./tmp.645vXPrkDJ 2024-03-04T16:04:01.769 INFO:tasks.workunit.client.0.smithi116.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.645vXPrkDJ -l 1 -n 1000 -p 10 -v 2024-03-04T16:04:01.774 INFO:tasks.workunit.client.0.smithi116.stdout:seed = 1710105401 2024-03-04T16:04:01.782 INFO:tasks.workunit.client.0.smithi116.stdout:0/0: dwrite - no filename 2024-03-04T16:04:01.782 INFO:tasks.workunit.client.0.smithi116.stdout:0/1: chown . 41078 1 2024-03-04T16:04:01.782 INFO:tasks.workunit.client.0.smithi116.stdout:0/2: dread - no filename 2024-03-04T16:04:01.782 INFO:tasks.workunit.client.0.smithi116.stdout:0/3: readlink - no filename 2024-03-04T16:04:01.789 INFO:tasks.workunit.client.0.smithi116.stdout:2/0: dwrite - no filename 2024-03-04T16:04:01.789 INFO:tasks.workunit.client.0.smithi116.stdout:2/1: chown . 1007865 1 2024-03-04T16:04:01.789 INFO:tasks.workunit.client.0.smithi116.stdout:2/2: dwrite - no filename 2024-03-04T16:04:01.789 INFO:tasks.workunit.client.0.smithi116.stdout:2/3: rename - no filename 2024-03-04T16:04:01.789 INFO:tasks.workunit.client.0.smithi116.stdout:2/4: dwrite - no filename 2024-03-04T16:04:01.789 INFO:tasks.workunit.client.0.smithi116.stdout:2/5: rename - no filename 2024-03-04T16:04:01.789 INFO:tasks.workunit.client.0.smithi116.stdout:2/6: write - no filename 2024-03-04T16:04:01.791 INFO:tasks.workunit.client.0.smithi116.stdout:0/4: symlink l0 0 2024-03-04T16:04:01.796 INFO:tasks.workunit.client.0.smithi116.stdout:1/0: mknod c0 0 2024-03-04T16:04:01.798 INFO:tasks.workunit.client.0.smithi116.stdout:0/5: symlink l1 0 2024-03-04T16:04:01.798 INFO:tasks.workunit.client.0.smithi116.stdout:0/6: truncate - no filename 2024-03-04T16:04:01.802 INFO:tasks.workunit.client.0.smithi116.stdout:2/7: mknod c0 0 2024-03-04T16:04:01.803 INFO:tasks.workunit.client.0.smithi116.stdout:0/7: unlink l0 0 2024-03-04T16:04:01.804 INFO:tasks.workunit.client.0.smithi116.stdout:0/8: truncate - no filename 2024-03-04T16:04:01.804 INFO:tasks.workunit.client.0.smithi116.stdout:0/9: truncate - no filename 2024-03-04T16:04:01.806 INFO:tasks.workunit.client.0.smithi116.stdout:5/0: write - no filename 2024-03-04T16:04:01.810 INFO:tasks.workunit.client.0.smithi116.stdout:3/0: symlink l0 0 2024-03-04T16:04:01.810 INFO:tasks.workunit.client.0.smithi116.stdout:3/1: stat l0 0 2024-03-04T16:04:01.810 INFO:tasks.workunit.client.0.smithi116.stdout:3/2: dread - no filename 2024-03-04T16:04:01.812 INFO:tasks.workunit.client.0.smithi116.stdout:2/8: creat f1 x:0 0 0 2024-03-04T16:04:01.815 INFO:tasks.workunit.client.0.smithi116.stdout:0/10: creat f2 x:0 0 0 2024-03-04T16:04:01.820 INFO:tasks.workunit.client.0.smithi116.stdout:6/0: write - no filename 2024-03-04T16:04:01.820 INFO:tasks.workunit.client.0.smithi116.stdout:6/1: dread - no filename 2024-03-04T16:04:01.820 INFO:tasks.workunit.client.0.smithi116.stdout:4/0: creat f0 x:0 0 0 2024-03-04T16:04:01.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:01 smithi152 ceph-mon[36876]: Reconfiguring prometheus.smithi116 (dependencies changed)... 2024-03-04T16:04:01.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:01 smithi152 ceph-mon[36876]: Reconfiguring daemon prometheus.smithi116 on smithi116 2024-03-04T16:04:01.939 INFO:tasks.workunit.client.0.smithi116.stdout:3/3: unlink l0 0 2024-03-04T16:04:01.940 INFO:tasks.workunit.client.0.smithi116.stdout:2/9: mknod c2 0 2024-03-04T16:04:01.940 INFO:tasks.workunit.client.0.smithi116.stdout:4/1: dread - f0 zero size 2024-03-04T16:04:01.940 INFO:tasks.workunit.client.0.smithi116.stdout:5/1: getdents . 0 2024-03-04T16:04:01.940 INFO:tasks.workunit.client.0.smithi116.stdout:5/2: rmdir - no directory 2024-03-04T16:04:01.940 INFO:tasks.workunit.client.0.smithi116.stdout:5/3: write - no filename 2024-03-04T16:04:01.940 INFO:tasks.workunit.client.0.smithi116.stdout:5/4: dread - no filename 2024-03-04T16:04:01.941 INFO:tasks.workunit.client.0.smithi116.stdout:2/10: chown c2 0 1 2024-03-04T16:04:01.941 INFO:tasks.workunit.client.0.smithi116.stdout:2/11: rmdir - no directory 2024-03-04T16:04:01.943 INFO:tasks.workunit.client.0.smithi116.stdout:6/2: creat f0 x:0 0 0 2024-03-04T16:04:01.943 INFO:tasks.workunit.client.0.smithi116.stdout:6/3: chown f0 7 1 2024-03-04T16:04:01.943 INFO:tasks.workunit.client.0.smithi116.stdout:6/4: dread - f0 zero size 2024-03-04T16:04:01.945 INFO:tasks.workunit.client.0.smithi116.stdout:3/4: creat f1 x:0 0 0 2024-03-04T16:04:01.948 INFO:tasks.workunit.client.0.smithi116.stdout:8/0: rename - no filename 2024-03-04T16:04:01.950 INFO:tasks.workunit.client.0.smithi116.stdout:4/2: mkdir d1 0 2024-03-04T16:04:01.974 INFO:tasks.workunit.client.0.smithi116.stdout:2/12: symlink l3 0 2024-03-04T16:04:01.974 INFO:tasks.workunit.client.0.smithi116.stdout:6/5: mknod c1 0 2024-03-04T16:04:01.974 INFO:tasks.workunit.client.0.smithi116.stdout:3/5: creat f2 x:0 0 0 2024-03-04T16:04:01.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:01 smithi116 ceph-mon[29120]: Reconfiguring prometheus.smithi116 (dependencies changed)... 2024-03-04T16:04:01.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:01 smithi116 ceph-mon[29120]: Reconfiguring daemon prometheus.smithi116 on smithi116 2024-03-04T16:04:01.975 INFO:tasks.workunit.client.0.smithi116.stdout:2/13: chown c2 4 1 2024-03-04T16:04:01.975 INFO:tasks.workunit.client.0.smithi116.stdout:7/0: creat f0 x:0 0 0 2024-03-04T16:04:01.976 INFO:tasks.workunit.client.0.smithi116.stdout:9/0: unlink - no file 2024-03-04T16:04:01.976 INFO:tasks.workunit.client.0.smithi116.stdout:9/1: unlink - no file 2024-03-04T16:04:01.976 INFO:tasks.workunit.client.0.smithi116.stdout:9/2: dread - no filename 2024-03-04T16:04:01.976 INFO:tasks.workunit.client.0.smithi116.stdout:9/3: rmdir - no directory 2024-03-04T16:04:01.976 INFO:tasks.workunit.client.0.smithi116.stdout:6/6: truncate f0 1047506 0 2024-03-04T16:04:01.976 INFO:tasks.workunit.client.0.smithi116.stdout:5/5: mknod c0 0 2024-03-04T16:04:01.978 INFO:tasks.workunit.client.0.smithi116.stdout:6/7: write f0 [1443992,65297] 0 2024-03-04T16:04:01.978 INFO:tasks.workunit.client.0.smithi116.stdout:3/6: mknod c3 0 2024-03-04T16:04:01.980 INFO:tasks.workunit.client.0.smithi116.stdout:4/3: dwrite f0 [0,4194304] 0 2024-03-04T16:04:01.981 INFO:tasks.workunit.client.0.smithi116.stdout:2/14: creat f4 x:0 0 0 2024-03-04T16:04:01.981 INFO:tasks.workunit.client.0.smithi116.stdout:4/4: write f0 [628192,31863] 0 2024-03-04T16:04:01.982 INFO:tasks.workunit.client.0.smithi116.stdout:2/15: write f4 [59588,52777] 0 2024-03-04T16:04:01.983 INFO:tasks.workunit.client.0.smithi116.stdout:2/16: write f4 [172712,46935] 0 2024-03-04T16:04:01.992 INFO:tasks.workunit.client.0.smithi116.stdout:3/7: mknod c4 0 2024-03-04T16:04:01.993 INFO:tasks.workunit.client.0.smithi116.stdout:3/8: write f2 [838264,57689] 0 2024-03-04T16:04:01.994 INFO:tasks.workunit.client.0.smithi116.stdout:3/9: write f1 [324203,102533] 0 2024-03-04T16:04:01.994 INFO:tasks.workunit.client.0.smithi116.stdout:3/10: chown c4 127998 1 2024-03-04T16:04:01.996 INFO:tasks.workunit.client.0.smithi116.stdout:7/1: link f0 f1 0 2024-03-04T16:04:02.001 INFO:tasks.workunit.client.0.smithi116.stdout:9/4: creat f0 x:0 0 0 2024-03-04T16:04:02.006 INFO:tasks.workunit.client.0.smithi116.stdout:3/11: creat f5 x:0 0 0 2024-03-04T16:04:02.006 INFO:tasks.workunit.client.0.smithi116.stdout:3/12: truncate f2 1789245 0 2024-03-04T16:04:02.009 INFO:tasks.workunit.client.0.smithi116.stdout:4/5: creat d1/f2 x:0 0 0 2024-03-04T16:04:02.009 INFO:tasks.workunit.client.0.smithi116.stdout:4/6: chown d1 2 1 2024-03-04T16:04:02.010 INFO:tasks.workunit.client.0.smithi116.stdout:2/17: dread f4 [0,4194304] 0 2024-03-04T16:04:02.010 INFO:tasks.workunit.client.0.smithi116.stdout:9/5: unlink f0 0 2024-03-04T16:04:02.011 INFO:tasks.workunit.client.0.smithi116.stdout:9/6: rename - no filename 2024-03-04T16:04:02.013 INFO:tasks.workunit.client.0.smithi116.stdout:5/6: link c0 c1 0 2024-03-04T16:04:02.013 INFO:tasks.workunit.client.0.smithi116.stdout:5/7: dwrite - no filename 2024-03-04T16:04:02.016 INFO:tasks.workunit.client.0.smithi116.stdout:3/13: unlink f1 0 2024-03-04T16:04:02.016 INFO:tasks.workunit.client.0.smithi116.stdout:4/7: dwrite d1/f2 [0,4194304] 0 2024-03-04T16:04:02.018 INFO:tasks.workunit.client.0.smithi116.stdout:4/8: write d1/f2 [1774352,59780] 0 2024-03-04T16:04:02.019 INFO:tasks.workunit.client.0.smithi116.stdout:2/18: mknod c5 0 2024-03-04T16:04:02.019 INFO:tasks.workunit.client.0.smithi116.stdout:2/19: chown f4 109 1 2024-03-04T16:04:02.023 INFO:tasks.workunit.client.0.smithi116.stdout:9/7: creat f1 x:0 0 0 2024-03-04T16:04:02.033 INFO:tasks.workunit.client.0.smithi116.stdout:3/14: symlink l6 0 2024-03-04T16:04:02.033 INFO:tasks.workunit.client.0.smithi116.stdout:3/15: readlink l6 0 2024-03-04T16:04:02.033 INFO:tasks.workunit.client.0.smithi116.stdout:3/16: write f5 [164681,80591] 0 2024-03-04T16:04:02.038 INFO:tasks.workunit.client.0.smithi116.stdout:4/9: mkdir d1/d3 0 2024-03-04T16:04:02.039 INFO:tasks.workunit.client.0.smithi116.stdout:2/20: creat f6 x:0 0 0 2024-03-04T16:04:02.041 INFO:tasks.workunit.client.0.smithi116.stdout:9/8: mkdir d2 0 2024-03-04T16:04:02.045 INFO:tasks.workunit.client.0.smithi116.stdout:3/17: creat f7 x:0 0 0 2024-03-04T16:04:02.051 INFO:tasks.workunit.client.0.smithi116.stdout:5/8: link c0 c2 0 2024-03-04T16:04:02.053 INFO:tasks.workunit.client.0.smithi116.stdout:3/18: unlink f7 0 2024-03-04T16:04:02.057 INFO:tasks.workunit.client.0.smithi116.stdout:9/9: mknod d2/c3 0 2024-03-04T16:04:02.060 INFO:tasks.workunit.client.0.smithi116.stdout:3/19: dwrite f5 [0,4194304] 0 2024-03-04T16:04:02.062 INFO:tasks.workunit.client.0.smithi116.stdout:3/20: dread f5 [0,4194304] 0 2024-03-04T16:04:02.066 INFO:tasks.workunit.client.0.smithi116.stdout:9/10: rename f1 to d2/f4 0 2024-03-04T16:04:02.066 INFO:tasks.workunit.client.0.smithi116.stdout:9/11: chown d2/f4 69135 1 2024-03-04T16:04:02.068 INFO:tasks.workunit.client.0.smithi116.stdout:5/9: rename c0 to c3 0 2024-03-04T16:04:02.068 INFO:tasks.workunit.client.0.smithi116.stdout:5/10: dwrite - no filename 2024-03-04T16:04:02.068 INFO:tasks.workunit.client.0.smithi116.stdout:5/11: dread - no filename 2024-03-04T16:04:02.078 INFO:tasks.workunit.client.0.smithi116.stdout:5/12: creat f4 x:0 0 0 2024-03-04T16:04:02.079 INFO:tasks.workunit.client.0.smithi116.stdout:5/13: truncate f4 843106 0 2024-03-04T16:04:02.082 INFO:tasks.workunit.client.0.smithi116.stdout:3/21: link c4 c8 0 2024-03-04T16:04:02.082 INFO:tasks.workunit.client.0.smithi116.stdout:9/12: dwrite d2/f4 [0,4194304] 0 2024-03-04T16:04:02.083 INFO:tasks.workunit.client.0.smithi116.stdout:5/14: creat f5 x:0 0 0 2024-03-04T16:04:02.083 INFO:tasks.workunit.client.0.smithi116.stdout:5/15: dread - f5 zero size 2024-03-04T16:04:02.087 INFO:tasks.workunit.client.0.smithi116.stdout:9/13: unlink d2/c3 0 2024-03-04T16:04:02.090 INFO:tasks.workunit.client.0.smithi116.stdout:9/14: dread d2/f4 [0,4194304] 0 2024-03-04T16:04:02.090 INFO:tasks.workunit.client.0.smithi116.stdout:9/15: readlink - no filename 2024-03-04T16:04:02.090 INFO:tasks.workunit.client.0.smithi116.stdout:3/22: chown c4 0 1 2024-03-04T16:04:02.091 INFO:tasks.workunit.client.0.smithi116.stdout:3/23: truncate f2 2422466 0 2024-03-04T16:04:02.095 INFO:tasks.workunit.client.0.smithi116.stdout:3/24: mknod c9 0 2024-03-04T16:04:02.096 INFO:tasks.workunit.client.0.smithi116.stdout:3/25: chown c3 43269 1 2024-03-04T16:04:02.098 INFO:tasks.workunit.client.0.smithi116.stdout:3/26: creat fa x:0 0 0 2024-03-04T16:04:02.102 INFO:tasks.workunit.client.0.smithi116.stdout:3/27: dread f2 [0,4194304] 0 2024-03-04T16:04:02.106 INFO:tasks.workunit.client.0.smithi116.stdout:3/28: link c3 cb 0 2024-03-04T16:04:02.108 INFO:tasks.workunit.client.0.smithi116.stdout:3/29: mkdir dc 0 2024-03-04T16:04:02.114 INFO:tasks.workunit.client.0.smithi116.stdout:3/30: dwrite f5 [0,4194304] 0 2024-03-04T16:04:02.117 INFO:tasks.workunit.client.0.smithi116.stdout:3/31: unlink c9 0 2024-03-04T16:04:02.120 INFO:tasks.workunit.client.0.smithi116.stdout:3/32: dread f2 [0,4194304] 0 2024-03-04T16:04:02.120 INFO:tasks.workunit.client.0.smithi116.stdout:3/33: readlink l6 0 2024-03-04T16:04:02.123 INFO:tasks.workunit.client.0.smithi116.stdout:3/34: symlink dc/ld 0 2024-03-04T16:04:02.123 INFO:tasks.workunit.client.0.smithi116.stdout:3/35: read f2 [1343747,62352] 0 2024-03-04T16:04:02.577 INFO:tasks.workunit.client.0.smithi116.stdout:6/8: getdents . 0 2024-03-04T16:04:02.579 INFO:tasks.workunit.client.0.smithi116.stdout:6/9: creat f2 x:0 0 0 2024-03-04T16:04:02.579 INFO:tasks.workunit.client.0.smithi116.stdout:6/10: read - f2 zero size 2024-03-04T16:04:02.585 INFO:tasks.workunit.client.0.smithi116.stdout:6/11: dwrite f0 [0,4194304] 0 2024-03-04T16:04:02.589 INFO:tasks.workunit.client.0.smithi116.stdout:6/12: unlink f0 0 2024-03-04T16:04:02.593 INFO:tasks.workunit.client.0.smithi116.stdout:6/13: creat f3 x:0 0 0 2024-03-04T16:04:02.593 INFO:tasks.workunit.client.0.smithi116.stdout:6/14: rmdir - no directory 2024-03-04T16:04:02.594 INFO:tasks.workunit.client.0.smithi116.stdout:6/15: stat c1 0 2024-03-04T16:04:02.594 INFO:tasks.workunit.client.0.smithi116.stdout:6/16: dread - f2 zero size 2024-03-04T16:04:02.594 INFO:tasks.workunit.client.0.smithi116.stdout:6/17: truncate f2 171078 0 2024-03-04T16:04:02.595 INFO:tasks.workunit.client.0.smithi116.stdout:6/18: chown f3 111068 1 2024-03-04T16:04:02.595 INFO:tasks.workunit.client.0.smithi116.stdout:6/19: chown c1 1727154 1 2024-03-04T16:04:02.595 INFO:tasks.workunit.client.0.smithi116.stdout:6/20: rmdir - no directory 2024-03-04T16:04:02.595 INFO:tasks.workunit.client.0.smithi116.stdout:6/21: write f3 [715593,112450] 0 2024-03-04T16:04:02.596 INFO:tasks.workunit.client.0.smithi116.stdout:2/21: fsync f4 0 2024-03-04T16:04:02.598 INFO:tasks.workunit.client.0.smithi116.stdout:6/22: symlink l4 0 2024-03-04T16:04:02.598 INFO:tasks.workunit.client.0.smithi116.stdout:6/23: rmdir - no directory 2024-03-04T16:04:02.602 INFO:tasks.workunit.client.0.smithi116.stdout:6/24: symlink l5 0 2024-03-04T16:04:02.603 INFO:tasks.workunit.client.0.smithi116.stdout:0/11: sync 2024-03-04T16:04:02.603 INFO:tasks.workunit.client.0.smithi116.stdout:1/1: sync 2024-03-04T16:04:02.603 INFO:tasks.workunit.client.0.smithi116.stdout:8/1: sync 2024-03-04T16:04:02.603 INFO:tasks.workunit.client.0.smithi116.stdout:8/2: rename - no filename 2024-03-04T16:04:02.603 INFO:tasks.workunit.client.0.smithi116.stdout:8/3: dwrite - no filename 2024-03-04T16:04:02.604 INFO:tasks.workunit.client.0.smithi116.stdout:2/22: dwrite f4 [0,4194304] 0 2024-03-04T16:04:02.604 INFO:tasks.workunit.client.0.smithi116.stdout:0/12: chown f2 112 1 2024-03-04T16:04:02.604 INFO:tasks.workunit.client.0.smithi116.stdout:1/2: chown c0 1628575 1 2024-03-04T16:04:02.605 INFO:tasks.workunit.client.0.smithi116.stdout:6/25: mkdir d6 0 2024-03-04T16:04:02.607 INFO:tasks.workunit.client.0.smithi116.stdout:2/23: mkdir d7 0 2024-03-04T16:04:02.607 INFO:tasks.workunit.client.0.smithi116.stdout:6/26: dread f2 [0,4194304] 0 2024-03-04T16:04:02.608 INFO:tasks.workunit.client.0.smithi116.stdout:6/27: read f2 [7270,104570] 0 2024-03-04T16:04:02.608 INFO:tasks.workunit.client.0.smithi116.stdout:0/13: rename l1 to l3 0 2024-03-04T16:04:02.609 INFO:tasks.workunit.client.0.smithi116.stdout:0/14: write f2 [779274,86691] 0 2024-03-04T16:04:02.612 INFO:tasks.workunit.client.0.smithi116.stdout:7/2: getdents . 0 2024-03-04T16:04:02.613 INFO:tasks.workunit.client.0.smithi116.stdout:8/4: symlink l0 0 2024-03-04T16:04:02.616 INFO:tasks.workunit.client.0.smithi116.stdout:0/15: dwrite f2 [0,4194304] 0 2024-03-04T16:04:02.616 INFO:tasks.workunit.client.0.smithi116.stdout:0/16: stat f2 0 2024-03-04T16:04:02.616 INFO:tasks.workunit.client.0.smithi116.stdout:6/28: rename l5 to d6/l7 0 2024-03-04T16:04:02.617 INFO:tasks.workunit.client.0.smithi116.stdout:0/17: chown f2 15154 1 2024-03-04T16:04:02.617 INFO:tasks.workunit.client.0.smithi116.stdout:6/29: chown l4 11 1 2024-03-04T16:04:02.620 INFO:tasks.workunit.client.0.smithi116.stdout:0/18: dread f2 [0,4194304] 0 2024-03-04T16:04:02.620 INFO:tasks.workunit.client.0.smithi116.stdout:6/30: dread f3 [0,4194304] 0 2024-03-04T16:04:02.625 INFO:tasks.workunit.client.0.smithi116.stdout:8/5: creat f1 x:0 0 0 2024-03-04T16:04:02.625 INFO:tasks.workunit.client.0.smithi116.stdout:8/6: readlink l0 0 2024-03-04T16:04:02.626 INFO:tasks.workunit.client.0.smithi116.stdout:8/7: chown l0 0 1 2024-03-04T16:04:02.632 INFO:tasks.workunit.client.0.smithi116.stdout:1/3: link c0 c1 0 2024-03-04T16:04:02.634 INFO:tasks.workunit.client.0.smithi116.stdout:6/31: creat d6/f8 x:0 0 0 2024-03-04T16:04:02.640 INFO:tasks.workunit.client.0.smithi116.stdout:8/8: unlink f1 0 2024-03-04T16:04:02.640 INFO:tasks.workunit.client.0.smithi116.stdout:8/9: dread - no filename 2024-03-04T16:04:02.640 INFO:tasks.workunit.client.0.smithi116.stdout:2/24: link c2 d7/c8 0 2024-03-04T16:04:02.640 INFO:tasks.workunit.client.0.smithi116.stdout:2/25: write f1 [843811,7083] 0 2024-03-04T16:04:02.643 INFO:tasks.workunit.client.0.smithi116.stdout:1/4: creat f2 x:0 0 0 2024-03-04T16:04:02.644 INFO:tasks.workunit.client.0.smithi116.stdout:6/32: creat d6/f9 x:0 0 0 2024-03-04T16:04:02.647 INFO:tasks.workunit.client.0.smithi116.stdout:8/10: mkdir d2 0 2024-03-04T16:04:02.648 INFO:tasks.workunit.client.0.smithi116.stdout:6/33: creat d6/fa x:0 0 0 2024-03-04T16:04:02.649 INFO:tasks.workunit.client.0.smithi116.stdout:6/34: truncate d6/f8 780704 0 2024-03-04T16:04:02.649 INFO:tasks.workunit.client.0.smithi116.stdout:6/35: truncate d6/f9 35283 0 2024-03-04T16:04:02.649 INFO:tasks.workunit.client.0.smithi116.stdout:6/36: chown d6/f8 966769 1 2024-03-04T16:04:02.650 INFO:tasks.workunit.client.0.smithi116.stdout:2/26: mknod d7/c9 0 2024-03-04T16:04:02.653 INFO:tasks.workunit.client.0.smithi116.stdout:2/27: mknod d7/ca 0 2024-03-04T16:04:02.654 INFO:tasks.workunit.client.0.smithi116.stdout:2/28: write f4 [922409,53339] 0 2024-03-04T16:04:02.655 INFO:tasks.workunit.client.0.smithi116.stdout:8/11: mkdir d2/d3 0 2024-03-04T16:04:02.655 INFO:tasks.workunit.client.0.smithi116.stdout:2/29: truncate f1 1344448 0 2024-03-04T16:04:02.656 INFO:tasks.workunit.client.0.smithi116.stdout:4/10: getdents d1 0 2024-03-04T16:04:02.663 INFO:tasks.workunit.client.0.smithi116.stdout:8/12: creat d2/d3/f4 x:0 0 0 2024-03-04T16:04:02.663 INFO:tasks.workunit.client.0.smithi116.stdout:8/13: dread - d2/d3/f4 zero size 2024-03-04T16:04:02.664 INFO:tasks.workunit.client.0.smithi116.stdout:8/14: stat d2/d3 0 2024-03-04T16:04:02.664 INFO:tasks.workunit.client.0.smithi116.stdout:4/11: link d1/f2 d1/d3/f4 0 2024-03-04T16:04:02.665 INFO:tasks.workunit.client.0.smithi116.stdout:4/12: write f0 [3387521,112721] 0 2024-03-04T16:04:02.665 INFO:tasks.workunit.client.0.smithi116.stdout:4/13: read d1/f2 [1204104,71688] 0 2024-03-04T16:04:02.666 INFO:tasks.workunit.client.0.smithi116.stdout:8/15: symlink d2/d3/l5 0 2024-03-04T16:04:02.668 INFO:tasks.workunit.client.0.smithi116.stdout:4/14: creat d1/f5 x:0 0 0 2024-03-04T16:04:02.671 INFO:tasks.workunit.client.0.smithi116.stdout:8/16: rename d2/d3/f4 to d2/f6 0 2024-03-04T16:04:02.671 INFO:tasks.workunit.client.0.smithi116.stdout:4/15: rename d1 to d1/d3/d6 22 2024-03-04T16:04:02.671 INFO:tasks.workunit.client.0.smithi116.stdout:4/16: dread - d1/f5 zero size 2024-03-04T16:04:02.674 INFO:tasks.workunit.client.0.smithi116.stdout:8/17: fdatasync d2/f6 0 2024-03-04T16:04:02.675 INFO:tasks.workunit.client.0.smithi116.stdout:4/17: mkdir d1/d7 0 2024-03-04T16:04:02.680 INFO:tasks.workunit.client.0.smithi116.stdout:4/18: link f0 d1/f8 0 2024-03-04T16:04:02.683 INFO:tasks.workunit.client.0.smithi116.stdout:4/19: write d1/d3/f4 [2416521,24540] 0 2024-03-04T16:04:02.683 INFO:tasks.workunit.client.0.smithi116.stdout:4/20: chown d1/f8 6293666 1 2024-03-04T16:04:02.689 INFO:tasks.workunit.client.0.smithi116.stdout:9/16: rename d2/f4 to d2/f5 0 2024-03-04T16:04:02.689 INFO:tasks.workunit.client.0.smithi116.stdout:4/21: dwrite d1/f2 [0,4194304] 0 2024-03-04T16:04:02.691 INFO:tasks.workunit.client.0.smithi116.stdout:9/17: write d2/f5 [3849943,33696] 0 2024-03-04T16:04:02.698 INFO:tasks.workunit.client.0.smithi116.stdout:4/22: dwrite d1/d3/f4 [0,4194304] 0 2024-03-04T16:04:02.699 INFO:tasks.workunit.client.0.smithi116.stdout:4/23: chown d1/f2 51 1 2024-03-04T16:04:02.701 INFO:tasks.workunit.client.0.smithi116.stdout:5/16: fsync f5 0 2024-03-04T16:04:02.703 INFO:tasks.workunit.client.0.smithi116.stdout:9/18: dwrite d2/f5 [0,4194304] 0 2024-03-04T16:04:02.703 INFO:tasks.workunit.client.0.smithi116.stdout:5/17: write f4 [888911,62177] 0 2024-03-04T16:04:02.703 INFO:tasks.workunit.client.0.smithi116.stdout:4/24: truncate d1/d3/f4 4829273 0 2024-03-04T16:04:02.707 INFO:tasks.workunit.client.0.smithi116.stdout:9/19: dread d2/f5 [0,4194304] 0 2024-03-04T16:04:02.717 INFO:tasks.workunit.client.0.smithi116.stdout:4/25: creat d1/d7/f9 x:0 0 0 2024-03-04T16:04:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:3/36: getdents dc 0 2024-03-04T16:04:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:3/37: write fa [1033211,113361] 0 2024-03-04T16:04:02.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:02 smithi116 ceph-mon[29120]: pgmap v17: 65 pgs: 65 active+clean; 180 MiB data, 843 MiB used, 536 GiB / 536 GiB avail; 4.0 MiB/s wr, 422 op/s 2024-03-04T16:04:02.728 INFO:tasks.workunit.client.0.smithi116.stdout:3/38: getdents dc 0 2024-03-04T16:04:02.728 INFO:tasks.workunit.client.0.smithi116.stdout:3/39: write fa [477479,49589] 0 2024-03-04T16:04:02.733 INFO:tasks.workunit.client.0.smithi116.stdout:3/40: mknod dc/ce 0 2024-03-04T16:04:02.737 INFO:tasks.workunit.client.0.smithi116.stdout:3/41: chown c4 1063783 1 2024-03-04T16:04:02.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:02 smithi152 ceph-mon[36876]: pgmap v17: 65 pgs: 65 active+clean; 180 MiB data, 843 MiB used, 536 GiB / 536 GiB avail; 4.0 MiB/s wr, 422 op/s 2024-03-04T16:04:02.898 INFO:tasks.workunit.client.0.smithi116.stdout:3/42: sync 2024-03-04T16:04:02.899 INFO:tasks.workunit.client.0.smithi116.stdout:1/5: sync 2024-03-04T16:04:02.902 INFO:tasks.workunit.client.0.smithi116.stdout:3/43: write f2 [1959305,113033] 0 2024-03-04T16:04:02.903 INFO:tasks.workunit.client.0.smithi116.stdout:1/6: stat c0 0 2024-03-04T16:04:02.905 INFO:tasks.workunit.client.0.smithi116.stdout:3/44: mkdir dc/df 0 2024-03-04T16:04:02.906 INFO:tasks.workunit.client.0.smithi116.stdout:3/45: fsync fa 0 2024-03-04T16:04:02.911 INFO:tasks.workunit.client.0.smithi116.stdout:3/46: dwrite fa [0,4194304] 0 2024-03-04T16:04:02.912 INFO:tasks.workunit.client.0.smithi116.stdout:1/7: link c1 c3 0 2024-03-04T16:04:02.913 INFO:tasks.workunit.client.0.smithi116.stdout:1/8: write f2 [249550,74257] 0 2024-03-04T16:04:02.914 INFO:tasks.workunit.client.0.smithi116.stdout:1/9: write f2 [964721,57188] 0 2024-03-04T16:04:02.915 INFO:tasks.workunit.client.0.smithi116.stdout:3/47: dread f2 [0,4194304] 0 2024-03-04T16:04:02.916 INFO:tasks.workunit.client.0.smithi116.stdout:3/48: write fa [2947505,98917] 0 2024-03-04T16:04:02.919 INFO:tasks.workunit.client.0.smithi116.stdout:1/10: creat f4 x:0 0 0 2024-03-04T16:04:02.920 INFO:tasks.workunit.client.0.smithi116.stdout:1/11: read - f4 zero size 2024-03-04T16:04:02.923 INFO:tasks.workunit.client.0.smithi116.stdout:3/49: dwrite f5 [0,4194304] 0 2024-03-04T16:04:02.928 INFO:tasks.workunit.client.0.smithi116.stdout:1/12: dwrite f2 [0,4194304] 0 2024-03-04T16:04:02.933 INFO:tasks.workunit.client.0.smithi116.stdout:1/13: mkdir d5 0 2024-03-04T16:04:02.937 INFO:tasks.workunit.client.0.smithi116.stdout:3/50: link cb dc/df/c10 0 2024-03-04T16:04:02.940 INFO:tasks.workunit.client.0.smithi116.stdout:3/51: symlink dc/l11 0 2024-03-04T16:04:02.943 INFO:tasks.workunit.client.0.smithi116.stdout:3/52: rename dc/l11 to dc/df/l12 0 2024-03-04T16:04:02.970 INFO:tasks.workunit.client.0.smithi116.stdout:6/37: stat d6/l7 0 2024-03-04T16:04:02.970 INFO:tasks.workunit.client.0.smithi116.stdout:6/38: rename d6 to d6/db 22 2024-03-04T16:04:02.972 INFO:tasks.workunit.client.0.smithi116.stdout:7/3: write f0 [204571,122859] 0 2024-03-04T16:04:02.976 INFO:tasks.workunit.client.0.smithi116.stdout:7/4: rename f0 to f2 0 2024-03-04T16:04:02.977 INFO:tasks.workunit.client.0.smithi116.stdout:6/39: dwrite d6/fa [0,4194304] 0 2024-03-04T16:04:02.983 INFO:tasks.workunit.client.0.smithi116.stdout:0/19: dwrite f2 [4194304,4194304] 0 2024-03-04T16:04:02.986 INFO:tasks.workunit.client.0.smithi116.stdout:6/40: dwrite d6/f8 [0,4194304] 0 2024-03-04T16:04:02.987 INFO:tasks.workunit.client.0.smithi116.stdout:6/41: write d6/fa [2674566,29353] 0 2024-03-04T16:04:02.996 INFO:tasks.workunit.client.0.smithi116.stdout:0/20: mknod c4 0 2024-03-04T16:04:03.000 INFO:tasks.workunit.client.0.smithi116.stdout:6/42: mknod d6/cc 0 2024-03-04T16:04:03.002 INFO:tasks.workunit.client.0.smithi116.stdout:0/21: readlink l3 0 2024-03-04T16:04:03.005 INFO:tasks.workunit.client.0.smithi116.stdout:6/43: rename c1 to d6/cd 0 2024-03-04T16:04:03.006 INFO:tasks.workunit.client.0.smithi116.stdout:0/22: mknod c5 0 2024-03-04T16:04:03.007 INFO:tasks.workunit.client.0.smithi116.stdout:2/30: dwrite f4 [4194304,4194304] 0 2024-03-04T16:04:03.008 INFO:tasks.workunit.client.0.smithi116.stdout:2/31: fsync f6 0 2024-03-04T16:04:03.008 INFO:tasks.workunit.client.0.smithi116.stdout:0/23: write f2 [2940025,31257] 0 2024-03-04T16:04:03.008 INFO:tasks.workunit.client.0.smithi116.stdout:0/24: stat c4 0 2024-03-04T16:04:03.010 INFO:tasks.workunit.client.0.smithi116.stdout:4/26: fsync d1/f2 0 2024-03-04T16:04:03.018 INFO:tasks.workunit.client.0.smithi116.stdout:0/25: rename c4 to c6 0 2024-03-04T16:04:03.021 INFO:tasks.workunit.client.0.smithi116.stdout:8/18: unlink d2/f6 0 2024-03-04T16:04:03.021 INFO:tasks.workunit.client.0.smithi116.stdout:8/19: fsync - no filename 2024-03-04T16:04:03.021 INFO:tasks.workunit.client.0.smithi116.stdout:8/20: write - no filename 2024-03-04T16:04:03.021 INFO:tasks.workunit.client.0.smithi116.stdout:8/21: read - no filename 2024-03-04T16:04:03.021 INFO:tasks.workunit.client.0.smithi116.stdout:8/22: stat d2/d3 0 2024-03-04T16:04:03.021 INFO:tasks.workunit.client.0.smithi116.stdout:8/23: readlink d2/d3/l5 0 2024-03-04T16:04:03.021 INFO:tasks.workunit.client.0.smithi116.stdout:8/24: dread - no filename 2024-03-04T16:04:03.024 INFO:tasks.workunit.client.0.smithi116.stdout:0/26: creat f7 x:0 0 0 2024-03-04T16:04:03.026 INFO:tasks.workunit.client.0.smithi116.stdout:8/25: creat d2/f7 x:0 0 0 2024-03-04T16:04:03.027 INFO:tasks.workunit.client.0.smithi116.stdout:8/26: write d2/f7 [771578,54] 0 2024-03-04T16:04:03.027 INFO:tasks.workunit.client.0.smithi116.stdout:2/32: fsync f4 0 2024-03-04T16:04:03.027 INFO:tasks.workunit.client.0.smithi116.stdout:2/33: dread - f6 zero size 2024-03-04T16:04:03.027 INFO:tasks.workunit.client.0.smithi116.stdout:0/27: unlink c5 0 2024-03-04T16:04:03.030 INFO:tasks.workunit.client.0.smithi116.stdout:8/27: dread d2/f7 [0,4194304] 0 2024-03-04T16:04:03.032 INFO:tasks.workunit.client.0.smithi116.stdout:2/34: creat d7/fb x:0 0 0 2024-03-04T16:04:03.033 INFO:tasks.workunit.client.0.smithi116.stdout:1/14: sync 2024-03-04T16:04:03.033 INFO:tasks.workunit.client.0.smithi116.stdout:9/20: rename d2/f5 to d2/f6 0 2024-03-04T16:04:03.034 INFO:tasks.workunit.client.0.smithi116.stdout:1/15: dread - f4 zero size 2024-03-04T16:04:03.034 INFO:tasks.workunit.client.0.smithi116.stdout:1/16: fdatasync f4 0 2024-03-04T16:04:03.036 INFO:tasks.workunit.client.0.smithi116.stdout:4/27: getdents d1 0 2024-03-04T16:04:03.037 INFO:tasks.workunit.client.0.smithi116.stdout:8/28: creat d2/d3/f8 x:0 0 0 2024-03-04T16:04:03.038 INFO:tasks.workunit.client.0.smithi116.stdout:4/28: write d1/d7/f9 [638327,15900] 0 2024-03-04T16:04:03.038 INFO:tasks.workunit.client.0.smithi116.stdout:8/29: dread - d2/d3/f8 zero size 2024-03-04T16:04:03.040 INFO:tasks.workunit.client.0.smithi116.stdout:1/17: dwrite f2 [0,4194304] 0 2024-03-04T16:04:03.042 INFO:tasks.workunit.client.0.smithi116.stdout:1/18: write f4 [710636,102316] 0 2024-03-04T16:04:03.043 INFO:tasks.workunit.client.0.smithi116.stdout:2/35: mknod d7/cc 0 2024-03-04T16:04:03.045 INFO:tasks.workunit.client.0.smithi116.stdout:9/21: symlink d2/l7 0 2024-03-04T16:04:03.052 INFO:tasks.workunit.client.0.smithi116.stdout:4/29: mknod d1/d7/ca 0 2024-03-04T16:04:03.052 INFO:tasks.workunit.client.0.smithi116.stdout:8/30: symlink d2/d3/l9 0 2024-03-04T16:04:03.052 INFO:tasks.workunit.client.0.smithi116.stdout:5/18: truncate f4 640108 0 2024-03-04T16:04:03.057 INFO:tasks.workunit.client.0.smithi116.stdout:8/31: creat d2/d3/fa x:0 0 0 2024-03-04T16:04:03.058 INFO:tasks.workunit.client.0.smithi116.stdout:1/19: creat d5/f6 x:0 0 0 2024-03-04T16:04:03.061 INFO:tasks.workunit.client.0.smithi116.stdout:5/19: mknod c6 0 2024-03-04T16:04:03.062 INFO:tasks.workunit.client.0.smithi116.stdout:1/20: dread f4 [0,4194304] 0 2024-03-04T16:04:03.064 INFO:tasks.workunit.client.0.smithi116.stdout:2/36: unlink d7/c8 0 2024-03-04T16:04:03.066 INFO:tasks.workunit.client.0.smithi116.stdout:8/32: creat d2/d3/fb x:0 0 0 2024-03-04T16:04:03.066 INFO:tasks.workunit.client.0.smithi116.stdout:2/37: dread f4 [4194304,4194304] 0 2024-03-04T16:04:03.067 INFO:tasks.workunit.client.0.smithi116.stdout:2/38: write d7/fb [417252,32254] 0 2024-03-04T16:04:03.070 INFO:tasks.workunit.client.0.smithi116.stdout:5/20: creat f7 x:0 0 0 2024-03-04T16:04:03.075 INFO:tasks.workunit.client.0.smithi116.stdout:2/39: rename d7/c9 to d7/cd 0 2024-03-04T16:04:03.077 INFO:tasks.workunit.client.0.smithi116.stdout:5/21: creat f8 x:0 0 0 2024-03-04T16:04:03.080 INFO:tasks.workunit.client.0.smithi116.stdout:1/21: rename c3 to d5/c7 0 2024-03-04T16:04:03.080 INFO:tasks.workunit.client.0.smithi116.stdout:8/33: dwrite d2/f7 [0,4194304] 0 2024-03-04T16:04:03.084 INFO:tasks.workunit.client.0.smithi116.stdout:2/40: dwrite f6 [0,4194304] 0 2024-03-04T16:04:03.086 INFO:tasks.workunit.client.0.smithi116.stdout:1/22: rename d5/f6 to d5/f8 0 2024-03-04T16:04:03.091 INFO:tasks.workunit.client.0.smithi116.stdout:1/23: write d5/f8 [360099,16907] 0 2024-03-04T16:04:03.091 INFO:tasks.workunit.client.0.smithi116.stdout:2/41: dwrite f4 [4194304,4194304] 0 2024-03-04T16:04:03.092 INFO:tasks.workunit.client.0.smithi116.stdout:2/42: stat d7/cc 0 2024-03-04T16:04:03.097 INFO:tasks.workunit.client.0.smithi116.stdout:1/24: mkdir d5/d9 0 2024-03-04T16:04:03.097 INFO:tasks.workunit.client.0.smithi116.stdout:1/25: write d5/f8 [660927,85929] 0 2024-03-04T16:04:03.171 INFO:tasks.workunit.client.0.smithi116.stdout:3/53: sync 2024-03-04T16:04:03.173 INFO:tasks.workunit.client.0.smithi116.stdout:3/54: dread f5 [0,4194304] 0 2024-03-04T16:04:03.176 INFO:tasks.workunit.client.0.smithi116.stdout:3/55: rename l6 to dc/df/l13 0 2024-03-04T16:04:03.178 INFO:tasks.workunit.client.0.smithi116.stdout:3/56: mkdir dc/df/d14 0 2024-03-04T16:04:03.180 INFO:tasks.workunit.client.0.smithi116.stdout:3/57: mkdir dc/d15 0 2024-03-04T16:04:03.182 INFO:tasks.workunit.client.0.smithi116.stdout:3/58: dread f2 [0,4194304] 0 2024-03-04T16:04:03.183 INFO:tasks.workunit.client.0.smithi116.stdout:3/59: readlink dc/ld 0 2024-03-04T16:04:03.185 INFO:tasks.workunit.client.0.smithi116.stdout:3/60: rename f5 to dc/f16 0 2024-03-04T16:04:03.196 INFO:tasks.workunit.client.0.smithi116.stdout:7/5: fsync f2 0 2024-03-04T16:04:03.197 INFO:tasks.workunit.client.0.smithi116.stdout:4/30: sync 2024-03-04T16:04:03.200 INFO:tasks.workunit.client.0.smithi116.stdout:4/31: creat d1/fb x:0 0 0 2024-03-04T16:04:03.203 INFO:tasks.workunit.client.0.smithi116.stdout:7/6: dwrite f1 [0,4194304] 0 2024-03-04T16:04:03.205 INFO:tasks.workunit.client.0.smithi116.stdout:4/32: unlink d1/d3/f4 0 2024-03-04T16:04:03.206 INFO:tasks.workunit.client.0.smithi116.stdout:4/33: chown d1 10 1 2024-03-04T16:04:03.206 INFO:tasks.workunit.client.0.smithi116.stdout:4/34: truncate f0 4776836 0 2024-03-04T16:04:03.207 INFO:tasks.workunit.client.0.smithi116.stdout:7/7: mkdir d3 0 2024-03-04T16:04:03.213 INFO:tasks.workunit.client.0.smithi116.stdout:4/35: creat d1/d7/fc x:0 0 0 2024-03-04T16:04:03.214 INFO:tasks.workunit.client.0.smithi116.stdout:4/36: write d1/d7/f9 [1463489,123661] 0 2024-03-04T16:04:03.216 INFO:tasks.workunit.client.0.smithi116.stdout:6/44: write f3 [759007,86920] 0 2024-03-04T16:04:03.217 INFO:tasks.workunit.client.0.smithi116.stdout:7/8: mkdir d3/d4 0 2024-03-04T16:04:03.218 INFO:tasks.workunit.client.0.smithi116.stdout:6/45: write f3 [1434305,41009] 0 2024-03-04T16:04:03.222 INFO:tasks.workunit.client.0.smithi116.stdout:7/9: creat d3/f5 x:0 0 0 2024-03-04T16:04:03.225 INFO:tasks.workunit.client.0.smithi116.stdout:6/46: dread f3 [0,4194304] 0 2024-03-04T16:04:03.228 INFO:tasks.workunit.client.0.smithi116.stdout:0/28: rename c6 to c8 0 2024-03-04T16:04:03.228 INFO:tasks.workunit.client.0.smithi116.stdout:0/29: dread - f7 zero size 2024-03-04T16:04:03.228 INFO:tasks.workunit.client.0.smithi116.stdout:0/30: chown f7 17 1 2024-03-04T16:04:03.229 INFO:tasks.workunit.client.0.smithi116.stdout:0/31: dread - f7 zero size 2024-03-04T16:04:03.229 INFO:tasks.workunit.client.0.smithi116.stdout:0/32: write f2 [2040983,86810] 0 2024-03-04T16:04:03.232 INFO:tasks.workunit.client.0.smithi116.stdout:6/47: unlink d6/l7 0 2024-03-04T16:04:03.232 INFO:tasks.workunit.client.0.smithi116.stdout:2/43: fsync d7/fb 0 2024-03-04T16:04:03.236 INFO:tasks.workunit.client.0.smithi116.stdout:0/33: dwrite f7 [0,4194304] 0 2024-03-04T16:04:03.236 INFO:tasks.workunit.client.0.smithi116.stdout:6/48: mkdir d6/de 0 2024-03-04T16:04:03.236 INFO:tasks.workunit.client.0.smithi116.stdout:0/34: chown f7 4 1 2024-03-04T16:04:03.237 INFO:tasks.workunit.client.0.smithi116.stdout:0/35: write f7 [3003048,48404] 0 2024-03-04T16:04:03.241 INFO:tasks.workunit.client.0.smithi116.stdout:2/44: mknod d7/ce 0 2024-03-04T16:04:03.244 INFO:tasks.workunit.client.0.smithi116.stdout:9/22: rmdir d2 39 2024-03-04T16:04:03.245 INFO:tasks.workunit.client.0.smithi116.stdout:6/49: mknod d6/cf 0 2024-03-04T16:04:03.248 INFO:tasks.workunit.client.0.smithi116.stdout:0/36: mknod c9 0 2024-03-04T16:04:03.251 INFO:tasks.workunit.client.0.smithi116.stdout:2/45: creat d7/ff x:0 0 0 2024-03-04T16:04:03.256 INFO:tasks.workunit.client.0.smithi116.stdout:5/22: dwrite f4 [0,4194304] 0 2024-03-04T16:04:03.257 INFO:tasks.workunit.client.0.smithi116.stdout:2/46: creat d7/f10 x:0 0 0 2024-03-04T16:04:03.257 INFO:tasks.workunit.client.0.smithi116.stdout:2/47: readlink l3 0 2024-03-04T16:04:03.258 INFO:tasks.workunit.client.0.smithi116.stdout:2/48: readlink l3 0 2024-03-04T16:04:03.258 INFO:tasks.workunit.client.0.smithi116.stdout:9/23: mkdir d2/d8 0 2024-03-04T16:04:03.263 INFO:tasks.workunit.client.0.smithi116.stdout:8/34: getdents d2/d3 0 2024-03-04T16:04:03.263 INFO:tasks.workunit.client.0.smithi116.stdout:8/35: fsync d2/d3/fb 0 2024-03-04T16:04:03.270 INFO:tasks.workunit.client.0.smithi116.stdout:2/49: symlink d7/l11 0 2024-03-04T16:04:03.271 INFO:tasks.workunit.client.0.smithi116.stdout:2/50: write f1 [875755,54980] 0 2024-03-04T16:04:03.275 INFO:tasks.workunit.client.0.smithi116.stdout:8/36: creat d2/d3/fc x:0 0 0 2024-03-04T16:04:03.279 INFO:tasks.workunit.client.0.smithi116.stdout:2/51: mknod d7/c12 0 2024-03-04T16:04:03.281 INFO:tasks.workunit.client.0.smithi116.stdout:2/52: readlink l3 0 2024-03-04T16:04:03.281 INFO:tasks.workunit.client.0.smithi116.stdout:9/24: dwrite d2/f6 [0,4194304] 0 2024-03-04T16:04:03.282 INFO:tasks.workunit.client.0.smithi116.stdout:9/25: chown d2 33 1 2024-03-04T16:04:03.284 INFO:tasks.workunit.client.0.smithi116.stdout:8/37: dwrite d2/d3/fc [0,4194304] 0 2024-03-04T16:04:03.288 INFO:tasks.workunit.client.0.smithi116.stdout:1/26: link d5/c7 d5/d9/ca 0 2024-03-04T16:04:03.292 INFO:tasks.workunit.client.0.smithi116.stdout:8/38: creat d2/fd x:0 0 0 2024-03-04T16:04:03.293 INFO:tasks.workunit.client.0.smithi116.stdout:8/39: dread - d2/d3/fb zero size 2024-03-04T16:04:03.303 INFO:tasks.workunit.client.0.smithi116.stdout:1/27: creat d5/d9/fb x:0 0 0 2024-03-04T16:04:03.315 INFO:tasks.workunit.client.0.smithi116.stdout:3/61: truncate dc/f16 2220145 0 2024-03-04T16:04:03.315 INFO:tasks.workunit.client.0.smithi116.stdout:3/62: write fa [4702395,48903] 0 2024-03-04T16:04:03.318 INFO:tasks.workunit.client.0.smithi116.stdout:3/63: chown dc/df/l12 214164 1 2024-03-04T16:04:03.320 INFO:tasks.workunit.client.0.smithi116.stdout:3/64: rmdir dc 39 2024-03-04T16:04:03.326 INFO:tasks.workunit.client.0.smithi116.stdout:3/65: mkdir dc/df/d14/d17 0 2024-03-04T16:04:03.326 INFO:tasks.workunit.client.0.smithi116.stdout:3/66: fsync fa 0 2024-03-04T16:04:03.329 INFO:tasks.workunit.client.0.smithi116.stdout:4/37: rename d1/d7 to d1/d3/dd 0 2024-03-04T16:04:03.329 INFO:tasks.workunit.client.0.smithi116.stdout:1/28: rename d5 to d5/d9/dc 22 2024-03-04T16:04:03.330 INFO:tasks.workunit.client.0.smithi116.stdout:1/29: readlink - no filename 2024-03-04T16:04:03.333 INFO:tasks.workunit.client.0.smithi116.stdout:1/30: mknod d5/d9/cd 0 2024-03-04T16:04:03.334 INFO:tasks.workunit.client.0.smithi116.stdout:7/10: getdents d3 0 2024-03-04T16:04:03.336 INFO:tasks.workunit.client.0.smithi116.stdout:4/38: dwrite d1/fb [0,4194304] 0 2024-03-04T16:04:03.340 INFO:tasks.workunit.client.0.smithi116.stdout:3/67: rmdir dc/d15 0 2024-03-04T16:04:03.343 INFO:tasks.workunit.client.0.smithi116.stdout:7/11: dwrite f2 [0,4194304] 0 2024-03-04T16:04:03.346 INFO:tasks.workunit.client.0.smithi116.stdout:7/12: dread f1 [0,4194304] 0 2024-03-04T16:04:03.348 INFO:tasks.workunit.client.0.smithi116.stdout:6/50: getdents d6 0 2024-03-04T16:04:03.356 INFO:tasks.workunit.client.0.smithi116.stdout:3/68: dwrite f2 [0,4194304] 0 2024-03-04T16:04:03.360 INFO:tasks.workunit.client.0.smithi116.stdout:7/13: dwrite f2 [0,4194304] 0 2024-03-04T16:04:03.360 INFO:tasks.workunit.client.0.smithi116.stdout:3/69: write fa [1054332,57351] 0 2024-03-04T16:04:03.361 INFO:tasks.workunit.client.0.smithi116.stdout:5/23: write f4 [4944689,89223] 0 2024-03-04T16:04:03.362 INFO:tasks.workunit.client.0.smithi116.stdout:3/70: stat fa 0 2024-03-04T16:04:03.362 INFO:tasks.workunit.client.0.smithi116.stdout:0/37: truncate f2 2502472 0 2024-03-04T16:04:03.363 INFO:tasks.workunit.client.0.smithi116.stdout:7/14: write f1 [924615,82052] 0 2024-03-04T16:04:03.364 INFO:tasks.workunit.client.0.smithi116.stdout:6/51: dwrite d6/f8 [4194304,4194304] 0 2024-03-04T16:04:03.364 INFO:tasks.workunit.client.0.smithi116.stdout:5/24: write f5 [555118,101566] 0 2024-03-04T16:04:03.366 INFO:tasks.workunit.client.0.smithi116.stdout:0/38: mkdir da 0 2024-03-04T16:04:03.373 INFO:tasks.workunit.client.0.smithi116.stdout:8/40: rename d2/d3 to d2/de 0 2024-03-04T16:04:03.377 INFO:tasks.workunit.client.0.smithi116.stdout:7/15: creat d3/f6 x:0 0 0 2024-03-04T16:04:03.377 INFO:tasks.workunit.client.0.smithi116.stdout:8/41: write d2/de/fa [987205,66076] 0 2024-03-04T16:04:03.384 INFO:tasks.workunit.client.0.smithi116.stdout:3/71: rename dc/df/l13 to dc/l18 0 2024-03-04T16:04:03.386 INFO:tasks.workunit.client.0.smithi116.stdout:2/53: truncate f6 576333 0 2024-03-04T16:04:03.387 INFO:tasks.workunit.client.0.smithi116.stdout:2/54: read f1 [272085,22844] 0 2024-03-04T16:04:03.388 INFO:tasks.workunit.client.0.smithi116.stdout:9/26: truncate d2/f6 3762335 0 2024-03-04T16:04:03.388 INFO:tasks.workunit.client.0.smithi116.stdout:9/27: chown d2/d8 84656 1 2024-03-04T16:04:03.389 INFO:tasks.workunit.client.0.smithi116.stdout:8/42: symlink d2/lf 0 2024-03-04T16:04:03.391 INFO:tasks.workunit.client.0.smithi116.stdout:3/72: dwrite fa [0,4194304] 0 2024-03-04T16:04:03.391 INFO:tasks.workunit.client.0.smithi116.stdout:6/52: mknod d6/de/c10 0 2024-03-04T16:04:03.392 INFO:tasks.workunit.client.0.smithi116.stdout:5/25: dread f5 [0,4194304] 0 2024-03-04T16:04:03.399 INFO:tasks.workunit.client.0.smithi116.stdout:9/28: symlink d2/l9 0 2024-03-04T16:04:03.401 INFO:tasks.workunit.client.0.smithi116.stdout:8/43: creat d2/de/f10 x:0 0 0 2024-03-04T16:04:03.404 INFO:tasks.workunit.client.0.smithi116.stdout:6/53: fsync f3 0 2024-03-04T16:04:03.405 INFO:tasks.workunit.client.0.smithi116.stdout:6/54: write d6/f9 [333501,28159] 0 2024-03-04T16:04:03.405 INFO:tasks.workunit.client.0.smithi116.stdout:6/55: fsync d6/fa 0 2024-03-04T16:04:03.406 INFO:tasks.workunit.client.0.smithi116.stdout:6/56: write d6/f9 [1397426,12004] 0 2024-03-04T16:04:03.409 INFO:tasks.workunit.client.0.smithi116.stdout:5/26: unlink f7 0 2024-03-04T16:04:03.415 INFO:tasks.workunit.client.0.smithi116.stdout:8/44: mknod d2/c11 0 2024-03-04T16:04:03.417 INFO:tasks.workunit.client.0.smithi116.stdout:1/31: truncate d5/f8 7082 0 2024-03-04T16:04:03.418 INFO:tasks.workunit.client.0.smithi116.stdout:4/39: write d1/f2 [2262526,63174] 0 2024-03-04T16:04:03.420 INFO:tasks.workunit.client.0.smithi116.stdout:3/73: symlink dc/df/d14/d17/l19 0 2024-03-04T16:04:03.421 INFO:tasks.workunit.client.0.smithi116.stdout:6/57: creat d6/de/f11 x:0 0 0 2024-03-04T16:04:03.422 INFO:tasks.workunit.client.0.smithi116.stdout:6/58: chown d6/fa 92130451 1 2024-03-04T16:04:03.423 INFO:tasks.workunit.client.0.smithi116.stdout:3/74: dread fa [0,4194304] 0 2024-03-04T16:04:03.424 INFO:tasks.workunit.client.0.smithi116.stdout:5/27: mknod c9 0 2024-03-04T16:04:03.426 INFO:tasks.workunit.client.0.smithi116.stdout:0/39: fsync f2 0 2024-03-04T16:04:03.428 INFO:tasks.workunit.client.0.smithi116.stdout:5/28: dread f4 [0,4194304] 0 2024-03-04T16:04:03.431 INFO:tasks.workunit.client.0.smithi116.stdout:4/40: mkdir d1/d3/de 0 2024-03-04T16:04:03.432 INFO:tasks.workunit.client.0.smithi116.stdout:4/41: truncate d1/d3/dd/fc 662674 0 2024-03-04T16:04:03.434 INFO:tasks.workunit.client.0.smithi116.stdout:7/16: rmdir d3 39 2024-03-04T16:04:03.439 INFO:tasks.workunit.client.0.smithi116.stdout:2/55: dread f6 [0,4194304] 0 2024-03-04T16:04:03.441 INFO:tasks.workunit.client.0.smithi116.stdout:8/45: link d2/f7 d2/de/f12 0 2024-03-04T16:04:03.442 INFO:tasks.workunit.client.0.smithi116.stdout:8/46: read d2/de/f12 [74944,49078] 0 2024-03-04T16:04:03.444 INFO:tasks.workunit.client.0.smithi116.stdout:5/29: rename f8 to fa 0 2024-03-04T16:04:03.445 INFO:tasks.workunit.client.0.smithi116.stdout:5/30: read - fa zero size 2024-03-04T16:04:03.446 INFO:tasks.workunit.client.0.smithi116.stdout:9/29: dwrite d2/f6 [0,4194304] 0 2024-03-04T16:04:03.447 INFO:tasks.workunit.client.0.smithi116.stdout:5/31: dread f5 [0,4194304] 0 2024-03-04T16:04:03.447 INFO:tasks.workunit.client.0.smithi116.stdout:9/30: read d2/f6 [2737452,107280] 0 2024-03-04T16:04:03.448 INFO:tasks.workunit.client.0.smithi116.stdout:4/42: mknod d1/d3/dd/cf 0 2024-03-04T16:04:03.451 INFO:tasks.workunit.client.0.smithi116.stdout:7/17: truncate d3/f6 75311 0 2024-03-04T16:04:03.461 INFO:tasks.workunit.client.0.smithi116.stdout:2/56: mkdir d7/d13 0 2024-03-04T16:04:03.465 INFO:tasks.workunit.client.0.smithi116.stdout:1/32: read d5/f8 [3766,68892] 0 2024-03-04T16:04:03.466 INFO:tasks.workunit.client.0.smithi116.stdout:2/57: dwrite d7/ff [0,4194304] 0 2024-03-04T16:04:03.468 INFO:tasks.workunit.client.0.smithi116.stdout:8/47: mkdir d2/d13 0 2024-03-04T16:04:03.469 INFO:tasks.workunit.client.0.smithi116.stdout:5/32: write f5 [402797,11438] 0 2024-03-04T16:04:03.472 INFO:tasks.workunit.client.0.smithi116.stdout:9/31: rename d2/l9 to d2/la 0 2024-03-04T16:04:03.476 INFO:tasks.workunit.client.0.smithi116.stdout:5/33: dwrite f5 [0,4194304] 0 2024-03-04T16:04:03.480 INFO:tasks.workunit.client.0.smithi116.stdout:4/43: rename d1/f8 to d1/d3/de/f10 0 2024-03-04T16:04:03.481 INFO:tasks.workunit.client.0.smithi116.stdout:7/18: symlink d3/l7 0 2024-03-04T16:04:03.481 INFO:tasks.workunit.client.0.smithi116.stdout:0/40: dwrite f2 [0,4194304] 0 2024-03-04T16:04:03.497 INFO:tasks.workunit.client.0.smithi116.stdout:6/59: getdents d6/de 0 2024-03-04T16:04:03.498 INFO:tasks.workunit.client.0.smithi116.stdout:6/60: write d6/de/f11 [447697,90288] 0 2024-03-04T16:04:03.500 INFO:tasks.workunit.client.0.smithi116.stdout:3/75: truncate f2 1478780 0 2024-03-04T16:04:03.503 INFO:tasks.workunit.client.0.smithi116.stdout:2/58: rename d7/ff to d7/d13/f14 0 2024-03-04T16:04:03.504 INFO:tasks.workunit.client.0.smithi116.stdout:4/44: mkdir d1/d3/de/d11 0 2024-03-04T16:04:03.504 INFO:tasks.workunit.client.0.smithi116.stdout:2/59: chown d7/ca 96677 1 2024-03-04T16:04:03.505 INFO:tasks.workunit.client.0.smithi116.stdout:4/45: chown f0 123576 1 2024-03-04T16:04:03.506 INFO:tasks.workunit.client.0.smithi116.stdout:2/60: write d7/fb [988646,440] 0 2024-03-04T16:04:03.506 INFO:tasks.workunit.client.0.smithi116.stdout:6/61: dwrite d6/fa [0,4194304] 0 2024-03-04T16:04:03.507 INFO:tasks.workunit.client.0.smithi116.stdout:2/61: stat d7/l11 0 2024-03-04T16:04:03.507 INFO:tasks.workunit.client.0.smithi116.stdout:8/48: symlink d2/d13/l14 0 2024-03-04T16:04:03.509 INFO:tasks.workunit.client.0.smithi116.stdout:9/32: symlink d2/d8/lb 0 2024-03-04T16:04:03.511 INFO:tasks.workunit.client.0.smithi116.stdout:1/33: write d5/f8 [17913,36581] 0 2024-03-04T16:04:03.513 INFO:tasks.workunit.client.0.smithi116.stdout:1/34: chown f2 14974560 1 2024-03-04T16:04:03.515 INFO:tasks.workunit.client.0.smithi116.stdout:0/41: symlink da/lb 0 2024-03-04T16:04:03.516 INFO:tasks.workunit.client.0.smithi116.stdout:6/62: dwrite d6/f9 [0,4194304] 0 2024-03-04T16:04:03.517 INFO:tasks.workunit.client.0.smithi116.stdout:0/42: fsync f2 0 2024-03-04T16:04:03.518 INFO:tasks.workunit.client.0.smithi116.stdout:4/46: creat d1/d3/de/f12 x:0 0 0 2024-03-04T16:04:03.520 INFO:tasks.workunit.client.0.smithi116.stdout:8/49: dwrite d2/de/fa [0,4194304] 0 2024-03-04T16:04:03.520 INFO:tasks.workunit.client.0.smithi116.stdout:0/43: write f7 [5030212,18432] 0 2024-03-04T16:04:03.522 INFO:tasks.workunit.client.0.smithi116.stdout:8/50: write d2/fd [165394,49180] 0 2024-03-04T16:04:03.522 INFO:tasks.workunit.client.0.smithi116.stdout:4/47: write d1/fb [2908494,180] 0 2024-03-04T16:04:03.523 INFO:tasks.workunit.client.0.smithi116.stdout:2/62: mkdir d7/d15 0 2024-03-04T16:04:03.525 INFO:tasks.workunit.client.0.smithi116.stdout:3/76: dread dc/f16 [0,4194304] 0 2024-03-04T16:04:03.531 INFO:tasks.workunit.client.0.smithi116.stdout:8/51: dwrite d2/de/f10 [0,4194304] 0 2024-03-04T16:04:03.532 INFO:tasks.workunit.client.0.smithi116.stdout:6/63: symlink d6/l12 0 2024-03-04T16:04:03.537 INFO:tasks.workunit.client.0.smithi116.stdout:0/44: mknod da/cc 0 2024-03-04T16:04:03.538 INFO:tasks.workunit.client.0.smithi116.stdout:4/48: creat d1/d3/de/f13 x:0 0 0 2024-03-04T16:04:03.539 INFO:tasks.workunit.client.0.smithi116.stdout:0/45: chown da/lb 27 1 2024-03-04T16:04:03.541 INFO:tasks.workunit.client.0.smithi116.stdout:4/49: stat d1/d3/de/d11 0 2024-03-04T16:04:03.543 INFO:tasks.workunit.client.0.smithi116.stdout:1/35: rename d5/d9/ca to d5/d9/ce 0 2024-03-04T16:04:03.544 INFO:tasks.workunit.client.0.smithi116.stdout:6/64: dwrite d6/f8 [8388608,4194304] 0 2024-03-04T16:04:03.544 INFO:tasks.workunit.client.0.smithi116.stdout:5/34: dwrite f4 [0,4194304] 0 2024-03-04T16:04:03.547 INFO:tasks.workunit.client.0.smithi116.stdout:3/77: creat dc/df/d14/f1a x:0 0 0 2024-03-04T16:04:03.548 INFO:tasks.workunit.client.0.smithi116.stdout:3/78: truncate dc/df/d14/f1a 752738 0 2024-03-04T16:04:03.551 INFO:tasks.workunit.client.0.smithi116.stdout:8/52: creat d2/de/f15 x:0 0 0 2024-03-04T16:04:03.559 INFO:tasks.workunit.client.0.smithi116.stdout:0/46: creat da/fd x:0 0 0 2024-03-04T16:04:03.559 INFO:tasks.workunit.client.0.smithi116.stdout:0/47: write f2 [4856600,338] 0 2024-03-04T16:04:03.560 INFO:tasks.workunit.client.0.smithi116.stdout:0/48: write da/fd [843640,4796] 0 2024-03-04T16:04:03.561 INFO:tasks.workunit.client.0.smithi116.stdout:0/49: chown da/fd 336294436 1 2024-03-04T16:04:03.561 INFO:tasks.workunit.client.0.smithi116.stdout:5/35: mknod cb 0 2024-03-04T16:04:03.566 INFO:tasks.workunit.client.0.smithi116.stdout:1/36: creat d5/d9/ff x:0 0 0 2024-03-04T16:04:03.567 INFO:tasks.workunit.client.0.smithi116.stdout:3/79: unlink dc/df/d14/d17/l19 0 2024-03-04T16:04:03.571 INFO:tasks.workunit.client.0.smithi116.stdout:2/63: link d7/cc d7/d15/c16 0 2024-03-04T16:04:03.572 INFO:tasks.workunit.client.0.smithi116.stdout:4/50: mknod d1/d3/de/d11/c14 0 2024-03-04T16:04:03.573 INFO:tasks.workunit.client.0.smithi116.stdout:0/50: creat da/fe x:0 0 0 2024-03-04T16:04:03.574 INFO:tasks.workunit.client.0.smithi116.stdout:0/51: write da/fd [959961,68444] 0 2024-03-04T16:04:03.577 INFO:tasks.workunit.client.0.smithi116.stdout:1/37: creat d5/f10 x:0 0 0 2024-03-04T16:04:03.578 INFO:tasks.workunit.client.0.smithi116.stdout:3/80: creat dc/df/d14/d17/f1b x:0 0 0 2024-03-04T16:04:03.580 INFO:tasks.workunit.client.0.smithi116.stdout:2/64: rmdir d7/d15 39 2024-03-04T16:04:03.581 INFO:tasks.workunit.client.0.smithi116.stdout:4/51: mkdir d1/d3/d15 0 2024-03-04T16:04:03.587 INFO:tasks.workunit.client.0.smithi116.stdout:1/38: mkdir d5/d11 0 2024-03-04T16:04:03.587 INFO:tasks.workunit.client.0.smithi116.stdout:6/65: write f2 [721987,29574] 0 2024-03-04T16:04:03.588 INFO:tasks.workunit.client.0.smithi116.stdout:6/66: chown d6/cc 124564 1 2024-03-04T16:04:03.588 INFO:tasks.workunit.client.0.smithi116.stdout:6/67: chown d6/de 0 1 2024-03-04T16:04:03.589 INFO:tasks.workunit.client.0.smithi116.stdout:3/81: mkdir dc/df/d14/d17/d1c 0 2024-03-04T16:04:03.591 INFO:tasks.workunit.client.0.smithi116.stdout:8/53: write d2/de/f12 [2062274,10582] 0 2024-03-04T16:04:03.592 INFO:tasks.workunit.client.0.smithi116.stdout:2/65: rename d7/ce to d7/d13/c17 0 2024-03-04T16:04:03.593 INFO:tasks.workunit.client.0.smithi116.stdout:8/54: write d2/fd [594102,31987] 0 2024-03-04T16:04:03.593 INFO:tasks.workunit.client.0.smithi116.stdout:8/55: readlink d2/lf 0 2024-03-04T16:04:03.594 INFO:tasks.workunit.client.0.smithi116.stdout:4/52: unlink d1/d3/dd/ca 0 2024-03-04T16:04:03.595 INFO:tasks.workunit.client.0.smithi116.stdout:6/68: dwrite d6/fa [0,4194304] 0 2024-03-04T16:04:03.596 INFO:tasks.workunit.client.0.smithi116.stdout:6/69: fsync d6/f8 0 2024-03-04T16:04:03.599 INFO:tasks.workunit.client.0.smithi116.stdout:1/39: unlink d5/d9/cd 0 2024-03-04T16:04:03.601 INFO:tasks.workunit.client.0.smithi116.stdout:4/53: dwrite d1/f2 [0,4194304] 0 2024-03-04T16:04:03.601 INFO:tasks.workunit.client.0.smithi116.stdout:4/54: dread - d1/f5 zero size 2024-03-04T16:04:03.602 INFO:tasks.workunit.client.0.smithi116.stdout:3/82: stat cb 0 2024-03-04T16:04:03.603 INFO:tasks.workunit.client.0.smithi116.stdout:3/83: truncate dc/df/d14/d17/f1b 225330 0 2024-03-04T16:04:03.604 INFO:tasks.workunit.client.0.smithi116.stdout:3/84: rename dc/df/d14/d17/d1c to dc/df/d14/d17/d1c/d1d 22 2024-03-04T16:04:03.605 INFO:tasks.workunit.client.0.smithi116.stdout:3/85: fsync dc/df/d14/f1a 0 2024-03-04T16:04:03.606 INFO:tasks.workunit.client.0.smithi116.stdout:2/66: creat d7/d13/f18 x:0 0 0 2024-03-04T16:04:03.607 INFO:tasks.workunit.client.0.smithi116.stdout:8/56: mkdir d2/de/d16 0 2024-03-04T16:04:03.613 INFO:tasks.workunit.client.0.smithi116.stdout:6/70: rmdir d6 39 2024-03-04T16:04:03.614 INFO:tasks.workunit.client.0.smithi116.stdout:6/71: write f2 [1107241,8020] 0 2024-03-04T16:04:03.616 INFO:tasks.workunit.client.0.smithi116.stdout:5/36: getdents . 0 2024-03-04T16:04:03.618 INFO:tasks.workunit.client.0.smithi116.stdout:4/55: creat d1/d3/de/f16 x:0 0 0 2024-03-04T16:04:03.622 INFO:tasks.workunit.client.0.smithi116.stdout:5/37: dwrite f4 [4194304,4194304] 0 2024-03-04T16:04:03.623 INFO:tasks.workunit.client.0.smithi116.stdout:0/52: truncate f7 3177349 0 2024-03-04T16:04:03.628 INFO:tasks.workunit.client.0.smithi116.stdout:5/38: dwrite f4 [0,4194304] 0 2024-03-04T16:04:03.628 INFO:tasks.workunit.client.0.smithi116.stdout:5/39: dread - fa zero size 2024-03-04T16:04:03.629 INFO:tasks.workunit.client.0.smithi116.stdout:5/40: fdatasync f5 0 2024-03-04T16:04:03.630 INFO:tasks.workunit.client.0.smithi116.stdout:5/41: read f4 [1853317,28427] 0 2024-03-04T16:04:03.633 INFO:tasks.workunit.client.0.smithi116.stdout:0/53: dread da/fd [0,4194304] 0 2024-03-04T16:04:03.634 INFO:tasks.workunit.client.0.smithi116.stdout:1/40: mknod d5/d11/c12 0 2024-03-04T16:04:03.635 INFO:tasks.workunit.client.0.smithi116.stdout:4/56: symlink d1/d3/dd/l17 0 2024-03-04T16:04:03.638 INFO:tasks.workunit.client.0.smithi116.stdout:1/41: write f2 [3242111,27044] 0 2024-03-04T16:04:03.641 INFO:tasks.workunit.client.0.smithi116.stdout:8/57: creat d2/de/d16/f17 x:0 0 0 2024-03-04T16:04:03.642 INFO:tasks.workunit.client.0.smithi116.stdout:5/42: dwrite f5 [0,4194304] 0 2024-03-04T16:04:03.645 INFO:tasks.workunit.client.0.smithi116.stdout:2/67: write f6 [65966,61034] 0 2024-03-04T16:04:03.648 INFO:tasks.workunit.client.0.smithi116.stdout:8/58: write d2/de/f12 [2502290,81749] 0 2024-03-04T16:04:03.648 INFO:tasks.workunit.client.0.smithi116.stdout:6/72: creat d6/f13 x:0 0 0 2024-03-04T16:04:03.650 INFO:tasks.workunit.client.0.smithi116.stdout:0/54: dwrite da/fd [0,4194304] 0 2024-03-04T16:04:03.651 INFO:tasks.workunit.client.0.smithi116.stdout:4/57: dread d1/f2 [0,4194304] 0 2024-03-04T16:04:03.653 INFO:tasks.workunit.client.0.smithi116.stdout:6/73: write d6/f9 [2760360,113200] 0 2024-03-04T16:04:03.653 INFO:tasks.workunit.client.0.smithi116.stdout:3/86: dwrite fa [4194304,4194304] 0 2024-03-04T16:04:03.654 INFO:tasks.workunit.client.0.smithi116.stdout:6/74: write f2 [1095640,30914] 0 2024-03-04T16:04:03.655 INFO:tasks.workunit.client.0.smithi116.stdout:1/42: rename d5/f8 to d5/d11/f13 0 2024-03-04T16:04:03.657 INFO:tasks.workunit.client.0.smithi116.stdout:3/87: dread dc/f16 [0,4194304] 0 2024-03-04T16:04:03.658 INFO:tasks.workunit.client.0.smithi116.stdout:5/43: symlink lc 0 2024-03-04T16:04:03.659 INFO:tasks.workunit.client.0.smithi116.stdout:3/88: write dc/df/d14/d17/f1b [437590,103240] 0 2024-03-04T16:04:03.673 INFO:tasks.workunit.client.0.smithi116.stdout:4/58: unlink d1/d3/dd/fc 0 2024-03-04T16:04:03.674 INFO:tasks.workunit.client.0.smithi116.stdout:6/75: creat d6/f14 x:0 0 0 2024-03-04T16:04:03.675 INFO:tasks.workunit.client.0.smithi116.stdout:1/43: dread f4 [0,4194304] 0 2024-03-04T16:04:03.675 INFO:tasks.workunit.client.0.smithi116.stdout:1/44: chown d5/d11/c12 7309827 1 2024-03-04T16:04:03.676 INFO:tasks.workunit.client.0.smithi116.stdout:2/68: rename c5 to d7/d15/c19 0 2024-03-04T16:04:03.682 INFO:tasks.workunit.client.0.smithi116.stdout:5/44: dwrite fa [0,4194304] 0 2024-03-04T16:04:03.690 INFO:tasks.workunit.client.0.smithi116.stdout:1/45: mkdir d5/d14 0 2024-03-04T16:04:03.690 INFO:tasks.workunit.client.0.smithi116.stdout:6/76: write d6/f9 [4579258,75083] 0 2024-03-04T16:04:03.691 INFO:tasks.workunit.client.0.smithi116.stdout:1/46: truncate d5/f10 489831 0 2024-03-04T16:04:03.691 INFO:tasks.workunit.client.0.smithi116.stdout:6/77: write d6/f9 [2130832,90064] 0 2024-03-04T16:04:03.692 INFO:tasks.workunit.client.0.smithi116.stdout:6/78: truncate d6/f13 135529 0 2024-03-04T16:04:03.697 INFO:tasks.workunit.client.0.smithi116.stdout:5/45: mkdir dd 0 2024-03-04T16:04:03.702 INFO:tasks.workunit.client.0.smithi116.stdout:3/89: rename f2 to dc/df/f1e 0 2024-03-04T16:04:03.705 INFO:tasks.workunit.client.0.smithi116.stdout:6/79: mkdir d6/de/d15 0 2024-03-04T16:04:03.707 INFO:tasks.workunit.client.0.smithi116.stdout:8/59: truncate d2/de/fc 115580 0 2024-03-04T16:04:03.710 INFO:tasks.workunit.client.0.smithi116.stdout:6/80: dread d6/f8 [4194304,4194304] 0 2024-03-04T16:04:03.710 INFO:tasks.workunit.client.0.smithi116.stdout:3/90: unlink fa 0 2024-03-04T16:04:03.710 INFO:tasks.workunit.client.0.smithi116.stdout:6/81: stat d6/de 0 2024-03-04T16:04:03.712 INFO:tasks.workunit.client.0.smithi116.stdout:3/91: write dc/df/d14/d17/f1b [755198,3467] 0 2024-03-04T16:04:03.713 INFO:tasks.workunit.client.0.smithi116.stdout:5/46: dwrite f4 [4194304,4194304] 0 2024-03-04T16:04:03.713 INFO:tasks.workunit.client.0.smithi116.stdout:2/69: rename d7/f10 to d7/d13/f1a 0 2024-03-04T16:04:03.713 INFO:tasks.workunit.client.0.smithi116.stdout:4/59: write f0 [2148823,65688] 0 2024-03-04T16:04:03.716 INFO:tasks.workunit.client.0.smithi116.stdout:8/60: creat d2/de/d16/f18 x:0 0 0 2024-03-04T16:04:03.723 INFO:tasks.workunit.client.0.smithi116.stdout:8/61: dwrite d2/de/fa [0,4194304] 0 2024-03-04T16:04:03.725 INFO:tasks.workunit.client.0.smithi116.stdout:2/70: write d7/d13/f14 [1787834,17005] 0 2024-03-04T16:04:03.729 INFO:tasks.workunit.client.0.smithi116.stdout:2/71: dread d7/d13/f14 [0,4194304] 0 2024-03-04T16:04:03.730 INFO:tasks.workunit.client.0.smithi116.stdout:1/47: rename c0 to d5/d11/c15 0 2024-03-04T16:04:03.731 INFO:tasks.workunit.client.0.smithi116.stdout:0/55: write f7 [690842,44979] 0 2024-03-04T16:04:03.732 INFO:tasks.workunit.client.0.smithi116.stdout:5/47: creat dd/fe x:0 0 0 2024-03-04T16:04:03.733 INFO:tasks.workunit.client.0.smithi116.stdout:0/56: dread - da/fe zero size 2024-03-04T16:04:03.733 INFO:tasks.workunit.client.0.smithi116.stdout:5/48: chown f4 44 1 2024-03-04T16:04:03.733 INFO:tasks.workunit.client.0.smithi116.stdout:0/57: chown c9 5425 1 2024-03-04T16:04:03.733 INFO:tasks.workunit.client.0.smithi116.stdout:5/49: chown dd 675801321 1 2024-03-04T16:04:03.743 INFO:tasks.workunit.client.0.smithi116.stdout:6/82: rename d6/cc to d6/de/c16 0 2024-03-04T16:04:03.744 INFO:tasks.workunit.client.0.smithi116.stdout:5/50: symlink dd/lf 0 2024-03-04T16:04:03.744 INFO:tasks.workunit.client.0.smithi116.stdout:5/51: readlink lc 0 2024-03-04T16:04:03.744 INFO:tasks.workunit.client.0.smithi116.stdout:5/52: fdatasync fa 0 2024-03-04T16:04:03.748 INFO:tasks.workunit.client.0.smithi116.stdout:2/72: mkdir d7/d1b 0 2024-03-04T16:04:03.750 INFO:tasks.workunit.client.0.smithi116.stdout:4/60: rename f0 to d1/f18 0 2024-03-04T16:04:03.751 INFO:tasks.workunit.client.0.smithi116.stdout:4/61: dread - d1/d3/de/f16 zero size 2024-03-04T16:04:03.751 INFO:tasks.workunit.client.0.smithi116.stdout:4/62: stat d1/d3/dd 0 2024-03-04T16:04:03.752 INFO:tasks.workunit.client.0.smithi116.stdout:4/63: truncate d1/d3/de/f13 667498 0 2024-03-04T16:04:03.752 INFO:tasks.workunit.client.0.smithi116.stdout:6/83: creat d6/f17 x:0 0 0 2024-03-04T16:04:03.753 INFO:tasks.workunit.client.0.smithi116.stdout:6/84: fdatasync d6/f17 0 2024-03-04T16:04:03.754 INFO:tasks.workunit.client.0.smithi116.stdout:6/85: stat d6 0 2024-03-04T16:04:03.754 INFO:tasks.workunit.client.0.smithi116.stdout:5/53: rename dd/lf to dd/l10 0 2024-03-04T16:04:03.759 INFO:tasks.workunit.client.0.smithi116.stdout:3/92: dwrite dc/df/f1e [0,4194304] 0 2024-03-04T16:04:03.759 INFO:tasks.workunit.client.0.smithi116.stdout:8/62: fdatasync d2/de/fc 0 2024-03-04T16:04:03.760 INFO:tasks.workunit.client.0.smithi116.stdout:4/64: dread d1/fb [0,4194304] 0 2024-03-04T16:04:03.762 INFO:tasks.workunit.client.0.smithi116.stdout:1/48: rename d5/d9/ff to d5/d11/f16 0 2024-03-04T16:04:03.763 INFO:tasks.workunit.client.0.smithi116.stdout:3/93: chown dc/df/d14/d17 487 1 2024-03-04T16:04:03.765 INFO:tasks.workunit.client.0.smithi116.stdout:1/49: write d5/f10 [1090626,5430] 0 2024-03-04T16:04:03.766 INFO:tasks.workunit.client.0.smithi116.stdout:6/86: dwrite d6/f14 [0,4194304] 0 2024-03-04T16:04:03.766 INFO:tasks.workunit.client.0.smithi116.stdout:5/54: dread f4 [4194304,4194304] 0 2024-03-04T16:04:03.767 INFO:tasks.workunit.client.0.smithi116.stdout:2/73: creat d7/d1b/f1c x:0 0 0 2024-03-04T16:04:03.769 INFO:tasks.workunit.client.0.smithi116.stdout:8/63: symlink d2/de/l19 0 2024-03-04T16:04:03.770 INFO:tasks.workunit.client.0.smithi116.stdout:1/50: dread f2 [0,4194304] 0 2024-03-04T16:04:03.771 INFO:tasks.workunit.client.0.smithi116.stdout:1/51: stat d5/d11/c12 0 2024-03-04T16:04:03.771 INFO:tasks.workunit.client.0.smithi116.stdout:1/52: readlink - no filename 2024-03-04T16:04:03.771 INFO:tasks.workunit.client.0.smithi116.stdout:1/53: fsync f2 0 2024-03-04T16:04:03.774 INFO:tasks.workunit.client.0.smithi116.stdout:4/65: creat d1/d3/de/f19 x:0 0 0 2024-03-04T16:04:03.779 INFO:tasks.workunit.client.0.smithi116.stdout:5/55: fdatasync f4 0 2024-03-04T16:04:03.781 INFO:tasks.workunit.client.0.smithi116.stdout:6/87: mkdir d6/d18 0 2024-03-04T16:04:03.782 INFO:tasks.workunit.client.0.smithi116.stdout:2/74: symlink d7/d15/l1d 0 2024-03-04T16:04:03.783 INFO:tasks.workunit.client.0.smithi116.stdout:2/75: fdatasync f4 0 2024-03-04T16:04:03.784 INFO:tasks.workunit.client.0.smithi116.stdout:2/76: chown d7/d1b/f1c 98825320 1 2024-03-04T16:04:03.784 INFO:tasks.workunit.client.0.smithi116.stdout:6/88: dread d6/f9 [4194304,4194304] 0 2024-03-04T16:04:03.785 INFO:tasks.workunit.client.0.smithi116.stdout:5/56: dwrite fa [0,4194304] 0 2024-03-04T16:04:03.785 INFO:tasks.workunit.client.0.smithi116.stdout:8/64: rename d2/d13/l14 to d2/de/l1a 0 2024-03-04T16:04:03.786 INFO:tasks.workunit.client.0.smithi116.stdout:6/89: write d6/f17 [279337,112690] 0 2024-03-04T16:04:03.787 INFO:tasks.workunit.client.0.smithi116.stdout:4/66: write d1/f18 [3170356,86612] 0 2024-03-04T16:04:03.788 INFO:tasks.workunit.client.0.smithi116.stdout:4/67: rename d1/d3/dd to d1/d3/dd/d1a 22 2024-03-04T16:04:03.793 INFO:tasks.workunit.client.0.smithi116.stdout:2/77: creat d7/d13/f1e x:0 0 0 2024-03-04T16:04:03.795 INFO:tasks.workunit.client.0.smithi116.stdout:8/65: creat d2/de/d16/f1b x:0 0 0 2024-03-04T16:04:03.795 INFO:tasks.workunit.client.0.smithi116.stdout:4/68: dwrite d1/f18 [0,4194304] 0 2024-03-04T16:04:03.796 INFO:tasks.workunit.client.0.smithi116.stdout:8/66: read - d2/de/f8 zero size 2024-03-04T16:04:03.799 INFO:tasks.workunit.client.0.smithi116.stdout:6/90: creat d6/d18/f19 x:0 0 0 2024-03-04T16:04:03.801 INFO:tasks.workunit.client.0.smithi116.stdout:8/67: mkdir d2/de/d16/d1c 0 2024-03-04T16:04:03.801 INFO:tasks.workunit.client.0.smithi116.stdout:8/68: fdatasync d2/de/f12 0 2024-03-04T16:04:03.802 INFO:tasks.workunit.client.0.smithi116.stdout:8/69: readlink d2/de/l9 0 2024-03-04T16:04:03.803 INFO:tasks.workunit.client.0.smithi116.stdout:5/57: getdents dd 0 2024-03-04T16:04:03.809 INFO:tasks.workunit.client.0.smithi116.stdout:2/78: rename d7/l11 to d7/l1f 0 2024-03-04T16:04:03.809 INFO:tasks.workunit.client.0.smithi116.stdout:2/79: chown d7/fb 11 1 2024-03-04T16:04:03.815 INFO:tasks.workunit.client.0.smithi116.stdout:5/58: getdents dd 0 2024-03-04T16:04:03.815 INFO:tasks.workunit.client.0.smithi116.stdout:5/59: readlink lc 0 2024-03-04T16:04:03.817 INFO:tasks.workunit.client.0.smithi116.stdout:2/80: dread f1 [0,4194304] 0 2024-03-04T16:04:03.817 INFO:tasks.workunit.client.0.smithi116.stdout:2/81: write d7/fb [288620,75942] 0 2024-03-04T16:04:03.819 INFO:tasks.workunit.client.0.smithi116.stdout:5/60: creat dd/f11 x:0 0 0 2024-03-04T16:04:03.847 INFO:tasks.workunit.client.0.smithi116.stdout:5/61: unlink dd/l10 0 2024-03-04T16:04:03.847 INFO:tasks.workunit.client.0.smithi116.stdout:2/82: dwrite f4 [4194304,4194304] 0 2024-03-04T16:04:03.847 INFO:tasks.workunit.client.0.smithi116.stdout:4/69: write d1/fb [4799952,94677] 0 2024-03-04T16:04:03.847 INFO:tasks.workunit.client.0.smithi116.stdout:8/70: rmdir d2 39 2024-03-04T16:04:03.848 INFO:tasks.workunit.client.0.smithi116.stdout:4/70: stat d1/d3/dd 0 2024-03-04T16:04:03.849 INFO:tasks.workunit.client.0.smithi116.stdout:5/62: fsync dd/f11 0 2024-03-04T16:04:03.850 INFO:tasks.workunit.client.0.smithi116.stdout:4/71: write d1/f5 [757598,118043] 0 2024-03-04T16:04:03.850 INFO:tasks.workunit.client.0.smithi116.stdout:8/71: dread - d2/de/f8 zero size 2024-03-04T16:04:03.851 INFO:tasks.workunit.client.0.smithi116.stdout:4/72: fdatasync d1/d3/de/f13 0 2024-03-04T16:04:03.853 INFO:tasks.workunit.client.0.smithi116.stdout:2/83: rename d7/d13/f14 to d7/d15/f20 0 2024-03-04T16:04:03.854 INFO:tasks.workunit.client.0.smithi116.stdout:2/84: fsync d7/d13/f18 0 2024-03-04T16:04:03.856 INFO:tasks.workunit.client.0.smithi116.stdout:2/85: write d7/d13/f18 [191688,78450] 0 2024-03-04T16:04:03.857 INFO:tasks.workunit.client.0.smithi116.stdout:8/72: dwrite d2/de/fb [0,4194304] 0 2024-03-04T16:04:03.859 INFO:tasks.workunit.client.0.smithi116.stdout:8/73: write d2/fd [1550789,82937] 0 2024-03-04T16:04:03.862 INFO:tasks.workunit.client.0.smithi116.stdout:2/86: dread d7/fb [0,4194304] 0 2024-03-04T16:04:03.869 INFO:tasks.workunit.client.0.smithi116.stdout:4/73: link d1/d3/de/f19 d1/d3/de/f1b 0 2024-03-04T16:04:03.870 INFO:tasks.workunit.client.0.smithi116.stdout:4/74: chown d1/d3/de/f16 4501 1 2024-03-04T16:04:03.875 INFO:tasks.workunit.client.0.smithi116.stdout:5/63: getdents dd 0 2024-03-04T16:04:03.877 INFO:tasks.workunit.client.0.smithi116.stdout:4/75: mkdir d1/d3/de/d11/d1c 0 2024-03-04T16:04:03.877 INFO:tasks.workunit.client.0.smithi116.stdout:4/76: chown d1/f5 1 1 2024-03-04T16:04:03.883 INFO:tasks.workunit.client.0.smithi116.stdout:4/77: symlink d1/d3/dd/l1d 0 2024-03-04T16:04:03.885 INFO:tasks.workunit.client.0.smithi116.stdout:5/64: getdents dd 0 2024-03-04T16:04:03.885 INFO:tasks.workunit.client.0.smithi116.stdout:5/65: write f5 [1601288,3957] 0 2024-03-04T16:04:03.888 INFO:tasks.workunit.client.0.smithi116.stdout:4/78: unlink d1/f18 0 2024-03-04T16:04:03.888 INFO:tasks.workunit.client.0.smithi116.stdout:4/79: dread d1/f2 [4194304,4194304] 0 2024-03-04T16:04:03.889 INFO:tasks.workunit.client.0.smithi116.stdout:5/66: dread f5 [0,4194304] 0 2024-03-04T16:04:03.895 INFO:tasks.workunit.client.0.smithi116.stdout:4/80: creat d1/d3/de/d11/f1e x:0 0 0 2024-03-04T16:04:03.898 INFO:tasks.workunit.client.0.smithi116.stdout:5/67: rename c2 to dd/c12 0 2024-03-04T16:04:03.902 INFO:tasks.workunit.client.0.smithi116.stdout:4/81: mknod d1/d3/d15/c1f 0 2024-03-04T16:04:03.903 INFO:tasks.workunit.client.0.smithi116.stdout:4/82: fdatasync d1/d3/de/d11/f1e 0 2024-03-04T16:04:03.904 INFO:tasks.workunit.client.0.smithi116.stdout:4/83: write d1/fb [1251979,30277] 0 2024-03-04T16:04:03.905 INFO:tasks.workunit.client.0.smithi116.stdout:4/84: write d1/d3/de/f16 [612757,52659] 0 2024-03-04T16:04:03.917 INFO:tasks.workunit.client.0.smithi116.stdout:2/87: truncate f4 466463 0 2024-03-04T16:04:03.918 INFO:tasks.workunit.client.0.smithi116.stdout:2/88: creat d7/d1b/f21 x:0 0 0 2024-03-04T16:04:03.922 INFO:tasks.workunit.client.0.smithi116.stdout:2/89: creat d7/d1b/f22 x:0 0 0 2024-03-04T16:04:03.926 INFO:tasks.workunit.client.0.smithi116.stdout:2/90: symlink d7/d13/l23 0 2024-03-04T16:04:03.932 INFO:tasks.workunit.client.0.smithi116.stdout:4/85: rename d1/d3/de/d11 to d1/d3/d15/d20 0 2024-03-04T16:04:03.934 INFO:tasks.workunit.client.0.smithi116.stdout:5/68: chown dd/c12 226795 1 2024-03-04T16:04:03.936 INFO:tasks.workunit.client.0.smithi116.stdout:2/91: creat d7/f24 x:0 0 0 2024-03-04T16:04:03.936 INFO:tasks.workunit.client.0.smithi116.stdout:0/58: sync 2024-03-04T16:04:03.937 INFO:tasks.workunit.client.0.smithi116.stdout:7/19: sync 2024-03-04T16:04:03.937 INFO:tasks.workunit.client.0.smithi116.stdout:9/33: sync 2024-03-04T16:04:03.937 INFO:tasks.workunit.client.0.smithi116.stdout:2/92: write d7/d1b/f21 [299890,128309] 0 2024-03-04T16:04:03.937 INFO:tasks.workunit.client.0.smithi116.stdout:5/69: creat dd/f13 x:0 0 0 2024-03-04T16:04:03.939 INFO:tasks.workunit.client.0.smithi116.stdout:0/59: creat da/ff x:0 0 0 2024-03-04T16:04:03.945 INFO:tasks.workunit.client.0.smithi116.stdout:5/70: mkdir dd/d14 0 2024-03-04T16:04:03.946 INFO:tasks.workunit.client.0.smithi116.stdout:0/60: symlink da/l10 0 2024-03-04T16:04:03.947 INFO:tasks.workunit.client.0.smithi116.stdout:0/61: chown da/ff 393942954 1 2024-03-04T16:04:03.948 INFO:tasks.workunit.client.0.smithi116.stdout:5/71: dread f4 [0,4194304] 0 2024-03-04T16:04:03.948 INFO:tasks.workunit.client.0.smithi116.stdout:2/93: mkdir d7/d25 0 2024-03-04T16:04:03.949 INFO:tasks.workunit.client.0.smithi116.stdout:5/72: write f4 [7811495,55490] 0 2024-03-04T16:04:03.950 INFO:tasks.workunit.client.0.smithi116.stdout:7/20: mkdir d3/d4/d8 0 2024-03-04T16:04:03.951 INFO:tasks.workunit.client.0.smithi116.stdout:7/21: read - d3/f5 zero size 2024-03-04T16:04:03.951 INFO:tasks.workunit.client.0.smithi116.stdout:7/22: readlink d3/l7 0 2024-03-04T16:04:03.957 INFO:tasks.workunit.client.0.smithi116.stdout:5/73: truncate f5 4449336 0 2024-03-04T16:04:03.958 INFO:tasks.workunit.client.0.smithi116.stdout:5/74: truncate dd/f13 563983 0 2024-03-04T16:04:03.960 INFO:tasks.workunit.client.0.smithi116.stdout:5/75: creat dd/f15 x:0 0 0 2024-03-04T16:04:03.961 INFO:tasks.workunit.client.0.smithi116.stdout:5/76: write dd/f15 [537457,25957] 0 2024-03-04T16:04:03.962 INFO:tasks.workunit.client.0.smithi116.stdout:5/77: dread - dd/fe zero size 2024-03-04T16:04:03.962 INFO:tasks.workunit.client.0.smithi116.stdout:7/23: mknod d3/d4/d8/c9 0 2024-03-04T16:04:03.963 INFO:tasks.workunit.client.0.smithi116.stdout:7/24: read f1 [1983278,28411] 0 2024-03-04T16:04:03.963 INFO:tasks.workunit.client.0.smithi116.stdout:7/25: read f2 [731559,10545] 0 2024-03-04T16:04:03.964 INFO:tasks.workunit.client.0.smithi116.stdout:7/26: read d3/f6 [44400,67119] 0 2024-03-04T16:04:03.965 INFO:tasks.workunit.client.0.smithi116.stdout:7/27: dread d3/f6 [0,4194304] 0 2024-03-04T16:04:03.965 INFO:tasks.workunit.client.0.smithi116.stdout:7/28: readlink d3/l7 0 2024-03-04T16:04:03.968 INFO:tasks.workunit.client.0.smithi116.stdout:2/94: fsync d7/f24 0 2024-03-04T16:04:03.972 INFO:tasks.workunit.client.0.smithi116.stdout:5/78: symlink dd/d14/l16 0 2024-03-04T16:04:03.974 INFO:tasks.workunit.client.0.smithi116.stdout:2/95: creat d7/d1b/f26 x:0 0 0 2024-03-04T16:04:03.976 INFO:tasks.workunit.client.0.smithi116.stdout:9/34: dwrite d2/f6 [0,4194304] 0 2024-03-04T16:04:03.981 INFO:tasks.workunit.client.0.smithi116.stdout:5/79: symlink dd/l17 0 2024-03-04T16:04:03.981 INFO:tasks.workunit.client.0.smithi116.stdout:5/80: readlink lc 0 2024-03-04T16:04:03.984 INFO:tasks.workunit.client.0.smithi116.stdout:2/96: creat d7/d1b/f27 x:0 0 0 2024-03-04T16:04:03.985 INFO:tasks.workunit.client.0.smithi116.stdout:0/62: write da/fd [4469622,40358] 0 2024-03-04T16:04:03.990 INFO:tasks.workunit.client.0.smithi116.stdout:5/81: symlink dd/d14/l18 0 2024-03-04T16:04:03.991 INFO:tasks.workunit.client.0.smithi116.stdout:5/82: rename dd to dd/d19 22 2024-03-04T16:04:03.993 INFO:tasks.workunit.client.0.smithi116.stdout:5/83: dread f4 [0,4194304] 0 2024-03-04T16:04:03.994 INFO:tasks.workunit.client.0.smithi116.stdout:2/97: symlink d7/d15/l28 0 2024-03-04T16:04:03.996 INFO:tasks.workunit.client.0.smithi116.stdout:2/98: dread d7/d13/f18 [0,4194304] 0 2024-03-04T16:04:04.000 INFO:tasks.workunit.client.0.smithi116.stdout:0/63: symlink da/l11 0 2024-03-04T16:04:04.001 INFO:tasks.workunit.client.0.smithi116.stdout:7/29: getdents d3/d4/d8 0 2024-03-04T16:04:04.007 INFO:tasks.workunit.client.0.smithi116.stdout:1/54: sync 2024-03-04T16:04:04.007 INFO:tasks.workunit.client.0.smithi116.stdout:3/94: sync 2024-03-04T16:04:04.011 INFO:tasks.workunit.client.0.smithi116.stdout:0/64: creat da/f12 x:0 0 0 2024-03-04T16:04:04.018 INFO:tasks.workunit.client.0.smithi116.stdout:4/86: mknod d1/d3/de/c21 0 2024-03-04T16:04:04.019 INFO:tasks.workunit.client.0.smithi116.stdout:9/35: truncate d2/f6 3856880 0 2024-03-04T16:04:04.021 INFO:tasks.workunit.client.0.smithi116.stdout:5/84: link dd/l17 dd/l1a 0 2024-03-04T16:04:04.024 INFO:tasks.workunit.client.0.smithi116.stdout:4/87: creat d1/d3/f22 x:0 0 0 2024-03-04T16:04:04.026 INFO:tasks.workunit.client.0.smithi116.stdout:5/85: dwrite dd/f13 [0,4194304] 0 2024-03-04T16:04:04.026 INFO:tasks.workunit.client.0.smithi116.stdout:0/65: link da/fd da/f13 0 2024-03-04T16:04:04.027 INFO:tasks.workunit.client.0.smithi116.stdout:0/66: readlink da/l11 0 2024-03-04T16:04:04.030 INFO:tasks.workunit.client.0.smithi116.stdout:3/95: rmdir dc/df/d14/d17/d1c 0 2024-03-04T16:04:04.030 INFO:tasks.workunit.client.0.smithi116.stdout:0/67: dread da/fd [0,4194304] 0 2024-03-04T16:04:04.032 INFO:tasks.workunit.client.0.smithi116.stdout:0/68: write da/f12 [288454,121928] 0 2024-03-04T16:04:04.037 INFO:tasks.workunit.client.0.smithi116.stdout:5/86: creat dd/d14/f1b x:0 0 0 2024-03-04T16:04:04.037 INFO:tasks.workunit.client.0.smithi116.stdout:3/96: dwrite dc/df/d14/f1a [0,4194304] 0 2024-03-04T16:04:04.038 INFO:tasks.workunit.client.0.smithi116.stdout:5/87: truncate dd/fe 559484 0 2024-03-04T16:04:04.045 INFO:tasks.workunit.client.0.smithi116.stdout:0/69: dwrite da/f13 [4194304,4194304] 0 2024-03-04T16:04:04.045 INFO:tasks.workunit.client.0.smithi116.stdout:5/88: mknod dd/d14/c1c 0 2024-03-04T16:04:04.051 INFO:tasks.workunit.client.0.smithi116.stdout:0/70: mknod da/c14 0 2024-03-04T16:04:04.054 INFO:tasks.workunit.client.0.smithi116.stdout:5/89: mknod dd/d14/c1d 0 2024-03-04T16:04:04.056 INFO:tasks.workunit.client.0.smithi116.stdout:3/97: link c4 dc/c1f 0 2024-03-04T16:04:04.057 INFO:tasks.workunit.client.0.smithi116.stdout:3/98: truncate dc/df/d14/f1a 5080975 0 2024-03-04T16:04:04.060 INFO:tasks.workunit.client.0.smithi116.stdout:0/71: mknod da/c15 0 2024-03-04T16:04:04.064 INFO:tasks.workunit.client.0.smithi116.stdout:5/90: symlink dd/d14/l1e 0 2024-03-04T16:04:04.064 INFO:tasks.workunit.client.0.smithi116.stdout:5/91: stat dd/f11 0 2024-03-04T16:04:04.066 INFO:tasks.workunit.client.0.smithi116.stdout:3/99: rename dc/df/f1e to dc/f20 0 2024-03-04T16:04:04.070 INFO:tasks.workunit.client.0.smithi116.stdout:0/72: mkdir da/d16 0 2024-03-04T16:04:04.071 INFO:tasks.workunit.client.0.smithi116.stdout:5/92: mkdir dd/d1f 0 2024-03-04T16:04:04.076 INFO:tasks.workunit.client.0.smithi116.stdout:3/100: write dc/f20 [3449149,53844] 0 2024-03-04T16:04:04.077 INFO:tasks.workunit.client.0.smithi116.stdout:4/88: link d1/f2 d1/d3/de/f23 0 2024-03-04T16:04:04.081 INFO:tasks.workunit.client.0.smithi116.stdout:3/101: creat dc/df/d14/d17/f21 x:0 0 0 2024-03-04T16:04:04.083 INFO:tasks.workunit.client.0.smithi116.stdout:0/73: creat da/d16/f17 x:0 0 0 2024-03-04T16:04:04.083 INFO:tasks.workunit.client.0.smithi116.stdout:4/89: dwrite d1/d3/f22 [0,4194304] 0 2024-03-04T16:04:04.083 INFO:tasks.workunit.client.0.smithi116.stdout:0/74: write f2 [2898991,37841] 0 2024-03-04T16:04:04.084 INFO:tasks.workunit.client.0.smithi116.stdout:5/93: fdatasync f5 0 2024-03-04T16:04:04.084 INFO:tasks.workunit.client.0.smithi116.stdout:0/75: readlink da/l11 0 2024-03-04T16:04:04.085 INFO:tasks.workunit.client.0.smithi116.stdout:4/90: write d1/d3/d15/d20/f1e [1008716,130281] 0 2024-03-04T16:04:04.087 INFO:tasks.workunit.client.0.smithi116.stdout:4/91: write d1/fb [457508,120346] 0 2024-03-04T16:04:04.089 INFO:tasks.workunit.client.0.smithi116.stdout:4/92: stat d1/d3/dd/l17 0 2024-03-04T16:04:04.090 INFO:tasks.workunit.client.0.smithi116.stdout:3/102: creat dc/df/d14/d17/f22 x:0 0 0 2024-03-04T16:04:04.090 INFO:tasks.workunit.client.0.smithi116.stdout:9/36: truncate d2/f6 951952 0 2024-03-04T16:04:04.093 INFO:tasks.workunit.client.0.smithi116.stdout:0/76: dwrite da/f13 [8388608,4194304] 0 2024-03-04T16:04:04.094 INFO:tasks.workunit.client.0.smithi116.stdout:0/77: write f7 [2814447,74208] 0 2024-03-04T16:04:04.097 INFO:tasks.workunit.client.0.smithi116.stdout:5/94: mkdir dd/d14/d20 0 2024-03-04T16:04:04.097 INFO:tasks.workunit.client.0.smithi116.stdout:5/95: stat f5 0 2024-03-04T16:04:04.104 INFO:tasks.workunit.client.0.smithi116.stdout:4/93: mkdir d1/d3/dd/d24 0 2024-03-04T16:04:04.114 INFO:tasks.workunit.client.0.smithi116.stdout:9/37: fdatasync d2/f6 0 2024-03-04T16:04:04.115 INFO:tasks.workunit.client.0.smithi116.stdout:5/96: mknod dd/d14/d20/c21 0 2024-03-04T16:04:04.116 INFO:tasks.workunit.client.0.smithi116.stdout:5/97: chown dd/d14/c1c 108005 1 2024-03-04T16:04:04.117 INFO:tasks.workunit.client.0.smithi116.stdout:4/94: unlink d1/d3/de/f19 0 2024-03-04T16:04:04.118 INFO:tasks.workunit.client.0.smithi116.stdout:3/103: rename dc/df/d14/d17 to dc/d23 0 2024-03-04T16:04:04.118 INFO:tasks.workunit.client.0.smithi116.stdout:4/95: chown d1/d3/de/c21 58 1 2024-03-04T16:04:04.118 INFO:tasks.workunit.client.0.smithi116.stdout:0/78: rename da to da/d16/d18 22 2024-03-04T16:04:04.119 INFO:tasks.workunit.client.0.smithi116.stdout:0/79: write da/fe [1669,123133] 0 2024-03-04T16:04:04.139 INFO:tasks.workunit.client.0.smithi116.stdout:6/91: sync 2024-03-04T16:04:04.140 INFO:tasks.workunit.client.0.smithi116.stdout:8/74: sync 2024-03-04T16:04:04.152 INFO:tasks.workunit.client.0.smithi116.stdout:5/98: creat dd/f22 x:0 0 0 2024-03-04T16:04:04.153 INFO:tasks.workunit.client.0.smithi116.stdout:0/80: read - da/ff zero size 2024-03-04T16:04:04.154 INFO:tasks.workunit.client.0.smithi116.stdout:0/81: truncate f2 4991536 0 2024-03-04T16:04:04.154 INFO:tasks.workunit.client.0.smithi116.stdout:9/38: write d2/f6 [1304324,23624] 0 2024-03-04T16:04:04.156 INFO:tasks.workunit.client.0.smithi116.stdout:3/104: dread dc/f20 [0,4194304] 0 2024-03-04T16:04:04.156 INFO:tasks.workunit.client.0.smithi116.stdout:4/96: getdents d1/d3/d15/d20/d1c 0 2024-03-04T16:04:04.158 INFO:tasks.workunit.client.0.smithi116.stdout:8/75: mkdir d2/d13/d1d 0 2024-03-04T16:04:04.159 INFO:tasks.workunit.client.0.smithi116.stdout:5/99: dwrite dd/f15 [0,4194304] 0 2024-03-04T16:04:04.160 INFO:tasks.workunit.client.0.smithi116.stdout:6/92: creat d6/d18/f1a x:0 0 0 2024-03-04T16:04:04.166 INFO:tasks.workunit.client.0.smithi116.stdout:0/82: mkdir da/d16/d19 0 2024-03-04T16:04:04.167 INFO:tasks.workunit.client.0.smithi116.stdout:3/105: symlink dc/l24 0 2024-03-04T16:04:04.171 INFO:tasks.workunit.client.0.smithi116.stdout:8/76: mkdir d2/d13/d1e 0 2024-03-04T16:04:04.172 INFO:tasks.workunit.client.0.smithi116.stdout:8/77: read d2/de/f12 [3099804,106198] 0 2024-03-04T16:04:04.173 INFO:tasks.workunit.client.0.smithi116.stdout:8/78: dread - d2/de/d16/f1b zero size 2024-03-04T16:04:04.173 INFO:tasks.workunit.client.0.smithi116.stdout:8/79: dread - d2/de/f8 zero size 2024-03-04T16:04:04.174 INFO:tasks.workunit.client.0.smithi116.stdout:6/93: unlink d6/d18/f19 0 2024-03-04T16:04:04.174 INFO:tasks.workunit.client.0.smithi116.stdout:8/80: stat d2/c11 0 2024-03-04T16:04:04.174 INFO:tasks.workunit.client.0.smithi116.stdout:6/94: write d6/f13 [1002712,45584] 0 2024-03-04T16:04:04.175 INFO:tasks.workunit.client.0.smithi116.stdout:6/95: fsync d6/f17 0 2024-03-04T16:04:04.176 INFO:tasks.workunit.client.0.smithi116.stdout:0/83: creat da/f1a x:0 0 0 2024-03-04T16:04:04.177 INFO:tasks.workunit.client.0.smithi116.stdout:3/106: mknod dc/c25 0 2024-03-04T16:04:04.178 INFO:tasks.workunit.client.0.smithi116.stdout:4/97: symlink d1/d3/dd/d24/l25 0 2024-03-04T16:04:04.186 INFO:tasks.workunit.client.0.smithi116.stdout:8/81: symlink d2/d13/l1f 0 2024-03-04T16:04:04.188 INFO:tasks.workunit.client.0.smithi116.stdout:6/96: creat d6/de/f1b x:0 0 0 2024-03-04T16:04:04.190 INFO:tasks.workunit.client.0.smithi116.stdout:6/97: write d6/f17 [312219,46970] 0 2024-03-04T16:04:04.190 INFO:tasks.workunit.client.0.smithi116.stdout:0/84: rename da/c15 to da/d16/d19/c1b 0 2024-03-04T16:04:04.195 INFO:tasks.workunit.client.0.smithi116.stdout:9/39: getdents d2 0 2024-03-04T16:04:04.195 INFO:tasks.workunit.client.0.smithi116.stdout:9/40: readlink d2/la 0 2024-03-04T16:04:04.196 INFO:tasks.workunit.client.0.smithi116.stdout:9/41: chown d2/d8/lb 457231 1 2024-03-04T16:04:04.199 INFO:tasks.workunit.client.0.smithi116.stdout:6/98: unlink d6/cd 0 2024-03-04T16:04:04.199 INFO:tasks.workunit.client.0.smithi116.stdout:6/99: stat d6/de 0 2024-03-04T16:04:04.200 INFO:tasks.workunit.client.0.smithi116.stdout:0/85: rename da/l10 to da/l1c 0 2024-03-04T16:04:04.201 INFO:tasks.workunit.client.0.smithi116.stdout:6/100: write d6/de/f1b [611721,40596] 0 2024-03-04T16:04:04.202 INFO:tasks.workunit.client.0.smithi116.stdout:9/42: dwrite d2/f6 [0,4194304] 0 2024-03-04T16:04:04.204 INFO:tasks.workunit.client.0.smithi116.stdout:5/100: truncate dd/f15 3956069 0 2024-03-04T16:04:04.207 INFO:tasks.workunit.client.0.smithi116.stdout:6/101: dread d6/fa [0,4194304] 0 2024-03-04T16:04:04.207 INFO:tasks.workunit.client.0.smithi116.stdout:0/86: creat da/d16/d19/f1d x:0 0 0 2024-03-04T16:04:04.208 INFO:tasks.workunit.client.0.smithi116.stdout:3/107: link c3 dc/df/c26 0 2024-03-04T16:04:04.209 INFO:tasks.workunit.client.0.smithi116.stdout:0/87: truncate da/ff 592669 0 2024-03-04T16:04:04.211 INFO:tasks.workunit.client.0.smithi116.stdout:9/43: dwrite d2/f6 [0,4194304] 0 2024-03-04T16:04:04.212 INFO:tasks.workunit.client.0.smithi116.stdout:0/88: write f2 [568461,11861] 0 2024-03-04T16:04:04.213 INFO:tasks.workunit.client.0.smithi116.stdout:5/101: symlink dd/l23 0 2024-03-04T16:04:04.216 INFO:tasks.workunit.client.0.smithi116.stdout:3/108: creat dc/df/f27 x:0 0 0 2024-03-04T16:04:04.216 INFO:tasks.workunit.client.0.smithi116.stdout:6/102: dwrite d6/f8 [8388608,4194304] 0 2024-03-04T16:04:04.218 INFO:tasks.workunit.client.0.smithi116.stdout:3/109: truncate dc/df/f27 281112 0 2024-03-04T16:04:04.222 INFO:tasks.workunit.client.0.smithi116.stdout:4/98: link d1/fb d1/d3/de/f26 0 2024-03-04T16:04:04.225 INFO:tasks.workunit.client.0.smithi116.stdout:9/44: creat d2/fc x:0 0 0 2024-03-04T16:04:04.227 INFO:tasks.workunit.client.0.smithi116.stdout:0/89: creat da/f1e x:0 0 0 2024-03-04T16:04:04.231 INFO:tasks.workunit.client.0.smithi116.stdout:6/103: rename d6/cf to d6/de/c1c 0 2024-03-04T16:04:04.233 INFO:tasks.workunit.client.0.smithi116.stdout:3/110: fdatasync dc/f16 0 2024-03-04T16:04:04.237 INFO:tasks.workunit.client.0.smithi116.stdout:9/45: mknod d2/d8/cd 0 2024-03-04T16:04:04.237 INFO:tasks.workunit.client.0.smithi116.stdout:9/46: read - d2/fc zero size 2024-03-04T16:04:04.239 INFO:tasks.workunit.client.0.smithi116.stdout:0/90: creat da/f1f x:0 0 0 2024-03-04T16:04:04.240 INFO:tasks.workunit.client.0.smithi116.stdout:5/102: mknod dd/d1f/c24 0 2024-03-04T16:04:04.242 INFO:tasks.workunit.client.0.smithi116.stdout:6/104: truncate d6/f9 3238011 0 2024-03-04T16:04:04.244 INFO:tasks.workunit.client.0.smithi116.stdout:3/111: symlink dc/d23/l28 0 2024-03-04T16:04:04.245 INFO:tasks.workunit.client.0.smithi116.stdout:9/47: mknod d2/d8/ce 0 2024-03-04T16:04:04.247 INFO:tasks.workunit.client.0.smithi116.stdout:5/103: dwrite dd/f22 [0,4194304] 0 2024-03-04T16:04:04.247 INFO:tasks.workunit.client.0.smithi116.stdout:0/91: mkdir da/d16/d20 0 2024-03-04T16:04:04.250 INFO:tasks.workunit.client.0.smithi116.stdout:6/105: unlink d6/fa 0 2024-03-04T16:04:04.253 INFO:tasks.workunit.client.0.smithi116.stdout:0/92: dwrite da/ff [0,4194304] 0 2024-03-04T16:04:04.254 INFO:tasks.workunit.client.0.smithi116.stdout:3/112: mknod dc/df/d14/c29 0 2024-03-04T16:04:04.257 INFO:tasks.workunit.client.0.smithi116.stdout:9/48: rmdir d2/d8 39 2024-03-04T16:04:04.260 INFO:tasks.workunit.client.0.smithi116.stdout:5/104: mknod dd/d14/d20/c25 0 2024-03-04T16:04:04.261 INFO:tasks.workunit.client.0.smithi116.stdout:2/99: sync 2024-03-04T16:04:04.262 INFO:tasks.workunit.client.0.smithi116.stdout:5/105: write dd/fe [1217725,25753] 0 2024-03-04T16:04:04.263 INFO:tasks.workunit.client.0.smithi116.stdout:6/106: creat d6/d18/f1d x:0 0 0 2024-03-04T16:04:04.265 INFO:tasks.workunit.client.0.smithi116.stdout:6/107: fsync f3 0 2024-03-04T16:04:04.267 INFO:tasks.workunit.client.0.smithi116.stdout:2/100: truncate d7/d13/f1a 944816 0 2024-03-04T16:04:04.268 INFO:tasks.workunit.client.0.smithi116.stdout:3/113: symlink dc/l2a 0 2024-03-04T16:04:04.270 INFO:tasks.workunit.client.0.smithi116.stdout:9/49: dwrite d2/fc [0,4194304] 0 2024-03-04T16:04:04.272 INFO:tasks.workunit.client.0.smithi116.stdout:4/99: dread d1/d3/de/f13 [0,4194304] 0 2024-03-04T16:04:04.272 INFO:tasks.workunit.client.0.smithi116.stdout:6/108: write d6/f17 [309777,40388] 0 2024-03-04T16:04:04.273 INFO:tasks.workunit.client.0.smithi116.stdout:9/50: fdatasync d2/f6 0 2024-03-04T16:04:04.274 INFO:tasks.workunit.client.0.smithi116.stdout:4/100: readlink d1/d3/dd/l17 0 2024-03-04T16:04:04.277 INFO:tasks.workunit.client.0.smithi116.stdout:9/51: write d2/fc [1429037,102352] 0 2024-03-04T16:04:04.280 INFO:tasks.workunit.client.0.smithi116.stdout:5/106: dwrite fa [0,4194304] 0 2024-03-04T16:04:04.280 INFO:tasks.workunit.client.0.smithi116.stdout:0/93: creat da/d16/d20/f21 x:0 0 0 2024-03-04T16:04:04.281 INFO:tasks.workunit.client.0.smithi116.stdout:0/94: write da/f12 [1038098,93158] 0 2024-03-04T16:04:04.283 INFO:tasks.workunit.client.0.smithi116.stdout:0/95: write da/d16/d19/f1d [585942,67199] 0 2024-03-04T16:04:04.284 INFO:tasks.workunit.client.0.smithi116.stdout:6/109: dwrite f3 [0,4194304] 0 2024-03-04T16:04:04.284 INFO:tasks.workunit.client.0.smithi116.stdout:0/96: fdatasync da/f13 0 2024-03-04T16:04:04.286 INFO:tasks.workunit.client.0.smithi116.stdout:3/114: rename dc/df/f27 to dc/df/d14/f2b 0 2024-03-04T16:04:04.290 INFO:tasks.workunit.client.0.smithi116.stdout:2/101: mkdir d7/d29 0 2024-03-04T16:04:04.292 INFO:tasks.workunit.client.0.smithi116.stdout:0/97: dwrite f2 [0,4194304] 0 2024-03-04T16:04:04.293 INFO:tasks.workunit.client.0.smithi116.stdout:0/98: chown da/d16/d19 1590 1 2024-03-04T16:04:04.295 INFO:tasks.workunit.client.0.smithi116.stdout:0/99: dread da/fe [0,4194304] 0 2024-03-04T16:04:04.299 INFO:tasks.workunit.client.0.smithi116.stdout:5/107: symlink dd/d14/d20/l26 0 2024-03-04T16:04:04.304 INFO:tasks.workunit.client.0.smithi116.stdout:6/110: mkdir d6/de/d1e 0 2024-03-04T16:04:04.305 INFO:tasks.workunit.client.0.smithi116.stdout:3/115: mkdir dc/d23/d2c 0 2024-03-04T16:04:04.306 INFO:tasks.workunit.client.0.smithi116.stdout:3/116: chown dc/ld 33777 1 2024-03-04T16:04:04.307 INFO:tasks.workunit.client.0.smithi116.stdout:4/101: creat d1/d3/d15/d20/d1c/f27 x:0 0 0 2024-03-04T16:04:04.309 INFO:tasks.workunit.client.0.smithi116.stdout:9/52: creat d2/d8/ff x:0 0 0 2024-03-04T16:04:04.309 INFO:tasks.workunit.client.0.smithi116.stdout:0/100: creat da/d16/d20/f22 x:0 0 0 2024-03-04T16:04:04.311 INFO:tasks.workunit.client.0.smithi116.stdout:7/30: sync 2024-03-04T16:04:04.311 INFO:tasks.workunit.client.0.smithi116.stdout:1/55: sync 2024-03-04T16:04:04.311 INFO:tasks.workunit.client.0.smithi116.stdout:0/101: dread - da/d16/d20/f22 zero size 2024-03-04T16:04:04.312 INFO:tasks.workunit.client.0.smithi116.stdout:5/108: mknod dd/c27 0 2024-03-04T16:04:04.314 INFO:tasks.workunit.client.0.smithi116.stdout:4/102: stat d1/d3/de/f16 0 2024-03-04T16:04:04.315 INFO:tasks.workunit.client.0.smithi116.stdout:6/111: dwrite d6/de/f11 [0,4194304] 0 2024-03-04T16:04:04.316 INFO:tasks.workunit.client.0.smithi116.stdout:0/102: truncate da/d16/f17 564813 0 2024-03-04T16:04:04.317 INFO:tasks.workunit.client.0.smithi116.stdout:3/117: rename dc/f20 to dc/d23/d2c/f2d 0 2024-03-04T16:04:04.319 INFO:tasks.workunit.client.0.smithi116.stdout:9/53: dread d2/f6 [0,4194304] 0 2024-03-04T16:04:04.319 INFO:tasks.workunit.client.0.smithi116.stdout:3/118: truncate dc/d23/f1b 1429119 0 2024-03-04T16:04:04.320 INFO:tasks.workunit.client.0.smithi116.stdout:9/54: chown d2/f6 4692 1 2024-03-04T16:04:04.320 INFO:tasks.workunit.client.0.smithi116.stdout:3/119: readlink dc/l2a 0 2024-03-04T16:04:04.320 INFO:tasks.workunit.client.0.smithi116.stdout:7/31: dread d3/f6 [0,4194304] 0 2024-03-04T16:04:04.321 INFO:tasks.workunit.client.0.smithi116.stdout:3/120: fdatasync dc/df/d14/f1a 0 2024-03-04T16:04:04.321 INFO:tasks.workunit.client.0.smithi116.stdout:7/32: readlink d3/l7 0 2024-03-04T16:04:04.321 INFO:tasks.workunit.client.0.smithi116.stdout:5/109: symlink dd/d14/d20/l28 0 2024-03-04T16:04:04.321 INFO:tasks.workunit.client.0.smithi116.stdout:7/33: dread - d3/f5 zero size 2024-03-04T16:04:04.322 INFO:tasks.workunit.client.0.smithi116.stdout:5/110: truncate dd/f11 815119 0 2024-03-04T16:04:04.328 INFO:tasks.workunit.client.0.smithi116.stdout:2/102: link d7/cc d7/d13/c2a 0 2024-03-04T16:04:04.330 INFO:tasks.workunit.client.0.smithi116.stdout:1/56: rename d5/d11/f13 to d5/f17 0 2024-03-04T16:04:04.334 INFO:tasks.workunit.client.0.smithi116.stdout:3/121: rmdir dc/d23 39 2024-03-04T16:04:04.343 INFO:tasks.workunit.client.0.smithi116.stdout:1/57: creat d5/d9/f18 x:0 0 0 2024-03-04T16:04:04.346 INFO:tasks.workunit.client.0.smithi116.stdout:9/55: symlink d2/l10 0 2024-03-04T16:04:04.347 INFO:tasks.workunit.client.0.smithi116.stdout:0/103: rmdir da/d16/d20 39 2024-03-04T16:04:04.347 INFO:tasks.workunit.client.0.smithi116.stdout:1/58: dread f4 [0,4194304] 0 2024-03-04T16:04:04.348 INFO:tasks.workunit.client.0.smithi116.stdout:3/122: mknod dc/df/d14/c2e 0 2024-03-04T16:04:04.356 INFO:tasks.workunit.client.0.smithi116.stdout:5/111: rmdir dd/d14 39 2024-03-04T16:04:04.357 INFO:tasks.workunit.client.0.smithi116.stdout:2/103: creat d7/f2b x:0 0 0 2024-03-04T16:04:04.360 INFO:tasks.workunit.client.0.smithi116.stdout:4/103: truncate d1/d3/f22 3967258 0 2024-03-04T16:04:04.361 INFO:tasks.workunit.client.0.smithi116.stdout:3/123: fsync dc/d23/d2c/f2d 0 2024-03-04T16:04:04.363 INFO:tasks.workunit.client.0.smithi116.stdout:2/104: mknod d7/d1b/c2c 0 2024-03-04T16:04:04.363 INFO:tasks.workunit.client.0.smithi116.stdout:5/112: dread - dd/d14/f1b zero size 2024-03-04T16:04:04.364 INFO:tasks.workunit.client.0.smithi116.stdout:9/56: link d2/la d2/d8/l11 0 2024-03-04T16:04:04.365 INFO:tasks.workunit.client.0.smithi116.stdout:1/59: dwrite d5/f17 [0,4194304] 0 2024-03-04T16:04:04.367 INFO:tasks.workunit.client.0.smithi116.stdout:0/104: getdents da 0 2024-03-04T16:04:04.369 INFO:tasks.workunit.client.0.smithi116.stdout:2/105: truncate f6 1181942 0 2024-03-04T16:04:04.370 INFO:tasks.workunit.client.0.smithi116.stdout:7/34: dwrite f1 [4194304,4194304] 0 2024-03-04T16:04:04.370 INFO:tasks.workunit.client.0.smithi116.stdout:4/104: mknod d1/d3/dd/c28 0 2024-03-04T16:04:04.371 INFO:tasks.workunit.client.0.smithi116.stdout:0/105: write da/f1e [345010,47718] 0 2024-03-04T16:04:04.372 INFO:tasks.workunit.client.0.smithi116.stdout:5/113: creat dd/d1f/f29 x:0 0 0 2024-03-04T16:04:04.372 INFO:tasks.workunit.client.0.smithi116.stdout:7/35: chown f1 11 1 2024-03-04T16:04:04.372 INFO:tasks.workunit.client.0.smithi116.stdout:9/57: unlink d2/f6 0 2024-03-04T16:04:04.373 INFO:tasks.workunit.client.0.smithi116.stdout:5/114: chown dd/f22 3709969 1 2024-03-04T16:04:04.373 INFO:tasks.workunit.client.0.smithi116.stdout:4/105: write d1/d3/d15/d20/d1c/f27 [79643,63846] 0 2024-03-04T16:04:04.374 INFO:tasks.workunit.client.0.smithi116.stdout:7/36: stat d3/d4/d8/c9 0 2024-03-04T16:04:04.387 INFO:tasks.workunit.client.0.smithi116.stdout:1/60: fsync d5/f17 0 2024-03-04T16:04:04.387 INFO:tasks.workunit.client.0.smithi116.stdout:3/124: symlink dc/l2f 0 2024-03-04T16:04:04.388 INFO:tasks.workunit.client.0.smithi116.stdout:1/61: stat f2 0 2024-03-04T16:04:04.389 INFO:tasks.workunit.client.0.smithi116.stdout:5/115: symlink dd/d14/d20/l2a 0 2024-03-04T16:04:04.390 INFO:tasks.workunit.client.0.smithi116.stdout:4/106: creat d1/d3/dd/d24/f29 x:0 0 0 2024-03-04T16:04:04.393 INFO:tasks.workunit.client.0.smithi116.stdout:9/58: dwrite d2/fc [4194304,4194304] 0 2024-03-04T16:04:04.394 INFO:tasks.workunit.client.0.smithi116.stdout:7/37: dwrite d3/f5 [0,4194304] 0 2024-03-04T16:04:04.394 INFO:tasks.workunit.client.0.smithi116.stdout:3/125: dread dc/f16 [0,4194304] 0 2024-03-04T16:04:04.395 INFO:tasks.workunit.client.0.smithi116.stdout:9/59: dread - d2/d8/ff zero size 2024-03-04T16:04:04.395 INFO:tasks.workunit.client.0.smithi116.stdout:9/60: stat d2/l10 0 2024-03-04T16:04:04.396 INFO:tasks.workunit.client.0.smithi116.stdout:9/61: rename d2 to d2/d8/d12 22 2024-03-04T16:04:04.396 INFO:tasks.workunit.client.0.smithi116.stdout:3/126: truncate dc/df/d14/f2b 301470 0 2024-03-04T16:04:04.397 INFO:tasks.workunit.client.0.smithi116.stdout:3/127: readlink dc/d23/l28 0 2024-03-04T16:04:04.434 INFO:tasks.workunit.client.0.smithi116.stdout:8/82: sync 2024-03-04T16:04:04.486 INFO:tasks.workunit.client.0.smithi116.stdout:6/112: sync 2024-03-04T16:04:04.502 INFO:tasks.workunit.client.0.smithi116.stdout:2/106: creat d7/d29/f2d x:0 0 0 2024-03-04T16:04:04.502 INFO:tasks.workunit.client.0.smithi116.stdout:0/106: symlink da/d16/l23 0 2024-03-04T16:04:04.503 INFO:tasks.workunit.client.0.smithi116.stdout:1/62: mkdir d5/d19 0 2024-03-04T16:04:04.503 INFO:tasks.workunit.client.0.smithi116.stdout:0/107: write f7 [1084240,35804] 0 2024-03-04T16:04:04.504 INFO:tasks.workunit.client.0.smithi116.stdout:0/108: fdatasync da/fd 0 2024-03-04T16:04:04.505 INFO:tasks.workunit.client.0.smithi116.stdout:4/107: dread - d1/d3/de/f12 zero size 2024-03-04T16:04:04.506 INFO:tasks.workunit.client.0.smithi116.stdout:5/116: rename dd/d14/d20/c21 to dd/d14/d20/c2b 0 2024-03-04T16:04:04.507 INFO:tasks.workunit.client.0.smithi116.stdout:5/117: chown dd/d14/d20/l2a 686751590 1 2024-03-04T16:04:04.508 INFO:tasks.workunit.client.0.smithi116.stdout:5/118: chown f4 2 1 2024-03-04T16:04:04.509 INFO:tasks.workunit.client.0.smithi116.stdout:7/38: creat d3/d4/d8/fa x:0 0 0 2024-03-04T16:04:04.511 INFO:tasks.workunit.client.0.smithi116.stdout:3/128: rmdir dc/df 39 2024-03-04T16:04:04.514 INFO:tasks.workunit.client.0.smithi116.stdout:6/113: creat d6/de/f1f x:0 0 0 2024-03-04T16:04:04.515 INFO:tasks.workunit.client.0.smithi116.stdout:2/107: creat d7/d15/f2e x:0 0 0 2024-03-04T16:04:04.516 INFO:tasks.workunit.client.0.smithi116.stdout:2/108: stat d7/d13/l23 0 2024-03-04T16:04:04.517 INFO:tasks.workunit.client.0.smithi116.stdout:2/109: truncate d7/d1b/f21 855549 0 2024-03-04T16:04:04.518 INFO:tasks.workunit.client.0.smithi116.stdout:6/114: dread d6/de/f1b [0,4194304] 0 2024-03-04T16:04:04.518 INFO:tasks.workunit.client.0.smithi116.stdout:6/115: chown l4 7410365 1 2024-03-04T16:04:04.518 INFO:tasks.workunit.client.0.smithi116.stdout:1/63: rmdir d5/d9 39 2024-03-04T16:04:04.519 INFO:tasks.workunit.client.0.smithi116.stdout:6/116: fsync d6/f17 0 2024-03-04T16:04:04.519 INFO:tasks.workunit.client.0.smithi116.stdout:6/117: stat f2 0 2024-03-04T16:04:04.520 INFO:tasks.workunit.client.0.smithi116.stdout:0/109: rename da/fd to da/f24 0 2024-03-04T16:04:04.521 INFO:tasks.workunit.client.0.smithi116.stdout:0/110: dread da/fe [0,4194304] 0 2024-03-04T16:04:04.525 INFO:tasks.workunit.client.0.smithi116.stdout:0/111: dread f7 [0,4194304] 0 2024-03-04T16:04:04.526 INFO:tasks.workunit.client.0.smithi116.stdout:5/119: symlink dd/d1f/l2c 0 2024-03-04T16:04:04.526 INFO:tasks.workunit.client.0.smithi116.stdout:7/39: mknod d3/d4/cb 0 2024-03-04T16:04:04.526 INFO:tasks.workunit.client.0.smithi116.stdout:9/62: symlink d2/l13 0 2024-03-04T16:04:04.528 INFO:tasks.workunit.client.0.smithi116.stdout:8/83: symlink d2/de/d16/d1c/l20 0 2024-03-04T16:04:04.533 INFO:tasks.workunit.client.0.smithi116.stdout:1/64: mkdir d5/d11/d1a 0 2024-03-04T16:04:04.535 INFO:tasks.workunit.client.0.smithi116.stdout:8/84: dread d2/fd [0,4194304] 0 2024-03-04T16:04:04.536 INFO:tasks.workunit.client.0.smithi116.stdout:6/118: rename d6/f8 to d6/de/d1e/f20 0 2024-03-04T16:04:04.543 INFO:tasks.workunit.client.0.smithi116.stdout:7/40: fsync d3/f6 0 2024-03-04T16:04:04.545 INFO:tasks.workunit.client.0.smithi116.stdout:7/41: dread f2 [0,4194304] 0 2024-03-04T16:04:04.546 INFO:tasks.workunit.client.0.smithi116.stdout:7/42: write f1 [8545663,3292] 0 2024-03-04T16:04:04.547 INFO:tasks.workunit.client.0.smithi116.stdout:3/129: truncate dc/d23/d2c/f2d 644955 0 2024-03-04T16:04:04.551 INFO:tasks.workunit.client.0.smithi116.stdout:2/110: mkdir d7/d2f 0 2024-03-04T16:04:04.556 INFO:tasks.workunit.client.0.smithi116.stdout:8/85: symlink d2/de/d16/d1c/l21 0 2024-03-04T16:04:04.556 INFO:tasks.workunit.client.0.smithi116.stdout:8/86: chown d2/de/d16 60589 1 2024-03-04T16:04:04.557 INFO:tasks.workunit.client.0.smithi116.stdout:8/87: chown d2/d13/d1e 706100 1 2024-03-04T16:04:04.557 INFO:tasks.workunit.client.0.smithi116.stdout:6/119: creat d6/f21 x:0 0 0 2024-03-04T16:04:04.558 INFO:tasks.workunit.client.0.smithi116.stdout:6/120: chown d6/d18/f1a 177273271 1 2024-03-04T16:04:04.566 INFO:tasks.workunit.client.0.smithi116.stdout:9/63: symlink d2/l14 0 2024-03-04T16:04:04.566 INFO:tasks.workunit.client.0.smithi116.stdout:9/64: chown d2/d8/lb 3850715 1 2024-03-04T16:04:04.567 INFO:tasks.workunit.client.0.smithi116.stdout:7/43: unlink d3/d4/cb 0 2024-03-04T16:04:04.569 INFO:tasks.workunit.client.0.smithi116.stdout:9/65: dread d2/fc [4194304,4194304] 0 2024-03-04T16:04:04.570 INFO:tasks.workunit.client.0.smithi116.stdout:9/66: chown d2/l10 182619 1 2024-03-04T16:04:04.573 INFO:tasks.workunit.client.0.smithi116.stdout:8/88: creat d2/de/d16/f22 x:0 0 0 2024-03-04T16:04:04.575 INFO:tasks.workunit.client.0.smithi116.stdout:6/121: creat d6/f22 x:0 0 0 2024-03-04T16:04:04.584 INFO:tasks.workunit.client.0.smithi116.stdout:6/122: creat d6/de/f23 x:0 0 0 2024-03-04T16:04:04.588 INFO:tasks.workunit.client.0.smithi116.stdout:8/89: dwrite d2/fd [0,4194304] 0 2024-03-04T16:04:04.590 INFO:tasks.workunit.client.0.smithi116.stdout:6/123: rename d6/f22 to d6/f24 0 2024-03-04T16:04:04.590 INFO:tasks.workunit.client.0.smithi116.stdout:3/130: link dc/l24 dc/df/d14/l30 0 2024-03-04T16:04:04.592 INFO:tasks.workunit.client.0.smithi116.stdout:9/67: creat d2/f15 x:0 0 0 2024-03-04T16:04:04.593 INFO:tasks.workunit.client.0.smithi116.stdout:9/68: chown d2/l7 9 1 2024-03-04T16:04:04.594 INFO:tasks.workunit.client.0.smithi116.stdout:8/90: creat d2/de/d16/f23 x:0 0 0 2024-03-04T16:04:04.594 INFO:tasks.workunit.client.0.smithi116.stdout:9/69: chown d2/d8/lb 336 1 2024-03-04T16:04:04.595 INFO:tasks.workunit.client.0.smithi116.stdout:8/91: fdatasync d2/de/fb 0 2024-03-04T16:04:04.596 INFO:tasks.workunit.client.0.smithi116.stdout:8/92: write d2/fd [2753487,48483] 0 2024-03-04T16:04:04.601 INFO:tasks.workunit.client.0.smithi116.stdout:9/70: dwrite d2/f15 [0,4194304] 0 2024-03-04T16:04:04.604 INFO:tasks.workunit.client.0.smithi116.stdout:8/93: dread d2/fd [0,4194304] 0 2024-03-04T16:04:04.605 INFO:tasks.workunit.client.0.smithi116.stdout:8/94: truncate d2/de/f15 898078 0 2024-03-04T16:04:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:04 smithi152 ceph-mon[36876]: pgmap v18: 65 pgs: 65 active+clean; 180 MiB data, 843 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s wr, 325 op/s 2024-03-04T16:04:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:04 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:04.608 INFO:tasks.workunit.client.0.smithi116.stdout:9/71: rename d2/d8/lb to d2/d8/l16 0 2024-03-04T16:04:04.612 INFO:tasks.workunit.client.0.smithi116.stdout:8/95: dwrite d2/de/d16/f22 [0,4194304] 0 2024-03-04T16:04:04.620 INFO:tasks.workunit.client.0.smithi116.stdout:8/96: dwrite d2/de/d16/f17 [0,4194304] 0 2024-03-04T16:04:04.622 INFO:tasks.workunit.client.0.smithi116.stdout:8/97: truncate d2/de/f8 198557 0 2024-03-04T16:04:04.624 INFO:tasks.workunit.client.0.smithi116.stdout:8/98: symlink d2/de/d16/l24 0 2024-03-04T16:04:04.627 INFO:tasks.workunit.client.0.smithi116.stdout:8/99: symlink d2/d13/d1e/l25 0 2024-03-04T16:04:04.639 INFO:tasks.workunit.client.0.smithi116.stdout:8/100: fsync d2/de/d16/f17 0 2024-03-04T16:04:04.641 INFO:tasks.workunit.client.0.smithi116.stdout:8/101: creat d2/de/d16/f26 x:0 0 0 2024-03-04T16:04:04.644 INFO:tasks.workunit.client.0.smithi116.stdout:8/102: read d2/de/fc [52428,11641] 0 2024-03-04T16:04:04.645 INFO:tasks.workunit.client.0.smithi116.stdout:8/103: read - d2/de/d16/f26 zero size 2024-03-04T16:04:04.645 INFO:tasks.workunit.client.0.smithi116.stdout:8/104: stat d2/de/fb 0 2024-03-04T16:04:04.647 INFO:tasks.workunit.client.0.smithi116.stdout:8/105: creat d2/d13/d1e/f27 x:0 0 0 2024-03-04T16:04:04.653 INFO:tasks.workunit.client.0.smithi116.stdout:8/106: dwrite d2/de/f12 [0,4194304] 0 2024-03-04T16:04:04.653 INFO:tasks.workunit.client.0.smithi116.stdout:8/107: chown d2/de/f15 0 1 2024-03-04T16:04:04.656 INFO:tasks.workunit.client.0.smithi116.stdout:8/108: symlink d2/de/l28 0 2024-03-04T16:04:04.656 INFO:tasks.workunit.client.0.smithi116.stdout:8/109: chown d2/lf 210501 1 2024-03-04T16:04:04.657 INFO:tasks.workunit.client.0.smithi116.stdout:8/110: write d2/de/fc [178959,39774] 0 2024-03-04T16:04:04.659 INFO:tasks.workunit.client.0.smithi116.stdout:8/111: write d2/de/f12 [344262,53647] 0 2024-03-04T16:04:04.661 INFO:tasks.workunit.client.0.smithi116.stdout:8/112: rmdir d2 39 2024-03-04T16:04:04.664 INFO:tasks.workunit.client.0.smithi116.stdout:8/113: read d2/de/f12 [1049455,83440] 0 2024-03-04T16:04:04.665 INFO:tasks.workunit.client.0.smithi116.stdout:8/114: dread d2/de/f8 [0,4194304] 0 2024-03-04T16:04:04.671 INFO:tasks.workunit.client.0.smithi116.stdout:8/115: dwrite d2/d13/d1e/f27 [0,4194304] 0 2024-03-04T16:04:04.679 INFO:tasks.workunit.client.0.smithi116.stdout:5/120: rename dd/d14/d20 to dd/d14/d2d 0 2024-03-04T16:04:04.679 INFO:tasks.workunit.client.0.smithi116.stdout:5/121: read fa [3151848,69910] 0 2024-03-04T16:04:04.679 INFO:tasks.workunit.client.0.smithi116.stdout:4/108: stat d1/d3/f22 0 2024-03-04T16:04:04.682 INFO:tasks.workunit.client.0.smithi116.stdout:4/109: symlink d1/d3/d15/d20/d1c/l2a 0 2024-03-04T16:04:04.683 INFO:tasks.workunit.client.0.smithi116.stdout:4/110: write d1/f5 [1842060,125953] 0 2024-03-04T16:04:04.685 INFO:tasks.workunit.client.0.smithi116.stdout:5/122: rename dd/d1f/c24 to dd/d14/d2d/c2e 0 2024-03-04T16:04:04.693 INFO:tasks.workunit.client.0.smithi116.stdout:4/111: mknod d1/d3/de/c2b 0 2024-03-04T16:04:04.694 INFO:tasks.workunit.client.0.smithi116.stdout:4/112: write d1/f5 [2375202,121616] 0 2024-03-04T16:04:04.697 INFO:tasks.workunit.client.0.smithi116.stdout:4/113: chown d1/d3/de/f1b 25641 1 2024-03-04T16:04:04.698 INFO:tasks.workunit.client.0.smithi116.stdout:5/123: dwrite fa [4194304,4194304] 0 2024-03-04T16:04:04.699 INFO:tasks.workunit.client.0.smithi116.stdout:4/114: fsync d1/d3/de/f12 0 2024-03-04T16:04:04.701 INFO:tasks.workunit.client.0.smithi116.stdout:5/124: mknod dd/d1f/c2f 0 2024-03-04T16:04:04.702 INFO:tasks.workunit.client.0.smithi116.stdout:5/125: dread - dd/d1f/f29 zero size 2024-03-04T16:04:04.704 INFO:tasks.workunit.client.0.smithi116.stdout:7/44: write d3/f6 [952915,65157] 0 2024-03-04T16:04:04.711 INFO:tasks.workunit.client.0.smithi116.stdout:6/124: truncate d6/de/f1b 624944 0 2024-03-04T16:04:04.711 INFO:tasks.workunit.client.0.smithi116.stdout:7/45: dwrite d3/f6 [0,4194304] 0 2024-03-04T16:04:04.714 INFO:tasks.workunit.client.0.smithi116.stdout:6/125: mkdir d6/de/d25 0 2024-03-04T16:04:04.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:04 smithi116 ceph-mon[29120]: pgmap v18: 65 pgs: 65 active+clean; 180 MiB data, 843 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s wr, 325 op/s 2024-03-04T16:04:04.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:04 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:04.756 INFO:tasks.workunit.client.0.smithi116.stdout:7/46: creat d3/fc x:0 0 0 2024-03-04T16:04:04.757 INFO:tasks.workunit.client.0.smithi116.stdout:0/112: truncate da/f13 4575013 0 2024-03-04T16:04:04.758 INFO:tasks.workunit.client.0.smithi116.stdout:2/111: write d7/d13/f18 [281067,129685] 0 2024-03-04T16:04:04.762 INFO:tasks.workunit.client.0.smithi116.stdout:0/113: dread da/d16/f17 [0,4194304] 0 2024-03-04T16:04:04.763 INFO:tasks.workunit.client.0.smithi116.stdout:3/131: truncate dc/df/d14/f2b 22340 0 2024-03-04T16:04:04.764 INFO:tasks.workunit.client.0.smithi116.stdout:9/72: rmdir d2/d8 39 2024-03-04T16:04:04.764 INFO:tasks.workunit.client.0.smithi116.stdout:1/65: dwrite f4 [0,4194304] 0 2024-03-04T16:04:04.769 INFO:tasks.workunit.client.0.smithi116.stdout:8/116: truncate d2/de/fa 1303010 0 2024-03-04T16:04:04.769 INFO:tasks.workunit.client.0.smithi116.stdout:7/47: dwrite d3/f5 [0,4194304] 0 2024-03-04T16:04:04.769 INFO:tasks.workunit.client.0.smithi116.stdout:2/112: creat d7/d15/f30 x:0 0 0 2024-03-04T16:04:04.772 INFO:tasks.workunit.client.0.smithi116.stdout:3/132: creat dc/df/d14/f31 x:0 0 0 2024-03-04T16:04:04.776 INFO:tasks.workunit.client.0.smithi116.stdout:6/126: creat d6/de/d15/f26 x:0 0 0 2024-03-04T16:04:04.781 INFO:tasks.workunit.client.0.smithi116.stdout:5/126: dwrite f5 [0,4194304] 0 2024-03-04T16:04:04.788 INFO:tasks.workunit.client.0.smithi116.stdout:0/114: creat da/d16/d20/f25 x:0 0 0 2024-03-04T16:04:04.794 INFO:tasks.workunit.client.0.smithi116.stdout:9/73: creat d2/d8/f17 x:0 0 0 2024-03-04T16:04:04.795 INFO:tasks.workunit.client.0.smithi116.stdout:9/74: fdatasync d2/d8/f17 0 2024-03-04T16:04:04.799 INFO:tasks.workunit.client.0.smithi116.stdout:3/133: symlink dc/d23/l32 0 2024-03-04T16:04:04.800 INFO:tasks.workunit.client.0.smithi116.stdout:6/127: creat d6/d18/f27 x:0 0 0 2024-03-04T16:04:04.801 INFO:tasks.workunit.client.0.smithi116.stdout:3/134: fdatasync dc/df/d14/f31 0 2024-03-04T16:04:04.802 INFO:tasks.workunit.client.0.smithi116.stdout:9/75: dwrite d2/d8/f17 [0,4194304] 0 2024-03-04T16:04:04.803 INFO:tasks.workunit.client.0.smithi116.stdout:9/76: readlink d2/l7 0 2024-03-04T16:04:04.809 INFO:tasks.workunit.client.0.smithi116.stdout:0/115: symlink da/d16/d19/l26 0 2024-03-04T16:04:04.809 INFO:tasks.workunit.client.0.smithi116.stdout:6/128: dwrite f2 [0,4194304] 0 2024-03-04T16:04:04.812 INFO:tasks.workunit.client.0.smithi116.stdout:5/127: dwrite dd/f15 [0,4194304] 0 2024-03-04T16:04:04.815 INFO:tasks.workunit.client.0.smithi116.stdout:6/129: dread f2 [0,4194304] 0 2024-03-04T16:04:04.816 INFO:tasks.workunit.client.0.smithi116.stdout:6/130: dread - d6/f21 zero size 2024-03-04T16:04:04.819 INFO:tasks.workunit.client.0.smithi116.stdout:3/135: readlink dc/df/d14/l30 0 2024-03-04T16:04:04.821 INFO:tasks.workunit.client.0.smithi116.stdout:8/117: dread d2/de/fa [0,4194304] 0 2024-03-04T16:04:04.822 INFO:tasks.workunit.client.0.smithi116.stdout:9/77: symlink d2/d8/l18 0 2024-03-04T16:04:04.832 INFO:tasks.workunit.client.0.smithi116.stdout:0/116: mkdir da/d16/d19/d27 0 2024-03-04T16:04:04.833 INFO:tasks.workunit.client.0.smithi116.stdout:0/117: write da/f12 [1757143,96395] 0 2024-03-04T16:04:04.835 INFO:tasks.workunit.client.0.smithi116.stdout:7/48: link f2 d3/fd 0 2024-03-04T16:04:04.835 INFO:tasks.workunit.client.0.smithi116.stdout:7/49: chown d3/l7 4851 1 2024-03-04T16:04:04.836 INFO:tasks.workunit.client.0.smithi116.stdout:1/66: write f2 [1591410,81512] 0 2024-03-04T16:04:04.839 INFO:tasks.workunit.client.0.smithi116.stdout:1/67: dread d5/f17 [0,4194304] 0 2024-03-04T16:04:04.845 INFO:tasks.workunit.client.0.smithi116.stdout:8/118: read d2/de/f8 [168745,56544] 0 2024-03-04T16:04:04.846 INFO:tasks.workunit.client.0.smithi116.stdout:2/113: link d7/cc d7/d13/c31 0 2024-03-04T16:04:04.859 INFO:tasks.workunit.client.0.smithi116.stdout:9/78: mkdir d2/d19 0 2024-03-04T16:04:04.865 INFO:tasks.workunit.client.0.smithi116.stdout:0/118: mkdir da/d16/d28 0 2024-03-04T16:04:04.868 INFO:tasks.workunit.client.0.smithi116.stdout:5/128: link c1 dd/d14/d2d/c30 0 2024-03-04T16:04:04.869 INFO:tasks.workunit.client.0.smithi116.stdout:6/131: link l4 d6/de/d15/l28 0 2024-03-04T16:04:04.870 INFO:tasks.workunit.client.0.smithi116.stdout:6/132: write d6/de/f1f [34762,8237] 0 2024-03-04T16:04:04.871 INFO:tasks.workunit.client.0.smithi116.stdout:1/68: rename d5/d11/c15 to d5/d11/c1b 0 2024-03-04T16:04:04.872 INFO:tasks.workunit.client.0.smithi116.stdout:1/69: write f4 [2011102,129015] 0 2024-03-04T16:04:04.880 INFO:tasks.workunit.client.0.smithi116.stdout:0/119: mknod da/d16/d19/c29 0 2024-03-04T16:04:04.882 INFO:tasks.workunit.client.0.smithi116.stdout:5/129: mkdir dd/d14/d2d/d31 0 2024-03-04T16:04:04.884 INFO:tasks.workunit.client.0.smithi116.stdout:6/133: mkdir d6/de/d29 0 2024-03-04T16:04:04.885 INFO:tasks.workunit.client.0.smithi116.stdout:6/134: truncate d6/f17 749124 0 2024-03-04T16:04:04.885 INFO:tasks.workunit.client.0.smithi116.stdout:6/135: fdatasync d6/f14 0 2024-03-04T16:04:04.886 INFO:tasks.workunit.client.0.smithi116.stdout:6/136: dread - d6/d18/f27 zero size 2024-03-04T16:04:04.888 INFO:tasks.workunit.client.0.smithi116.stdout:1/70: dread - d5/d9/f18 zero size 2024-03-04T16:04:04.900 INFO:tasks.workunit.client.0.smithi116.stdout:7/50: write d3/fd [756032,97863] 0 2024-03-04T16:04:04.902 INFO:tasks.workunit.client.0.smithi116.stdout:1/71: unlink d5/f10 0 2024-03-04T16:04:04.904 INFO:tasks.workunit.client.0.smithi116.stdout:3/136: dread dc/df/d14/f2b [0,4194304] 0 2024-03-04T16:04:04.905 INFO:tasks.workunit.client.0.smithi116.stdout:9/79: creat d2/f1a x:0 0 0 2024-03-04T16:04:04.907 INFO:tasks.workunit.client.0.smithi116.stdout:8/119: truncate d2/fd 4159254 0 2024-03-04T16:04:04.907 INFO:tasks.workunit.client.0.smithi116.stdout:2/114: getdents d7/d15 0 2024-03-04T16:04:04.909 INFO:tasks.workunit.client.0.smithi116.stdout:0/120: rmdir da/d16 39 2024-03-04T16:04:04.911 INFO:tasks.workunit.client.0.smithi116.stdout:4/115: sync 2024-03-04T16:04:04.911 INFO:tasks.workunit.client.0.smithi116.stdout:7/51: mknod d3/d4/d8/ce 0 2024-03-04T16:04:04.911 INFO:tasks.workunit.client.0.smithi116.stdout:7/52: chown d3/d4 1174 1 2024-03-04T16:04:04.921 INFO:tasks.workunit.client.0.smithi116.stdout:5/130: link dd/d14/d2d/l26 dd/d14/d2d/l32 0 2024-03-04T16:04:04.924 INFO:tasks.workunit.client.0.smithi116.stdout:4/116: creat d1/f2c x:0 0 0 2024-03-04T16:04:04.927 INFO:tasks.workunit.client.0.smithi116.stdout:7/53: mknod d3/d4/d8/cf 0 2024-03-04T16:04:04.927 INFO:tasks.workunit.client.0.smithi116.stdout:5/131: dwrite dd/d14/f1b [0,4194304] 0 2024-03-04T16:04:04.933 INFO:tasks.workunit.client.0.smithi116.stdout:6/137: truncate d6/de/f11 1818893 0 2024-03-04T16:04:04.934 INFO:tasks.workunit.client.0.smithi116.stdout:9/80: mknod d2/d19/c1b 0 2024-03-04T16:04:04.935 INFO:tasks.workunit.client.0.smithi116.stdout:9/81: write d2/f1a [943850,124388] 0 2024-03-04T16:04:04.939 INFO:tasks.workunit.client.0.smithi116.stdout:8/120: truncate d2/de/d16/f17 2263881 0 2024-03-04T16:04:04.945 INFO:tasks.workunit.client.0.smithi116.stdout:8/121: dwrite d2/de/d16/f1b [0,4194304] 0 2024-03-04T16:04:04.945 INFO:tasks.workunit.client.0.smithi116.stdout:4/117: symlink d1/d3/d15/l2d 0 2024-03-04T16:04:04.946 INFO:tasks.workunit.client.0.smithi116.stdout:7/54: creat d3/f10 x:0 0 0 2024-03-04T16:04:04.948 INFO:tasks.workunit.client.0.smithi116.stdout:5/132: rename dd/d14/l16 to dd/d14/d2d/l33 0 2024-03-04T16:04:04.951 INFO:tasks.workunit.client.0.smithi116.stdout:1/72: rmdir d5/d11/d1a 0 2024-03-04T16:04:04.952 INFO:tasks.workunit.client.0.smithi116.stdout:4/118: dwrite d1/f5 [0,4194304] 0 2024-03-04T16:04:04.955 INFO:tasks.workunit.client.0.smithi116.stdout:3/137: creat dc/f33 x:0 0 0 2024-03-04T16:04:04.958 INFO:tasks.workunit.client.0.smithi116.stdout:6/138: creat d6/de/d1e/f2a x:0 0 0 2024-03-04T16:04:04.959 INFO:tasks.workunit.client.0.smithi116.stdout:6/139: dread - d6/d18/f1a zero size 2024-03-04T16:04:04.959 INFO:tasks.workunit.client.0.smithi116.stdout:1/73: dwrite d5/d9/fb [0,4194304] 0 2024-03-04T16:04:04.960 INFO:tasks.workunit.client.0.smithi116.stdout:6/140: fdatasync d6/f13 0 2024-03-04T16:04:04.971 INFO:tasks.workunit.client.0.smithi116.stdout:7/55: mkdir d3/d4/d11 0 2024-03-04T16:04:04.972 INFO:tasks.workunit.client.0.smithi116.stdout:8/122: dwrite d2/f7 [0,4194304] 0 2024-03-04T16:04:04.977 INFO:tasks.workunit.client.0.smithi116.stdout:7/56: dread f2 [4194304,4194304] 0 2024-03-04T16:04:04.981 INFO:tasks.workunit.client.0.smithi116.stdout:9/82: mknod d2/c1c 0 2024-03-04T16:04:04.988 INFO:tasks.workunit.client.0.smithi116.stdout:6/141: mknod d6/de/d1e/c2b 0 2024-03-04T16:04:04.990 INFO:tasks.workunit.client.0.smithi116.stdout:0/121: link da/l11 da/l2a 0 2024-03-04T16:04:04.990 INFO:tasks.workunit.client.0.smithi116.stdout:6/142: read d6/de/f1f [27752,74726] 0 2024-03-04T16:04:04.996 INFO:tasks.workunit.client.0.smithi116.stdout:8/123: rmdir d2/de 39 2024-03-04T16:04:04.999 INFO:tasks.workunit.client.0.smithi116.stdout:3/138: mkdir dc/d34 0 2024-03-04T16:04:05.001 INFO:tasks.workunit.client.0.smithi116.stdout:7/57: creat d3/d4/f12 x:0 0 0 2024-03-04T16:04:05.003 INFO:tasks.workunit.client.0.smithi116.stdout:9/83: mkdir d2/d8/d1d 0 2024-03-04T16:04:05.013 INFO:tasks.workunit.client.0.smithi116.stdout:6/143: symlink d6/l2c 0 2024-03-04T16:04:05.014 INFO:tasks.workunit.client.0.smithi116.stdout:5/133: truncate fa 1584227 0 2024-03-04T16:04:05.014 INFO:tasks.workunit.client.0.smithi116.stdout:5/134: fsync f5 0 2024-03-04T16:04:05.017 INFO:tasks.workunit.client.0.smithi116.stdout:4/119: rename d1/d3/d15/c1f to d1/d3/de/c2e 0 2024-03-04T16:04:05.017 INFO:tasks.workunit.client.0.smithi116.stdout:4/120: readlink d1/d3/dd/l17 0 2024-03-04T16:04:05.018 INFO:tasks.workunit.client.0.smithi116.stdout:3/139: truncate dc/f16 1935561 0 2024-03-04T16:04:05.018 INFO:tasks.workunit.client.0.smithi116.stdout:4/121: chown d1/d3/f22 3075016 1 2024-03-04T16:04:05.020 INFO:tasks.workunit.client.0.smithi116.stdout:9/84: unlink d2/d8/cd 0 2024-03-04T16:04:05.020 INFO:tasks.workunit.client.0.smithi116.stdout:9/85: read - d2/d8/ff zero size 2024-03-04T16:04:05.021 INFO:tasks.workunit.client.0.smithi116.stdout:9/86: chown d2/l14 8241 1 2024-03-04T16:04:05.021 INFO:tasks.workunit.client.0.smithi116.stdout:1/74: creat d5/f1c x:0 0 0 2024-03-04T16:04:05.021 INFO:tasks.workunit.client.0.smithi116.stdout:1/75: dread - d5/d9/f18 zero size 2024-03-04T16:04:05.022 INFO:tasks.workunit.client.0.smithi116.stdout:0/122: rmdir da/d16/d19 39 2024-03-04T16:04:05.023 INFO:tasks.workunit.client.0.smithi116.stdout:0/123: chown da/f1e 5 1 2024-03-04T16:04:05.023 INFO:tasks.workunit.client.0.smithi116.stdout:0/124: chown f2 154 1 2024-03-04T16:04:05.023 INFO:tasks.workunit.client.0.smithi116.stdout:0/125: dread - da/f1f zero size 2024-03-04T16:04:05.024 INFO:tasks.workunit.client.0.smithi116.stdout:6/144: symlink d6/d18/l2d 0 2024-03-04T16:04:05.026 INFO:tasks.workunit.client.0.smithi116.stdout:8/124: fsync d2/fd 0 2024-03-04T16:04:05.028 INFO:tasks.workunit.client.0.smithi116.stdout:3/140: mknod dc/df/c35 0 2024-03-04T16:04:05.030 INFO:tasks.workunit.client.0.smithi116.stdout:6/145: dwrite d6/de/f23 [0,4194304] 0 2024-03-04T16:04:05.030 INFO:tasks.workunit.client.0.smithi116.stdout:4/122: symlink d1/d3/l2f 0 2024-03-04T16:04:05.033 INFO:tasks.workunit.client.0.smithi116.stdout:9/87: rename d2/l10 to d2/d19/l1e 0 2024-03-04T16:04:05.035 INFO:tasks.workunit.client.0.smithi116.stdout:9/88: chown d2/d8/f17 710106 1 2024-03-04T16:04:05.035 INFO:tasks.workunit.client.0.smithi116.stdout:4/123: dread d1/f2 [0,4194304] 0 2024-03-04T16:04:05.037 INFO:tasks.workunit.client.0.smithi116.stdout:0/126: unlink da/lb 0 2024-03-04T16:04:05.038 INFO:tasks.workunit.client.0.smithi116.stdout:5/135: symlink dd/l34 0 2024-03-04T16:04:05.040 INFO:tasks.workunit.client.0.smithi116.stdout:9/89: dwrite d2/d8/f17 [0,4194304] 0 2024-03-04T16:04:05.063 INFO:tasks.workunit.client.0.smithi116.stdout:4/124: fdatasync d1/d3/de/f13 0 2024-03-04T16:04:05.063 INFO:tasks.workunit.client.0.smithi116.stdout:4/125: chown d1/fb 7182 1 2024-03-04T16:04:05.071 INFO:tasks.workunit.client.0.smithi116.stdout:7/58: getdents d3/d4/d8 0 2024-03-04T16:04:05.073 INFO:tasks.workunit.client.0.smithi116.stdout:3/141: mknod dc/d34/c36 0 2024-03-04T16:04:05.074 INFO:tasks.workunit.client.0.smithi116.stdout:6/146: truncate d6/de/f1b 1666710 0 2024-03-04T16:04:05.075 INFO:tasks.workunit.client.0.smithi116.stdout:6/147: fdatasync f3 0 2024-03-04T16:04:05.075 INFO:tasks.workunit.client.0.smithi116.stdout:6/148: chown d6/f13 77715 1 2024-03-04T16:04:05.076 INFO:tasks.workunit.client.0.smithi116.stdout:1/76: rmdir d5/d14 0 2024-03-04T16:04:05.079 INFO:tasks.workunit.client.0.smithi116.stdout:0/127: chown da/d16/d19/c1b 2 1 2024-03-04T16:04:05.079 INFO:tasks.workunit.client.0.smithi116.stdout:6/149: dread d6/f13 [0,4194304] 0 2024-03-04T16:04:05.079 INFO:tasks.workunit.client.0.smithi116.stdout:6/150: read f2 [2882991,41439] 0 2024-03-04T16:04:05.081 INFO:tasks.workunit.client.0.smithi116.stdout:5/136: rename dd/d14/d2d/l26 to dd/d1f/l35 0 2024-03-04T16:04:05.083 INFO:tasks.workunit.client.0.smithi116.stdout:0/128: dwrite f2 [4194304,4194304] 0 2024-03-04T16:04:05.084 INFO:tasks.workunit.client.0.smithi116.stdout:0/129: fdatasync da/f12 0 2024-03-04T16:04:05.086 INFO:tasks.workunit.client.0.smithi116.stdout:7/59: creat d3/d4/d8/f13 x:0 0 0 2024-03-04T16:04:05.088 INFO:tasks.workunit.client.0.smithi116.stdout:3/142: mknod dc/df/c37 0 2024-03-04T16:04:05.097 INFO:tasks.workunit.client.0.smithi116.stdout:5/137: mknod dd/d1f/c36 0 2024-03-04T16:04:05.098 INFO:tasks.workunit.client.0.smithi116.stdout:5/138: truncate dd/f15 4294113 0 2024-03-04T16:04:05.100 INFO:tasks.workunit.client.0.smithi116.stdout:0/130: mkdir da/d16/d20/d2b 0 2024-03-04T16:04:05.101 INFO:tasks.workunit.client.0.smithi116.stdout:9/90: write d2/fc [3997460,129603] 0 2024-03-04T16:04:05.104 INFO:tasks.workunit.client.0.smithi116.stdout:5/139: dwrite dd/d14/f1b [0,4194304] 0 2024-03-04T16:04:05.104 INFO:tasks.workunit.client.0.smithi116.stdout:7/60: rename d3/d4/f12 to d3/f14 0 2024-03-04T16:04:05.106 INFO:tasks.workunit.client.0.smithi116.stdout:3/143: unlink dc/ld 0 2024-03-04T16:04:05.112 INFO:tasks.workunit.client.0.smithi116.stdout:9/91: dwrite d2/d8/ff [0,4194304] 0 2024-03-04T16:04:05.113 INFO:tasks.workunit.client.0.smithi116.stdout:6/151: mknod d6/de/d29/c2e 0 2024-03-04T16:04:05.113 INFO:tasks.workunit.client.0.smithi116.stdout:8/125: dwrite d2/de/f8 [0,4194304] 0 2024-03-04T16:04:05.122 INFO:tasks.workunit.client.0.smithi116.stdout:7/61: dwrite d3/fc [0,4194304] 0 2024-03-04T16:04:05.122 INFO:tasks.workunit.client.0.smithi116.stdout:5/140: unlink dd/d14/f1b 0 2024-03-04T16:04:05.126 INFO:tasks.workunit.client.0.smithi116.stdout:5/141: truncate dd/d1f/f29 859542 0 2024-03-04T16:04:05.127 INFO:tasks.workunit.client.0.smithi116.stdout:3/144: symlink dc/d34/l38 0 2024-03-04T16:04:05.128 INFO:tasks.workunit.client.0.smithi116.stdout:5/142: fsync dd/d1f/f29 0 2024-03-04T16:04:05.130 INFO:tasks.workunit.client.0.smithi116.stdout:3/145: dread - dc/d23/f21 zero size 2024-03-04T16:04:05.130 INFO:tasks.workunit.client.0.smithi116.stdout:6/152: dwrite d6/de/f23 [0,4194304] 0 2024-03-04T16:04:05.130 INFO:tasks.workunit.client.0.smithi116.stdout:5/143: chown dd/f15 10710978 1 2024-03-04T16:04:05.133 INFO:tasks.workunit.client.0.smithi116.stdout:7/62: dwrite d3/f6 [0,4194304] 0 2024-03-04T16:04:05.133 INFO:tasks.workunit.client.0.smithi116.stdout:1/77: creat d5/f1d x:0 0 0 2024-03-04T16:04:05.135 INFO:tasks.workunit.client.0.smithi116.stdout:6/153: rename d6/de to d6/de/d25/d2f 22 2024-03-04T16:04:05.139 INFO:tasks.workunit.client.0.smithi116.stdout:6/154: dread f2 [0,4194304] 0 2024-03-04T16:04:05.140 INFO:tasks.workunit.client.0.smithi116.stdout:6/155: read d6/f14 [2933462,107261] 0 2024-03-04T16:04:05.145 INFO:tasks.workunit.client.0.smithi116.stdout:0/131: mkdir da/d2c 0 2024-03-04T16:04:05.149 INFO:tasks.workunit.client.0.smithi116.stdout:3/146: dread dc/d23/f1b [0,4194304] 0 2024-03-04T16:04:05.160 INFO:tasks.workunit.client.0.smithi116.stdout:8/126: dwrite d2/de/f8 [4194304,4194304] 0 2024-03-04T16:04:05.161 INFO:tasks.workunit.client.0.smithi116.stdout:9/92: mkdir d2/d8/d1d/d1f 0 2024-03-04T16:04:05.168 INFO:tasks.workunit.client.0.smithi116.stdout:8/127: dwrite d2/de/d16/f18 [0,4194304] 0 2024-03-04T16:04:05.169 INFO:tasks.workunit.client.0.smithi116.stdout:3/147: unlink dc/df/c37 0 2024-03-04T16:04:05.169 INFO:tasks.workunit.client.0.smithi116.stdout:9/93: dwrite d2/d8/ff [4194304,4194304] 0 2024-03-04T16:04:05.170 INFO:tasks.workunit.client.0.smithi116.stdout:3/148: readlink dc/l2f 0 2024-03-04T16:04:05.171 INFO:tasks.workunit.client.0.smithi116.stdout:9/94: read d2/f1a [94345,49151] 0 2024-03-04T16:04:05.182 INFO:tasks.workunit.client.0.smithi116.stdout:1/78: rename d5/c7 to d5/d19/c1e 0 2024-03-04T16:04:05.188 INFO:tasks.workunit.client.0.smithi116.stdout:1/79: dwrite f4 [0,4194304] 0 2024-03-04T16:04:05.193 INFO:tasks.workunit.client.0.smithi116.stdout:9/95: readlink d2/d8/l11 0 2024-03-04T16:04:05.193 INFO:tasks.workunit.client.0.smithi116.stdout:7/63: dwrite f1 [0,4194304] 0 2024-03-04T16:04:05.193 INFO:tasks.workunit.client.0.smithi116.stdout:3/149: dread dc/d23/d2c/f2d [0,4194304] 0 2024-03-04T16:04:05.194 INFO:tasks.workunit.client.0.smithi116.stdout:9/96: write d2/fc [6080430,83760] 0 2024-03-04T16:04:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:6/156: link d6/de/f1f d6/de/f30 0 2024-03-04T16:04:05.203 INFO:tasks.workunit.client.0.smithi116.stdout:0/132: rename da/f1f to da/f2d 0 2024-03-04T16:04:05.203 INFO:tasks.workunit.client.0.smithi116.stdout:0/133: truncate da/f2d 952512 0 2024-03-04T16:04:05.209 INFO:tasks.workunit.client.0.smithi116.stdout:5/144: truncate dd/f13 887808 0 2024-03-04T16:04:05.209 INFO:tasks.workunit.client.0.smithi116.stdout:8/128: truncate d2/de/fb 736215 0 2024-03-04T16:04:05.210 INFO:tasks.workunit.client.0.smithi116.stdout:5/145: write dd/f11 [105557,61797] 0 2024-03-04T16:04:05.210 INFO:tasks.workunit.client.0.smithi116.stdout:0/134: dwrite da/f1a [0,4194304] 0 2024-03-04T16:04:05.211 INFO:tasks.workunit.client.0.smithi116.stdout:1/80: symlink d5/d19/l1f 0 2024-03-04T16:04:05.214 INFO:tasks.workunit.client.0.smithi116.stdout:7/64: fdatasync d3/fd 0 2024-03-04T16:04:05.226 INFO:tasks.workunit.client.0.smithi116.stdout:5/146: rmdir dd/d14 39 2024-03-04T16:04:05.231 INFO:tasks.workunit.client.0.smithi116.stdout:5/147: dwrite dd/f11 [0,4194304] 0 2024-03-04T16:04:05.232 INFO:tasks.workunit.client.0.smithi116.stdout:9/97: rename d2/d8/l16 to d2/d8/d1d/d1f/l20 0 2024-03-04T16:04:05.234 INFO:tasks.workunit.client.0.smithi116.stdout:6/157: link d6/d18/f1d d6/de/d15/f31 0 2024-03-04T16:04:05.235 INFO:tasks.workunit.client.0.smithi116.stdout:5/148: dread dd/f11 [0,4194304] 0 2024-03-04T16:04:05.236 INFO:tasks.workunit.client.0.smithi116.stdout:5/149: stat dd/d1f/l2c 0 2024-03-04T16:04:05.241 INFO:tasks.workunit.client.0.smithi116.stdout:0/135: link da/d16/d20/f21 da/d16/d28/f2e 0 2024-03-04T16:04:05.243 INFO:tasks.workunit.client.0.smithi116.stdout:7/65: symlink d3/d4/d11/l15 0 2024-03-04T16:04:05.244 INFO:tasks.workunit.client.0.smithi116.stdout:7/66: write f1 [2485150,90183] 0 2024-03-04T16:04:05.246 INFO:tasks.workunit.client.0.smithi116.stdout:9/98: fsync d2/f1a 0 2024-03-04T16:04:05.251 INFO:tasks.workunit.client.0.smithi116.stdout:0/136: rename f2 to da/d16/d19/f2f 0 2024-03-04T16:04:05.253 INFO:tasks.workunit.client.0.smithi116.stdout:3/150: dwrite dc/df/d14/f2b [0,4194304] 0 2024-03-04T16:04:05.255 INFO:tasks.workunit.client.0.smithi116.stdout:9/99: dwrite d2/f15 [0,4194304] 0 2024-03-04T16:04:05.259 INFO:tasks.workunit.client.0.smithi116.stdout:9/100: write d2/d8/f17 [3555442,98999] 0 2024-03-04T16:04:05.261 INFO:tasks.workunit.client.0.smithi116.stdout:5/150: truncate fa 1228480 0 2024-03-04T16:04:05.263 INFO:tasks.workunit.client.0.smithi116.stdout:3/151: dwrite dc/f33 [0,4194304] 0 2024-03-04T16:04:05.263 INFO:tasks.workunit.client.0.smithi116.stdout:0/137: stat c8 0 2024-03-04T16:04:05.264 INFO:tasks.workunit.client.0.smithi116.stdout:1/81: truncate f2 3476045 0 2024-03-04T16:04:05.266 INFO:tasks.workunit.client.0.smithi116.stdout:7/67: rename d3/fc to d3/d4/f16 0 2024-03-04T16:04:05.268 INFO:tasks.workunit.client.0.smithi116.stdout:1/82: dread d5/d9/fb [0,4194304] 0 2024-03-04T16:04:05.272 INFO:tasks.workunit.client.0.smithi116.stdout:9/101: write d2/f1a [1558033,50483] 0 2024-03-04T16:04:05.279 INFO:tasks.workunit.client.0.smithi116.stdout:3/152: chown c4 960 1 2024-03-04T16:04:05.279 INFO:tasks.workunit.client.0.smithi116.stdout:2/115: sync 2024-03-04T16:04:05.279 INFO:tasks.workunit.client.0.smithi116.stdout:9/102: dwrite d2/f1a [0,4194304] 0 2024-03-04T16:04:05.280 INFO:tasks.workunit.client.0.smithi116.stdout:2/116: read - d7/d1b/f1c zero size 2024-03-04T16:04:05.288 INFO:tasks.workunit.client.0.smithi116.stdout:7/68: dwrite d3/d4/f16 [4194304,4194304] 0 2024-03-04T16:04:05.293 INFO:tasks.workunit.client.0.smithi116.stdout:0/138: mknod da/d16/d20/d2b/c30 0 2024-03-04T16:04:05.293 INFO:tasks.workunit.client.0.smithi116.stdout:0/139: write da/f1e [131555,98659] 0 2024-03-04T16:04:05.293 INFO:tasks.workunit.client.0.smithi116.stdout:0/140: stat da/cc 0 2024-03-04T16:04:05.294 INFO:tasks.workunit.client.0.smithi116.stdout:0/141: stat da/d16/d19/c29 0 2024-03-04T16:04:05.297 INFO:tasks.workunit.client.0.smithi116.stdout:3/153: rmdir dc/d23 39 2024-03-04T16:04:05.297 INFO:tasks.workunit.client.0.smithi116.stdout:3/154: rename dc/df to dc/df/d39 22 2024-03-04T16:04:05.299 INFO:tasks.workunit.client.0.smithi116.stdout:9/103: creat d2/d8/f21 x:0 0 0 2024-03-04T16:04:05.300 INFO:tasks.workunit.client.0.smithi116.stdout:2/117: symlink d7/d29/l32 0 2024-03-04T16:04:05.302 INFO:tasks.workunit.client.0.smithi116.stdout:2/118: write d7/d15/f30 [516211,102237] 0 2024-03-04T16:04:05.304 INFO:tasks.workunit.client.0.smithi116.stdout:3/155: dwrite dc/df/d14/f31 [0,4194304] 0 2024-03-04T16:04:05.305 INFO:tasks.workunit.client.0.smithi116.stdout:9/104: dread d2/f15 [0,4194304] 0 2024-03-04T16:04:05.312 INFO:tasks.workunit.client.0.smithi116.stdout:0/142: mknod da/d16/d19/c31 0 2024-03-04T16:04:05.312 INFO:tasks.workunit.client.0.smithi116.stdout:7/69: dwrite d3/f14 [0,4194304] 0 2024-03-04T16:04:05.314 INFO:tasks.workunit.client.0.smithi116.stdout:7/70: write d3/d4/d8/fa [398424,121233] 0 2024-03-04T16:04:05.315 INFO:tasks.workunit.client.0.smithi116.stdout:0/143: dread da/d16/d19/f1d [0,4194304] 0 2024-03-04T16:04:05.316 INFO:tasks.workunit.client.0.smithi116.stdout:7/71: write d3/d4/d8/f13 [266872,73329] 0 2024-03-04T16:04:05.322 INFO:tasks.workunit.client.0.smithi116.stdout:3/156: symlink dc/d34/l3a 0 2024-03-04T16:04:05.323 INFO:tasks.workunit.client.0.smithi116.stdout:9/105: symlink d2/d8/d1d/d1f/l22 0 2024-03-04T16:04:05.326 INFO:tasks.workunit.client.0.smithi116.stdout:9/106: dread d2/f1a [0,4194304] 0 2024-03-04T16:04:05.330 INFO:tasks.workunit.client.0.smithi116.stdout:5/151: write dd/f13 [1187282,101675] 0 2024-03-04T16:04:05.330 INFO:tasks.workunit.client.0.smithi116.stdout:2/119: creat d7/d25/f33 x:0 0 0 2024-03-04T16:04:05.331 INFO:tasks.workunit.client.0.smithi116.stdout:2/120: dread - d7/d25/f33 zero size 2024-03-04T16:04:05.331 INFO:tasks.workunit.client.0.smithi116.stdout:0/144: creat da/d16/d20/d2b/f32 x:0 0 0 2024-03-04T16:04:05.336 INFO:tasks.workunit.client.0.smithi116.stdout:9/107: mknod d2/d8/d1d/d1f/c23 0 2024-03-04T16:04:05.337 INFO:tasks.workunit.client.0.smithi116.stdout:5/152: dwrite dd/d1f/f29 [0,4194304] 0 2024-03-04T16:04:05.340 INFO:tasks.workunit.client.0.smithi116.stdout:2/121: rename d7/d25/f33 to d7/d25/f34 0 2024-03-04T16:04:05.342 INFO:tasks.workunit.client.0.smithi116.stdout:9/108: dread d2/fc [0,4194304] 0 2024-03-04T16:04:05.342 INFO:tasks.workunit.client.0.smithi116.stdout:9/109: fsync d2/fc 0 2024-03-04T16:04:05.344 INFO:tasks.workunit.client.0.smithi116.stdout:0/145: rename da/d16/d20/f25 to da/d16/d19/f33 0 2024-03-04T16:04:05.346 INFO:tasks.workunit.client.0.smithi116.stdout:1/83: dread f2 [0,4194304] 0 2024-03-04T16:04:05.347 INFO:tasks.workunit.client.0.smithi116.stdout:1/84: read f4 [530721,113250] 0 2024-03-04T16:04:05.354 INFO:tasks.workunit.client.0.smithi116.stdout:9/110: creat d2/d8/d1d/d1f/f24 x:0 0 0 2024-03-04T16:04:05.358 INFO:tasks.workunit.client.0.smithi116.stdout:7/72: getdents d3/d4/d11 0 2024-03-04T16:04:05.359 INFO:tasks.workunit.client.0.smithi116.stdout:3/157: getdents dc 0 2024-03-04T16:04:05.362 INFO:tasks.workunit.client.0.smithi116.stdout:0/146: creat da/d16/d19/d27/f34 x:0 0 0 2024-03-04T16:04:05.363 INFO:tasks.workunit.client.0.smithi116.stdout:9/111: creat d2/d8/d1d/d1f/f25 x:0 0 0 2024-03-04T16:04:05.363 INFO:tasks.workunit.client.0.smithi116.stdout:0/147: dread - da/d16/d19/f33 zero size 2024-03-04T16:04:05.363 INFO:tasks.workunit.client.0.smithi116.stdout:5/153: dread f4 [0,4194304] 0 2024-03-04T16:04:05.371 INFO:tasks.workunit.client.0.smithi116.stdout:0/148: truncate da/fe 1110130 0 2024-03-04T16:04:05.372 INFO:tasks.workunit.client.0.smithi116.stdout:0/149: fsync da/ff 0 2024-03-04T16:04:05.374 INFO:tasks.workunit.client.0.smithi116.stdout:7/73: getdents d3/d4/d11 0 2024-03-04T16:04:05.374 INFO:tasks.workunit.client.0.smithi116.stdout:9/112: dread d2/f15 [0,4194304] 0 2024-03-04T16:04:05.376 INFO:tasks.workunit.client.0.smithi116.stdout:2/122: dwrite d7/fb [0,4194304] 0 2024-03-04T16:04:05.379 INFO:tasks.workunit.client.0.smithi116.stdout:1/85: getdents d5/d11 0 2024-03-04T16:04:05.382 INFO:tasks.workunit.client.0.smithi116.stdout:3/158: write dc/d23/f1b [660710,14063] 0 2024-03-04T16:04:05.385 INFO:tasks.workunit.client.0.smithi116.stdout:9/113: dwrite d2/d8/d1d/d1f/f24 [0,4194304] 0 2024-03-04T16:04:05.385 INFO:tasks.workunit.client.0.smithi116.stdout:3/159: chown dc/df/d14/l30 28931 1 2024-03-04T16:04:05.385 INFO:tasks.workunit.client.0.smithi116.stdout:2/123: mkdir d7/d15/d35 0 2024-03-04T16:04:05.392 INFO:tasks.workunit.client.0.smithi116.stdout:5/154: write fa [1405548,120564] 0 2024-03-04T16:04:05.393 INFO:tasks.workunit.client.0.smithi116.stdout:7/74: link d3/f5 d3/f17 0 2024-03-04T16:04:05.394 INFO:tasks.workunit.client.0.smithi116.stdout:7/75: write f2 [255468,32271] 0 2024-03-04T16:04:05.395 INFO:tasks.workunit.client.0.smithi116.stdout:7/76: write d3/f6 [3324970,40699] 0 2024-03-04T16:04:05.397 INFO:tasks.workunit.client.0.smithi116.stdout:3/160: creat dc/d23/d2c/f3b x:0 0 0 2024-03-04T16:04:05.397 INFO:tasks.workunit.client.0.smithi116.stdout:1/86: dread f4 [0,4194304] 0 2024-03-04T16:04:05.399 INFO:tasks.workunit.client.0.smithi116.stdout:3/161: read dc/d23/d2c/f2d [511462,41107] 0 2024-03-04T16:04:05.401 INFO:tasks.workunit.client.0.smithi116.stdout:3/162: truncate dc/d23/d2c/f3b 469963 0 2024-03-04T16:04:05.402 INFO:tasks.workunit.client.0.smithi116.stdout:7/77: dwrite d3/fd [4194304,4194304] 0 2024-03-04T16:04:05.403 INFO:tasks.workunit.client.0.smithi116.stdout:7/78: write d3/f10 [857058,64568] 0 2024-03-04T16:04:05.409 INFO:tasks.workunit.client.0.smithi116.stdout:3/163: dwrite dc/df/d14/f1a [4194304,4194304] 0 2024-03-04T16:04:05.410 INFO:tasks.workunit.client.0.smithi116.stdout:7/79: dwrite d3/f14 [0,4194304] 0 2024-03-04T16:04:05.414 INFO:tasks.workunit.client.0.smithi116.stdout:4/126: sync 2024-03-04T16:04:05.415 INFO:tasks.workunit.client.0.smithi116.stdout:4/127: truncate d1/f2c 1006053 0 2024-03-04T16:04:05.416 INFO:tasks.workunit.client.0.smithi116.stdout:4/128: write d1/f2c [1936638,260] 0 2024-03-04T16:04:05.416 INFO:tasks.workunit.client.0.smithi116.stdout:4/129: fsync d1/d3/dd/d24/f29 0 2024-03-04T16:04:05.417 INFO:tasks.workunit.client.0.smithi116.stdout:4/130: write d1/d3/de/f12 [30063,36219] 0 2024-03-04T16:04:05.448 INFO:tasks.workunit.client.0.smithi116.stdout:6/158: sync 2024-03-04T16:04:05.448 INFO:tasks.workunit.client.0.smithi116.stdout:8/129: sync 2024-03-04T16:04:05.451 INFO:tasks.workunit.client.0.smithi116.stdout:6/159: dread d6/f17 [0,4194304] 0 2024-03-04T16:04:05.462 INFO:tasks.workunit.client.0.smithi116.stdout:6/160: dread d6/de/d1e/f20 [4194304,4194304] 0 2024-03-04T16:04:05.463 INFO:tasks.workunit.client.0.smithi116.stdout:6/161: truncate d6/f14 4225000 0 2024-03-04T16:04:05.466 INFO:tasks.workunit.client.0.smithi116.stdout:0/150: creat da/f35 x:0 0 0 2024-03-04T16:04:05.468 INFO:tasks.workunit.client.0.smithi116.stdout:9/114: mkdir d2/d26 0 2024-03-04T16:04:05.472 INFO:tasks.workunit.client.0.smithi116.stdout:7/80: creat d3/f18 x:0 0 0 2024-03-04T16:04:05.476 INFO:tasks.workunit.client.0.smithi116.stdout:6/162: symlink d6/d18/l32 0 2024-03-04T16:04:05.477 INFO:tasks.workunit.client.0.smithi116.stdout:0/151: symlink da/d16/d19/l36 0 2024-03-04T16:04:05.479 INFO:tasks.workunit.client.0.smithi116.stdout:5/155: mknod dd/d14/c37 0 2024-03-04T16:04:05.479 INFO:tasks.workunit.client.0.smithi116.stdout:5/156: stat dd/d14/l18 0 2024-03-04T16:04:05.482 INFO:tasks.workunit.client.0.smithi116.stdout:9/115: rename d2/d8/d1d to d2/d19/d27 0 2024-03-04T16:04:05.483 INFO:tasks.workunit.client.0.smithi116.stdout:9/116: fdatasync d2/d8/f21 0 2024-03-04T16:04:05.483 INFO:tasks.workunit.client.0.smithi116.stdout:2/124: creat d7/f36 x:0 0 0 2024-03-04T16:04:05.483 INFO:tasks.workunit.client.0.smithi116.stdout:9/117: fdatasync d2/d8/ff 0 2024-03-04T16:04:05.486 INFO:tasks.workunit.client.0.smithi116.stdout:1/87: truncate d5/f17 2675996 0 2024-03-04T16:04:05.491 INFO:tasks.workunit.client.0.smithi116.stdout:7/81: mknod d3/d4/d8/c19 0 2024-03-04T16:04:05.493 INFO:tasks.workunit.client.0.smithi116.stdout:8/130: mkdir d2/d13/d1d/d29 0 2024-03-04T16:04:05.494 INFO:tasks.workunit.client.0.smithi116.stdout:8/131: read d2/f7 [3875968,67120] 0 2024-03-04T16:04:05.496 INFO:tasks.workunit.client.0.smithi116.stdout:0/152: dread da/d16/f17 [0,4194304] 0 2024-03-04T16:04:05.496 INFO:tasks.workunit.client.0.smithi116.stdout:0/153: readlink da/d16/l23 0 2024-03-04T16:04:05.503 INFO:tasks.workunit.client.0.smithi116.stdout:4/131: creat d1/d3/de/f30 x:0 0 0 2024-03-04T16:04:05.504 INFO:tasks.workunit.client.0.smithi116.stdout:7/82: mkdir d3/d4/d8/d1a 0 2024-03-04T16:04:05.504 INFO:tasks.workunit.client.0.smithi116.stdout:7/83: chown d3/d4/d11 16301 1 2024-03-04T16:04:05.506 INFO:tasks.workunit.client.0.smithi116.stdout:4/132: dread d1/d3/d15/d20/d1c/f27 [0,4194304] 0 2024-03-04T16:04:05.510 INFO:tasks.workunit.client.0.smithi116.stdout:7/84: dwrite d3/f14 [0,4194304] 0 2024-03-04T16:04:05.510 INFO:tasks.workunit.client.0.smithi116.stdout:2/125: mknod d7/c37 0 2024-03-04T16:04:05.510 INFO:tasks.workunit.client.0.smithi116.stdout:7/85: readlink d3/l7 0 2024-03-04T16:04:05.511 INFO:tasks.workunit.client.0.smithi116.stdout:9/118: mkdir d2/d26/d28 0 2024-03-04T16:04:05.512 INFO:tasks.workunit.client.0.smithi116.stdout:9/119: chown d2/d8/f17 211055275 1 2024-03-04T16:04:05.512 INFO:tasks.workunit.client.0.smithi116.stdout:9/120: read d2/f15 [2393070,114564] 0 2024-03-04T16:04:05.519 INFO:tasks.workunit.client.0.smithi116.stdout:5/157: creat dd/f38 x:0 0 0 2024-03-04T16:04:05.520 INFO:tasks.workunit.client.0.smithi116.stdout:4/133: creat d1/d3/dd/f31 x:0 0 0 2024-03-04T16:04:05.523 INFO:tasks.workunit.client.0.smithi116.stdout:2/126: creat d7/d15/f38 x:0 0 0 2024-03-04T16:04:05.526 INFO:tasks.workunit.client.0.smithi116.stdout:6/163: link d6/de/c16 d6/de/c33 0 2024-03-04T16:04:05.527 INFO:tasks.workunit.client.0.smithi116.stdout:6/164: dread - d6/d18/f1d zero size 2024-03-04T16:04:05.528 INFO:tasks.workunit.client.0.smithi116.stdout:5/158: readlink dd/d14/d2d/l33 0 2024-03-04T16:04:05.529 INFO:tasks.workunit.client.0.smithi116.stdout:2/127: dwrite d7/d29/f2d [0,4194304] 0 2024-03-04T16:04:05.531 INFO:tasks.workunit.client.0.smithi116.stdout:7/86: getdents d3/d4/d11 0 2024-03-04T16:04:05.535 INFO:tasks.workunit.client.0.smithi116.stdout:5/159: dwrite dd/f22 [0,4194304] 0 2024-03-04T16:04:05.536 INFO:tasks.workunit.client.0.smithi116.stdout:5/160: fdatasync fa 0 2024-03-04T16:04:05.537 INFO:tasks.workunit.client.0.smithi116.stdout:2/128: dwrite d7/d15/f2e [0,4194304] 0 2024-03-04T16:04:05.538 INFO:tasks.workunit.client.0.smithi116.stdout:4/134: link d1/d3/de/f13 d1/f32 0 2024-03-04T16:04:05.539 INFO:tasks.workunit.client.0.smithi116.stdout:4/135: dread - d1/d3/dd/d24/f29 zero size 2024-03-04T16:04:05.543 INFO:tasks.workunit.client.0.smithi116.stdout:6/165: creat d6/de/d1e/f34 x:0 0 0 2024-03-04T16:04:05.544 INFO:tasks.workunit.client.0.smithi116.stdout:0/154: getdents da/d16/d20 0 2024-03-04T16:04:05.544 INFO:tasks.workunit.client.0.smithi116.stdout:0/155: dread - da/f35 zero size 2024-03-04T16:04:05.548 INFO:tasks.workunit.client.0.smithi116.stdout:7/87: rename d3/l7 to d3/d4/d8/l1b 0 2024-03-04T16:04:05.554 INFO:tasks.workunit.client.0.smithi116.stdout:2/129: creat d7/d1b/f39 x:0 0 0 2024-03-04T16:04:05.557 INFO:tasks.workunit.client.0.smithi116.stdout:6/166: creat d6/de/d1e/f35 x:0 0 0 2024-03-04T16:04:05.558 INFO:tasks.workunit.client.0.smithi116.stdout:6/167: truncate d6/de/d1e/f35 924990 0 2024-03-04T16:04:05.559 INFO:tasks.workunit.client.0.smithi116.stdout:0/156: rename c9 to da/d16/d28/c37 0 2024-03-04T16:04:05.559 INFO:tasks.workunit.client.0.smithi116.stdout:6/168: write d6/f13 [1064890,120655] 0 2024-03-04T16:04:05.560 INFO:tasks.workunit.client.0.smithi116.stdout:7/88: creat d3/d4/d8/f1c x:0 0 0 2024-03-04T16:04:05.562 INFO:tasks.workunit.client.0.smithi116.stdout:5/161: mknod dd/d14/d2d/d31/c39 0 2024-03-04T16:04:05.564 INFO:tasks.workunit.client.0.smithi116.stdout:4/136: mkdir d1/d3/de/d33 0 2024-03-04T16:04:05.568 INFO:tasks.workunit.client.0.smithi116.stdout:5/162: dwrite dd/f15 [0,4194304] 0 2024-03-04T16:04:05.571 INFO:tasks.workunit.client.0.smithi116.stdout:2/130: dwrite d7/d13/f1a [0,4194304] 0 2024-03-04T16:04:05.572 INFO:tasks.workunit.client.0.smithi116.stdout:2/131: truncate d7/d1b/f26 240959 0 2024-03-04T16:04:05.572 INFO:tasks.workunit.client.0.smithi116.stdout:2/132: chown d7/d1b/f1c 0 1 2024-03-04T16:04:05.573 INFO:tasks.workunit.client.0.smithi116.stdout:2/133: dread - d7/d25/f34 zero size 2024-03-04T16:04:05.573 INFO:tasks.workunit.client.0.smithi116.stdout:4/137: creat d1/d3/d15/f34 x:0 0 0 2024-03-04T16:04:05.584 INFO:tasks.workunit.client.0.smithi116.stdout:4/138: creat d1/d3/d15/d20/f35 x:0 0 0 2024-03-04T16:04:05.585 INFO:tasks.workunit.client.0.smithi116.stdout:4/139: dread - d1/d3/dd/f31 zero size 2024-03-04T16:04:05.593 INFO:tasks.workunit.client.0.smithi116.stdout:5/163: link dd/l1a dd/d14/d2d/l3a 0 2024-03-04T16:04:05.596 INFO:tasks.workunit.client.0.smithi116.stdout:5/164: dread f5 [0,4194304] 0 2024-03-04T16:04:05.602 INFO:tasks.workunit.client.0.smithi116.stdout:5/165: dwrite dd/f38 [0,4194304] 0 2024-03-04T16:04:05.603 INFO:tasks.workunit.client.0.smithi116.stdout:3/164: truncate dc/d23/f1b 782339 0 2024-03-04T16:04:05.603 INFO:tasks.workunit.client.0.smithi116.stdout:3/165: chown dc/f16 1657 1 2024-03-04T16:04:05.607 INFO:tasks.workunit.client.0.smithi116.stdout:5/166: dread dd/f15 [0,4194304] 0 2024-03-04T16:04:05.613 INFO:tasks.workunit.client.0.smithi116.stdout:5/167: creat dd/d14/d2d/f3b x:0 0 0 2024-03-04T16:04:05.613 INFO:tasks.workunit.client.0.smithi116.stdout:1/88: write f4 [1877148,129863] 0 2024-03-04T16:04:05.614 INFO:tasks.workunit.client.0.smithi116.stdout:1/89: write d5/f1d [609838,57326] 0 2024-03-04T16:04:05.617 INFO:tasks.workunit.client.0.smithi116.stdout:5/168: unlink dd/d1f/l2c 0 2024-03-04T16:04:05.618 INFO:tasks.workunit.client.0.smithi116.stdout:5/169: chown dd/d14/l18 1 1 2024-03-04T16:04:05.619 INFO:tasks.workunit.client.0.smithi116.stdout:1/90: creat d5/d11/f20 x:0 0 0 2024-03-04T16:04:05.622 INFO:tasks.workunit.client.0.smithi116.stdout:5/170: write dd/f15 [531902,80911] 0 2024-03-04T16:04:05.623 INFO:tasks.workunit.client.0.smithi116.stdout:1/91: mkdir d5/d9/d21 0 2024-03-04T16:04:05.623 INFO:tasks.workunit.client.0.smithi116.stdout:5/171: write dd/d1f/f29 [321141,42650] 0 2024-03-04T16:04:05.624 INFO:tasks.workunit.client.0.smithi116.stdout:5/172: write dd/f38 [2715629,15456] 0 2024-03-04T16:04:05.629 INFO:tasks.workunit.client.0.smithi116.stdout:5/173: dwrite dd/fe [0,4194304] 0 2024-03-04T16:04:05.630 INFO:tasks.workunit.client.0.smithi116.stdout:5/174: fdatasync f5 0 2024-03-04T16:04:05.635 INFO:tasks.workunit.client.0.smithi116.stdout:5/175: dwrite dd/fe [0,4194304] 0 2024-03-04T16:04:05.636 INFO:tasks.workunit.client.0.smithi116.stdout:1/92: getdents d5/d19 0 2024-03-04T16:04:05.641 INFO:tasks.workunit.client.0.smithi116.stdout:1/93: creat d5/d19/f22 x:0 0 0 2024-03-04T16:04:05.642 INFO:tasks.workunit.client.0.smithi116.stdout:1/94: readlink d5/d19/l1f 0 2024-03-04T16:04:05.645 INFO:tasks.workunit.client.0.smithi116.stdout:1/95: creat d5/d9/f23 x:0 0 0 2024-03-04T16:04:05.652 INFO:tasks.workunit.client.0.smithi116.stdout:7/89: readlink d3/d4/d8/l1b 0 2024-03-04T16:04:05.657 INFO:tasks.workunit.client.0.smithi116.stdout:7/90: dwrite f1 [4194304,4194304] 0 2024-03-04T16:04:05.661 INFO:tasks.workunit.client.0.smithi116.stdout:7/91: chown d3 8 1 2024-03-04T16:04:05.661 INFO:tasks.workunit.client.0.smithi116.stdout:2/134: getdents d7/d1b 0 2024-03-04T16:04:05.664 INFO:tasks.workunit.client.0.smithi116.stdout:2/135: creat d7/d29/f3a x:0 0 0 2024-03-04T16:04:05.667 INFO:tasks.workunit.client.0.smithi116.stdout:7/92: rename d3/d4/d11/l15 to d3/d4/d8/d1a/l1d 0 2024-03-04T16:04:05.670 INFO:tasks.workunit.client.0.smithi116.stdout:0/157: write da/d16/d20/f21 [912885,85629] 0 2024-03-04T16:04:05.671 INFO:tasks.workunit.client.0.smithi116.stdout:7/93: unlink d3/d4/d8/fa 0 2024-03-04T16:04:05.673 INFO:tasks.workunit.client.0.smithi116.stdout:2/136: truncate f4 1491286 0 2024-03-04T16:04:05.676 INFO:tasks.workunit.client.0.smithi116.stdout:0/158: symlink da/d16/d28/l38 0 2024-03-04T16:04:05.679 INFO:tasks.workunit.client.0.smithi116.stdout:7/94: dwrite d3/f18 [0,4194304] 0 2024-03-04T16:04:05.680 INFO:tasks.workunit.client.0.smithi116.stdout:0/159: write da/d16/d19/d27/f34 [610386,52499] 0 2024-03-04T16:04:05.681 INFO:tasks.workunit.client.0.smithi116.stdout:2/137: creat d7/d15/d35/f3b x:0 0 0 2024-03-04T16:04:05.682 INFO:tasks.workunit.client.0.smithi116.stdout:6/169: dwrite d6/de/f11 [0,4194304] 0 2024-03-04T16:04:05.684 INFO:tasks.workunit.client.0.smithi116.stdout:4/140: write d1/d3/de/f10 [1713282,11982] 0 2024-03-04T16:04:05.685 INFO:tasks.workunit.client.0.smithi116.stdout:4/141: fsync d1/d3/de/f30 0 2024-03-04T16:04:05.686 INFO:tasks.workunit.client.0.smithi116.stdout:7/95: rename d3/f17 to d3/f1e 0 2024-03-04T16:04:05.687 INFO:tasks.workunit.client.0.smithi116.stdout:4/142: truncate d1/d3/de/f1b 757143 0 2024-03-04T16:04:05.690 INFO:tasks.workunit.client.0.smithi116.stdout:0/160: rename da/d16/d20/d2b to da/d16/d28/d39 0 2024-03-04T16:04:05.695 INFO:tasks.workunit.client.0.smithi116.stdout:7/96: creat d3/f1f x:0 0 0 2024-03-04T16:04:05.696 INFO:tasks.workunit.client.0.smithi116.stdout:4/143: symlink d1/d3/d15/d20/d1c/l36 0 2024-03-04T16:04:05.700 INFO:tasks.workunit.client.0.smithi116.stdout:6/170: rename d6/d18 to d6/de/d25/d36 0 2024-03-04T16:04:05.700 INFO:tasks.workunit.client.0.smithi116.stdout:3/166: truncate dc/df/d14/f2b 149832 0 2024-03-04T16:04:05.701 INFO:tasks.workunit.client.0.smithi116.stdout:0/161: symlink da/d16/d28/l3a 0 2024-03-04T16:04:05.702 INFO:tasks.workunit.client.0.smithi116.stdout:0/162: read da/f1a [3946111,115705] 0 2024-03-04T16:04:05.706 INFO:tasks.workunit.client.0.smithi116.stdout:6/171: creat d6/f37 x:0 0 0 2024-03-04T16:04:05.707 INFO:tasks.workunit.client.0.smithi116.stdout:3/167: creat dc/df/f3c x:0 0 0 2024-03-04T16:04:05.708 INFO:tasks.workunit.client.0.smithi116.stdout:3/168: fdatasync dc/df/d14/f31 0 2024-03-04T16:04:05.709 INFO:tasks.workunit.client.0.smithi116.stdout:3/169: write dc/df/d14/f1a [694869,61430] 0 2024-03-04T16:04:05.710 INFO:tasks.workunit.client.0.smithi116.stdout:3/170: dread - dc/d23/f21 zero size 2024-03-04T16:04:05.711 INFO:tasks.workunit.client.0.smithi116.stdout:3/171: chown dc/d23/d2c/f2d 12 1 2024-03-04T16:04:05.711 INFO:tasks.workunit.client.0.smithi116.stdout:3/172: stat dc/d23/d2c 0 2024-03-04T16:04:05.713 INFO:tasks.workunit.client.0.smithi116.stdout:4/144: dread d1/fb [0,4194304] 0 2024-03-04T16:04:05.713 INFO:tasks.workunit.client.0.smithi116.stdout:0/163: unlink da/d16/d20/f22 0 2024-03-04T16:04:05.716 INFO:tasks.workunit.client.0.smithi116.stdout:6/172: truncate d6/de/d25/d36/f1d 205843 0 2024-03-04T16:04:05.720 INFO:tasks.workunit.client.0.smithi116.stdout:1/96: getdents d5/d9 0 2024-03-04T16:04:05.723 INFO:tasks.workunit.client.0.smithi116.stdout:3/173: mknod dc/df/d14/c3d 0 2024-03-04T16:04:05.725 INFO:tasks.workunit.client.0.smithi116.stdout:3/174: dread dc/d23/d2c/f3b [0,4194304] 0 2024-03-04T16:04:05.726 INFO:tasks.workunit.client.0.smithi116.stdout:1/97: dwrite d5/f1c [0,4194304] 0 2024-03-04T16:04:05.726 INFO:tasks.workunit.client.0.smithi116.stdout:0/164: chown da/l11 202 1 2024-03-04T16:04:05.727 INFO:tasks.workunit.client.0.smithi116.stdout:3/175: dread - dc/d23/f21 zero size 2024-03-04T16:04:05.727 INFO:tasks.workunit.client.0.smithi116.stdout:6/173: creat d6/de/f38 x:0 0 0 2024-03-04T16:04:05.727 INFO:tasks.workunit.client.0.smithi116.stdout:1/98: chown d5/d19/l1f 12612669 1 2024-03-04T16:04:05.728 INFO:tasks.workunit.client.0.smithi116.stdout:6/174: dread - d6/de/f38 zero size 2024-03-04T16:04:05.729 INFO:tasks.workunit.client.0.smithi116.stdout:1/99: dread - d5/d11/f20 zero size 2024-03-04T16:04:05.734 INFO:tasks.workunit.client.0.smithi116.stdout:6/175: dwrite d6/de/d15/f26 [0,4194304] 0 2024-03-04T16:04:05.735 INFO:tasks.workunit.client.0.smithi116.stdout:0/165: creat da/d16/d20/f3b x:0 0 0 2024-03-04T16:04:05.736 INFO:tasks.workunit.client.0.smithi116.stdout:8/132: sync 2024-03-04T16:04:05.737 INFO:tasks.workunit.client.0.smithi116.stdout:3/176: creat dc/d34/f3e x:0 0 0 2024-03-04T16:04:05.737 INFO:tasks.workunit.client.0.smithi116.stdout:8/133: write d2/de/d16/f22 [4260964,8506] 0 2024-03-04T16:04:05.744 INFO:tasks.workunit.client.0.smithi116.stdout:1/100: dwrite d5/d11/f16 [0,4194304] 0 2024-03-04T16:04:05.747 INFO:tasks.workunit.client.0.smithi116.stdout:6/176: unlink d6/de/f23 0 2024-03-04T16:04:05.751 INFO:tasks.workunit.client.0.smithi116.stdout:3/177: mknod dc/df/c3f 0 2024-03-04T16:04:05.753 INFO:tasks.workunit.client.0.smithi116.stdout:8/134: creat d2/de/d16/d1c/f2a x:0 0 0 2024-03-04T16:04:05.761 INFO:tasks.workunit.client.0.smithi116.stdout:6/177: mknod d6/de/d25/c39 0 2024-03-04T16:04:05.763 INFO:tasks.workunit.client.0.smithi116.stdout:3/178: rename dc/ce to dc/c40 0 2024-03-04T16:04:05.764 INFO:tasks.workunit.client.0.smithi116.stdout:8/135: rmdir d2/d13/d1e 39 2024-03-04T16:04:05.766 INFO:tasks.workunit.client.0.smithi116.stdout:1/101: mkdir d5/d9/d21/d24 0 2024-03-04T16:04:05.767 INFO:tasks.workunit.client.0.smithi116.stdout:7/97: truncate d3/f6 1019523 0 2024-03-04T16:04:05.767 INFO:tasks.workunit.client.0.smithi116.stdout:1/102: chown d5/d19/l1f 201804591 1 2024-03-04T16:04:05.770 INFO:tasks.workunit.client.0.smithi116.stdout:6/178: mkdir d6/de/d29/d3a 0 2024-03-04T16:04:05.771 INFO:tasks.workunit.client.0.smithi116.stdout:3/179: chown cb 3 1 2024-03-04T16:04:05.772 INFO:tasks.workunit.client.0.smithi116.stdout:8/136: mkdir d2/de/d16/d2b 0 2024-03-04T16:04:05.775 INFO:tasks.workunit.client.0.smithi116.stdout:6/179: dwrite d6/de/d1e/f35 [0,4194304] 0 2024-03-04T16:04:05.778 INFO:tasks.workunit.client.0.smithi116.stdout:8/137: dread d2/de/f8 [0,4194304] 0 2024-03-04T16:04:05.782 INFO:tasks.workunit.client.0.smithi116.stdout:6/180: dwrite d6/de/f11 [0,4194304] 0 2024-03-04T16:04:05.788 INFO:tasks.workunit.client.0.smithi116.stdout:1/103: unlink c1 0 2024-03-04T16:04:05.792 INFO:tasks.workunit.client.0.smithi116.stdout:1/104: dread f4 [0,4194304] 0 2024-03-04T16:04:05.798 INFO:tasks.workunit.client.0.smithi116.stdout:3/180: dread dc/d23/d2c/f3b [0,4194304] 0 2024-03-04T16:04:05.799 INFO:tasks.workunit.client.0.smithi116.stdout:3/181: stat c8 0 2024-03-04T16:04:05.800 INFO:tasks.workunit.client.0.smithi116.stdout:6/181: unlink f3 0 2024-03-04T16:04:05.813 INFO:tasks.workunit.client.0.smithi116.stdout:6/182: write d6/f24 [533566,94001] 0 2024-03-04T16:04:05.813 INFO:tasks.workunit.client.0.smithi116.stdout:6/183: stat d6/l12 0 2024-03-04T16:04:05.814 INFO:tasks.workunit.client.0.smithi116.stdout:6/184: write d6/f14 [1348959,55328] 0 2024-03-04T16:04:05.815 INFO:tasks.workunit.client.0.smithi116.stdout:6/185: stat d6/de/d25/d36/l32 0 2024-03-04T16:04:05.817 INFO:tasks.workunit.client.0.smithi116.stdout:1/105: mknod d5/d9/d21/d24/c25 0 2024-03-04T16:04:05.817 INFO:tasks.workunit.client.0.smithi116.stdout:1/106: chown d5/d11/f16 19 1 2024-03-04T16:04:05.818 INFO:tasks.workunit.client.0.smithi116.stdout:7/98: fsync d3/f6 0 2024-03-04T16:04:05.824 INFO:tasks.workunit.client.0.smithi116.stdout:6/186: creat d6/de/f3b x:0 0 0 2024-03-04T16:04:05.826 INFO:tasks.workunit.client.0.smithi116.stdout:7/99: creat d3/d4/d8/d1a/f20 x:0 0 0 2024-03-04T16:04:05.830 INFO:tasks.workunit.client.0.smithi116.stdout:7/100: symlink d3/d4/l21 0 2024-03-04T16:04:05.833 INFO:tasks.workunit.client.0.smithi116.stdout:7/101: dread d3/fd [4194304,4194304] 0 2024-03-04T16:04:05.833 INFO:tasks.workunit.client.0.smithi116.stdout:9/121: sync 2024-03-04T16:04:05.835 INFO:tasks.workunit.client.0.smithi116.stdout:3/182: getdents dc/df/d14 0 2024-03-04T16:04:05.835 INFO:tasks.workunit.client.0.smithi116.stdout:1/107: getdents d5/d9/d21 0 2024-03-04T16:04:05.836 INFO:tasks.workunit.client.0.smithi116.stdout:1/108: fdatasync d5/d9/f23 0 2024-03-04T16:04:05.837 INFO:tasks.workunit.client.0.smithi116.stdout:1/109: write d5/f1d [151521,102599] 0 2024-03-04T16:04:05.839 INFO:tasks.workunit.client.0.smithi116.stdout:8/138: dwrite d2/f7 [0,4194304] 0 2024-03-04T16:04:05.843 INFO:tasks.workunit.client.0.smithi116.stdout:3/183: truncate dc/f16 1084329 0 2024-03-04T16:04:05.843 INFO:tasks.workunit.client.0.smithi116.stdout:1/110: dread d5/f1d [0,4194304] 0 2024-03-04T16:04:05.851 INFO:tasks.workunit.client.0.smithi116.stdout:3/184: creat dc/d23/f41 x:0 0 0 2024-03-04T16:04:05.852 INFO:tasks.workunit.client.0.smithi116.stdout:3/185: write dc/df/d14/f1a [7008036,83456] 0 2024-03-04T16:04:05.862 INFO:tasks.workunit.client.0.smithi116.stdout:3/186: dread dc/d23/f1b [0,4194304] 0 2024-03-04T16:04:05.867 INFO:tasks.workunit.client.0.smithi116.stdout:3/187: creat dc/d23/f42 x:0 0 0 2024-03-04T16:04:05.868 INFO:tasks.workunit.client.0.smithi116.stdout:9/122: truncate d2/f1a 3006936 0 2024-03-04T16:04:05.871 INFO:tasks.workunit.client.0.smithi116.stdout:3/188: unlink dc/d23/l28 0 2024-03-04T16:04:05.875 INFO:tasks.workunit.client.0.smithi116.stdout:8/139: write d2/de/fa [1624130,79755] 0 2024-03-04T16:04:05.877 INFO:tasks.workunit.client.0.smithi116.stdout:9/123: mknod d2/d26/d28/c29 0 2024-03-04T16:04:05.878 INFO:tasks.workunit.client.0.smithi116.stdout:3/189: dread dc/f33 [0,4194304] 0 2024-03-04T16:04:05.879 INFO:tasks.workunit.client.0.smithi116.stdout:6/187: dread d6/de/d1e/f20 [0,4194304] 0 2024-03-04T16:04:05.880 INFO:tasks.workunit.client.0.smithi116.stdout:6/188: write d6/f14 [1880141,124183] 0 2024-03-04T16:04:05.880 INFO:tasks.workunit.client.0.smithi116.stdout:8/140: creat d2/de/d16/f2c x:0 0 0 2024-03-04T16:04:05.881 INFO:tasks.workunit.client.0.smithi116.stdout:8/141: chown d2/c11 4979 1 2024-03-04T16:04:05.884 INFO:tasks.workunit.client.0.smithi116.stdout:3/190: mknod dc/d23/c43 0 2024-03-04T16:04:05.889 INFO:tasks.workunit.client.0.smithi116.stdout:1/111: dwrite d5/f17 [0,4194304] 0 2024-03-04T16:04:05.892 INFO:tasks.workunit.client.0.smithi116.stdout:6/189: dread d6/de/f1b [0,4194304] 0 2024-03-04T16:04:05.893 INFO:tasks.workunit.client.0.smithi116.stdout:9/124: dread d2/d8/ff [0,4194304] 0 2024-03-04T16:04:05.894 INFO:tasks.workunit.client.0.smithi116.stdout:6/190: chown d6/f17 22604468 1 2024-03-04T16:04:05.894 INFO:tasks.workunit.client.0.smithi116.stdout:8/142: mknod d2/de/d16/d2b/c2d 0 2024-03-04T16:04:05.896 INFO:tasks.workunit.client.0.smithi116.stdout:8/143: truncate d2/de/d16/f23 611894 0 2024-03-04T16:04:05.896 INFO:tasks.workunit.client.0.smithi116.stdout:3/191: mkdir dc/d34/d44 0 2024-03-04T16:04:05.902 INFO:tasks.workunit.client.0.smithi116.stdout:9/125: rename d2/d8/f17 to d2/d19/f2a 0 2024-03-04T16:04:05.902 INFO:tasks.workunit.client.0.smithi116.stdout:9/126: chown d2/d8/l18 11 1 2024-03-04T16:04:05.912 INFO:tasks.workunit.client.0.smithi116.stdout:8/144: rename d2/de/d16/d1c to d2/d13/d1d/d29/d2e 0 2024-03-04T16:04:05.919 INFO:tasks.workunit.client.0.smithi116.stdout:3/192: mknod dc/d34/d44/c45 0 2024-03-04T16:04:05.920 INFO:tasks.workunit.client.0.smithi116.stdout:3/193: dread - dc/d23/f21 zero size 2024-03-04T16:04:05.921 INFO:tasks.workunit.client.0.smithi116.stdout:1/112: getdents d5 0 2024-03-04T16:04:05.922 INFO:tasks.workunit.client.0.smithi116.stdout:1/113: chown f2 1343652 1 2024-03-04T16:04:05.928 INFO:tasks.workunit.client.0.smithi116.stdout:8/145: dread d2/de/d16/f1b [0,4194304] 0 2024-03-04T16:04:05.930 INFO:tasks.workunit.client.0.smithi116.stdout:3/194: creat dc/d23/f46 x:0 0 0 2024-03-04T16:04:05.931 INFO:tasks.workunit.client.0.smithi116.stdout:6/191: truncate d6/f13 141163 0 2024-03-04T16:04:05.932 INFO:tasks.workunit.client.0.smithi116.stdout:1/114: creat d5/d9/d21/f26 x:0 0 0 2024-03-04T16:04:05.935 INFO:tasks.workunit.client.0.smithi116.stdout:9/127: creat d2/f2b x:0 0 0 2024-03-04T16:04:05.938 INFO:tasks.workunit.client.0.smithi116.stdout:3/195: mkdir dc/df/d47 0 2024-03-04T16:04:05.939 INFO:tasks.workunit.client.0.smithi116.stdout:9/128: symlink d2/d19/d27/l2c 0 2024-03-04T16:04:05.940 INFO:tasks.workunit.client.0.smithi116.stdout:9/129: truncate d2/d8/f21 672438 0 2024-03-04T16:04:05.940 INFO:tasks.workunit.client.0.smithi116.stdout:8/146: mknod d2/d13/d1e/c2f 0 2024-03-04T16:04:05.942 INFO:tasks.workunit.client.0.smithi116.stdout:3/196: unlink dc/df/d14/f31 0 2024-03-04T16:04:05.942 INFO:tasks.workunit.client.0.smithi116.stdout:8/147: readlink d2/de/l1a 0 2024-03-04T16:04:05.944 INFO:tasks.workunit.client.0.smithi116.stdout:3/197: mknod dc/d34/d44/c48 0 2024-03-04T16:04:05.947 INFO:tasks.workunit.client.0.smithi116.stdout:3/198: creat dc/df/d14/f49 x:0 0 0 2024-03-04T16:04:05.966 INFO:tasks.workunit.client.0.smithi116.stdout:5/176: sync 2024-03-04T16:04:05.969 INFO:tasks.workunit.client.0.smithi116.stdout:5/177: creat dd/d14/d2d/d31/f3c x:0 0 0 2024-03-04T16:04:05.970 INFO:tasks.workunit.client.0.smithi116.stdout:8/148: write d2/de/d16/f17 [1453009,60535] 0 2024-03-04T16:04:05.971 INFO:tasks.workunit.client.0.smithi116.stdout:8/149: write d2/de/d16/f23 [193199,60681] 0 2024-03-04T16:04:05.972 INFO:tasks.workunit.client.0.smithi116.stdout:8/150: chown d2/de 109727542 1 2024-03-04T16:04:05.972 INFO:tasks.workunit.client.0.smithi116.stdout:3/199: write dc/d23/d2c/f3b [1334441,16590] 0 2024-03-04T16:04:05.972 INFO:tasks.workunit.client.0.smithi116.stdout:3/200: chown dc/df/d47 1 1 2024-03-04T16:04:05.973 INFO:tasks.workunit.client.0.smithi116.stdout:5/178: symlink dd/d1f/l3d 0 2024-03-04T16:04:05.975 INFO:tasks.workunit.client.0.smithi116.stdout:8/151: creat d2/de/d16/f30 x:0 0 0 2024-03-04T16:04:05.980 INFO:tasks.workunit.client.0.smithi116.stdout:8/152: mknod d2/d13/d1d/d29/c31 0 2024-03-04T16:04:05.980 INFO:tasks.workunit.client.0.smithi116.stdout:8/153: chown d2/de/l19 16096818 1 2024-03-04T16:04:05.991 INFO:tasks.workunit.client.0.smithi116.stdout:8/154: creat d2/f32 x:0 0 0 2024-03-04T16:04:05.994 INFO:tasks.workunit.client.0.smithi116.stdout:8/155: symlink d2/de/d16/l33 0 2024-03-04T16:04:05.998 INFO:tasks.workunit.client.0.smithi116.stdout:8/156: dwrite d2/de/f10 [0,4194304] 0 2024-03-04T16:04:05.999 INFO:tasks.workunit.client.0.smithi116.stdout:8/157: write d2/de/f10 [3524198,30760] 0 2024-03-04T16:04:05.999 INFO:tasks.workunit.client.0.smithi116.stdout:8/158: chown d2/c11 3903280 1 2024-03-04T16:04:06.008 INFO:tasks.workunit.client.0.smithi116.stdout:3/201: dwrite dc/d23/f1b [0,4194304] 0 2024-03-04T16:04:06.012 INFO:tasks.workunit.client.0.smithi116.stdout:3/202: symlink dc/d23/d2c/l4a 0 2024-03-04T16:04:06.014 INFO:tasks.workunit.client.0.smithi116.stdout:5/179: truncate dd/f38 3466383 0 2024-03-04T16:04:06.017 INFO:tasks.workunit.client.0.smithi116.stdout:3/203: dwrite dc/d23/f42 [0,4194304] 0 2024-03-04T16:04:06.035 INFO:tasks.workunit.client.0.smithi116.stdout:3/204: fsync dc/d23/f42 0 2024-03-04T16:04:06.041 INFO:tasks.workunit.client.0.smithi116.stdout:3/205: creat dc/f4b x:0 0 0 2024-03-04T16:04:06.045 INFO:tasks.workunit.client.0.smithi116.stdout:3/206: creat dc/d23/d2c/f4c x:0 0 0 2024-03-04T16:04:06.048 INFO:tasks.workunit.client.0.smithi116.stdout:3/207: mknod dc/df/d14/c4d 0 2024-03-04T16:04:06.053 INFO:tasks.workunit.client.0.smithi116.stdout:3/208: unlink dc/l18 0 2024-03-04T16:04:06.059 INFO:tasks.workunit.client.0.smithi116.stdout:3/209: dwrite dc/df/d14/f49 [0,4194304] 0 2024-03-04T16:04:06.059 INFO:tasks.workunit.client.0.smithi116.stdout:2/138: sync 2024-03-04T16:04:06.059 INFO:tasks.workunit.client.0.smithi116.stdout:4/145: sync 2024-03-04T16:04:06.061 INFO:tasks.workunit.client.0.smithi116.stdout:2/139: truncate d7/f2b 284110 0 2024-03-04T16:04:06.062 INFO:tasks.workunit.client.0.smithi116.stdout:4/146: dread d1/f32 [0,4194304] 0 2024-03-04T16:04:06.063 INFO:tasks.workunit.client.0.smithi116.stdout:2/140: write d7/d15/d35/f3b [459760,90572] 0 2024-03-04T16:04:06.066 INFO:tasks.workunit.client.0.smithi116.stdout:3/210: dread dc/f33 [0,4194304] 0 2024-03-04T16:04:06.067 INFO:tasks.workunit.client.0.smithi116.stdout:5/180: write dd/f38 [3258639,71618] 0 2024-03-04T16:04:06.073 INFO:tasks.workunit.client.0.smithi116.stdout:5/181: stat dd/l1a 0 2024-03-04T16:04:06.074 INFO:tasks.workunit.client.0.smithi116.stdout:5/182: chown dd/f22 12 1 2024-03-04T16:04:06.074 INFO:tasks.workunit.client.0.smithi116.stdout:5/183: chown dd/f38 1398 1 2024-03-04T16:04:06.084 INFO:tasks.workunit.client.0.smithi116.stdout:5/184: symlink dd/d1f/l3e 0 2024-03-04T16:04:06.087 INFO:tasks.workunit.client.0.smithi116.stdout:4/147: getdents d1/d3/d15 0 2024-03-04T16:04:06.094 INFO:tasks.workunit.client.0.smithi116.stdout:2/141: truncate d7/d15/f2e 824344 0 2024-03-04T16:04:06.095 INFO:tasks.workunit.client.0.smithi116.stdout:4/148: dwrite d1/fb [0,4194304] 0 2024-03-04T16:04:06.097 INFO:tasks.workunit.client.0.smithi116.stdout:2/142: mknod d7/d25/c3c 0 2024-03-04T16:04:06.101 INFO:tasks.workunit.client.0.smithi116.stdout:2/143: write d7/d15/f30 [200921,71541] 0 2024-03-04T16:04:06.103 INFO:tasks.workunit.client.0.smithi116.stdout:5/185: write dd/f11 [853715,69515] 0 2024-03-04T16:04:06.105 INFO:tasks.workunit.client.0.smithi116.stdout:2/144: write d7/d15/f30 [154954,42054] 0 2024-03-04T16:04:06.106 INFO:tasks.workunit.client.0.smithi116.stdout:4/149: dwrite d1/d3/d15/f34 [0,4194304] 0 2024-03-04T16:04:06.107 INFO:tasks.workunit.client.0.smithi116.stdout:3/211: dwrite dc/df/d14/f2b [0,4194304] 0 2024-03-04T16:04:06.110 INFO:tasks.workunit.client.0.smithi116.stdout:6/192: sync 2024-03-04T16:04:06.110 INFO:tasks.workunit.client.0.smithi116.stdout:0/166: sync 2024-03-04T16:04:06.114 INFO:tasks.workunit.client.0.smithi116.stdout:6/193: truncate d6/de/f38 723536 0 2024-03-04T16:04:06.115 INFO:tasks.workunit.client.0.smithi116.stdout:0/167: write da/d16/d19/d27/f34 [1393111,30921] 0 2024-03-04T16:04:06.116 INFO:tasks.workunit.client.0.smithi116.stdout:5/186: dwrite dd/f15 [0,4194304] 0 2024-03-04T16:04:06.117 INFO:tasks.workunit.client.0.smithi116.stdout:0/168: readlink da/l11 0 2024-03-04T16:04:06.117 INFO:tasks.workunit.client.0.smithi116.stdout:5/187: stat c6 0 2024-03-04T16:04:06.117 INFO:tasks.workunit.client.0.smithi116.stdout:3/212: rename dc/df/d14/c4d to dc/d34/c4e 0 2024-03-04T16:04:06.118 INFO:tasks.workunit.client.0.smithi116.stdout:3/213: truncate dc/d23/f41 260665 0 2024-03-04T16:04:06.121 INFO:tasks.workunit.client.0.smithi116.stdout:3/214: dread dc/d23/d2c/f3b [0,4194304] 0 2024-03-04T16:04:06.125 INFO:tasks.workunit.client.0.smithi116.stdout:6/194: symlink d6/de/d29/l3c 0 2024-03-04T16:04:06.125 INFO:tasks.workunit.client.0.smithi116.stdout:6/195: chown d6/de/d25/d36 98 1 2024-03-04T16:04:06.126 INFO:tasks.workunit.client.0.smithi116.stdout:6/196: stat d6/de/d25 0 2024-03-04T16:04:06.128 INFO:tasks.workunit.client.0.smithi116.stdout:0/169: creat da/d16/d28/f3c x:0 0 0 2024-03-04T16:04:06.129 INFO:tasks.workunit.client.0.smithi116.stdout:0/170: chown da/d16/d19/l36 7300368 1 2024-03-04T16:04:06.130 INFO:tasks.workunit.client.0.smithi116.stdout:5/188: creat dd/d14/f3f x:0 0 0 2024-03-04T16:04:06.131 INFO:tasks.workunit.client.0.smithi116.stdout:5/189: write dd/d14/f3f [970560,32818] 0 2024-03-04T16:04:06.133 INFO:tasks.workunit.client.0.smithi116.stdout:3/215: truncate dc/f33 1254502 0 2024-03-04T16:04:06.134 INFO:tasks.workunit.client.0.smithi116.stdout:3/216: chown dc/d34/d44/c45 55700 1 2024-03-04T16:04:06.135 INFO:tasks.workunit.client.0.smithi116.stdout:6/197: creat d6/de/f3d x:0 0 0 2024-03-04T16:04:06.137 INFO:tasks.workunit.client.0.smithi116.stdout:0/171: mkdir da/d16/d28/d3d 0 2024-03-04T16:04:06.137 INFO:tasks.workunit.client.0.smithi116.stdout:0/172: write da/d16/d28/f2e [1909753,66748] 0 2024-03-04T16:04:06.144 INFO:tasks.workunit.client.0.smithi116.stdout:3/217: rename c8 to dc/df/c4f 0 2024-03-04T16:04:06.144 INFO:tasks.workunit.client.0.smithi116.stdout:4/150: dread d1/d3/de/f10 [0,4194304] 0 2024-03-04T16:04:06.146 INFO:tasks.workunit.client.0.smithi116.stdout:6/198: mknod d6/c3e 0 2024-03-04T16:04:06.146 INFO:tasks.workunit.client.0.smithi116.stdout:5/190: dread dd/f13 [0,4194304] 0 2024-03-04T16:04:06.149 INFO:tasks.workunit.client.0.smithi116.stdout:3/218: dwrite dc/d23/f22 [0,4194304] 0 2024-03-04T16:04:06.153 INFO:tasks.workunit.client.0.smithi116.stdout:4/151: symlink d1/d3/d15/d20/l37 0 2024-03-04T16:04:06.155 INFO:tasks.workunit.client.0.smithi116.stdout:5/191: unlink f5 0 2024-03-04T16:04:06.158 INFO:tasks.workunit.client.0.smithi116.stdout:6/199: creat d6/de/d1e/f3f x:0 0 0 2024-03-04T16:04:06.162 INFO:tasks.workunit.client.0.smithi116.stdout:0/173: symlink da/d16/l3e 0 2024-03-04T16:04:06.163 INFO:tasks.workunit.client.0.smithi116.stdout:3/219: creat dc/d23/f50 x:0 0 0 2024-03-04T16:04:06.167 INFO:tasks.workunit.client.0.smithi116.stdout:5/192: creat dd/d14/d2d/d31/f40 x:0 0 0 2024-03-04T16:04:06.167 INFO:tasks.workunit.client.0.smithi116.stdout:3/220: dread dc/d23/f22 [0,4194304] 0 2024-03-04T16:04:06.169 INFO:tasks.workunit.client.0.smithi116.stdout:4/152: dwrite d1/d3/de/f16 [0,4194304] 0 2024-03-04T16:04:06.178 INFO:tasks.workunit.client.0.smithi116.stdout:6/200: truncate d6/de/f30 194806 0 2024-03-04T16:04:06.179 INFO:tasks.workunit.client.0.smithi116.stdout:6/201: write d6/de/f11 [5049036,48896] 0 2024-03-04T16:04:06.182 INFO:tasks.workunit.client.0.smithi116.stdout:5/193: dread dd/f11 [0,4194304] 0 2024-03-04T16:04:06.182 INFO:tasks.workunit.client.0.smithi116.stdout:4/153: truncate d1/d3/de/f13 277589 0 2024-03-04T16:04:06.184 INFO:tasks.workunit.client.0.smithi116.stdout:3/221: mknod dc/c51 0 2024-03-04T16:04:06.201 INFO:tasks.workunit.client.0.smithi116.stdout:3/222: dwrite dc/d23/f22 [0,4194304] 0 2024-03-04T16:04:06.201 INFO:tasks.workunit.client.0.smithi116.stdout:3/223: write dc/d23/f50 [915824,116228] 0 2024-03-04T16:04:06.207 INFO:tasks.workunit.client.0.smithi116.stdout:3/224: dread dc/d23/d2c/f3b [0,4194304] 0 2024-03-04T16:04:06.208 INFO:tasks.workunit.client.0.smithi116.stdout:6/202: getdents d6/de/d29 0 2024-03-04T16:04:06.209 INFO:tasks.workunit.client.0.smithi116.stdout:6/203: readlink d6/l2c 0 2024-03-04T16:04:06.210 INFO:tasks.workunit.client.0.smithi116.stdout:6/204: dread d6/de/f1b [0,4194304] 0 2024-03-04T16:04:06.211 INFO:tasks.workunit.client.0.smithi116.stdout:6/205: chown d6/de/d25/c39 1942913 1 2024-03-04T16:04:06.219 INFO:tasks.workunit.client.0.smithi116.stdout:6/206: rename d6/de/f11 to d6/de/d25/f40 0 2024-03-04T16:04:06.223 INFO:tasks.workunit.client.0.smithi116.stdout:6/207: symlink d6/de/d25/d36/l41 0 2024-03-04T16:04:06.224 INFO:tasks.workunit.client.0.smithi116.stdout:6/208: truncate d6/de/d25/d36/f1a 549446 0 2024-03-04T16:04:06.234 INFO:tasks.workunit.client.0.smithi116.stdout:3/225: rmdir dc 39 2024-03-04T16:04:06.240 INFO:tasks.workunit.client.0.smithi116.stdout:4/154: write d1/d3/de/f1b [571765,45030] 0 2024-03-04T16:04:06.241 INFO:tasks.workunit.client.0.smithi116.stdout:4/155: dread - d1/d3/d15/d20/f35 zero size 2024-03-04T16:04:06.242 INFO:tasks.workunit.client.0.smithi116.stdout:3/226: symlink dc/l52 0 2024-03-04T16:04:06.245 INFO:tasks.workunit.client.0.smithi116.stdout:4/156: dread d1/d3/d15/d20/d1c/f27 [0,4194304] 0 2024-03-04T16:04:06.245 INFO:tasks.workunit.client.0.smithi116.stdout:4/157: stat d1/d3/d15/f34 0 2024-03-04T16:04:06.246 INFO:tasks.workunit.client.0.smithi116.stdout:4/158: write d1/d3/de/f26 [1612405,30004] 0 2024-03-04T16:04:06.247 INFO:tasks.workunit.client.0.smithi116.stdout:5/194: truncate dd/f38 2437133 0 2024-03-04T16:04:06.247 INFO:tasks.workunit.client.0.smithi116.stdout:1/115: sync 2024-03-04T16:04:06.247 INFO:tasks.workunit.client.0.smithi116.stdout:9/130: sync 2024-03-04T16:04:06.248 INFO:tasks.workunit.client.0.smithi116.stdout:7/102: sync 2024-03-04T16:04:06.248 INFO:tasks.workunit.client.0.smithi116.stdout:8/159: sync 2024-03-04T16:04:06.249 INFO:tasks.workunit.client.0.smithi116.stdout:8/160: write d2/de/d16/f17 [697045,42306] 0 2024-03-04T16:04:06.252 INFO:tasks.workunit.client.0.smithi116.stdout:5/195: creat dd/d1f/f41 x:0 0 0 2024-03-04T16:04:06.254 INFO:tasks.workunit.client.0.smithi116.stdout:3/227: fsync dc/f16 0 2024-03-04T16:04:06.257 INFO:tasks.workunit.client.0.smithi116.stdout:5/196: dwrite dd/d14/f3f [0,4194304] 0 2024-03-04T16:04:06.258 INFO:tasks.workunit.client.0.smithi116.stdout:7/103: creat d3/d4/d11/f22 x:0 0 0 2024-03-04T16:04:06.258 INFO:tasks.workunit.client.0.smithi116.stdout:5/197: chown dd/d1f 45403579 1 2024-03-04T16:04:06.259 INFO:tasks.workunit.client.0.smithi116.stdout:1/116: rmdir d5/d9/d21/d24 39 2024-03-04T16:04:06.262 INFO:tasks.workunit.client.0.smithi116.stdout:9/131: truncate d2/f15 2624227 0 2024-03-04T16:04:06.268 INFO:tasks.workunit.client.0.smithi116.stdout:4/159: creat d1/d3/de/d33/f38 x:0 0 0 2024-03-04T16:04:06.270 INFO:tasks.workunit.client.0.smithi116.stdout:3/228: rename dc/d23/d2c/f4c to dc/df/f53 0 2024-03-04T16:04:06.273 INFO:tasks.workunit.client.0.smithi116.stdout:7/104: symlink d3/d4/d8/l23 0 2024-03-04T16:04:06.274 INFO:tasks.workunit.client.0.smithi116.stdout:3/229: dread dc/d23/f22 [0,4194304] 0 2024-03-04T16:04:06.274 INFO:tasks.workunit.client.0.smithi116.stdout:1/117: read f4 [3113154,84148] 0 2024-03-04T16:04:06.286 INFO:tasks.workunit.client.0.smithi116.stdout:7/105: creat d3/f24 x:0 0 0 2024-03-04T16:04:06.288 INFO:tasks.workunit.client.0.smithi116.stdout:1/118: rmdir d5/d11 39 2024-03-04T16:04:06.290 INFO:tasks.workunit.client.0.smithi116.stdout:3/230: truncate dc/d23/d2c/f3b 1656030 0 2024-03-04T16:04:06.303 INFO:tasks.workunit.client.0.smithi116.stdout:3/231: symlink dc/d23/d2c/l54 0 2024-03-04T16:04:06.304 INFO:tasks.workunit.client.0.smithi116.stdout:3/232: fdatasync dc/d23/f21 0 2024-03-04T16:04:06.304 INFO:tasks.workunit.client.0.smithi116.stdout:3/233: fdatasync dc/d23/f41 0 2024-03-04T16:04:06.304 INFO:tasks.workunit.client.0.smithi116.stdout:7/106: rmdir d3/d4/d8 39 2024-03-04T16:04:06.307 INFO:tasks.workunit.client.0.smithi116.stdout:8/161: getdents d2/d13 0 2024-03-04T16:04:06.307 INFO:tasks.workunit.client.0.smithi116.stdout:8/162: readlink d2/de/d16/l24 0 2024-03-04T16:04:06.307 INFO:tasks.workunit.client.0.smithi116.stdout:8/163: fdatasync d2/de/f15 0 2024-03-04T16:04:06.309 INFO:tasks.workunit.client.0.smithi116.stdout:5/198: write dd/f38 [2866035,94285] 0 2024-03-04T16:04:06.311 INFO:tasks.workunit.client.0.smithi116.stdout:8/164: dread d2/de/f12 [0,4194304] 0 2024-03-04T16:04:06.317 INFO:tasks.workunit.client.0.smithi116.stdout:5/199: truncate f4 4861267 0 2024-03-04T16:04:06.319 INFO:tasks.workunit.client.0.smithi116.stdout:7/107: creat d3/d4/d8/f25 x:0 0 0 2024-03-04T16:04:06.322 INFO:tasks.workunit.client.0.smithi116.stdout:3/234: rename dc/d23/d2c/l4a to dc/l55 0 2024-03-04T16:04:06.323 INFO:tasks.workunit.client.0.smithi116.stdout:5/200: creat dd/d14/d2d/f42 x:0 0 0 2024-03-04T16:04:06.325 INFO:tasks.workunit.client.0.smithi116.stdout:9/132: dwrite d2/f15 [0,4194304] 0 2024-03-04T16:04:06.329 INFO:tasks.workunit.client.0.smithi116.stdout:8/165: rename d2/d13/d1e/f27 to d2/de/d16/f34 0 2024-03-04T16:04:06.331 INFO:tasks.workunit.client.0.smithi116.stdout:8/166: stat d2/de/f12 0 2024-03-04T16:04:06.331 INFO:tasks.workunit.client.0.smithi116.stdout:0/174: sync 2024-03-04T16:04:06.331 INFO:tasks.workunit.client.0.smithi116.stdout:2/145: sync 2024-03-04T16:04:06.331 INFO:tasks.workunit.client.0.smithi116.stdout:3/235: dwrite dc/df/d14/f49 [0,4194304] 0 2024-03-04T16:04:06.333 INFO:tasks.workunit.client.0.smithi116.stdout:7/108: rename d3/d4/d8/d1a/l1d to d3/d4/d8/d1a/l26 0 2024-03-04T16:04:06.334 INFO:tasks.workunit.client.0.smithi116.stdout:1/119: dwrite f4 [0,4194304] 0 2024-03-04T16:04:06.334 INFO:tasks.workunit.client.0.smithi116.stdout:8/167: readlink d2/de/d16/l24 0 2024-03-04T16:04:06.335 INFO:tasks.workunit.client.0.smithi116.stdout:7/109: chown d3/f10 654477 1 2024-03-04T16:04:06.337 INFO:tasks.workunit.client.0.smithi116.stdout:7/110: read d3/f6 [835385,124577] 0 2024-03-04T16:04:06.338 INFO:tasks.workunit.client.0.smithi116.stdout:9/133: rename d2/c1c to d2/d19/c2d 0 2024-03-04T16:04:06.340 INFO:tasks.workunit.client.0.smithi116.stdout:2/146: chown d7/d15/c16 4 1 2024-03-04T16:04:06.341 INFO:tasks.workunit.client.0.smithi116.stdout:2/147: write d7/f36 [592905,60401] 0 2024-03-04T16:04:06.348 INFO:tasks.workunit.client.0.smithi116.stdout:0/175: symlink da/d16/d28/d39/l3f 0 2024-03-04T16:04:06.351 INFO:tasks.workunit.client.0.smithi116.stdout:9/134: rmdir d2/d26 39 2024-03-04T16:04:06.353 INFO:tasks.workunit.client.0.smithi116.stdout:9/135: write d2/d19/d27/d1f/f24 [2965361,5018] 0 2024-03-04T16:04:06.354 INFO:tasks.workunit.client.0.smithi116.stdout:7/111: rename d3/f5 to d3/d4/d8/d1a/f27 0 2024-03-04T16:04:06.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:06 smithi152 ceph-mon[36876]: pgmap v19: 65 pgs: 65 active+clean; 357 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 743 KiB/s rd, 29 MiB/s wr, 461 op/s 2024-03-04T16:04:06.360 INFO:tasks.workunit.client.0.smithi116.stdout:0/176: mkdir da/d16/d28/d40 0 2024-03-04T16:04:06.363 INFO:tasks.workunit.client.0.smithi116.stdout:1/120: dwrite d5/d11/f16 [0,4194304] 0 2024-03-04T16:04:06.366 INFO:tasks.workunit.client.0.smithi116.stdout:2/148: symlink d7/d2f/l3d 0 2024-03-04T16:04:06.369 INFO:tasks.workunit.client.0.smithi116.stdout:7/112: creat d3/d4/d8/d1a/f28 x:0 0 0 2024-03-04T16:04:06.370 INFO:tasks.workunit.client.0.smithi116.stdout:7/113: write d3/f1f [63996,3472] 0 2024-03-04T16:04:06.374 INFO:tasks.workunit.client.0.smithi116.stdout:2/149: dread f1 [0,4194304] 0 2024-03-04T16:04:06.374 INFO:tasks.workunit.client.0.smithi116.stdout:9/136: mknod d2/c2e 0 2024-03-04T16:04:06.378 INFO:tasks.workunit.client.0.smithi116.stdout:8/168: dwrite d2/de/f8 [0,4194304] 0 2024-03-04T16:04:06.379 INFO:tasks.workunit.client.0.smithi116.stdout:7/114: readlink d3/d4/d8/d1a/l26 0 2024-03-04T16:04:06.381 INFO:tasks.workunit.client.0.smithi116.stdout:8/169: chown d2/de/l19 33099167 1 2024-03-04T16:04:06.382 INFO:tasks.workunit.client.0.smithi116.stdout:0/177: dwrite da/d16/d19/f2f [4194304,4194304] 0 2024-03-04T16:04:06.383 INFO:tasks.workunit.client.0.smithi116.stdout:2/150: mkdir d7/d13/d3e 0 2024-03-04T16:04:06.386 INFO:tasks.workunit.client.0.smithi116.stdout:9/137: dread d2/f15 [0,4194304] 0 2024-03-04T16:04:06.387 INFO:tasks.workunit.client.0.smithi116.stdout:3/236: dwrite dc/d23/d2c/f3b [0,4194304] 0 2024-03-04T16:04:06.397 INFO:tasks.workunit.client.0.smithi116.stdout:2/151: truncate d7/d15/f2e 883523 0 2024-03-04T16:04:06.399 INFO:tasks.workunit.client.0.smithi116.stdout:3/237: dwrite dc/df/d14/f1a [4194304,4194304] 0 2024-03-04T16:04:06.400 INFO:tasks.workunit.client.0.smithi116.stdout:2/152: write d7/d13/f1e [683909,119616] 0 2024-03-04T16:04:06.400 INFO:tasks.workunit.client.0.smithi116.stdout:2/153: chown d7/d29/l32 7 1 2024-03-04T16:04:06.400 INFO:tasks.workunit.client.0.smithi116.stdout:2/154: stat l3 0 2024-03-04T16:04:06.402 INFO:tasks.workunit.client.0.smithi116.stdout:1/121: rename d5/d19 to d5/d9/d21/d27 0 2024-03-04T16:04:06.404 INFO:tasks.workunit.client.0.smithi116.stdout:3/238: readlink dc/l2f 0 2024-03-04T16:04:06.406 INFO:tasks.workunit.client.0.smithi116.stdout:0/178: getdents da/d16 0 2024-03-04T16:04:06.408 INFO:tasks.workunit.client.0.smithi116.stdout:8/170: rename d2/d13/d1d/d29/c31 to d2/de/c35 0 2024-03-04T16:04:06.409 INFO:tasks.workunit.client.0.smithi116.stdout:2/155: dread d7/d1b/f21 [0,4194304] 0 2024-03-04T16:04:06.411 INFO:tasks.workunit.client.0.smithi116.stdout:9/138: link d2/f1a d2/d19/d27/d1f/f2f 0 2024-03-04T16:04:06.413 INFO:tasks.workunit.client.0.smithi116.stdout:3/239: dwrite dc/df/d14/f1a [4194304,4194304] 0 2024-03-04T16:04:06.415 INFO:tasks.workunit.client.0.smithi116.stdout:0/179: rename da/d2c to da/d16/d28/d3d/d41 0 2024-03-04T16:04:06.417 INFO:tasks.workunit.client.0.smithi116.stdout:1/122: unlink d5/d9/d21/d27/c1e 0 2024-03-04T16:04:06.421 INFO:tasks.workunit.client.0.smithi116.stdout:7/115: dwrite d3/f1e [0,4194304] 0 2024-03-04T16:04:06.422 INFO:tasks.workunit.client.0.smithi116.stdout:8/171: dwrite d2/de/fc [0,4194304] 0 2024-03-04T16:04:06.424 INFO:tasks.workunit.client.0.smithi116.stdout:2/156: dwrite d7/d13/f18 [0,4194304] 0 2024-03-04T16:04:06.424 INFO:tasks.workunit.client.0.smithi116.stdout:0/180: rmdir da/d16/d20 39 2024-03-04T16:04:06.428 INFO:tasks.workunit.client.0.smithi116.stdout:7/116: creat d3/d4/d8/d1a/f29 x:0 0 0 2024-03-04T16:04:06.431 INFO:tasks.workunit.client.0.smithi116.stdout:1/123: write d5/f1c [944681,43117] 0 2024-03-04T16:04:06.434 INFO:tasks.workunit.client.0.smithi116.stdout:2/157: creat d7/d15/f3f x:0 0 0 2024-03-04T16:04:06.435 INFO:tasks.workunit.client.0.smithi116.stdout:2/158: write d7/d13/f18 [3837756,80891] 0 2024-03-04T16:04:06.436 INFO:tasks.workunit.client.0.smithi116.stdout:7/117: dread d3/d4/d8/d1a/f27 [0,4194304] 0 2024-03-04T16:04:06.443 INFO:tasks.workunit.client.0.smithi116.stdout:8/172: rename d2/d13/d1d to d2/de/d16/d36 0 2024-03-04T16:04:06.444 INFO:tasks.workunit.client.0.smithi116.stdout:0/181: dread da/d16/f17 [0,4194304] 0 2024-03-04T16:04:06.445 INFO:tasks.workunit.client.0.smithi116.stdout:1/124: chown d5/f17 75260862 1 2024-03-04T16:04:06.445 INFO:tasks.workunit.client.0.smithi116.stdout:0/182: dread - da/f35 zero size 2024-03-04T16:04:06.447 INFO:tasks.workunit.client.0.smithi116.stdout:3/240: link dc/df/d14/f1a dc/f56 0 2024-03-04T16:04:06.455 INFO:tasks.workunit.client.0.smithi116.stdout:8/173: mkdir d2/de/d16/d2b/d37 0 2024-03-04T16:04:06.456 INFO:tasks.workunit.client.0.smithi116.stdout:6/209: sync 2024-03-04T16:04:06.456 INFO:tasks.workunit.client.0.smithi116.stdout:7/118: fsync d3/f1e 0 2024-03-04T16:04:06.461 INFO:tasks.workunit.client.0.smithi116.stdout:3/241: write dc/df/d14/f1a [7637323,117922] 0 2024-03-04T16:04:06.462 INFO:tasks.workunit.client.0.smithi116.stdout:9/139: dread d2/d19/f2a [0,4194304] 0 2024-03-04T16:04:06.464 INFO:tasks.workunit.client.0.smithi116.stdout:9/140: write d2/d19/d27/d1f/f24 [4466169,96218] 0 2024-03-04T16:04:06.468 INFO:tasks.workunit.client.0.smithi116.stdout:1/125: truncate d5/f1d 1164790 0 2024-03-04T16:04:06.468 INFO:tasks.workunit.client.0.smithi116.stdout:1/126: fsync d5/d11/f16 0 2024-03-04T16:04:06.469 INFO:tasks.workunit.client.0.smithi116.stdout:2/159: truncate d7/f36 536023 0 2024-03-04T16:04:06.470 INFO:tasks.workunit.client.0.smithi116.stdout:7/119: mknod d3/c2a 0 2024-03-04T16:04:06.471 INFO:tasks.workunit.client.0.smithi116.stdout:0/183: mkdir da/d16/d20/d42 0 2024-03-04T16:04:06.473 INFO:tasks.workunit.client.0.smithi116.stdout:9/141: rmdir d2/d19 39 2024-03-04T16:04:06.474 INFO:tasks.workunit.client.0.smithi116.stdout:8/174: mkdir d2/d38 0 2024-03-04T16:04:06.477 INFO:tasks.workunit.client.0.smithi116.stdout:2/160: mkdir d7/d2f/d40 0 2024-03-04T16:04:06.477 INFO:tasks.workunit.client.0.smithi116.stdout:7/120: mknod d3/d4/d8/c2b 0 2024-03-04T16:04:06.479 INFO:tasks.workunit.client.0.smithi116.stdout:7/121: truncate d3/f10 1045122 0 2024-03-04T16:04:06.479 INFO:tasks.workunit.client.0.smithi116.stdout:0/184: rmdir da/d16 39 2024-03-04T16:04:06.479 INFO:tasks.workunit.client.0.smithi116.stdout:0/185: readlink l3 0 2024-03-04T16:04:06.479 INFO:tasks.workunit.client.0.smithi116.stdout:9/142: symlink d2/d8/l30 0 2024-03-04T16:04:06.481 INFO:tasks.workunit.client.0.smithi116.stdout:1/127: mknod d5/d9/d21/d24/c28 0 2024-03-04T16:04:06.489 INFO:tasks.workunit.client.0.smithi116.stdout:5/201: sync 2024-03-04T16:04:06.489 INFO:tasks.workunit.client.0.smithi116.stdout:4/160: sync 2024-03-04T16:04:06.512 INFO:tasks.workunit.client.0.smithi116.stdout:7/122: symlink d3/d4/d8/d1a/l2c 0 2024-03-04T16:04:06.512 INFO:tasks.workunit.client.0.smithi116.stdout:6/210: link d6/de/c33 d6/c42 0 2024-03-04T16:04:06.513 INFO:tasks.workunit.client.0.smithi116.stdout:2/161: dwrite d7/f24 [0,4194304] 0 2024-03-04T16:04:06.514 INFO:tasks.workunit.client.0.smithi116.stdout:4/161: write d1/d3/dd/f9 [1053172,97971] 0 2024-03-04T16:04:06.515 INFO:tasks.workunit.client.0.smithi116.stdout:2/162: chown d7/fb 1622356 1 2024-03-04T16:04:06.515 INFO:tasks.workunit.client.0.smithi116.stdout:3/242: getdents dc/d34/d44 0 2024-03-04T16:04:06.515 INFO:tasks.workunit.client.0.smithi116.stdout:4/162: stat d1/d3/d15/d20/f1e 0 2024-03-04T16:04:06.516 INFO:tasks.workunit.client.0.smithi116.stdout:7/123: fdatasync f2 0 2024-03-04T16:04:06.517 INFO:tasks.workunit.client.0.smithi116.stdout:2/163: fsync d7/d29/f3a 0 2024-03-04T16:04:06.517 INFO:tasks.workunit.client.0.smithi116.stdout:6/211: creat d6/de/d1e/f43 x:0 0 0 2024-03-04T16:04:06.517 INFO:tasks.workunit.client.0.smithi116.stdout:3/243: dread - dc/df/f3c zero size 2024-03-04T16:04:06.518 INFO:tasks.workunit.client.0.smithi116.stdout:5/202: symlink dd/l43 0 2024-03-04T16:04:06.519 INFO:tasks.workunit.client.0.smithi116.stdout:4/163: chown d1/d3/d15/d20/f1e 2 1 2024-03-04T16:04:06.519 INFO:tasks.workunit.client.0.smithi116.stdout:6/212: fdatasync d6/de/f3b 0 2024-03-04T16:04:06.519 INFO:tasks.workunit.client.0.smithi116.stdout:2/164: fdatasync d7/d15/f3f 0 2024-03-04T16:04:06.520 INFO:tasks.workunit.client.0.smithi116.stdout:1/128: creat d5/f29 x:0 0 0 2024-03-04T16:04:06.521 INFO:tasks.workunit.client.0.smithi116.stdout:5/203: read - dd/d1f/f41 zero size 2024-03-04T16:04:06.521 INFO:tasks.workunit.client.0.smithi116.stdout:7/124: read d3/d4/d8/f13 [38935,115398] 0 2024-03-04T16:04:06.522 INFO:tasks.workunit.client.0.smithi116.stdout:6/213: chown d6/de/d29/l3c 217 1 2024-03-04T16:04:06.525 INFO:tasks.workunit.client.0.smithi116.stdout:4/164: creat d1/d3/f39 x:0 0 0 2024-03-04T16:04:06.528 INFO:tasks.workunit.client.0.smithi116.stdout:0/186: dwrite da/d16/d28/d39/f32 [0,4194304] 0 2024-03-04T16:04:06.528 INFO:tasks.workunit.client.0.smithi116.stdout:3/244: getdents dc/d34/d44 0 2024-03-04T16:04:06.531 INFO:tasks.workunit.client.0.smithi116.stdout:6/214: read d6/f13 [5472,102086] 0 2024-03-04T16:04:06.533 INFO:tasks.workunit.client.0.smithi116.stdout:4/165: creat d1/d3/de/d33/f3a x:0 0 0 2024-03-04T16:04:06.534 INFO:tasks.workunit.client.0.smithi116.stdout:3/245: stat dc/d23/d2c/f3b 0 2024-03-04T16:04:06.536 INFO:tasks.workunit.client.0.smithi116.stdout:7/125: dwrite d3/d4/d8/f1c [0,4194304] 0 2024-03-04T16:04:06.537 INFO:tasks.workunit.client.0.smithi116.stdout:4/166: write d1/d3/d15/d20/f35 [897090,42197] 0 2024-03-04T16:04:06.544 INFO:tasks.workunit.client.0.smithi116.stdout:5/204: getdents dd/d14/d2d/d31 0 2024-03-04T16:04:06.549 INFO:tasks.workunit.client.0.smithi116.stdout:7/126: symlink d3/d4/d8/l2d 0 2024-03-04T16:04:06.550 INFO:tasks.workunit.client.0.smithi116.stdout:4/167: mknod d1/c3b 0 2024-03-04T16:04:06.553 INFO:tasks.workunit.client.0.smithi116.stdout:0/187: rename da/d16/d28/d39/c30 to da/d16/d19/d27/c43 0 2024-03-04T16:04:06.554 INFO:tasks.workunit.client.0.smithi116.stdout:5/205: fdatasync dd/f11 0 2024-03-04T16:04:06.555 INFO:tasks.workunit.client.0.smithi116.stdout:5/206: readlink dd/d1f/l3e 0 2024-03-04T16:04:06.560 INFO:tasks.workunit.client.0.smithi116.stdout:7/127: creat d3/d4/d8/d1a/f2e x:0 0 0 2024-03-04T16:04:06.562 INFO:tasks.workunit.client.0.smithi116.stdout:1/129: write d5/f1d [1422983,4474] 0 2024-03-04T16:04:06.562 INFO:tasks.workunit.client.0.smithi116.stdout:2/165: dread d7/d15/f20 [0,4194304] 0 2024-03-04T16:04:06.563 INFO:tasks.workunit.client.0.smithi116.stdout:5/207: symlink dd/d1f/l44 0 2024-03-04T16:04:06.564 INFO:tasks.workunit.client.0.smithi116.stdout:1/130: write d5/f17 [2111626,92457] 0 2024-03-04T16:04:06.566 INFO:tasks.workunit.client.0.smithi116.stdout:7/128: dread d3/f18 [0,4194304] 0 2024-03-04T16:04:06.567 INFO:tasks.workunit.client.0.smithi116.stdout:7/129: write d3/f1f [827693,2537] 0 2024-03-04T16:04:06.568 INFO:tasks.workunit.client.0.smithi116.stdout:1/131: dread d5/f1d [0,4194304] 0 2024-03-04T16:04:06.572 INFO:tasks.workunit.client.0.smithi116.stdout:2/166: rename d7/d2f/d40 to d7/d25/d41 0 2024-03-04T16:04:06.572 INFO:tasks.workunit.client.0.smithi116.stdout:6/215: dwrite d6/de/f1b [0,4194304] 0 2024-03-04T16:04:06.575 INFO:tasks.workunit.client.0.smithi116.stdout:3/246: truncate dc/df/d14/f49 3109621 0 2024-03-04T16:04:06.578 INFO:tasks.workunit.client.0.smithi116.stdout:5/208: mkdir dd/d45 0 2024-03-04T16:04:06.579 INFO:tasks.workunit.client.0.smithi116.stdout:4/168: getdents d1/d3/dd/d24 0 2024-03-04T16:04:06.584 INFO:tasks.workunit.client.0.smithi116.stdout:1/132: creat d5/d9/d21/d27/f2a x:0 0 0 2024-03-04T16:04:06.584 INFO:tasks.workunit.client.0.smithi116.stdout:1/133: stat d5/d11/f20 0 2024-03-04T16:04:06.585 INFO:tasks.workunit.client.0.smithi116.stdout:1/134: chown d5/d9/d21/f26 17532 1 2024-03-04T16:04:06.585 INFO:tasks.workunit.client.0.smithi116.stdout:1/135: read - d5/d9/d21/f26 zero size 2024-03-04T16:04:06.589 INFO:tasks.workunit.client.0.smithi116.stdout:2/167: rmdir d7/d13 39 2024-03-04T16:04:06.590 INFO:tasks.workunit.client.0.smithi116.stdout:6/216: mkdir d6/de/d25/d36/d44 0 2024-03-04T16:04:06.593 INFO:tasks.workunit.client.0.smithi116.stdout:7/130: write d3/d4/d8/f13 [113900,3005] 0 2024-03-04T16:04:06.594 INFO:tasks.workunit.client.0.smithi116.stdout:3/247: creat dc/df/f57 x:0 0 0 2024-03-04T16:04:06.597 INFO:tasks.workunit.client.0.smithi116.stdout:4/169: creat d1/d3/dd/f3c x:0 0 0 2024-03-04T16:04:06.602 INFO:tasks.workunit.client.0.smithi116.stdout:7/131: unlink d3/f24 0 2024-03-04T16:04:06.604 INFO:tasks.workunit.client.0.smithi116.stdout:3/248: dread dc/d23/f22 [0,4194304] 0 2024-03-04T16:04:06.605 INFO:tasks.workunit.client.0.smithi116.stdout:3/249: truncate dc/df/f57 764385 0 2024-03-04T16:04:06.606 INFO:tasks.workunit.client.0.smithi116.stdout:3/250: write dc/d23/d2c/f3b [2605414,33557] 0 2024-03-04T16:04:06.607 INFO:tasks.workunit.client.0.smithi116.stdout:4/170: creat d1/d3/dd/d24/f3d x:0 0 0 2024-03-04T16:04:06.607 INFO:tasks.workunit.client.0.smithi116.stdout:1/136: mknod d5/d9/d21/c2b 0 2024-03-04T16:04:06.609 INFO:tasks.workunit.client.0.smithi116.stdout:1/137: write d5/d9/f18 [344274,101200] 0 2024-03-04T16:04:06.610 INFO:tasks.workunit.client.0.smithi116.stdout:6/217: write d6/f13 [261195,85208] 0 2024-03-04T16:04:06.612 INFO:tasks.workunit.client.0.smithi116.stdout:7/132: creat d3/d4/d11/f2f x:0 0 0 2024-03-04T16:04:06.614 INFO:tasks.workunit.client.0.smithi116.stdout:3/251: dwrite dc/f56 [4194304,4194304] 0 2024-03-04T16:04:06.614 INFO:tasks.workunit.client.0.smithi116.stdout:3/252: chown dc/d34/d44 308219848 1 2024-03-04T16:04:06.615 INFO:tasks.workunit.client.0.smithi116.stdout:2/168: creat d7/f42 x:0 0 0 2024-03-04T16:04:06.616 INFO:tasks.workunit.client.0.smithi116.stdout:6/218: creat d6/f45 x:0 0 0 2024-03-04T16:04:06.617 INFO:tasks.workunit.client.0.smithi116.stdout:6/219: write d6/f24 [1320308,95237] 0 2024-03-04T16:04:06.618 INFO:tasks.workunit.client.0.smithi116.stdout:5/209: link dd/d14/d2d/l32 dd/d45/l46 0 2024-03-04T16:04:06.618 INFO:tasks.workunit.client.0.smithi116.stdout:5/210: write dd/f38 [418601,53344] 0 2024-03-04T16:04:06.623 INFO:tasks.workunit.client.0.smithi116.stdout:2/169: stat d7/f36 0 2024-03-04T16:04:06.632 INFO:tasks.workunit.client.0.smithi116.stdout:6/220: rename d6/de/f30 to d6/de/d29/d3a/f46 0 2024-03-04T16:04:06.632 INFO:tasks.workunit.client.0.smithi116.stdout:6/221: stat d6/de/f1b 0 2024-03-04T16:04:06.633 INFO:tasks.workunit.client.0.smithi116.stdout:2/170: mknod d7/c43 0 2024-03-04T16:04:06.634 INFO:tasks.workunit.client.0.smithi116.stdout:6/222: readlink d6/de/d15/l28 0 2024-03-04T16:04:06.636 INFO:tasks.workunit.client.0.smithi116.stdout:2/171: creat d7/d2f/f44 x:0 0 0 2024-03-04T16:04:06.636 INFO:tasks.workunit.client.0.smithi116.stdout:2/172: dread - d7/d25/f34 zero size 2024-03-04T16:04:06.639 INFO:tasks.workunit.client.0.smithi116.stdout:6/223: fsync d6/de/d15/f31 0 2024-03-04T16:04:06.640 INFO:tasks.workunit.client.0.smithi116.stdout:6/224: chown d6/de/f3b 10148 1 2024-03-04T16:04:06.640 INFO:tasks.workunit.client.0.smithi116.stdout:6/225: chown d6/f17 1868 1 2024-03-04T16:04:06.644 INFO:tasks.workunit.client.0.smithi116.stdout:2/173: getdents d7/d13/d3e 0 2024-03-04T16:04:06.647 INFO:tasks.workunit.client.0.smithi116.stdout:3/253: dwrite dc/f33 [0,4194304] 0 2024-03-04T16:04:06.648 INFO:tasks.workunit.client.0.smithi116.stdout:3/254: dread - dc/d23/f46 zero size 2024-03-04T16:04:06.649 INFO:tasks.workunit.client.0.smithi116.stdout:2/174: creat d7/d13/f45 x:0 0 0 2024-03-04T16:04:06.649 INFO:tasks.workunit.client.0.smithi116.stdout:6/226: dwrite d6/de/d1e/f34 [0,4194304] 0 2024-03-04T16:04:06.650 INFO:tasks.workunit.client.0.smithi116.stdout:3/255: write dc/f56 [3957595,87020] 0 2024-03-04T16:04:06.652 INFO:tasks.workunit.client.0.smithi116.stdout:2/175: creat d7/d2f/f46 x:0 0 0 2024-03-04T16:04:06.653 INFO:tasks.workunit.client.0.smithi116.stdout:6/227: mkdir d6/de/d15/d47 0 2024-03-04T16:04:06.656 INFO:tasks.workunit.client.0.smithi116.stdout:3/256: link dc/df/f53 dc/df/d14/f58 0 2024-03-04T16:04:06.659 INFO:tasks.workunit.client.0.smithi116.stdout:2/176: dwrite d7/d1b/f22 [0,4194304] 0 2024-03-04T16:04:06.660 INFO:tasks.workunit.client.0.smithi116.stdout:6/228: rename d6/de/d25/d36/f1d to d6/de/d1e/f48 0 2024-03-04T16:04:06.663 INFO:tasks.workunit.client.0.smithi116.stdout:6/229: dread d6/f13 [0,4194304] 0 2024-03-04T16:04:06.664 INFO:tasks.workunit.client.0.smithi116.stdout:6/230: read d6/de/d1e/f34 [3252106,77048] 0 2024-03-04T16:04:06.666 INFO:tasks.workunit.client.0.smithi116.stdout:3/257: rename dc/c51 to dc/df/c59 0 2024-03-04T16:04:06.672 INFO:tasks.workunit.client.0.smithi116.stdout:3/258: dwrite dc/d23/f1b [0,4194304] 0 2024-03-04T16:04:06.673 INFO:tasks.workunit.client.0.smithi116.stdout:3/259: readlink dc/l2f 0 2024-03-04T16:04:06.678 INFO:tasks.workunit.client.0.smithi116.stdout:3/260: unlink dc/c40 0 2024-03-04T16:04:06.693 INFO:tasks.workunit.client.0.smithi116.stdout:8/175: sync 2024-03-04T16:04:06.694 INFO:tasks.workunit.client.0.smithi116.stdout:2/177: write d7/d15/f20 [5228013,1274] 0 2024-03-04T16:04:06.694 INFO:tasks.workunit.client.0.smithi116.stdout:6/231: write d6/de/f1f [1002337,30900] 0 2024-03-04T16:04:06.695 INFO:tasks.workunit.client.0.smithi116.stdout:3/261: dread dc/d23/d2c/f2d [0,4194304] 0 2024-03-04T16:04:06.696 INFO:tasks.workunit.client.0.smithi116.stdout:3/262: fdatasync dc/f4b 0 2024-03-04T16:04:06.697 INFO:tasks.workunit.client.0.smithi116.stdout:2/178: mkdir d7/d29/d47 0 2024-03-04T16:04:06.697 INFO:tasks.workunit.client.0.smithi116.stdout:6/232: symlink d6/de/d15/l49 0 2024-03-04T16:04:06.700 INFO:tasks.workunit.client.0.smithi116.stdout:3/263: mkdir dc/d23/d5a 0 2024-03-04T16:04:06.702 INFO:tasks.workunit.client.0.smithi116.stdout:2/179: dread d7/d15/f30 [0,4194304] 0 2024-03-04T16:04:06.703 INFO:tasks.workunit.client.0.smithi116.stdout:8/176: dread d2/de/fa [0,4194304] 0 2024-03-04T16:04:06.703 INFO:tasks.workunit.client.0.smithi116.stdout:3/264: readlink dc/l2a 0 2024-03-04T16:04:06.704 INFO:tasks.workunit.client.0.smithi116.stdout:3/265: truncate dc/d23/f21 735088 0 2024-03-04T16:04:06.709 INFO:tasks.workunit.client.0.smithi116.stdout:8/177: dwrite d2/de/f8 [0,4194304] 0 2024-03-04T16:04:06.709 INFO:tasks.workunit.client.0.smithi116.stdout:3/266: fdatasync dc/df/d14/f49 0 2024-03-04T16:04:06.712 INFO:tasks.workunit.client.0.smithi116.stdout:3/267: dread dc/d23/f41 [0,4194304] 0 2024-03-04T16:04:06.714 INFO:tasks.workunit.client.0.smithi116.stdout:8/178: creat d2/de/d16/d36/d29/f39 x:0 0 0 2024-03-04T16:04:06.715 INFO:tasks.workunit.client.0.smithi116.stdout:8/179: read d2/de/d16/f22 [4178391,60215] 0 2024-03-04T16:04:06.716 INFO:tasks.workunit.client.0.smithi116.stdout:3/268: mknod dc/d23/d2c/c5b 0 2024-03-04T16:04:06.721 INFO:tasks.workunit.client.0.smithi116.stdout:8/180: rename d2/c11 to d2/de/d16/d36/c3a 0 2024-03-04T16:04:06.722 INFO:tasks.workunit.client.0.smithi116.stdout:8/181: chown d2/de/fc 6756 1 2024-03-04T16:04:06.723 INFO:tasks.workunit.client.0.smithi116.stdout:3/269: getdents dc/d23/d5a 0 2024-03-04T16:04:06.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:06 smithi116 ceph-mon[29120]: pgmap v19: 65 pgs: 65 active+clean; 357 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 743 KiB/s rd, 29 MiB/s wr, 461 op/s 2024-03-04T16:04:06.725 INFO:tasks.workunit.client.0.smithi116.stdout:6/233: dwrite d6/f13 [0,4194304] 0 2024-03-04T16:04:06.730 INFO:tasks.workunit.client.0.smithi116.stdout:2/180: dwrite f1 [0,4194304] 0 2024-03-04T16:04:06.731 INFO:tasks.workunit.client.0.smithi116.stdout:2/181: stat d7/d15/f20 0 2024-03-04T16:04:06.731 INFO:tasks.workunit.client.0.smithi116.stdout:6/234: mkdir d6/de/d25/d4a 0 2024-03-04T16:04:06.732 INFO:tasks.workunit.client.0.smithi116.stdout:3/270: dread dc/d23/f22 [0,4194304] 0 2024-03-04T16:04:06.733 INFO:tasks.workunit.client.0.smithi116.stdout:6/235: chown d6/f24 40614 1 2024-03-04T16:04:06.733 INFO:tasks.workunit.client.0.smithi116.stdout:6/236: readlink d6/de/d25/d36/l41 0 2024-03-04T16:04:06.734 INFO:tasks.workunit.client.0.smithi116.stdout:2/182: creat d7/d15/d35/f48 x:0 0 0 2024-03-04T16:04:06.740 INFO:tasks.workunit.client.0.smithi116.stdout:3/271: creat dc/d23/d5a/f5c x:0 0 0 2024-03-04T16:04:06.741 INFO:tasks.workunit.client.0.smithi116.stdout:2/183: dwrite d7/d1b/f27 [0,4194304] 0 2024-03-04T16:04:06.743 INFO:tasks.workunit.client.0.smithi116.stdout:6/237: rename d6/de/d15/f31 to d6/de/d25/d36/d44/f4b 0 2024-03-04T16:04:06.748 INFO:tasks.workunit.client.0.smithi116.stdout:3/272: creat dc/df/d14/f5d x:0 0 0 2024-03-04T16:04:06.748 INFO:tasks.workunit.client.0.smithi116.stdout:2/184: creat d7/d29/d47/f49 x:0 0 0 2024-03-04T16:04:06.749 INFO:tasks.workunit.client.0.smithi116.stdout:8/182: truncate d2/de/d16/f1b 3768882 0 2024-03-04T16:04:06.752 INFO:tasks.workunit.client.0.smithi116.stdout:6/238: dwrite d6/de/d1e/f43 [0,4194304] 0 2024-03-04T16:04:06.752 INFO:tasks.workunit.client.0.smithi116.stdout:3/273: creat dc/d23/d2c/f5e x:0 0 0 2024-03-04T16:04:06.753 INFO:tasks.workunit.client.0.smithi116.stdout:3/274: dread - dc/df/d14/f5d zero size 2024-03-04T16:04:06.759 INFO:tasks.workunit.client.0.smithi116.stdout:6/239: creat d6/de/d15/d47/f4c x:0 0 0 2024-03-04T16:04:06.760 INFO:tasks.workunit.client.0.smithi116.stdout:6/240: chown d6/de/d25/d36/d44 665713059 1 2024-03-04T16:04:06.760 INFO:tasks.workunit.client.0.smithi116.stdout:8/183: readlink d2/d13/l1f 0 2024-03-04T16:04:06.767 INFO:tasks.workunit.client.0.smithi116.stdout:9/143: sync 2024-03-04T16:04:06.768 INFO:tasks.workunit.client.0.smithi116.stdout:3/275: creat dc/f5f x:0 0 0 2024-03-04T16:04:06.769 INFO:tasks.workunit.client.0.smithi116.stdout:6/241: dread d6/f24 [0,4194304] 0 2024-03-04T16:04:06.769 INFO:tasks.workunit.client.0.smithi116.stdout:9/144: chown d2/l14 7 1 2024-03-04T16:04:06.770 INFO:tasks.workunit.client.0.smithi116.stdout:8/184: creat d2/d38/f3b x:0 0 0 2024-03-04T16:04:06.772 INFO:tasks.workunit.client.0.smithi116.stdout:6/242: chown d6/de/d1e/f3f 15088 1 2024-03-04T16:04:06.773 INFO:tasks.workunit.client.0.smithi116.stdout:2/185: dwrite d7/f36 [0,4194304] 0 2024-03-04T16:04:06.773 INFO:tasks.workunit.client.0.smithi116.stdout:3/276: unlink dc/df/f57 0 2024-03-04T16:04:06.778 INFO:tasks.workunit.client.0.smithi116.stdout:9/145: unlink d2/d8/ce 0 2024-03-04T16:04:06.781 INFO:tasks.workunit.client.0.smithi116.stdout:6/243: creat d6/de/f4d x:0 0 0 2024-03-04T16:04:06.784 INFO:tasks.workunit.client.0.smithi116.stdout:8/185: rmdir d2 39 2024-03-04T16:04:06.789 INFO:tasks.workunit.client.0.smithi116.stdout:9/146: chown d2/d26/d28/c29 0 1 2024-03-04T16:04:06.793 INFO:tasks.workunit.client.0.smithi116.stdout:6/244: mkdir d6/de/d25/d36/d4e 0 2024-03-04T16:04:06.794 INFO:tasks.workunit.client.0.smithi116.stdout:2/186: dread d7/d13/f1a [0,4194304] 0 2024-03-04T16:04:06.796 INFO:tasks.workunit.client.0.smithi116.stdout:2/187: write d7/d15/d35/f3b [666990,83423] 0 2024-03-04T16:04:06.797 INFO:tasks.workunit.client.0.smithi116.stdout:6/245: creat d6/de/f4f x:0 0 0 2024-03-04T16:04:06.799 INFO:tasks.workunit.client.0.smithi116.stdout:6/246: dread - d6/de/f3b zero size 2024-03-04T16:04:06.800 INFO:tasks.workunit.client.0.smithi116.stdout:2/188: write d7/d29/d47/f49 [533510,56053] 0 2024-03-04T16:04:06.802 INFO:tasks.workunit.client.0.smithi116.stdout:8/186: mkdir d2/d13/d3c 0 2024-03-04T16:04:06.804 INFO:tasks.workunit.client.0.smithi116.stdout:3/277: dwrite dc/df/d14/f49 [0,4194304] 0 2024-03-04T16:04:06.805 INFO:tasks.workunit.client.0.smithi116.stdout:3/278: write dc/d23/f46 [856865,110306] 0 2024-03-04T16:04:06.806 INFO:tasks.workunit.client.0.smithi116.stdout:8/187: creat d2/d38/f3d x:0 0 0 2024-03-04T16:04:06.808 INFO:tasks.workunit.client.0.smithi116.stdout:2/189: link d7/d1b/f1c d7/d29/f4a 0 2024-03-04T16:04:06.810 INFO:tasks.workunit.client.0.smithi116.stdout:3/279: symlink dc/d34/l60 0 2024-03-04T16:04:06.812 INFO:tasks.workunit.client.0.smithi116.stdout:5/211: sync 2024-03-04T16:04:06.812 INFO:tasks.workunit.client.0.smithi116.stdout:0/188: sync 2024-03-04T16:04:06.812 INFO:tasks.workunit.client.0.smithi116.stdout:4/171: sync 2024-03-04T16:04:06.812 INFO:tasks.workunit.client.0.smithi116.stdout:7/133: sync 2024-03-04T16:04:06.812 INFO:tasks.workunit.client.0.smithi116.stdout:1/138: sync 2024-03-04T16:04:06.814 INFO:tasks.workunit.client.0.smithi116.stdout:8/188: rename d2/de/d16/d36/d29/d2e to d2/de/d3e 0 2024-03-04T16:04:06.814 INFO:tasks.workunit.client.0.smithi116.stdout:3/280: read dc/d23/f21 [84928,121498] 0 2024-03-04T16:04:06.815 INFO:tasks.workunit.client.0.smithi116.stdout:1/139: write d5/f1c [3187338,15158] 0 2024-03-04T16:04:06.815 INFO:tasks.workunit.client.0.smithi116.stdout:4/172: fsync d1/d3/dd/f31 0 2024-03-04T16:04:06.817 INFO:tasks.workunit.client.0.smithi116.stdout:2/190: dwrite d7/d13/f1e [0,4194304] 0 2024-03-04T16:04:06.819 INFO:tasks.workunit.client.0.smithi116.stdout:5/212: mknod dd/d14/c47 0 2024-03-04T16:04:06.823 INFO:tasks.workunit.client.0.smithi116.stdout:7/134: symlink d3/d4/d8/d1a/l30 0 2024-03-04T16:04:06.825 INFO:tasks.workunit.client.0.smithi116.stdout:5/213: write dd/d14/d2d/d31/f3c [434758,121320] 0 2024-03-04T16:04:06.825 INFO:tasks.workunit.client.0.smithi116.stdout:7/135: fdatasync d3/d4/d8/f1c 0 2024-03-04T16:04:06.827 INFO:tasks.workunit.client.0.smithi116.stdout:4/173: dwrite d1/d3/f39 [0,4194304] 0 2024-03-04T16:04:06.828 INFO:tasks.workunit.client.0.smithi116.stdout:9/147: dread d2/d19/d27/d1f/f2f [0,4194304] 0 2024-03-04T16:04:06.828 INFO:tasks.workunit.client.0.smithi116.stdout:2/191: rmdir d7/d29/d47 39 2024-03-04T16:04:06.831 INFO:tasks.workunit.client.0.smithi116.stdout:9/148: truncate d2/d19/d27/d1f/f25 566307 0 2024-03-04T16:04:06.833 INFO:tasks.workunit.client.0.smithi116.stdout:8/189: mknod d2/de/d16/d2b/d37/c3f 0 2024-03-04T16:04:06.834 INFO:tasks.workunit.client.0.smithi116.stdout:6/247: truncate d6/de/d25/f40 2171240 0 2024-03-04T16:04:06.835 INFO:tasks.workunit.client.0.smithi116.stdout:7/136: dread d3/f10 [0,4194304] 0 2024-03-04T16:04:06.839 INFO:tasks.workunit.client.0.smithi116.stdout:4/174: mknod d1/d3/c3e 0 2024-03-04T16:04:06.842 INFO:tasks.workunit.client.0.smithi116.stdout:2/192: unlink f1 0 2024-03-04T16:04:06.843 INFO:tasks.workunit.client.0.smithi116.stdout:8/190: chown d2/d13/d1e/c2f 0 1 2024-03-04T16:04:06.847 INFO:tasks.workunit.client.0.smithi116.stdout:1/140: creat d5/d9/d21/f2c x:0 0 0 2024-03-04T16:04:06.848 INFO:tasks.workunit.client.0.smithi116.stdout:5/214: mknod dd/d45/c48 0 2024-03-04T16:04:06.851 INFO:tasks.workunit.client.0.smithi116.stdout:4/175: chown d1/d3/de/f23 1 1 2024-03-04T16:04:06.856 INFO:tasks.workunit.client.0.smithi116.stdout:8/191: symlink d2/d38/l40 0 2024-03-04T16:04:06.856 INFO:tasks.workunit.client.0.smithi116.stdout:0/189: link da/d16/d28/c37 da/d16/d19/d27/c44 0 2024-03-04T16:04:06.856 INFO:tasks.workunit.client.0.smithi116.stdout:8/192: chown d2/de/d3e/l20 117542437 1 2024-03-04T16:04:06.858 INFO:tasks.workunit.client.0.smithi116.stdout:6/248: mkdir d6/de/d25/d36/d4e/d50 0 2024-03-04T16:04:06.861 INFO:tasks.workunit.client.0.smithi116.stdout:1/141: symlink d5/d9/d21/d27/l2d 0 2024-03-04T16:04:06.864 INFO:tasks.workunit.client.0.smithi116.stdout:5/215: rmdir dd/d14 39 2024-03-04T16:04:06.873 INFO:tasks.workunit.client.0.smithi116.stdout:3/281: truncate dc/f56 777376 0 2024-03-04T16:04:06.875 INFO:tasks.workunit.client.0.smithi116.stdout:3/282: write dc/df/d14/f2b [4654819,83523] 0 2024-03-04T16:04:06.876 INFO:tasks.workunit.client.0.smithi116.stdout:8/193: mkdir d2/d38/d41 0 2024-03-04T16:04:06.884 INFO:tasks.workunit.client.0.smithi116.stdout:2/193: creat d7/d29/f4b x:0 0 0 2024-03-04T16:04:06.885 INFO:tasks.workunit.client.0.smithi116.stdout:2/194: write d7/d29/f2d [903,4076] 0 2024-03-04T16:04:06.885 INFO:tasks.workunit.client.0.smithi116.stdout:9/149: link d2/d19/c2d d2/d19/d27/d1f/c31 0 2024-03-04T16:04:06.887 INFO:tasks.workunit.client.0.smithi116.stdout:5/216: dread f4 [4194304,4194304] 0 2024-03-04T16:04:06.890 INFO:tasks.workunit.client.0.smithi116.stdout:0/190: truncate da/fe 1591645 0 2024-03-04T16:04:06.893 INFO:tasks.workunit.client.0.smithi116.stdout:5/217: dwrite dd/d1f/f41 [0,4194304] 0 2024-03-04T16:04:06.895 INFO:tasks.workunit.client.0.smithi116.stdout:8/194: mkdir d2/d13/d1e/d42 0 2024-03-04T16:04:06.903 INFO:tasks.workunit.client.0.smithi116.stdout:9/150: dread d2/d19/d27/d1f/f24 [0,4194304] 0 2024-03-04T16:04:06.904 INFO:tasks.workunit.client.0.smithi116.stdout:9/151: write d2/d19/d27/d1f/f24 [5200457,54575] 0 2024-03-04T16:04:06.906 INFO:tasks.workunit.client.0.smithi116.stdout:2/195: symlink d7/d2f/l4c 0 2024-03-04T16:04:06.912 INFO:tasks.workunit.client.0.smithi116.stdout:0/191: mkdir da/d16/d19/d27/d45 0 2024-03-04T16:04:06.913 INFO:tasks.workunit.client.0.smithi116.stdout:0/192: chown da/d16/d20/f21 1162517 1 2024-03-04T16:04:06.914 INFO:tasks.workunit.client.0.smithi116.stdout:3/283: creat dc/df/d47/f61 x:0 0 0 2024-03-04T16:04:06.914 INFO:tasks.workunit.client.0.smithi116.stdout:0/193: write da/d16/d20/f21 [1437587,126754] 0 2024-03-04T16:04:06.917 INFO:tasks.workunit.client.0.smithi116.stdout:8/195: write d2/de/d16/f34 [896054,52871] 0 2024-03-04T16:04:06.918 INFO:tasks.workunit.client.0.smithi116.stdout:8/196: fdatasync d2/de/d16/f30 0 2024-03-04T16:04:06.920 INFO:tasks.workunit.client.0.smithi116.stdout:1/142: truncate f2 3212324 0 2024-03-04T16:04:06.922 INFO:tasks.workunit.client.0.smithi116.stdout:9/152: write d2/d19/f2a [229528,52771] 0 2024-03-04T16:04:06.922 INFO:tasks.workunit.client.0.smithi116.stdout:9/153: readlink d2/l14 0 2024-03-04T16:04:06.927 INFO:tasks.workunit.client.0.smithi116.stdout:3/284: creat dc/d23/d5a/f62 x:0 0 0 2024-03-04T16:04:06.929 INFO:tasks.workunit.client.0.smithi116.stdout:0/194: rename da/f2d to da/d16/d28/d3d/d41/f46 0 2024-03-04T16:04:06.930 INFO:tasks.workunit.client.0.smithi116.stdout:5/218: readlink dd/d14/d2d/l32 0 2024-03-04T16:04:06.930 INFO:tasks.workunit.client.0.smithi116.stdout:0/195: write da/d16/d19/d27/f34 [1623768,54984] 0 2024-03-04T16:04:06.930 INFO:tasks.workunit.client.0.smithi116.stdout:5/219: fsync dd/f15 0 2024-03-04T16:04:06.933 INFO:tasks.workunit.client.0.smithi116.stdout:1/143: symlink d5/d9/d21/d27/l2e 0 2024-03-04T16:04:06.934 INFO:tasks.workunit.client.0.smithi116.stdout:1/144: read - d5/d9/d21/f26 zero size 2024-03-04T16:04:06.936 INFO:tasks.workunit.client.0.smithi116.stdout:0/196: dwrite da/d16/d28/f3c [0,4194304] 0 2024-03-04T16:04:06.937 INFO:tasks.workunit.client.0.smithi116.stdout:3/285: symlink dc/df/d47/l63 0 2024-03-04T16:04:06.941 INFO:tasks.workunit.client.0.smithi116.stdout:5/220: creat dd/d45/f49 x:0 0 0 2024-03-04T16:04:06.952 INFO:tasks.workunit.client.0.smithi116.stdout:2/196: creat d7/f4d x:0 0 0 2024-03-04T16:04:06.953 INFO:tasks.workunit.client.0.smithi116.stdout:0/197: creat da/d16/d28/d39/f47 x:0 0 0 2024-03-04T16:04:06.959 INFO:tasks.workunit.client.0.smithi116.stdout:6/249: link d6/de/d25/f40 d6/de/d1e/f51 0 2024-03-04T16:04:06.961 INFO:tasks.workunit.client.0.smithi116.stdout:9/154: creat d2/f32 x:0 0 0 2024-03-04T16:04:06.968 INFO:tasks.workunit.client.0.smithi116.stdout:6/250: creat d6/de/f52 x:0 0 0 2024-03-04T16:04:06.972 INFO:tasks.workunit.client.0.smithi116.stdout:8/197: write d2/de/fb [1062726,53082] 0 2024-03-04T16:04:06.973 INFO:tasks.workunit.client.0.smithi116.stdout:1/145: creat d5/d9/d21/f2f x:0 0 0 2024-03-04T16:04:06.975 INFO:tasks.workunit.client.0.smithi116.stdout:0/198: mkdir da/d16/d28/d40/d48 0 2024-03-04T16:04:06.975 INFO:tasks.workunit.client.0.smithi116.stdout:0/199: chown da/l11 20127 1 2024-03-04T16:04:06.977 INFO:tasks.workunit.client.0.smithi116.stdout:3/286: creat dc/f64 x:0 0 0 2024-03-04T16:04:06.977 INFO:tasks.workunit.client.0.smithi116.stdout:3/287: dread - dc/df/f53 zero size 2024-03-04T16:04:06.978 INFO:tasks.workunit.client.0.smithi116.stdout:3/288: write dc/d23/d5a/f5c [516983,16593] 0 2024-03-04T16:04:06.978 INFO:tasks.workunit.client.0.smithi116.stdout:6/251: mkdir d6/de/d29/d53 0 2024-03-04T16:04:06.979 INFO:tasks.workunit.client.0.smithi116.stdout:3/289: write dc/d23/f46 [732678,116083] 0 2024-03-04T16:04:06.980 INFO:tasks.workunit.client.0.smithi116.stdout:9/155: mkdir d2/d33 0 2024-03-04T16:04:06.981 INFO:tasks.workunit.client.0.smithi116.stdout:7/137: sync 2024-03-04T16:04:06.981 INFO:tasks.workunit.client.0.smithi116.stdout:4/176: sync 2024-03-04T16:04:06.982 INFO:tasks.workunit.client.0.smithi116.stdout:7/138: truncate d3/f1e 4195721 0 2024-03-04T16:04:06.984 INFO:tasks.workunit.client.0.smithi116.stdout:0/200: mkdir da/d16/d28/d3d/d41/d49 0 2024-03-04T16:04:06.985 INFO:tasks.workunit.client.0.smithi116.stdout:5/221: write f4 [2976858,78035] 0 2024-03-04T16:04:06.994 INFO:tasks.workunit.client.0.smithi116.stdout:9/156: stat d2/d19/c2d 0 2024-03-04T16:04:06.998 INFO:tasks.workunit.client.0.smithi116.stdout:8/198: link d2/de/f8 d2/d13/d1e/f43 0 2024-03-04T16:04:06.998 INFO:tasks.workunit.client.0.smithi116.stdout:2/197: truncate d7/f36 2666925 0 2024-03-04T16:04:06.998 INFO:tasks.workunit.client.0.smithi116.stdout:8/199: dread - d2/f32 zero size 2024-03-04T16:04:06.999 INFO:tasks.workunit.client.0.smithi116.stdout:9/157: dwrite d2/d19/f2a [0,4194304] 0 2024-03-04T16:04:07.000 INFO:tasks.workunit.client.0.smithi116.stdout:9/158: chown d2/d19/d27/d1f/c31 21616119 1 2024-03-04T16:04:07.001 INFO:tasks.workunit.client.0.smithi116.stdout:4/177: rename d1/d3/de/c2b to d1/d3/d15/d20/c3f 0 2024-03-04T16:04:07.003 INFO:tasks.workunit.client.0.smithi116.stdout:7/139: rmdir d3/d4/d8/d1a 39 2024-03-04T16:04:07.007 INFO:tasks.workunit.client.0.smithi116.stdout:1/146: mknod d5/d9/d21/c30 0 2024-03-04T16:04:07.008 INFO:tasks.workunit.client.0.smithi116.stdout:7/140: dwrite f1 [0,4194304] 0 2024-03-04T16:04:07.012 INFO:tasks.workunit.client.0.smithi116.stdout:3/290: unlink c4 0 2024-03-04T16:04:07.013 INFO:tasks.workunit.client.0.smithi116.stdout:3/291: write dc/d23/f46 [665085,20499] 0 2024-03-04T16:04:07.014 INFO:tasks.workunit.client.0.smithi116.stdout:3/292: read - dc/df/d14/f58 zero size 2024-03-04T16:04:07.017 INFO:tasks.workunit.client.0.smithi116.stdout:6/252: link d6/f37 d6/de/d29/d53/f54 0 2024-03-04T16:04:07.023 INFO:tasks.workunit.client.0.smithi116.stdout:9/159: symlink d2/d19/d27/d1f/l34 0 2024-03-04T16:04:07.023 INFO:tasks.workunit.client.0.smithi116.stdout:9/160: read d2/f1a [227103,68448] 0 2024-03-04T16:04:07.024 INFO:tasks.workunit.client.0.smithi116.stdout:5/222: rename dd/l1a to dd/d45/l4a 0 2024-03-04T16:04:07.026 INFO:tasks.workunit.client.0.smithi116.stdout:4/178: mkdir d1/d3/d40 0 2024-03-04T16:04:07.028 INFO:tasks.workunit.client.0.smithi116.stdout:7/141: unlink d3/d4/d11/f22 0 2024-03-04T16:04:07.030 INFO:tasks.workunit.client.0.smithi116.stdout:0/201: link da/d16/d19/f33 da/d16/d28/d3d/f4a 0 2024-03-04T16:04:07.037 INFO:tasks.workunit.client.0.smithi116.stdout:6/253: symlink d6/l55 0 2024-03-04T16:04:07.038 INFO:tasks.workunit.client.0.smithi116.stdout:2/198: getdents d7/d13/d3e 0 2024-03-04T16:04:07.038 INFO:tasks.workunit.client.0.smithi116.stdout:2/199: chown d7/d29 51 1 2024-03-04T16:04:07.041 INFO:tasks.workunit.client.0.smithi116.stdout:8/200: rename d2/de/fb to d2/d13/d1e/f44 0 2024-03-04T16:04:07.044 INFO:tasks.workunit.client.0.smithi116.stdout:5/223: unlink dd/d14/c47 0 2024-03-04T16:04:07.046 INFO:tasks.workunit.client.0.smithi116.stdout:4/179: creat d1/f41 x:0 0 0 2024-03-04T16:04:07.049 INFO:tasks.workunit.client.0.smithi116.stdout:7/142: mknod d3/d4/d11/c31 0 2024-03-04T16:04:07.054 INFO:tasks.workunit.client.0.smithi116.stdout:7/143: dwrite d3/fd [4194304,4194304] 0 2024-03-04T16:04:07.058 INFO:tasks.workunit.client.0.smithi116.stdout:1/147: rename d5/d9/d21/d27/l2e to d5/d11/l31 0 2024-03-04T16:04:07.058 INFO:tasks.workunit.client.0.smithi116.stdout:1/148: write d5/d9/f23 [619333,56327] 0 2024-03-04T16:04:07.059 INFO:tasks.workunit.client.0.smithi116.stdout:8/201: creat d2/d13/d1e/f45 x:0 0 0 2024-03-04T16:04:07.061 INFO:tasks.workunit.client.0.smithi116.stdout:3/293: write dc/d23/f21 [976360,59604] 0 2024-03-04T16:04:07.068 INFO:tasks.workunit.client.0.smithi116.stdout:4/180: creat d1/d3/de/d33/f42 x:0 0 0 2024-03-04T16:04:07.068 INFO:tasks.workunit.client.0.smithi116.stdout:4/181: stat d1/d3 0 2024-03-04T16:04:07.071 INFO:tasks.workunit.client.0.smithi116.stdout:0/202: fsync da/fe 0 2024-03-04T16:04:07.073 INFO:tasks.workunit.client.0.smithi116.stdout:9/161: creat d2/f35 x:0 0 0 2024-03-04T16:04:07.074 INFO:tasks.workunit.client.0.smithi116.stdout:9/162: chown d2/d19/d27 7257 1 2024-03-04T16:04:07.085 INFO:tasks.workunit.client.0.smithi116.stdout:5/224: truncate dd/f15 2119920 0 2024-03-04T16:04:07.089 INFO:tasks.workunit.client.0.smithi116.stdout:4/182: creat d1/d3/dd/f43 x:0 0 0 2024-03-04T16:04:07.090 INFO:tasks.workunit.client.0.smithi116.stdout:4/183: dread - d1/d3/dd/f31 zero size 2024-03-04T16:04:07.090 INFO:tasks.workunit.client.0.smithi116.stdout:4/184: dread - d1/d3/dd/f31 zero size 2024-03-04T16:04:07.096 INFO:tasks.workunit.client.0.smithi116.stdout:8/202: mkdir d2/d13/d3c/d46 0 2024-03-04T16:04:07.098 INFO:tasks.workunit.client.0.smithi116.stdout:5/225: unlink c6 0 2024-03-04T16:04:07.100 INFO:tasks.workunit.client.0.smithi116.stdout:3/294: write dc/d34/f3e [784031,45251] 0 2024-03-04T16:04:07.104 INFO:tasks.workunit.client.0.smithi116.stdout:4/185: creat d1/d3/f44 x:0 0 0 2024-03-04T16:04:07.104 INFO:tasks.workunit.client.0.smithi116.stdout:4/186: dread - d1/d3/dd/f31 zero size 2024-03-04T16:04:07.106 INFO:tasks.workunit.client.0.smithi116.stdout:4/187: read d1/d3/de/f23 [4162470,75488] 0 2024-03-04T16:04:07.106 INFO:tasks.workunit.client.0.smithi116.stdout:2/200: rename d7/d1b/f27 to d7/f4e 0 2024-03-04T16:04:07.108 INFO:tasks.workunit.client.0.smithi116.stdout:8/203: mknod d2/de/d16/d36/d29/c47 0 2024-03-04T16:04:07.110 INFO:tasks.workunit.client.0.smithi116.stdout:1/149: truncate f4 3945493 0 2024-03-04T16:04:07.111 INFO:tasks.workunit.client.0.smithi116.stdout:4/188: dwrite d1/d3/f39 [0,4194304] 0 2024-03-04T16:04:07.114 INFO:tasks.workunit.client.0.smithi116.stdout:4/189: read d1/d3/dd/f9 [1436572,94835] 0 2024-03-04T16:04:07.117 INFO:tasks.workunit.client.0.smithi116.stdout:0/203: rename da/d16/d19/c1b to da/d16/d20/d42/c4b 0 2024-03-04T16:04:07.118 INFO:tasks.workunit.client.0.smithi116.stdout:2/201: symlink d7/d15/l4f 0 2024-03-04T16:04:07.119 INFO:tasks.workunit.client.0.smithi116.stdout:8/204: mknod d2/d38/c48 0 2024-03-04T16:04:07.122 INFO:tasks.workunit.client.0.smithi116.stdout:5/226: rename dd/d45/l4a to dd/d45/l4b 0 2024-03-04T16:04:07.124 INFO:tasks.workunit.client.0.smithi116.stdout:1/150: mkdir d5/d9/d21/d27/d32 0 2024-03-04T16:04:07.125 INFO:tasks.workunit.client.0.smithi116.stdout:1/151: read - d5/d9/d21/f2c zero size 2024-03-04T16:04:07.127 INFO:tasks.workunit.client.0.smithi116.stdout:5/227: dwrite dd/d45/f49 [0,4194304] 0 2024-03-04T16:04:07.129 INFO:tasks.workunit.client.0.smithi116.stdout:5/228: truncate dd/d14/d2d/f42 26144 0 2024-03-04T16:04:07.130 INFO:tasks.workunit.client.0.smithi116.stdout:6/254: sync 2024-03-04T16:04:07.132 INFO:tasks.workunit.client.0.smithi116.stdout:4/190: mkdir d1/d3/dd/d45 0 2024-03-04T16:04:07.139 INFO:tasks.workunit.client.0.smithi116.stdout:9/163: rename d2/c2e to d2/d33/c36 0 2024-03-04T16:04:07.140 INFO:tasks.workunit.client.0.smithi116.stdout:9/164: chown d2/d19/d27/d1f/c23 175591 1 2024-03-04T16:04:07.140 INFO:tasks.workunit.client.0.smithi116.stdout:9/165: chown d2/d8/f21 55013 1 2024-03-04T16:04:07.159 INFO:tasks.workunit.client.0.smithi116.stdout:2/202: dwrite d7/f4e [4194304,4194304] 0 2024-03-04T16:04:07.183 INFO:tasks.workunit.client.0.smithi116.stdout:8/205: creat d2/de/d16/d2b/d37/f49 x:0 0 0 2024-03-04T16:04:07.184 INFO:tasks.workunit.client.0.smithi116.stdout:8/206: truncate d2/de/fa 2196087 0 2024-03-04T16:04:07.188 INFO:tasks.workunit.client.0.smithi116.stdout:3/295: creat dc/d23/d5a/f65 x:0 0 0 2024-03-04T16:04:07.191 INFO:tasks.workunit.client.0.smithi116.stdout:4/191: creat d1/d3/f46 x:0 0 0 2024-03-04T16:04:07.195 INFO:tasks.workunit.client.0.smithi116.stdout:7/144: rename d3/f14 to d3/d4/d8/d1a/f32 0 2024-03-04T16:04:07.196 INFO:tasks.workunit.client.0.smithi116.stdout:7/145: chown f1 1527106 1 2024-03-04T16:04:07.197 INFO:tasks.workunit.client.0.smithi116.stdout:4/192: dwrite d1/d3/dd/d24/f29 [0,4194304] 0 2024-03-04T16:04:07.198 INFO:tasks.workunit.client.0.smithi116.stdout:4/193: chown d1/d3/dd/d24/f3d 0 1 2024-03-04T16:04:07.201 INFO:tasks.workunit.client.0.smithi116.stdout:0/204: dwrite f7 [0,4194304] 0 2024-03-04T16:04:07.213 INFO:tasks.workunit.client.0.smithi116.stdout:1/152: symlink d5/l33 0 2024-03-04T16:04:07.216 INFO:tasks.workunit.client.0.smithi116.stdout:5/229: symlink dd/l4c 0 2024-03-04T16:04:07.224 INFO:tasks.workunit.client.0.smithi116.stdout:6/255: rename d6/de/d15/d47 to d6/d56 0 2024-03-04T16:04:07.233 INFO:tasks.workunit.client.0.smithi116.stdout:2/203: link d7/d1b/f26 d7/d2f/f50 0 2024-03-04T16:04:07.235 INFO:tasks.workunit.client.0.smithi116.stdout:8/207: mkdir d2/d4a 0 2024-03-04T16:04:07.238 INFO:tasks.workunit.client.0.smithi116.stdout:8/208: dread d2/de/f8 [0,4194304] 0 2024-03-04T16:04:07.247 INFO:tasks.workunit.client.0.smithi116.stdout:1/153: symlink d5/d9/d21/l34 0 2024-03-04T16:04:07.250 INFO:tasks.workunit.client.0.smithi116.stdout:1/154: dread d5/d11/f16 [0,4194304] 0 2024-03-04T16:04:07.252 INFO:tasks.workunit.client.0.smithi116.stdout:6/256: creat d6/de/d29/d53/f57 x:0 0 0 2024-03-04T16:04:07.253 INFO:tasks.workunit.client.0.smithi116.stdout:2/204: creat d7/d29/d47/f51 x:0 0 0 2024-03-04T16:04:07.255 INFO:tasks.workunit.client.0.smithi116.stdout:0/205: rename da/d16/d19/f2f to da/d16/f4c 0 2024-03-04T16:04:07.256 INFO:tasks.workunit.client.0.smithi116.stdout:8/209: mknod d2/d38/d41/c4b 0 2024-03-04T16:04:07.259 INFO:tasks.workunit.client.0.smithi116.stdout:6/257: mkdir d6/d56/d58 0 2024-03-04T16:04:07.261 INFO:tasks.workunit.client.0.smithi116.stdout:2/205: creat d7/d13/f52 x:0 0 0 2024-03-04T16:04:07.261 INFO:tasks.workunit.client.0.smithi116.stdout:2/206: fsync d7/d2f/f44 0 2024-03-04T16:04:07.263 INFO:tasks.workunit.client.0.smithi116.stdout:7/146: rmdir d3 39 2024-03-04T16:04:07.267 INFO:tasks.workunit.client.0.smithi116.stdout:2/207: dwrite d7/d2f/f46 [0,4194304] 0 2024-03-04T16:04:07.276 INFO:tasks.workunit.client.0.smithi116.stdout:6/258: symlink d6/de/d15/l59 0 2024-03-04T16:04:07.280 INFO:tasks.workunit.client.0.smithi116.stdout:7/147: fdatasync d3/d4/d8/d1a/f28 0 2024-03-04T16:04:07.282 INFO:tasks.workunit.client.0.smithi116.stdout:6/259: dwrite d6/de/f38 [0,4194304] 0 2024-03-04T16:04:07.284 INFO:tasks.workunit.client.0.smithi116.stdout:3/296: rename dc/c1f to dc/df/c66 0 2024-03-04T16:04:07.288 INFO:tasks.workunit.client.0.smithi116.stdout:3/297: chown dc/d23/d5a/f65 10 1 2024-03-04T16:04:07.289 INFO:tasks.workunit.client.0.smithi116.stdout:7/148: dwrite d3/f1f [0,4194304] 0 2024-03-04T16:04:07.293 INFO:tasks.workunit.client.0.smithi116.stdout:7/149: dread d3/fd [0,4194304] 0 2024-03-04T16:04:07.299 INFO:tasks.workunit.client.0.smithi116.stdout:5/230: rename dd/d14/d2d/d31/f40 to dd/d1f/f4d 0 2024-03-04T16:04:07.302 INFO:tasks.workunit.client.0.smithi116.stdout:2/208: unlink d7/d15/f2e 0 2024-03-04T16:04:07.302 INFO:tasks.workunit.client.0.smithi116.stdout:9/166: truncate d2/d19/f2a 2748607 0 2024-03-04T16:04:07.304 INFO:tasks.workunit.client.0.smithi116.stdout:3/298: readlink dc/l2a 0 2024-03-04T16:04:07.305 INFO:tasks.workunit.client.0.smithi116.stdout:6/260: dread d6/f14 [0,4194304] 0 2024-03-04T16:04:07.306 INFO:tasks.workunit.client.0.smithi116.stdout:7/150: mknod d3/d4/d8/d1a/c33 0 2024-03-04T16:04:07.307 INFO:tasks.workunit.client.0.smithi116.stdout:6/261: write d6/de/d1e/f35 [2056274,83239] 0 2024-03-04T16:04:07.309 INFO:tasks.workunit.client.0.smithi116.stdout:2/209: dwrite d7/d29/d47/f51 [0,4194304] 0 2024-03-04T16:04:07.309 INFO:tasks.workunit.client.0.smithi116.stdout:7/151: write d3/d4/d8/d1a/f20 [95435,36701] 0 2024-03-04T16:04:07.310 INFO:tasks.workunit.client.0.smithi116.stdout:4/194: sync 2024-03-04T16:04:07.311 INFO:tasks.workunit.client.0.smithi116.stdout:7/152: chown d3/d4/d8/d1a/f27 128594 1 2024-03-04T16:04:07.312 INFO:tasks.workunit.client.0.smithi116.stdout:5/231: creat dd/d1f/f4e x:0 0 0 2024-03-04T16:04:07.313 INFO:tasks.workunit.client.0.smithi116.stdout:7/153: write d3/d4/d8/f13 [728830,89531] 0 2024-03-04T16:04:07.324 INFO:tasks.workunit.client.0.smithi116.stdout:6/262: mknod d6/de/d25/d36/d44/c5a 0 2024-03-04T16:04:07.326 INFO:tasks.workunit.client.0.smithi116.stdout:2/210: creat d7/d25/d41/f53 x:0 0 0 2024-03-04T16:04:07.328 INFO:tasks.workunit.client.0.smithi116.stdout:3/299: dread dc/df/d14/f2b [0,4194304] 0 2024-03-04T16:04:07.328 INFO:tasks.workunit.client.0.smithi116.stdout:0/206: rmdir da 39 2024-03-04T16:04:07.329 INFO:tasks.workunit.client.0.smithi116.stdout:3/300: truncate dc/d23/d5a/f65 113491 0 2024-03-04T16:04:07.331 INFO:tasks.workunit.client.0.smithi116.stdout:1/155: write d5/d9/fb [2299284,121226] 0 2024-03-04T16:04:07.337 INFO:tasks.workunit.client.0.smithi116.stdout:4/195: mkdir d1/d3/de/d47 0 2024-03-04T16:04:07.338 INFO:tasks.workunit.client.0.smithi116.stdout:4/196: readlink d1/d3/d15/d20/d1c/l36 0 2024-03-04T16:04:07.341 INFO:tasks.workunit.client.0.smithi116.stdout:2/211: rename d7/d13/f18 to d7/d15/d35/f54 0 2024-03-04T16:04:07.341 INFO:tasks.workunit.client.0.smithi116.stdout:2/212: chown d7/d25/d41/f53 20785253 1 2024-03-04T16:04:07.345 INFO:tasks.workunit.client.0.smithi116.stdout:3/301: dread - dc/f5f zero size 2024-03-04T16:04:07.349 INFO:tasks.workunit.client.0.smithi116.stdout:7/154: mkdir d3/d4/d34 0 2024-03-04T16:04:07.350 INFO:tasks.workunit.client.0.smithi116.stdout:7/155: write d3/d4/d11/f2f [429403,93635] 0 2024-03-04T16:04:07.354 INFO:tasks.workunit.client.0.smithi116.stdout:4/197: symlink d1/d3/d15/l48 0 2024-03-04T16:04:07.360 INFO:tasks.workunit.client.0.smithi116.stdout:0/207: fsync da/d16/d19/f33 0 2024-03-04T16:04:07.360 INFO:tasks.workunit.client.0.smithi116.stdout:3/302: mknod dc/d34/d44/c67 0 2024-03-04T16:04:07.360 INFO:tasks.workunit.client.0.smithi116.stdout:8/210: truncate d2/de/d16/f17 1062876 0 2024-03-04T16:04:07.363 INFO:tasks.workunit.client.0.smithi116.stdout:7/156: read f1 [44971,34396] 0 2024-03-04T16:04:07.365 INFO:tasks.workunit.client.0.smithi116.stdout:1/156: mkdir d5/d9/d21/d27/d32/d35 0 2024-03-04T16:04:07.366 INFO:tasks.workunit.client.0.smithi116.stdout:9/167: getdents d2/d33 0 2024-03-04T16:04:07.368 INFO:tasks.workunit.client.0.smithi116.stdout:4/198: rename d1/d3/d15/d20/l37 to d1/d3/d15/d20/d1c/l49 0 2024-03-04T16:04:07.372 INFO:tasks.workunit.client.0.smithi116.stdout:8/211: unlink d2/de/d16/f26 0 2024-03-04T16:04:07.376 INFO:tasks.workunit.client.0.smithi116.stdout:5/232: getdents dd 0 2024-03-04T16:04:07.382 INFO:tasks.workunit.client.0.smithi116.stdout:5/233: dwrite dd/d1f/f41 [4194304,4194304] 0 2024-03-04T16:04:07.383 INFO:tasks.workunit.client.0.smithi116.stdout:5/234: write dd/d1f/f4e [641013,103260] 0 2024-03-04T16:04:07.383 INFO:tasks.workunit.client.0.smithi116.stdout:8/212: truncate d2/de/f12 3448985 0 2024-03-04T16:04:07.391 INFO:tasks.workunit.client.0.smithi116.stdout:3/303: rename dc/l55 to dc/d23/d5a/l68 0 2024-03-04T16:04:07.393 INFO:tasks.workunit.client.0.smithi116.stdout:5/235: symlink dd/l4f 0 2024-03-04T16:04:07.399 INFO:tasks.workunit.client.0.smithi116.stdout:1/157: creat d5/d9/d21/f36 x:0 0 0 2024-03-04T16:04:07.399 INFO:tasks.workunit.client.0.smithi116.stdout:9/168: rename d2/f2b to d2/d19/f37 0 2024-03-04T16:04:07.407 INFO:tasks.workunit.client.0.smithi116.stdout:3/304: link dc/df/d14/f2b dc/d23/d5a/f69 0 2024-03-04T16:04:07.410 INFO:tasks.workunit.client.0.smithi116.stdout:3/305: dread dc/df/d14/f49 [0,4194304] 0 2024-03-04T16:04:07.411 INFO:tasks.workunit.client.0.smithi116.stdout:4/199: rename d1/d3/f22 to d1/d3/dd/f4a 0 2024-03-04T16:04:07.412 INFO:tasks.workunit.client.0.smithi116.stdout:4/200: write d1/d3/f39 [4405694,103690] 0 2024-03-04T16:04:07.416 INFO:tasks.workunit.client.0.smithi116.stdout:1/158: rename d5/d9/d21/f26 to d5/d9/d21/d24/f37 0 2024-03-04T16:04:07.417 INFO:tasks.workunit.client.0.smithi116.stdout:1/159: dread - d5/d9/d21/f2f zero size 2024-03-04T16:04:07.418 INFO:tasks.workunit.client.0.smithi116.stdout:1/160: stat d5/d9/d21/c2b 0 2024-03-04T16:04:07.418 INFO:tasks.workunit.client.0.smithi116.stdout:1/161: chown d5/d9/d21/d27/l2d 17 1 2024-03-04T16:04:07.423 INFO:tasks.workunit.client.0.smithi116.stdout:1/162: dwrite d5/f17 [0,4194304] 0 2024-03-04T16:04:07.424 INFO:tasks.workunit.client.0.smithi116.stdout:1/163: write d5/d9/d21/d27/f2a [605654,121765] 0 2024-03-04T16:04:07.425 INFO:tasks.workunit.client.0.smithi116.stdout:1/164: readlink d5/d9/d21/l34 0 2024-03-04T16:04:07.425 INFO:tasks.workunit.client.0.smithi116.stdout:4/201: rename d1/d3/de/d33/f3a to d1/d3/dd/d45/f4b 0 2024-03-04T16:04:07.436 INFO:tasks.workunit.client.0.smithi116.stdout:7/157: dwrite d3/f6 [0,4194304] 0 2024-03-04T16:04:07.437 INFO:tasks.workunit.client.0.smithi116.stdout:7/158: readlink d3/d4/l21 0 2024-03-04T16:04:07.451 INFO:tasks.workunit.client.0.smithi116.stdout:1/165: creat d5/f38 x:0 0 0 2024-03-04T16:04:07.457 INFO:tasks.workunit.client.0.smithi116.stdout:5/236: dread dd/d45/f49 [0,4194304] 0 2024-03-04T16:04:07.459 INFO:tasks.workunit.client.0.smithi116.stdout:4/202: getdents d1/d3/d15/d20/d1c 0 2024-03-04T16:04:07.463 INFO:tasks.workunit.client.0.smithi116.stdout:9/169: dwrite d2/fc [0,4194304] 0 2024-03-04T16:04:07.463 INFO:tasks.workunit.client.0.smithi116.stdout:7/159: creat d3/d4/d8/f35 x:0 0 0 2024-03-04T16:04:07.466 INFO:tasks.workunit.client.0.smithi116.stdout:5/237: truncate dd/d1f/f4d 108893 0 2024-03-04T16:04:07.470 INFO:tasks.workunit.client.0.smithi116.stdout:7/160: dwrite d3/d4/d8/d1a/f29 [0,4194304] 0 2024-03-04T16:04:07.473 INFO:tasks.workunit.client.0.smithi116.stdout:4/203: dread d1/d3/de/f26 [0,4194304] 0 2024-03-04T16:04:07.474 INFO:tasks.workunit.client.0.smithi116.stdout:4/204: truncate d1/d3/d15/d20/f1e 1374298 0 2024-03-04T16:04:07.477 INFO:tasks.workunit.client.0.smithi116.stdout:5/238: readlink dd/l4c 0 2024-03-04T16:04:07.483 INFO:tasks.workunit.client.0.smithi116.stdout:7/161: creat d3/d4/d11/f36 x:0 0 0 2024-03-04T16:04:07.484 INFO:tasks.workunit.client.0.smithi116.stdout:3/306: truncate dc/d23/d2c/f3b 1076123 0 2024-03-04T16:04:07.486 INFO:tasks.workunit.client.0.smithi116.stdout:9/170: unlink d2/d19/l1e 0 2024-03-04T16:04:07.487 INFO:tasks.workunit.client.0.smithi116.stdout:7/162: dread f1 [4194304,4194304] 0 2024-03-04T16:04:07.487 INFO:tasks.workunit.client.0.smithi116.stdout:4/205: fsync d1/d3/de/f26 0 2024-03-04T16:04:07.488 INFO:tasks.workunit.client.0.smithi116.stdout:4/206: chown d1/d3/de/f12 23682204 1 2024-03-04T16:04:07.489 INFO:tasks.workunit.client.0.smithi116.stdout:4/207: truncate d1/d3/f44 477899 0 2024-03-04T16:04:07.491 INFO:tasks.workunit.client.0.smithi116.stdout:4/208: fdatasync d1/d3/de/f12 0 2024-03-04T16:04:07.492 INFO:tasks.workunit.client.0.smithi116.stdout:3/307: rmdir dc 39 2024-03-04T16:04:07.492 INFO:tasks.workunit.client.0.smithi116.stdout:5/239: write dd/d14/d2d/d31/f3c [1534723,57296] 0 2024-03-04T16:04:07.495 INFO:tasks.workunit.client.0.smithi116.stdout:8/213: dwrite d2/de/f12 [0,4194304] 0 2024-03-04T16:04:07.495 INFO:tasks.workunit.client.0.smithi116.stdout:9/171: creat d2/d19/d27/f38 x:0 0 0 2024-03-04T16:04:07.496 INFO:tasks.workunit.client.0.smithi116.stdout:8/214: rename d2/de/d16 to d2/de/d16/d4c 22 2024-03-04T16:04:07.501 INFO:tasks.workunit.client.0.smithi116.stdout:1/166: rmdir d5 39 2024-03-04T16:04:07.501 INFO:tasks.workunit.client.0.smithi116.stdout:4/209: creat d1/d3/dd/d45/f4c x:0 0 0 2024-03-04T16:04:07.502 INFO:tasks.workunit.client.0.smithi116.stdout:4/210: read - d1/d3/dd/f3c zero size 2024-03-04T16:04:07.502 INFO:tasks.workunit.client.0.smithi116.stdout:8/215: dread d2/de/f12 [0,4194304] 0 2024-03-04T16:04:07.503 INFO:tasks.workunit.client.0.smithi116.stdout:5/240: creat dd/d14/d2d/f50 x:0 0 0 2024-03-04T16:04:07.504 INFO:tasks.workunit.client.0.smithi116.stdout:5/241: truncate dd/d14/d2d/d31/f3c 2168862 0 2024-03-04T16:04:07.510 INFO:tasks.workunit.client.0.smithi116.stdout:4/211: symlink d1/d3/de/d33/l4d 0 2024-03-04T16:04:07.510 INFO:tasks.workunit.client.0.smithi116.stdout:5/242: dwrite dd/d1f/f41 [0,4194304] 0 2024-03-04T16:04:07.517 INFO:tasks.workunit.client.0.smithi116.stdout:1/167: truncate f2 3910635 0 2024-03-04T16:04:07.518 INFO:tasks.workunit.client.0.smithi116.stdout:9/172: dread d2/f15 [0,4194304] 0 2024-03-04T16:04:07.519 INFO:tasks.workunit.client.0.smithi116.stdout:5/243: mkdir dd/d1f/d51 0 2024-03-04T16:04:07.520 INFO:tasks.workunit.client.0.smithi116.stdout:8/216: dwrite d2/d13/d1e/f44 [0,4194304] 0 2024-03-04T16:04:07.520 INFO:tasks.workunit.client.0.smithi116.stdout:4/212: fsync d1/d3/dd/d45/f4b 0 2024-03-04T16:04:07.527 INFO:tasks.workunit.client.0.smithi116.stdout:9/173: creat d2/d26/f39 x:0 0 0 2024-03-04T16:04:07.531 INFO:tasks.workunit.client.0.smithi116.stdout:4/213: creat d1/d3/d15/d20/f4e x:0 0 0 2024-03-04T16:04:07.532 INFO:tasks.workunit.client.0.smithi116.stdout:4/214: write d1/fb [1519633,63927] 0 2024-03-04T16:04:07.534 INFO:tasks.workunit.client.0.smithi116.stdout:8/217: creat d2/de/d16/d2b/d37/f4d x:0 0 0 2024-03-04T16:04:07.536 INFO:tasks.workunit.client.0.smithi116.stdout:4/215: dread d1/d3/d15/d20/d1c/f27 [0,4194304] 0 2024-03-04T16:04:07.542 INFO:tasks.workunit.client.0.smithi116.stdout:8/218: creat d2/de/d3e/f4e x:0 0 0 2024-03-04T16:04:07.542 INFO:tasks.workunit.client.0.smithi116.stdout:9/174: rmdir d2/d19/d27 39 2024-03-04T16:04:07.545 INFO:tasks.workunit.client.0.smithi116.stdout:5/244: dwrite dd/f13 [0,4194304] 0 2024-03-04T16:04:07.547 INFO:tasks.workunit.client.0.smithi116.stdout:4/216: creat d1/d3/d40/f4f x:0 0 0 2024-03-04T16:04:07.548 INFO:tasks.workunit.client.0.smithi116.stdout:8/219: dread d2/d13/d1e/f44 [0,4194304] 0 2024-03-04T16:04:07.565 INFO:tasks.workunit.client.0.smithi116.stdout:5/245: dwrite dd/d1f/f4d [0,4194304] 0 2024-03-04T16:04:07.565 INFO:tasks.workunit.client.0.smithi116.stdout:3/308: dwrite dc/f16 [0,4194304] 0 2024-03-04T16:04:07.566 INFO:tasks.workunit.client.0.smithi116.stdout:8/220: dwrite d2/de/d16/f30 [0,4194304] 0 2024-03-04T16:04:07.566 INFO:tasks.workunit.client.0.smithi116.stdout:3/309: dread - dc/df/d14/f58 zero size 2024-03-04T16:04:07.567 INFO:tasks.workunit.client.0.smithi116.stdout:3/310: chown cb 2345527 1 2024-03-04T16:04:07.569 INFO:tasks.workunit.client.0.smithi116.stdout:5/246: mkdir dd/d1f/d52 0 2024-03-04T16:04:07.570 INFO:tasks.workunit.client.0.smithi116.stdout:4/217: write d1/f2 [705907,96285] 0 2024-03-04T16:04:07.571 INFO:tasks.workunit.client.0.smithi116.stdout:9/175: dread d2/fc [4194304,4194304] 0 2024-03-04T16:04:07.575 INFO:tasks.workunit.client.0.smithi116.stdout:3/311: mkdir dc/d34/d44/d6a 0 2024-03-04T16:04:07.575 INFO:tasks.workunit.client.0.smithi116.stdout:6/263: sync 2024-03-04T16:04:07.575 INFO:tasks.workunit.client.0.smithi116.stdout:3/312: chown dc/d23/d5a 2161165 1 2024-03-04T16:04:07.575 INFO:tasks.workunit.client.0.smithi116.stdout:5/247: mknod dd/d45/c53 0 2024-03-04T16:04:07.577 INFO:tasks.workunit.client.0.smithi116.stdout:1/168: dwrite f2 [0,4194304] 0 2024-03-04T16:04:07.578 INFO:tasks.workunit.client.0.smithi116.stdout:5/248: write dd/d1f/f29 [499598,123786] 0 2024-03-04T16:04:07.580 INFO:tasks.workunit.client.0.smithi116.stdout:4/218: mkdir d1/d3/dd/d45/d50 0 2024-03-04T16:04:07.582 INFO:tasks.workunit.client.0.smithi116.stdout:6/264: rmdir d6/de/d25/d36 39 2024-03-04T16:04:07.583 INFO:tasks.workunit.client.0.smithi116.stdout:3/313: fdatasync dc/d23/d2c/f2d 0 2024-03-04T16:04:07.585 INFO:tasks.workunit.client.0.smithi116.stdout:3/314: write dc/df/d47/f61 [1039234,122171] 0 2024-03-04T16:04:07.586 INFO:tasks.workunit.client.0.smithi116.stdout:1/169: fsync d5/d9/f18 0 2024-03-04T16:04:07.587 INFO:tasks.workunit.client.0.smithi116.stdout:1/170: write d5/f1c [2637192,9212] 0 2024-03-04T16:04:07.590 INFO:tasks.workunit.client.0.smithi116.stdout:6/265: mkdir d6/de/d5b 0 2024-03-04T16:04:07.590 INFO:tasks.workunit.client.0.smithi116.stdout:5/249: dread dd/d14/d2d/d31/f3c [0,4194304] 0 2024-03-04T16:04:07.591 INFO:tasks.workunit.client.0.smithi116.stdout:3/315: dread dc/d23/f22 [0,4194304] 0 2024-03-04T16:04:07.595 INFO:tasks.workunit.client.0.smithi116.stdout:6/266: rmdir d6/de/d1e 39 2024-03-04T16:04:07.595 INFO:tasks.workunit.client.0.smithi116.stdout:6/267: read - d6/f21 zero size 2024-03-04T16:04:07.596 INFO:tasks.workunit.client.0.smithi116.stdout:5/250: creat dd/d1f/f54 x:0 0 0 2024-03-04T16:04:07.600 INFO:tasks.workunit.client.0.smithi116.stdout:6/268: mknod d6/de/d29/d53/c5c 0 2024-03-04T16:04:07.601 INFO:tasks.workunit.client.0.smithi116.stdout:5/251: mkdir dd/d1f/d55 0 2024-03-04T16:04:07.602 INFO:tasks.workunit.client.0.smithi116.stdout:3/316: creat dc/d34/d44/d6a/f6b x:0 0 0 2024-03-04T16:04:07.605 INFO:tasks.workunit.client.0.smithi116.stdout:6/269: rename d6/l12 to d6/d56/d58/l5d 0 2024-03-04T16:04:07.606 INFO:tasks.workunit.client.0.smithi116.stdout:6/270: fsync d6/de/f38 0 2024-03-04T16:04:07.606 INFO:tasks.workunit.client.0.smithi116.stdout:1/171: getdents d5/d9/d21 0 2024-03-04T16:04:07.608 INFO:tasks.workunit.client.0.smithi116.stdout:3/317: creat dc/df/d47/f6c x:0 0 0 2024-03-04T16:04:07.609 INFO:tasks.workunit.client.0.smithi116.stdout:4/219: dread d1/d3/de/f10 [0,4194304] 0 2024-03-04T16:04:07.609 INFO:tasks.workunit.client.0.smithi116.stdout:1/172: truncate d5/d9/d21/f2f 929360 0 2024-03-04T16:04:07.609 INFO:tasks.workunit.client.0.smithi116.stdout:5/252: rename dd/d1f/l35 to dd/d14/l56 0 2024-03-04T16:04:07.610 INFO:tasks.workunit.client.0.smithi116.stdout:4/220: fdatasync d1/f2c 0 2024-03-04T16:04:07.611 INFO:tasks.workunit.client.0.smithi116.stdout:8/221: dwrite d2/de/d16/f1b [0,4194304] 0 2024-03-04T16:04:07.612 INFO:tasks.workunit.client.0.smithi116.stdout:5/253: write dd/d14/d2d/f3b [812977,23655] 0 2024-03-04T16:04:07.613 INFO:tasks.workunit.client.0.smithi116.stdout:9/176: truncate d2/d8/f21 340320 0 2024-03-04T16:04:07.615 INFO:tasks.workunit.client.0.smithi116.stdout:6/271: chown d6/de/c1c 1 1 2024-03-04T16:04:07.616 INFO:tasks.workunit.client.0.smithi116.stdout:6/272: write d6/de/f3d [933434,118775] 0 2024-03-04T16:04:07.618 INFO:tasks.workunit.client.0.smithi116.stdout:0/208: sync 2024-03-04T16:04:07.618 INFO:tasks.workunit.client.0.smithi116.stdout:2/213: sync 2024-03-04T16:04:07.618 INFO:tasks.workunit.client.0.smithi116.stdout:0/209: chown da/d16/d28/d39/l3f 198380 1 2024-03-04T16:04:07.624 INFO:tasks.workunit.client.0.smithi116.stdout:4/221: mkdir d1/d3/dd/d24/d51 0 2024-03-04T16:04:07.626 INFO:tasks.workunit.client.0.smithi116.stdout:8/222: fsync d2/fd 0 2024-03-04T16:04:07.628 INFO:tasks.workunit.client.0.smithi116.stdout:5/254: mknod dd/d14/c57 0 2024-03-04T16:04:07.629 INFO:tasks.workunit.client.0.smithi116.stdout:9/177: write d2/f15 [5127502,25313] 0 2024-03-04T16:04:07.629 INFO:tasks.workunit.client.0.smithi116.stdout:5/255: truncate f4 5311513 0 2024-03-04T16:04:07.630 INFO:tasks.workunit.client.0.smithi116.stdout:4/222: dwrite d1/d3/d15/d20/f1e [0,4194304] 0 2024-03-04T16:04:07.632 INFO:tasks.workunit.client.0.smithi116.stdout:6/273: creat d6/d56/f5e x:0 0 0 2024-03-04T16:04:07.633 INFO:tasks.workunit.client.0.smithi116.stdout:8/223: dread d2/de/d16/f30 [0,4194304] 0 2024-03-04T16:04:07.634 INFO:tasks.workunit.client.0.smithi116.stdout:8/224: write d2/de/d16/f34 [2267988,96533] 0 2024-03-04T16:04:07.639 INFO:tasks.workunit.client.0.smithi116.stdout:0/210: read da/f13 [677641,27741] 0 2024-03-04T16:04:07.639 INFO:tasks.workunit.client.0.smithi116.stdout:1/173: rename d5/d11/c1b to d5/d9/d21/c39 0 2024-03-04T16:04:07.641 INFO:tasks.workunit.client.0.smithi116.stdout:3/318: write dc/d23/d5a/f69 [3661368,67841] 0 2024-03-04T16:04:07.641 INFO:tasks.workunit.client.0.smithi116.stdout:3/319: chown dc/d34 18142007 1 2024-03-04T16:04:07.643 INFO:tasks.workunit.client.0.smithi116.stdout:3/320: write dc/d34/f3e [1283858,113406] 0 2024-03-04T16:04:07.646 INFO:tasks.workunit.client.0.smithi116.stdout:4/223: mkdir d1/d3/d40/d52 0 2024-03-04T16:04:07.656 INFO:tasks.workunit.client.0.smithi116.stdout:9/178: getdents d2/d33 0 2024-03-04T16:04:07.656 INFO:tasks.workunit.client.0.smithi116.stdout:3/321: truncate dc/df/f53 89403 0 2024-03-04T16:04:07.657 INFO:tasks.workunit.client.0.smithi116.stdout:3/322: readlink dc/d23/d2c/l54 0 2024-03-04T16:04:07.657 INFO:tasks.workunit.client.0.smithi116.stdout:5/256: creat dd/d1f/d55/f58 x:0 0 0 2024-03-04T16:04:07.661 INFO:tasks.workunit.client.0.smithi116.stdout:5/257: dread dd/d1f/f4e [0,4194304] 0 2024-03-04T16:04:07.662 INFO:tasks.workunit.client.0.smithi116.stdout:8/225: mkdir d2/d13/d3c/d46/d4f 0 2024-03-04T16:04:07.664 INFO:tasks.workunit.client.0.smithi116.stdout:0/211: symlink da/d16/d28/d40/d48/l4d 0 2024-03-04T16:04:07.667 INFO:tasks.workunit.client.0.smithi116.stdout:5/258: dwrite dd/d14/d2d/f3b [0,4194304] 0 2024-03-04T16:04:07.669 INFO:tasks.workunit.client.0.smithi116.stdout:1/174: chown d5/d9/d21/c2b 25931 1 2024-03-04T16:04:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:6/274: creat d6/de/d25/d36/d4e/d50/f5f x:0 0 0 2024-03-04T16:04:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:2/214: getdents d7/d25 0 2024-03-04T16:04:07.674 INFO:tasks.workunit.client.0.smithi116.stdout:2/215: dread - d7/d15/f38 zero size 2024-03-04T16:04:07.674 INFO:tasks.workunit.client.0.smithi116.stdout:6/275: write d6/de/d25/d36/d4e/d50/f5f [313228,32677] 0 2024-03-04T16:04:07.676 INFO:tasks.workunit.client.0.smithi116.stdout:8/226: creat d2/de/d3e/f50 x:0 0 0 2024-03-04T16:04:07.678 INFO:tasks.workunit.client.0.smithi116.stdout:0/212: unlink da/d16/d19/f33 0 2024-03-04T16:04:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:9/179: fsync d2/d19/d27/d1f/f2f 0 2024-03-04T16:04:07.686 INFO:tasks.workunit.client.0.smithi116.stdout:5/259: dwrite dd/d1f/f4e [0,4194304] 0 2024-03-04T16:04:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:5/260: chown dd/d14/d2d/l28 325037267 1 2024-03-04T16:04:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:5/261: stat dd/l4f 0 2024-03-04T16:04:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:8/227: rmdir d2/de/d16/d36 39 2024-03-04T16:04:07.691 INFO:tasks.workunit.client.0.smithi116.stdout:8/228: write d2/de/d3e/f50 [972675,38024] 0 2024-03-04T16:04:07.691 INFO:tasks.workunit.client.0.smithi116.stdout:8/229: stat d2/de/d3e/f2a 0 2024-03-04T16:04:07.693 INFO:tasks.workunit.client.0.smithi116.stdout:0/213: mkdir da/d16/d20/d42/d4e 0 2024-03-04T16:04:07.694 INFO:tasks.workunit.client.0.smithi116.stdout:0/214: truncate da/d16/d20/f3b 85566 0 2024-03-04T16:04:07.697 INFO:tasks.workunit.client.0.smithi116.stdout:1/175: creat d5/d9/d21/d24/f3a x:0 0 0 2024-03-04T16:04:07.698 INFO:tasks.workunit.client.0.smithi116.stdout:4/224: getdents d1/d3/d40 0 2024-03-04T16:04:07.698 INFO:tasks.workunit.client.0.smithi116.stdout:1/176: truncate d5/d9/d21/d24/f37 403724 0 2024-03-04T16:04:07.740 INFO:tasks.workunit.client.0.smithi116.stdout:6/276: link d6/de/d25/d36/f27 d6/d56/f60 0 2024-03-04T16:04:07.741 INFO:tasks.workunit.client.0.smithi116.stdout:8/230: mknod d2/de/c51 0 2024-03-04T16:04:07.743 INFO:tasks.workunit.client.0.smithi116.stdout:4/225: chown d1/d3/dd/d24/f29 6952 1 2024-03-04T16:04:07.744 INFO:tasks.workunit.client.0.smithi116.stdout:0/215: mknod da/d16/d20/d42/d4e/c4f 0 2024-03-04T16:04:07.745 INFO:tasks.workunit.client.0.smithi116.stdout:6/277: symlink d6/d56/l61 0 2024-03-04T16:04:07.749 INFO:tasks.workunit.client.0.smithi116.stdout:8/231: write d2/d13/d1e/f43 [6906538,23594] 0 2024-03-04T16:04:07.749 INFO:tasks.workunit.client.0.smithi116.stdout:1/177: unlink d5/d9/d21/d27/f22 0 2024-03-04T16:04:07.750 INFO:tasks.workunit.client.0.smithi116.stdout:2/216: dwrite d7/d2f/f50 [0,4194304] 0 2024-03-04T16:04:07.750 INFO:tasks.workunit.client.0.smithi116.stdout:3/323: dwrite dc/df/f53 [0,4194304] 0 2024-03-04T16:04:07.751 INFO:tasks.workunit.client.0.smithi116.stdout:9/180: getdents d2/d26 0 2024-03-04T16:04:07.753 INFO:tasks.workunit.client.0.smithi116.stdout:9/181: readlink d2/la 0 2024-03-04T16:04:07.755 INFO:tasks.workunit.client.0.smithi116.stdout:2/217: dread - d7/d15/f38 zero size 2024-03-04T16:04:07.759 INFO:tasks.workunit.client.0.smithi116.stdout:0/216: dread da/d16/d20/f21 [0,4194304] 0 2024-03-04T16:04:07.759 INFO:tasks.workunit.client.0.smithi116.stdout:2/218: dread - d7/d25/f34 zero size 2024-03-04T16:04:07.759 INFO:tasks.workunit.client.0.smithi116.stdout:6/278: dwrite d6/de/f3d [0,4194304] 0 2024-03-04T16:04:07.761 INFO:tasks.workunit.client.0.smithi116.stdout:2/219: readlink d7/d29/l32 0 2024-03-04T16:04:07.763 INFO:tasks.workunit.client.0.smithi116.stdout:2/220: write d7/d2f/f44 [553721,127481] 0 2024-03-04T16:04:07.764 INFO:tasks.workunit.client.0.smithi116.stdout:4/226: rename d1/d3/dd/d24/l25 to d1/d3/d15/d20/l53 0 2024-03-04T16:04:07.765 INFO:tasks.workunit.client.0.smithi116.stdout:3/324: dwrite dc/d23/f42 [4194304,4194304] 0 2024-03-04T16:04:07.767 INFO:tasks.workunit.client.0.smithi116.stdout:4/227: chown d1/d3/d15/d20/d1c/l36 1 1 2024-03-04T16:04:07.767 INFO:tasks.workunit.client.0.smithi116.stdout:3/325: read dc/d34/f3e [853792,75536] 0 2024-03-04T16:04:07.767 INFO:tasks.workunit.client.0.smithi116.stdout:4/228: chown d1/d3/de 797489044 1 2024-03-04T16:04:07.768 INFO:tasks.workunit.client.0.smithi116.stdout:4/229: dread - d1/d3/f46 zero size 2024-03-04T16:04:07.769 INFO:tasks.workunit.client.0.smithi116.stdout:4/230: readlink d1/d3/d15/d20/d1c/l2a 0 2024-03-04T16:04:07.770 INFO:tasks.workunit.client.0.smithi116.stdout:9/182: truncate d2/d19/d27/d1f/f2f 3272892 0 2024-03-04T16:04:07.771 INFO:tasks.workunit.client.0.smithi116.stdout:9/183: chown d2/d8/l18 2055 1 2024-03-04T16:04:07.772 INFO:tasks.workunit.client.0.smithi116.stdout:9/184: dread - d2/f32 zero size 2024-03-04T16:04:07.772 INFO:tasks.workunit.client.0.smithi116.stdout:6/279: symlink d6/de/d29/d53/l62 0 2024-03-04T16:04:07.773 INFO:tasks.workunit.client.0.smithi116.stdout:9/185: read d2/f15 [2580709,121172] 0 2024-03-04T16:04:07.779 INFO:tasks.workunit.client.0.smithi116.stdout:0/217: dread da/f12 [0,4194304] 0 2024-03-04T16:04:07.781 INFO:tasks.workunit.client.0.smithi116.stdout:8/232: unlink d2/de/d16/f17 0 2024-03-04T16:04:07.783 INFO:tasks.workunit.client.0.smithi116.stdout:3/326: symlink dc/d34/d44/d6a/l6d 0 2024-03-04T16:04:07.784 INFO:tasks.workunit.client.0.smithi116.stdout:3/327: read dc/d23/d5a/f69 [2072357,116456] 0 2024-03-04T16:04:07.786 INFO:tasks.workunit.client.0.smithi116.stdout:4/231: fdatasync d1/d3/d15/d20/d1c/f27 0 2024-03-04T16:04:07.793 INFO:tasks.workunit.client.0.smithi116.stdout:9/186: creat d2/d33/f3a x:0 0 0 2024-03-04T16:04:07.793 INFO:tasks.workunit.client.0.smithi116.stdout:9/187: stat d2/d33/c36 0 2024-03-04T16:04:07.794 INFO:tasks.workunit.client.0.smithi116.stdout:1/178: creat d5/f3b x:0 0 0 2024-03-04T16:04:07.798 INFO:tasks.workunit.client.0.smithi116.stdout:7/163: sync 2024-03-04T16:04:07.804 INFO:tasks.workunit.client.0.smithi116.stdout:3/328: creat dc/d23/f6e x:0 0 0 2024-03-04T16:04:07.804 INFO:tasks.workunit.client.0.smithi116.stdout:3/329: chown dc/d34/c36 52780 1 2024-03-04T16:04:07.809 INFO:tasks.workunit.client.0.smithi116.stdout:6/280: mkdir d6/de/d25/d4a/d63 0 2024-03-04T16:04:07.809 INFO:tasks.workunit.client.0.smithi116.stdout:6/281: fsync d6/f13 0 2024-03-04T16:04:07.811 INFO:tasks.workunit.client.0.smithi116.stdout:2/221: link d7/ca d7/d2f/c55 0 2024-03-04T16:04:07.812 INFO:tasks.workunit.client.0.smithi116.stdout:0/218: dread - da/d16/d28/d3d/f4a zero size 2024-03-04T16:04:07.813 INFO:tasks.workunit.client.0.smithi116.stdout:8/233: symlink d2/de/d16/d36/d29/l52 0 2024-03-04T16:04:07.814 INFO:tasks.workunit.client.0.smithi116.stdout:8/234: read - d2/de/d16/f2c zero size 2024-03-04T16:04:07.834 INFO:tasks.workunit.client.0.smithi116.stdout:4/232: mknod d1/d3/dd/d24/d51/c54 0 2024-03-04T16:04:07.834 INFO:tasks.workunit.client.0.smithi116.stdout:0/219: symlink da/d16/d20/d42/d4e/l50 0 2024-03-04T16:04:07.835 INFO:tasks.workunit.client.0.smithi116.stdout:0/220: chown da/ff 1 1 2024-03-04T16:04:07.836 INFO:tasks.workunit.client.0.smithi116.stdout:1/179: dwrite d5/d9/d21/d24/f37 [0,4194304] 0 2024-03-04T16:04:07.838 INFO:tasks.workunit.client.0.smithi116.stdout:1/180: write d5/d9/f23 [1500473,80698] 0 2024-03-04T16:04:07.839 INFO:tasks.workunit.client.0.smithi116.stdout:9/188: getdents d2/d19 0 2024-03-04T16:04:07.839 INFO:tasks.workunit.client.0.smithi116.stdout:1/181: dread - d5/f38 zero size 2024-03-04T16:04:07.840 INFO:tasks.workunit.client.0.smithi116.stdout:8/235: creat d2/d13/d1e/d42/f53 x:0 0 0 2024-03-04T16:04:07.842 INFO:tasks.workunit.client.0.smithi116.stdout:8/236: write d2/de/f10 [1314507,66104] 0 2024-03-04T16:04:07.844 INFO:tasks.workunit.client.0.smithi116.stdout:8/237: chown d2/de/d16/l24 8969662 1 2024-03-04T16:04:07.844 INFO:tasks.workunit.client.0.smithi116.stdout:0/221: dwrite da/d16/d28/d39/f32 [0,4194304] 0 2024-03-04T16:04:07.846 INFO:tasks.workunit.client.0.smithi116.stdout:4/233: fsync d1/d3/de/f13 0 2024-03-04T16:04:07.848 INFO:tasks.workunit.client.0.smithi116.stdout:9/189: write d2/d19/f37 [979813,20088] 0 2024-03-04T16:04:07.849 INFO:tasks.workunit.client.0.smithi116.stdout:9/190: chown d2/d19/f37 1394 1 2024-03-04T16:04:07.849 INFO:tasks.workunit.client.0.smithi116.stdout:9/191: fsync d2/d33/f3a 0 2024-03-04T16:04:07.850 INFO:tasks.workunit.client.0.smithi116.stdout:8/238: fdatasync d2/f7 0 2024-03-04T16:04:07.851 INFO:tasks.workunit.client.0.smithi116.stdout:0/222: symlink da/d16/d19/l51 0 2024-03-04T16:04:07.852 INFO:tasks.workunit.client.0.smithi116.stdout:8/239: write d2/de/d3e/f4e [122541,123910] 0 2024-03-04T16:04:07.853 INFO:tasks.workunit.client.0.smithi116.stdout:4/234: symlink d1/d3/dd/d24/d51/l55 0 2024-03-04T16:04:07.857 INFO:tasks.workunit.client.0.smithi116.stdout:1/182: link d5/f3b d5/d9/d21/d27/f3c 0 2024-03-04T16:04:07.863 INFO:tasks.workunit.client.0.smithi116.stdout:4/235: rename d1/f2 to d1/d3/d15/d20/f56 0 2024-03-04T16:04:07.863 INFO:tasks.workunit.client.0.smithi116.stdout:1/183: mknod d5/d9/d21/d27/d32/c3d 0 2024-03-04T16:04:07.864 INFO:tasks.workunit.client.0.smithi116.stdout:1/184: write d5/f17 [1314720,60611] 0 2024-03-04T16:04:07.864 INFO:tasks.workunit.client.0.smithi116.stdout:1/185: readlink d5/l33 0 2024-03-04T16:04:07.865 INFO:tasks.workunit.client.0.smithi116.stdout:1/186: stat d5/d9/d21 0 2024-03-04T16:04:07.866 INFO:tasks.workunit.client.0.smithi116.stdout:1/187: write d5/d9/fb [3600799,35544] 0 2024-03-04T16:04:07.868 INFO:tasks.workunit.client.0.smithi116.stdout:9/192: getdents d2/d26 0 2024-03-04T16:04:07.872 INFO:tasks.workunit.client.0.smithi116.stdout:4/236: dwrite d1/d3/de/f10 [0,4194304] 0 2024-03-04T16:04:07.874 INFO:tasks.workunit.client.0.smithi116.stdout:8/240: getdents d2/de 0 2024-03-04T16:04:07.876 INFO:tasks.workunit.client.0.smithi116.stdout:9/193: mkdir d2/d19/d27/d3b 0 2024-03-04T16:04:07.876 INFO:tasks.workunit.client.0.smithi116.stdout:9/194: readlink d2/l14 0 2024-03-04T16:04:07.878 INFO:tasks.workunit.client.0.smithi116.stdout:9/195: read d2/d19/d27/d1f/f25 [510624,23709] 0 2024-03-04T16:04:07.878 INFO:tasks.workunit.client.0.smithi116.stdout:1/188: link d5/d9/d21/d24/f3a d5/d9/d21/d27/d32/f3e 0 2024-03-04T16:04:07.878 INFO:tasks.workunit.client.0.smithi116.stdout:0/223: write da/d16/d19/f1d [792037,125201] 0 2024-03-04T16:04:07.880 INFO:tasks.workunit.client.0.smithi116.stdout:4/237: creat d1/d3/dd/d24/d51/f57 x:0 0 0 2024-03-04T16:04:07.883 INFO:tasks.workunit.client.0.smithi116.stdout:1/189: fdatasync d5/f1d 0 2024-03-04T16:04:07.885 INFO:tasks.workunit.client.0.smithi116.stdout:9/196: truncate d2/f15 5132285 0 2024-03-04T16:04:07.886 INFO:tasks.workunit.client.0.smithi116.stdout:0/224: creat da/d16/d28/d40/d48/f52 x:0 0 0 2024-03-04T16:04:07.888 INFO:tasks.workunit.client.0.smithi116.stdout:0/225: read da/d16/d28/f3c [2134259,1078] 0 2024-03-04T16:04:07.888 INFO:tasks.workunit.client.0.smithi116.stdout:4/238: rename d1/d3/d15/d20/f35 to d1/f58 0 2024-03-04T16:04:07.890 INFO:tasks.workunit.client.0.smithi116.stdout:4/239: truncate d1/d3/f44 679104 0 2024-03-04T16:04:07.891 INFO:tasks.workunit.client.0.smithi116.stdout:9/197: dwrite d2/f35 [0,4194304] 0 2024-03-04T16:04:07.896 INFO:tasks.workunit.client.0.smithi116.stdout:1/190: write d5/d9/d21/f2c [1006549,127547] 0 2024-03-04T16:04:07.896 INFO:tasks.workunit.client.0.smithi116.stdout:9/198: dwrite d2/d8/ff [0,4194304] 0 2024-03-04T16:04:07.897 INFO:tasks.workunit.client.0.smithi116.stdout:1/191: chown d5/d11/c12 96 1 2024-03-04T16:04:07.904 INFO:tasks.workunit.client.0.smithi116.stdout:0/226: rmdir da/d16/d19/d27 39 2024-03-04T16:04:07.906 INFO:tasks.workunit.client.0.smithi116.stdout:8/241: link d2/de/d16/l24 d2/de/d16/d36/l54 0 2024-03-04T16:04:07.909 INFO:tasks.workunit.client.0.smithi116.stdout:4/240: unlink d1/d3/f46 0 2024-03-04T16:04:07.909 INFO:tasks.workunit.client.0.smithi116.stdout:4/241: fsync d1/d3/dd/d24/d51/f57 0 2024-03-04T16:04:07.911 INFO:tasks.workunit.client.0.smithi116.stdout:1/192: creat d5/d9/f3f x:0 0 0 2024-03-04T16:04:07.912 INFO:tasks.workunit.client.0.smithi116.stdout:1/193: chown d5/d9/d21/d27/d32/f3e 882609290 1 2024-03-04T16:04:07.913 INFO:tasks.workunit.client.0.smithi116.stdout:0/227: mkdir da/d16/d20/d53 0 2024-03-04T16:04:07.914 INFO:tasks.workunit.client.0.smithi116.stdout:0/228: rename da/d16/d20 to da/d16/d20/d54 22 2024-03-04T16:04:07.918 INFO:tasks.workunit.client.0.smithi116.stdout:4/242: unlink d1/d3/d15/l2d 0 2024-03-04T16:04:07.918 INFO:tasks.workunit.client.0.smithi116.stdout:4/243: read d1/d3/d15/d20/f1e [1875355,193] 0 2024-03-04T16:04:07.919 INFO:tasks.workunit.client.0.smithi116.stdout:1/194: symlink d5/d9/d21/d27/l40 0 2024-03-04T16:04:07.920 INFO:tasks.workunit.client.0.smithi116.stdout:1/195: dread - d5/f38 zero size 2024-03-04T16:04:07.932 INFO:tasks.workunit.client.0.smithi116.stdout:4/244: symlink d1/d3/dd/d45/l59 0 2024-03-04T16:04:07.936 INFO:tasks.workunit.client.0.smithi116.stdout:1/196: rename d5/d9/d21/d27/f2a to d5/d9/f41 0 2024-03-04T16:04:07.945 INFO:tasks.workunit.client.0.smithi116.stdout:0/229: rename da/f24 to da/d16/d19/d27/f55 0 2024-03-04T16:04:07.952 INFO:tasks.workunit.client.0.smithi116.stdout:4/245: creat d1/d3/de/f5a x:0 0 0 2024-03-04T16:04:07.952 INFO:tasks.workunit.client.0.smithi116.stdout:4/246: fsync d1/d3/de/d33/f38 0 2024-03-04T16:04:07.954 INFO:tasks.workunit.client.0.smithi116.stdout:0/230: unlink da/f13 0 2024-03-04T16:04:07.958 INFO:tasks.workunit.client.0.smithi116.stdout:4/247: dwrite d1/d3/de/f26 [0,4194304] 0 2024-03-04T16:04:07.959 INFO:tasks.workunit.client.0.smithi116.stdout:9/199: write d2/d19/d27/d1f/f2f [1502950,119790] 0 2024-03-04T16:04:07.960 INFO:tasks.workunit.client.0.smithi116.stdout:4/248: write d1/d3/d40/f4f [51230,102641] 0 2024-03-04T16:04:07.961 INFO:tasks.workunit.client.0.smithi116.stdout:9/200: chown d2/f32 33480 1 2024-03-04T16:04:07.962 INFO:tasks.workunit.client.0.smithi116.stdout:0/231: dwrite da/f1e [0,4194304] 0 2024-03-04T16:04:07.964 INFO:tasks.workunit.client.0.smithi116.stdout:0/232: fsync da/d16/d19/d27/f34 0 2024-03-04T16:04:07.966 INFO:tasks.workunit.client.0.smithi116.stdout:9/201: rmdir d2 39 2024-03-04T16:04:07.968 INFO:tasks.workunit.client.0.smithi116.stdout:8/242: truncate d2/de/d16/f18 2077406 0 2024-03-04T16:04:07.969 INFO:tasks.workunit.client.0.smithi116.stdout:4/249: symlink d1/d3/dd/l5b 0 2024-03-04T16:04:07.970 INFO:tasks.workunit.client.0.smithi116.stdout:4/250: rename d1 to d1/d3/dd/d24/d51/d5c 22 2024-03-04T16:04:07.979 INFO:tasks.workunit.client.0.smithi116.stdout:8/243: mkdir d2/d38/d55 0 2024-03-04T16:04:07.980 INFO:tasks.workunit.client.0.smithi116.stdout:0/233: dwrite da/d16/f4c [0,4194304] 0 2024-03-04T16:04:07.980 INFO:tasks.workunit.client.0.smithi116.stdout:5/262: sync 2024-03-04T16:04:07.981 INFO:tasks.workunit.client.0.smithi116.stdout:4/251: creat d1/d3/dd/f5d x:0 0 0 2024-03-04T16:04:07.985 INFO:tasks.workunit.client.0.smithi116.stdout:8/244: unlink d2/de/d16/d2b/d37/f4d 0 2024-03-04T16:04:07.992 INFO:tasks.workunit.client.0.smithi116.stdout:0/234: rename da/d16/d28/d39/f32 to da/d16/d19/d27/f56 0 2024-03-04T16:04:07.993 INFO:tasks.workunit.client.0.smithi116.stdout:4/252: creat d1/d3/d15/d20/f5e x:0 0 0 2024-03-04T16:04:07.997 INFO:tasks.workunit.client.0.smithi116.stdout:4/253: read d1/d3/d15/d20/f56 [145546,103115] 0 2024-03-04T16:04:07.998 INFO:tasks.workunit.client.0.smithi116.stdout:8/245: symlink d2/d13/d1e/l56 0 2024-03-04T16:04:07.999 INFO:tasks.workunit.client.0.smithi116.stdout:8/246: write d2/de/d16/d2b/d37/f49 [1047907,59124] 0 2024-03-04T16:04:07.999 INFO:tasks.workunit.client.0.smithi116.stdout:8/247: chown d2/d13/d3c 132977035 1 2024-03-04T16:04:08.005 INFO:tasks.workunit.client.0.smithi116.stdout:8/248: dwrite d2/de/d16/d36/d29/f39 [0,4194304] 0 2024-03-04T16:04:08.006 INFO:tasks.workunit.client.0.smithi116.stdout:8/249: fdatasync d2/de/f12 0 2024-03-04T16:04:08.007 INFO:tasks.workunit.client.0.smithi116.stdout:0/235: truncate da/d16/d28/d3d/d41/f46 1615835 0 2024-03-04T16:04:08.016 INFO:tasks.workunit.client.0.smithi116.stdout:0/236: mkdir da/d16/d28/d40/d48/d57 0 2024-03-04T16:04:08.018 INFO:tasks.workunit.client.0.smithi116.stdout:8/250: symlink d2/d13/d3c/d46/d4f/l57 0 2024-03-04T16:04:08.018 INFO:tasks.workunit.client.0.smithi116.stdout:8/251: dread - d2/de/d3e/f2a zero size 2024-03-04T16:04:08.022 INFO:tasks.workunit.client.0.smithi116.stdout:4/254: dread d1/f5 [0,4194304] 0 2024-03-04T16:04:08.024 INFO:tasks.workunit.client.0.smithi116.stdout:9/202: truncate d2/d19/d27/d1f/f24 3316481 0 2024-03-04T16:04:08.025 INFO:tasks.workunit.client.0.smithi116.stdout:9/203: write d2/f35 [3895465,107533] 0 2024-03-04T16:04:08.029 INFO:tasks.workunit.client.0.smithi116.stdout:5/263: truncate f4 3947918 0 2024-03-04T16:04:08.030 INFO:tasks.workunit.client.0.smithi116.stdout:8/252: link d2/de/c51 d2/d13/d1e/c58 0 2024-03-04T16:04:08.030 INFO:tasks.workunit.client.0.smithi116.stdout:8/253: write d2/de/f10 [1316440,54869] 0 2024-03-04T16:04:08.032 INFO:tasks.workunit.client.0.smithi116.stdout:9/204: mkdir d2/d8/d3c 0 2024-03-04T16:04:08.032 INFO:tasks.workunit.client.0.smithi116.stdout:9/205: read d2/f35 [2751756,8348] 0 2024-03-04T16:04:08.033 INFO:tasks.workunit.client.0.smithi116.stdout:4/255: mknod d1/d3/de/c5f 0 2024-03-04T16:04:08.038 INFO:tasks.workunit.client.0.smithi116.stdout:8/254: mknod d2/de/c59 0 2024-03-04T16:04:08.041 INFO:tasks.workunit.client.0.smithi116.stdout:0/237: rmdir da/d16/d28/d40/d48 39 2024-03-04T16:04:08.041 INFO:tasks.workunit.client.0.smithi116.stdout:6/282: sync 2024-03-04T16:04:08.041 INFO:tasks.workunit.client.0.smithi116.stdout:7/164: sync 2024-03-04T16:04:08.045 INFO:tasks.workunit.client.0.smithi116.stdout:4/256: unlink d1/d3/d15/f34 0 2024-03-04T16:04:08.049 INFO:tasks.workunit.client.0.smithi116.stdout:4/257: fsync d1/d3/de/f16 0 2024-03-04T16:04:08.050 INFO:tasks.workunit.client.0.smithi116.stdout:0/238: dwrite da/f12 [0,4194304] 0 2024-03-04T16:04:08.050 INFO:tasks.workunit.client.0.smithi116.stdout:8/255: mkdir d2/d13/d1e/d5a 0 2024-03-04T16:04:08.053 INFO:tasks.workunit.client.0.smithi116.stdout:6/283: dwrite d6/de/d15/f26 [0,4194304] 0 2024-03-04T16:04:08.054 INFO:tasks.workunit.client.0.smithi116.stdout:7/165: dwrite d3/d4/d8/d1a/f20 [0,4194304] 0 2024-03-04T16:04:08.054 INFO:tasks.workunit.client.0.smithi116.stdout:4/258: chown d1/d3/dd/d24/f29 57 1 2024-03-04T16:04:08.061 INFO:tasks.workunit.client.0.smithi116.stdout:9/206: creat d2/d19/d27/d3b/f3d x:0 0 0 2024-03-04T16:04:08.066 INFO:tasks.workunit.client.0.smithi116.stdout:9/207: dwrite d2/f32 [0,4194304] 0 2024-03-04T16:04:08.076 INFO:tasks.workunit.client.0.smithi116.stdout:8/256: creat d2/d13/d3c/d46/d4f/f5b x:0 0 0 2024-03-04T16:04:08.076 INFO:tasks.workunit.client.0.smithi116.stdout:8/257: stat d2/de/d16/d2b/c2d 0 2024-03-04T16:04:08.077 INFO:tasks.workunit.client.0.smithi116.stdout:7/166: stat d3/d4/d8/l1b 0 2024-03-04T16:04:08.079 INFO:tasks.workunit.client.0.smithi116.stdout:7/167: read d3/f18 [1386956,117529] 0 2024-03-04T16:04:08.080 INFO:tasks.workunit.client.0.smithi116.stdout:7/168: chown d3/f18 0 1 2024-03-04T16:04:08.081 INFO:tasks.workunit.client.0.smithi116.stdout:6/284: symlink d6/de/d25/d36/d44/l64 0 2024-03-04T16:04:08.081 INFO:tasks.workunit.client.0.smithi116.stdout:6/285: stat d6/d56 0 2024-03-04T16:04:08.082 INFO:tasks.workunit.client.0.smithi116.stdout:6/286: stat d6/de/d29/d53 0 2024-03-04T16:04:08.085 INFO:tasks.workunit.client.0.smithi116.stdout:4/259: rename d1/d3/dd/d24/d51/f57 to d1/d3/de/d47/f60 0 2024-03-04T16:04:08.091 INFO:tasks.workunit.client.0.smithi116.stdout:2/222: sync 2024-03-04T16:04:08.091 INFO:tasks.workunit.client.0.smithi116.stdout:1/197: sync 2024-03-04T16:04:08.091 INFO:tasks.workunit.client.0.smithi116.stdout:3/330: sync 2024-03-04T16:04:08.092 INFO:tasks.workunit.client.0.smithi116.stdout:8/258: unlink d2/d13/d1e/d42/f53 0 2024-03-04T16:04:08.093 INFO:tasks.workunit.client.0.smithi116.stdout:2/223: truncate d7/d15/d35/f3b 843067 0 2024-03-04T16:04:08.093 INFO:tasks.workunit.client.0.smithi116.stdout:8/259: chown d2/d13/d3c/d46/d4f/f5b 3339 1 2024-03-04T16:04:08.095 INFO:tasks.workunit.client.0.smithi116.stdout:8/260: dread - d2/d13/d3c/d46/d4f/f5b zero size 2024-03-04T16:04:08.096 INFO:tasks.workunit.client.0.smithi116.stdout:8/261: chown d2/de/d16/d2b/d37/f49 1213627715 1 2024-03-04T16:04:08.101 INFO:tasks.workunit.client.0.smithi116.stdout:4/260: creat d1/d3/f61 x:0 0 0 2024-03-04T16:04:08.102 INFO:tasks.workunit.client.0.smithi116.stdout:1/198: dread d5/d9/fb [0,4194304] 0 2024-03-04T16:04:08.105 INFO:tasks.workunit.client.0.smithi116.stdout:3/331: mknod dc/d23/d2c/c6f 0 2024-03-04T16:04:08.106 INFO:tasks.workunit.client.0.smithi116.stdout:3/332: write dc/df/f53 [1417363,50546] 0 2024-03-04T16:04:08.106 INFO:tasks.workunit.client.0.smithi116.stdout:0/239: link da/d16/d20/d42/d4e/c4f da/d16/d20/d42/c58 0 2024-03-04T16:04:08.109 INFO:tasks.workunit.client.0.smithi116.stdout:2/224: chown c2 6383037 1 2024-03-04T16:04:08.110 INFO:tasks.workunit.client.0.smithi116.stdout:7/169: mknod d3/d4/d34/c37 0 2024-03-04T16:04:08.111 INFO:tasks.workunit.client.0.smithi116.stdout:7/170: write d3/d4/d8/f1c [4306135,45697] 0 2024-03-04T16:04:08.116 INFO:tasks.workunit.client.0.smithi116.stdout:9/208: dwrite d2/f32 [4194304,4194304] 0 2024-03-04T16:04:08.117 INFO:tasks.workunit.client.0.smithi116.stdout:4/261: readlink d1/d3/d15/d20/l53 0 2024-03-04T16:04:08.119 INFO:tasks.workunit.client.0.smithi116.stdout:4/262: readlink d1/d3/d15/d20/d1c/l2a 0 2024-03-04T16:04:08.120 INFO:tasks.workunit.client.0.smithi116.stdout:1/199: mkdir d5/d11/d42 0 2024-03-04T16:04:08.121 INFO:tasks.workunit.client.0.smithi116.stdout:6/287: dwrite d6/de/d1e/f20 [0,4194304] 0 2024-03-04T16:04:08.126 INFO:tasks.workunit.client.0.smithi116.stdout:1/200: dwrite d5/d9/f18 [0,4194304] 0 2024-03-04T16:04:08.128 INFO:tasks.workunit.client.0.smithi116.stdout:1/201: write d5/f38 [147733,107969] 0 2024-03-04T16:04:08.128 INFO:tasks.workunit.client.0.smithi116.stdout:3/333: rmdir dc/d34 39 2024-03-04T16:04:08.128 INFO:tasks.workunit.client.0.smithi116.stdout:1/202: chown d5/d9/f18 12875 1 2024-03-04T16:04:08.132 INFO:tasks.workunit.client.0.smithi116.stdout:0/240: truncate da/d16/f17 1364349 0 2024-03-04T16:04:08.135 INFO:tasks.workunit.client.0.smithi116.stdout:2/225: creat d7/d13/f56 x:0 0 0 2024-03-04T16:04:08.137 INFO:tasks.workunit.client.0.smithi116.stdout:7/171: creat d3/d4/d11/f38 x:0 0 0 2024-03-04T16:04:08.140 INFO:tasks.workunit.client.0.smithi116.stdout:2/226: write d7/d15/f3f [936308,128885] 0 2024-03-04T16:04:08.141 INFO:tasks.workunit.client.0.smithi116.stdout:3/334: dwrite dc/df/d14/f5d [0,4194304] 0 2024-03-04T16:04:08.141 INFO:tasks.workunit.client.0.smithi116.stdout:7/172: fdatasync d3/d4/d8/f25 0 2024-03-04T16:04:08.143 INFO:tasks.workunit.client.0.smithi116.stdout:1/203: dwrite d5/d11/f16 [0,4194304] 0 2024-03-04T16:04:08.149 INFO:tasks.workunit.client.0.smithi116.stdout:4/263: mkdir d1/d3/d15/d20/d1c/d62 0 2024-03-04T16:04:08.149 INFO:tasks.workunit.client.0.smithi116.stdout:3/335: write dc/d23/f6e [609293,22044] 0 2024-03-04T16:04:08.149 INFO:tasks.workunit.client.0.smithi116.stdout:5/264: truncate f4 4181262 0 2024-03-04T16:04:08.149 INFO:tasks.workunit.client.0.smithi116.stdout:4/264: fdatasync d1/f2c 0 2024-03-04T16:04:08.149 INFO:tasks.workunit.client.0.smithi116.stdout:6/288: rmdir d6/de/d25/d36 39 2024-03-04T16:04:08.153 INFO:tasks.workunit.client.0.smithi116.stdout:4/265: chown d1/d3/dd/d45/d50 10409 1 2024-03-04T16:04:08.153 INFO:tasks.workunit.client.0.smithi116.stdout:7/173: rename d3/d4 to d3/d4/d11/d39 22 2024-03-04T16:04:08.153 INFO:tasks.workunit.client.0.smithi116.stdout:0/241: write da/d16/d28/d40/d48/f52 [970032,51] 0 2024-03-04T16:04:08.157 INFO:tasks.workunit.client.0.smithi116.stdout:9/209: mkdir d2/d8/d3c/d3e 0 2024-03-04T16:04:08.159 INFO:tasks.workunit.client.0.smithi116.stdout:7/174: dwrite d3/d4/f16 [0,4194304] 0 2024-03-04T16:04:08.164 INFO:tasks.workunit.client.0.smithi116.stdout:5/265: mkdir dd/d14/d2d/d59 0 2024-03-04T16:04:08.164 INFO:tasks.workunit.client.0.smithi116.stdout:9/210: dwrite d2/d19/d27/d1f/f2f [0,4194304] 0 2024-03-04T16:04:08.202 INFO:tasks.workunit.client.0.smithi116.stdout:4/266: mknod d1/d3/de/d33/c63 0 2024-03-04T16:04:08.203 INFO:tasks.workunit.client.0.smithi116.stdout:0/242: unlink da/f12 0 2024-03-04T16:04:08.204 INFO:tasks.workunit.client.0.smithi116.stdout:9/211: write d2/f32 [3355683,119063] 0 2024-03-04T16:04:08.206 INFO:tasks.workunit.client.0.smithi116.stdout:6/289: dwrite d6/f24 [0,4194304] 0 2024-03-04T16:04:08.206 INFO:tasks.workunit.client.0.smithi116.stdout:1/204: creat d5/d9/d21/d27/d32/d35/f43 x:0 0 0 2024-03-04T16:04:08.206 INFO:tasks.workunit.client.0.smithi116.stdout:0/243: stat da/d16/d28/l38 0 2024-03-04T16:04:08.210 INFO:tasks.workunit.client.0.smithi116.stdout:8/262: dwrite d2/de/d16/f18 [0,4194304] 0 2024-03-04T16:04:08.212 INFO:tasks.workunit.client.0.smithi116.stdout:5/266: rename dd/d1f/c2f to dd/d14/d2d/c5a 0 2024-03-04T16:04:08.212 INFO:tasks.workunit.client.0.smithi116.stdout:0/244: fdatasync da/d16/d28/d40/d48/f52 0 2024-03-04T16:04:08.212 INFO:tasks.workunit.client.0.smithi116.stdout:3/336: link dc/d23/d5a/f65 dc/f70 0 2024-03-04T16:04:08.215 INFO:tasks.workunit.client.0.smithi116.stdout:5/267: stat dd/d14/d2d/d31/f3c 0 2024-03-04T16:04:08.216 INFO:tasks.workunit.client.0.smithi116.stdout:3/337: readlink dc/d23/d2c/l54 0 2024-03-04T16:04:08.218 INFO:tasks.workunit.client.0.smithi116.stdout:9/212: fdatasync d2/d19/d27/d1f/f25 0 2024-03-04T16:04:08.218 INFO:tasks.workunit.client.0.smithi116.stdout:5/268: stat dd/d14/d2d/l33 0 2024-03-04T16:04:08.221 INFO:tasks.workunit.client.0.smithi116.stdout:4/267: dwrite d1/d3/de/d33/f42 [0,4194304] 0 2024-03-04T16:04:08.222 INFO:tasks.workunit.client.0.smithi116.stdout:5/269: fdatasync dd/d14/d2d/f3b 0 2024-03-04T16:04:08.222 INFO:tasks.workunit.client.0.smithi116.stdout:2/227: dwrite d7/d29/f4a [0,4194304] 0 2024-03-04T16:04:08.224 INFO:tasks.workunit.client.0.smithi116.stdout:9/213: truncate d2/d26/f39 119473 0 2024-03-04T16:04:08.224 INFO:tasks.workunit.client.0.smithi116.stdout:8/263: mknod d2/d38/d41/c5c 0 2024-03-04T16:04:08.226 INFO:tasks.workunit.client.0.smithi116.stdout:6/290: rename d6/d56/f5e to d6/de/d25/d4a/d63/f65 0 2024-03-04T16:04:08.227 INFO:tasks.workunit.client.0.smithi116.stdout:4/268: chown d1/d3/d15/d20/d1c/l2a 1968694905 1 2024-03-04T16:04:08.227 INFO:tasks.workunit.client.0.smithi116.stdout:2/228: write d7/d2f/f44 [452532,49207] 0 2024-03-04T16:04:08.228 INFO:tasks.workunit.client.0.smithi116.stdout:6/291: chown d6/f45 0 1 2024-03-04T16:04:08.231 INFO:tasks.workunit.client.0.smithi116.stdout:0/245: creat da/d16/d28/d3d/d41/f59 x:0 0 0 2024-03-04T16:04:08.233 INFO:tasks.workunit.client.0.smithi116.stdout:3/338: dwrite dc/df/d14/f5d [0,4194304] 0 2024-03-04T16:04:08.239 INFO:tasks.workunit.client.0.smithi116.stdout:2/229: dwrite d7/d29/d47/f51 [0,4194304] 0 2024-03-04T16:04:08.240 INFO:tasks.workunit.client.0.smithi116.stdout:2/230: stat d7/d13/c2a 0 2024-03-04T16:04:08.241 INFO:tasks.workunit.client.0.smithi116.stdout:2/231: chown d7/d15/f20 1 1 2024-03-04T16:04:08.242 INFO:tasks.workunit.client.0.smithi116.stdout:2/232: write d7/f4e [518836,54841] 0 2024-03-04T16:04:08.246 INFO:tasks.workunit.client.0.smithi116.stdout:8/264: creat d2/d13/d1e/d42/f5d x:0 0 0 2024-03-04T16:04:08.252 INFO:tasks.workunit.client.0.smithi116.stdout:8/265: read d2/de/fc [1758107,127361] 0 2024-03-04T16:04:08.252 INFO:tasks.workunit.client.0.smithi116.stdout:4/269: rename d1/d3/l2f to d1/d3/d15/d20/l64 0 2024-03-04T16:04:08.253 INFO:tasks.workunit.client.0.smithi116.stdout:6/292: creat d6/de/d15/f66 x:0 0 0 2024-03-04T16:04:08.254 INFO:tasks.workunit.client.0.smithi116.stdout:6/293: write d6/de/f4f [729003,29068] 0 2024-03-04T16:04:08.254 INFO:tasks.workunit.client.0.smithi116.stdout:6/294: stat d6/de/d29/d3a 0 2024-03-04T16:04:08.255 INFO:tasks.workunit.client.0.smithi116.stdout:0/246: unlink da/d16/d19/l51 0 2024-03-04T16:04:08.260 INFO:tasks.workunit.client.0.smithi116.stdout:5/270: rmdir dd/d14 39 2024-03-04T16:04:08.264 INFO:tasks.workunit.client.0.smithi116.stdout:4/270: chown d1/d3/de/c2e 191 1 2024-03-04T16:04:08.265 INFO:tasks.workunit.client.0.smithi116.stdout:4/271: readlink d1/d3/dd/l17 0 2024-03-04T16:04:08.266 INFO:tasks.workunit.client.0.smithi116.stdout:6/295: symlink d6/d56/d58/l67 0 2024-03-04T16:04:08.271 INFO:tasks.workunit.client.0.smithi116.stdout:4/272: dread d1/d3/de/f1b [0,4194304] 0 2024-03-04T16:04:08.275 INFO:tasks.workunit.client.0.smithi116.stdout:2/233: rename d7/cd to d7/d13/d3e/c57 0 2024-03-04T16:04:08.276 INFO:tasks.workunit.client.0.smithi116.stdout:1/205: truncate d5/f17 146451 0 2024-03-04T16:04:08.277 INFO:tasks.workunit.client.0.smithi116.stdout:8/266: mkdir d2/d13/d5e 0 2024-03-04T16:04:08.282 INFO:tasks.workunit.client.0.smithi116.stdout:9/214: getdents d2/d19/d27/d1f 0 2024-03-04T16:04:08.287 INFO:tasks.workunit.client.0.smithi116.stdout:2/234: rename d7/d13/c31 to d7/d13/c58 0 2024-03-04T16:04:08.289 INFO:tasks.workunit.client.0.smithi116.stdout:1/206: readlink d5/d11/l31 0 2024-03-04T16:04:08.290 INFO:tasks.workunit.client.0.smithi116.stdout:8/267: symlink d2/d13/d1e/d42/l5f 0 2024-03-04T16:04:08.291 INFO:tasks.workunit.client.0.smithi116.stdout:8/268: readlink d2/d13/d1e/l56 0 2024-03-04T16:04:08.292 INFO:tasks.workunit.client.0.smithi116.stdout:8/269: write d2/de/d16/d36/d29/f39 [1487247,124430] 0 2024-03-04T16:04:08.295 INFO:tasks.workunit.client.0.smithi116.stdout:9/215: mknod d2/d26/d28/c3f 0 2024-03-04T16:04:08.298 INFO:tasks.workunit.client.0.smithi116.stdout:4/273: getdents d1/d3/d40/d52 0 2024-03-04T16:04:08.299 INFO:tasks.workunit.client.0.smithi116.stdout:8/270: dwrite d2/de/d16/f18 [4194304,4194304] 0 2024-03-04T16:04:08.301 INFO:tasks.workunit.client.0.smithi116.stdout:5/271: dread dd/fe [0,4194304] 0 2024-03-04T16:04:08.302 INFO:tasks.workunit.client.0.smithi116.stdout:1/207: creat d5/d9/d21/d24/f44 x:0 0 0 2024-03-04T16:04:08.306 INFO:tasks.workunit.client.0.smithi116.stdout:1/208: stat d5/d9/d21/d27/d32 0 2024-03-04T16:04:08.306 INFO:tasks.workunit.client.0.smithi116.stdout:0/247: fsync da/d16/d28/d3d/d41/f46 0 2024-03-04T16:04:08.307 INFO:tasks.workunit.client.0.smithi116.stdout:3/339: link dc/c25 dc/d34/c71 0 2024-03-04T16:04:08.307 INFO:tasks.workunit.client.0.smithi116.stdout:0/248: chown da/d16/d20/d42 2198614 1 2024-03-04T16:04:08.308 INFO:tasks.workunit.client.0.smithi116.stdout:9/216: dwrite d2/d33/f3a [0,4194304] 0 2024-03-04T16:04:08.309 INFO:tasks.workunit.client.0.smithi116.stdout:0/249: write da/d16/d28/d40/d48/f52 [1854725,109977] 0 2024-03-04T16:04:08.314 INFO:tasks.workunit.client.0.smithi116.stdout:4/274: mkdir d1/d3/d15/d20/d65 0 2024-03-04T16:04:08.316 INFO:tasks.workunit.client.0.smithi116.stdout:0/250: dread da/d16/d20/f21 [0,4194304] 0 2024-03-04T16:04:08.317 INFO:tasks.workunit.client.0.smithi116.stdout:0/251: readlink da/d16/d19/l26 0 2024-03-04T16:04:08.318 INFO:tasks.workunit.client.0.smithi116.stdout:3/340: dwrite dc/df/d47/f6c [0,4194304] 0 2024-03-04T16:04:08.318 INFO:tasks.workunit.client.0.smithi116.stdout:1/209: creat d5/d11/f45 x:0 0 0 2024-03-04T16:04:08.326 INFO:tasks.workunit.client.0.smithi116.stdout:4/275: symlink d1/d3/d40/l66 0 2024-03-04T16:04:08.328 INFO:tasks.workunit.client.0.smithi116.stdout:3/341: creat dc/df/d14/f72 x:0 0 0 2024-03-04T16:04:08.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.327+0000 7fe2fdc13700 1 -- 172.21.15.116:0/818407900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f80ff430 msgr2=0x7fe2f8101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:08.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.327+0000 7fe2fdc13700 1 --2- 172.21.15.116:0/818407900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f80ff430 0x7fe2f8101860 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009910 tx=0x7fe2e0009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:08.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.327+0000 7fe2fdc13700 1 -- 172.21.15.116:0/818407900 shutdown_connections 2024-03-04T16:04:08.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.327+0000 7fe2fdc13700 1 --2- 172.21.15.116:0/818407900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2f8101da0 0x7fe2f8104230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:08.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.327+0000 7fe2fdc13700 1 --2- 172.21.15.116:0/818407900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f80ff430 0x7fe2f8101860 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:08.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.327+0000 7fe2fdc13700 1 -- 172.21.15.116:0/818407900 >> 172.21.15.116:0/818407900 conn(0x7fe2f80fb030 msgr2=0x7fe2f80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:08.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.328+0000 7fe2fdc13700 1 -- 172.21.15.116:0/818407900 shutdown_connections 2024-03-04T16:04:08.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.328+0000 7fe2fdc13700 1 -- 172.21.15.116:0/818407900 wait complete. 2024-03-04T16:04:08.329 INFO:tasks.workunit.client.0.smithi116.stdout:2/235: dwrite d7/d1b/f21 [0,4194304] 0 2024-03-04T16:04:08.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.329+0000 7fe2fdc13700 1 Processor -- start 2024-03-04T16:04:08.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.329+0000 7fe2fdc13700 1 -- start start 2024-03-04T16:04:08.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.329+0000 7fe2fdc13700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2f80ff430 0x7fe2f819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:08.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.329+0000 7fe2fdc13700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8101da0 0x7fe2f819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:08.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.329+0000 7fe2fdc13700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f819d410 con 0x7fe2f8101da0 2024-03-04T16:04:08.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.329+0000 7fe2fdc13700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f819d550 con 0x7fe2f80ff430 2024-03-04T16:04:08.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.330+0000 7fe2f6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8101da0 0x7fe2f819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:08.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.330+0000 7fe2f6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8101da0 0x7fe2f819ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50238/0 (socket says 172.21.15.116:50238) 2024-03-04T16:04:08.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.330+0000 7fe2f6ffd700 1 -- 172.21.15.116:0/1358027220 learned_addr learned my addr 172.21.15.116:0/1358027220 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:08.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.330+0000 7fe2f77fe700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2f80ff430 0x7fe2f819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:08.332 INFO:tasks.workunit.client.0.smithi116.stdout:0/252: dread da/d16/d19/d27/f34 [0,4194304] 0 2024-03-04T16:04:08.333 INFO:tasks.workunit.client.0.smithi116.stdout:1/210: mkdir d5/d9/d46 0 2024-03-04T16:04:08.333 INFO:tasks.workunit.client.0.smithi116.stdout:9/217: dread d2/f35 [0,4194304] 0 2024-03-04T16:04:08.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.330+0000 7fe2f6ffd700 1 -- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2f80ff430 msgr2=0x7fe2f819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:08.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.330+0000 7fe2f6ffd700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2f80ff430 0x7fe2f819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:08.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.330+0000 7fe2f6ffd700 1 -- 172.21.15.116:0/1358027220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e00095f0 con 0x7fe2f8101da0 2024-03-04T16:04:08.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.330+0000 7fe2f77fe700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2f80ff430 0x7fe2f819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:04:08.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.330+0000 7fe2f6ffd700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8101da0 0x7fe2f819ce80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800cfd0 tx=0x7fe2e800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:08.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.331+0000 7fe2f4ff9700 1 -- 172.21.15.116:0/1358027220 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2e800ad70 con 0x7fe2f8101da0 2024-03-04T16:04:08.334 INFO:tasks.workunit.client.0.smithi116.stdout:8/271: creat d2/f60 x:0 0 0 2024-03-04T16:04:08.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.331+0000 7fe2fdc13700 1 -- 172.21.15.116:0/1358027220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f81a2010 con 0x7fe2f8101da0 2024-03-04T16:04:08.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.331+0000 7fe2fdc13700 1 -- 172.21.15.116:0/1358027220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f81a2530 con 0x7fe2f8101da0 2024-03-04T16:04:08.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.331+0000 7fe2f4ff9700 1 -- 172.21.15.116:0/1358027220 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe2e8007bf0 con 0x7fe2f8101da0 2024-03-04T16:04:08.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.331+0000 7fe2f4ff9700 1 -- 172.21.15.116:0/1358027220 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2e800ad70 con 0x7fe2f8101da0 2024-03-04T16:04:08.335 INFO:tasks.workunit.client.0.smithi116.stdout:9/218: chown d2/l14 82441896 1 2024-03-04T16:04:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.332+0000 7fe2fdc13700 1 -- 172.21.15.116:0/1358027220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d8005320 con 0x7fe2f8101da0 2024-03-04T16:04:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.333+0000 7fe2f4ff9700 1 -- 172.21.15.116:0/1358027220 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) v1 ==== 90928+0+0 (secure 0 0 0) 0x7fe2e801b600 con 0x7fe2f8101da0 2024-03-04T16:04:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.333+0000 7fe2f4ff9700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fe2e406cee0 0x7fe2e406f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.334+0000 7fe2f4ff9700 1 -- 172.21.15.116:0/1358027220 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fe2e808c550 con 0x7fe2f8101da0 2024-03-04T16:04:08.336 INFO:tasks.workunit.client.0.smithi116.stdout:9/219: fdatasync d2/f32 0 2024-03-04T16:04:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.334+0000 7fe2f77fe700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fe2e406cee0 0x7fe2e406f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:08.337 INFO:tasks.workunit.client.0.smithi116.stdout:8/272: read d2/de/d16/f23 [555902,72400] 0 2024-03-04T16:04:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.337+0000 7fe2f77fe700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fe2e406cee0 0x7fe2e406f3a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00059d0 tx=0x7fe2e0005920 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:08.339 INFO:tasks.workunit.client.0.smithi116.stdout:8/273: read - d2/d13/d3c/d46/d4f/f5b zero size 2024-03-04T16:04:08.340 INFO:tasks.workunit.client.0.smithi116.stdout:8/274: stat d2/d13/d1e/d42/l5f 0 2024-03-04T16:04:08.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.341+0000 7fe2f4ff9700 1 -- 172.21.15.116:0/1358027220 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fe2e8059d00 con 0x7fe2f8101da0 2024-03-04T16:04:08.342 INFO:tasks.workunit.client.0.smithi116.stdout:1/211: creat d5/d9/d21/d27/d32/f47 x:0 0 0 2024-03-04T16:04:08.346 INFO:tasks.workunit.client.0.smithi116.stdout:3/342: rename dc/d23/f46 to dc/f73 0 2024-03-04T16:04:08.347 INFO:tasks.workunit.client.0.smithi116.stdout:0/253: dwrite da/d16/d20/f21 [0,4194304] 0 2024-03-04T16:04:08.347 INFO:tasks.workunit.client.0.smithi116.stdout:5/272: write dd/d45/f49 [1305650,47962] 0 2024-03-04T16:04:08.348 INFO:tasks.workunit.client.0.smithi116.stdout:0/254: chown da 6 1 2024-03-04T16:04:08.348 INFO:tasks.workunit.client.0.smithi116.stdout:1/212: truncate d5/d9/d21/d27/f3c 655232 0 2024-03-04T16:04:08.352 INFO:tasks.workunit.client.0.smithi116.stdout:8/275: dread d2/de/d16/f30 [0,4194304] 0 2024-03-04T16:04:08.352 INFO:tasks.workunit.client.0.smithi116.stdout:9/220: rename d2/d8/d3c/d3e to d2/d33/d40 0 2024-03-04T16:04:08.355 INFO:tasks.workunit.client.0.smithi116.stdout:3/343: rmdir dc/d34/d44 39 2024-03-04T16:04:08.362 INFO:tasks.workunit.client.0.smithi116.stdout:2/236: dread d7/d13/f1a [0,4194304] 0 2024-03-04T16:04:08.363 INFO:tasks.workunit.client.0.smithi116.stdout:0/255: mknod da/d16/d20/d42/c5a 0 2024-03-04T16:04:08.363 INFO:tasks.workunit.client.0.smithi116.stdout:8/276: rmdir d2/d38/d41 39 2024-03-04T16:04:08.365 INFO:tasks.workunit.client.0.smithi116.stdout:9/221: symlink d2/d19/d27/d3b/l41 0 2024-03-04T16:04:08.366 INFO:tasks.workunit.client.0.smithi116.stdout:3/344: rmdir dc/d23/d2c 39 2024-03-04T16:04:08.368 INFO:tasks.workunit.client.0.smithi116.stdout:5/273: symlink dd/d1f/l5b 0 2024-03-04T16:04:08.368 INFO:tasks.workunit.client.0.smithi116.stdout:2/237: dread d7/d29/d47/f49 [0,4194304] 0 2024-03-04T16:04:08.369 INFO:tasks.workunit.client.0.smithi116.stdout:4/276: dwrite d1/f32 [0,4194304] 0 2024-03-04T16:04:08.372 INFO:tasks.workunit.client.0.smithi116.stdout:4/277: chown d1/d3/de/d33/c63 6463 1 2024-03-04T16:04:08.374 INFO:tasks.workunit.client.0.smithi116.stdout:9/222: creat d2/d8/f42 x:0 0 0 2024-03-04T16:04:08.374 INFO:tasks.workunit.client.0.smithi116.stdout:4/278: stat d1/f2c 0 2024-03-04T16:04:08.375 INFO:tasks.workunit.client.0.smithi116.stdout:8/277: dwrite d2/d13/d1e/f45 [0,4194304] 0 2024-03-04T16:04:08.377 INFO:tasks.workunit.client.0.smithi116.stdout:8/278: truncate d2/f32 46049 0 2024-03-04T16:04:08.379 INFO:tasks.workunit.client.0.smithi116.stdout:2/238: creat d7/d15/f59 x:0 0 0 2024-03-04T16:04:08.384 INFO:tasks.workunit.client.0.smithi116.stdout:1/213: write d5/d9/d21/d24/f3a [695382,94009] 0 2024-03-04T16:04:08.384 INFO:tasks.workunit.client.0.smithi116.stdout:1/214: write d5/f1c [1016157,66477] 0 2024-03-04T16:04:08.386 INFO:tasks.workunit.client.0.smithi116.stdout:9/223: creat d2/d19/d27/d1f/f43 x:0 0 0 2024-03-04T16:04:08.389 INFO:tasks.workunit.client.0.smithi116.stdout:4/279: mknod d1/d3/de/d47/c67 0 2024-03-04T16:04:08.391 INFO:tasks.workunit.client.0.smithi116.stdout:8/279: mknod d2/d13/d1e/c61 0 2024-03-04T16:04:08.394 INFO:tasks.workunit.client.0.smithi116.stdout:2/239: creat d7/d25/d41/f5a x:0 0 0 2024-03-04T16:04:08.398 INFO:tasks.workunit.client.0.smithi116.stdout:9/224: rmdir d2/d19/d27 39 2024-03-04T16:04:08.403 INFO:tasks.workunit.client.0.smithi116.stdout:2/240: rename d7/d2f/f46 to d7/d29/d47/f5b 0 2024-03-04T16:04:08.406 INFO:tasks.workunit.client.0.smithi116.stdout:3/345: getdents dc/d23/d5a 0 2024-03-04T16:04:08.407 INFO:tasks.workunit.client.0.smithi116.stdout:5/274: dwrite f4 [4194304,4194304] 0 2024-03-04T16:04:08.410 INFO:tasks.workunit.client.0.smithi116.stdout:8/280: mknod d2/d4a/c62 0 2024-03-04T16:04:08.413 INFO:tasks.workunit.client.0.smithi116.stdout:2/241: creat d7/d13/d3e/f5c x:0 0 0 2024-03-04T16:04:08.416 INFO:tasks.workunit.client.0.smithi116.stdout:4/280: truncate d1/d3/dd/d24/f29 1450217 0 2024-03-04T16:04:08.425 INFO:tasks.workunit.client.0.smithi116.stdout:2/242: dwrite d7/d13/f45 [0,4194304] 0 2024-03-04T16:04:08.425 INFO:tasks.workunit.client.0.smithi116.stdout:1/215: rename d5/f17 to d5/f48 0 2024-03-04T16:04:08.430 INFO:tasks.workunit.client.0.smithi116.stdout:8/281: truncate d2/de/f12 3909408 0 2024-03-04T16:04:08.430 INFO:tasks.workunit.client.0.smithi116.stdout:4/281: dwrite d1/d3/f44 [0,4194304] 0 2024-03-04T16:04:08.431 INFO:tasks.workunit.client.0.smithi116.stdout:3/346: dread dc/df/d14/f2b [0,4194304] 0 2024-03-04T16:04:08.433 INFO:tasks.workunit.client.0.smithi116.stdout:4/282: fsync d1/d3/dd/f31 0 2024-03-04T16:04:08.433 INFO:tasks.workunit.client.0.smithi116.stdout:8/282: write d2/de/d16/f2c [84633,51482] 0 2024-03-04T16:04:08.433 INFO:tasks.workunit.client.0.smithi116.stdout:2/243: mknod d7/d15/d35/c5d 0 2024-03-04T16:04:08.435 INFO:tasks.workunit.client.0.smithi116.stdout:4/283: readlink d1/d3/d15/d20/d1c/l49 0 2024-03-04T16:04:08.444 INFO:tasks.workunit.client.0.smithi116.stdout:8/283: dwrite d2/d13/d1e/f45 [0,4194304] 0 2024-03-04T16:04:08.445 INFO:tasks.workunit.client.0.smithi116.stdout:4/284: dwrite d1/d3/de/f16 [0,4194304] 0 2024-03-04T16:04:08.446 INFO:tasks.workunit.client.0.smithi116.stdout:2/244: mkdir d7/d15/d5e 0 2024-03-04T16:04:08.447 INFO:tasks.workunit.client.0.smithi116.stdout:4/285: stat d1/d3/dd/d45/f4c 0 2024-03-04T16:04:08.448 INFO:tasks.workunit.client.0.smithi116.stdout:4/286: write d1/f32 [935944,106886] 0 2024-03-04T16:04:08.452 INFO:tasks.workunit.client.0.smithi116.stdout:9/225: dread d2/d19/f2a [0,4194304] 0 2024-03-04T16:04:08.455 INFO:tasks.workunit.client.0.smithi116.stdout:4/287: creat d1/d3/d15/d20/f68 x:0 0 0 2024-03-04T16:04:08.457 INFO:tasks.workunit.client.0.smithi116.stdout:8/284: symlink d2/d13/d1e/d5a/l63 0 2024-03-04T16:04:08.460 INFO:tasks.workunit.client.0.smithi116.stdout:5/275: truncate dd/f22 4156608 0 2024-03-04T16:04:08.460 INFO:tasks.workunit.client.0.smithi116.stdout:8/285: dread d2/de/d16/f2c [0,4194304] 0 2024-03-04T16:04:08.461 INFO:tasks.workunit.client.0.smithi116.stdout:2/245: truncate f4 2239161 0 2024-03-04T16:04:08.461 INFO:tasks.workunit.client.0.smithi116.stdout:2/246: dread - d7/d25/d41/f5a zero size 2024-03-04T16:04:08.462 INFO:tasks.workunit.client.0.smithi116.stdout:2/247: fsync d7/d13/f52 0 2024-03-04T16:04:08.463 INFO:tasks.workunit.client.0.smithi116.stdout:2/248: fdatasync d7/d15/f20 0 2024-03-04T16:04:08.463 INFO:tasks.workunit.client.0.smithi116.stdout:9/226: rename d2/f32 to d2/d26/f44 0 2024-03-04T16:04:08.464 INFO:tasks.workunit.client.0.smithi116.stdout:1/216: truncate f2 1202296 0 2024-03-04T16:04:08.465 INFO:tasks.workunit.client.0.smithi116.stdout:4/288: creat d1/d3/dd/d24/f69 x:0 0 0 2024-03-04T16:04:08.469 INFO:tasks.workunit.client.0.smithi116.stdout:2/249: mkdir d7/d13/d5f 0 2024-03-04T16:04:08.470 INFO:tasks.workunit.client.0.smithi116.stdout:3/347: truncate dc/f33 1395640 0 2024-03-04T16:04:08.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:08 smithi116 ceph-mon[29120]: pgmap v20: 65 pgs: 65 active+clean; 357 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 743 KiB/s rd, 27 MiB/s wr, 289 op/s 2024-03-04T16:04:08.480 INFO:tasks.workunit.client.0.smithi116.stdout:7/175: sync 2024-03-04T16:04:08.495 INFO:tasks.workunit.client.0.smithi116.stdout:1/217: mkdir d5/d9/d21/d27/d32/d49 0 2024-03-04T16:04:08.496 INFO:tasks.workunit.client.0.smithi116.stdout:4/289: creat d1/d3/d15/d20/d1c/f6a x:0 0 0 2024-03-04T16:04:08.496 INFO:tasks.workunit.client.0.smithi116.stdout:9/227: write d2/d19/d27/d1f/f43 [65670,60734] 0 2024-03-04T16:04:08.497 INFO:tasks.workunit.client.0.smithi116.stdout:2/250: symlink d7/d15/d5e/l60 0 2024-03-04T16:04:08.499 INFO:tasks.workunit.client.0.smithi116.stdout:2/251: write d7/d29/f2d [4338093,2247] 0 2024-03-04T16:04:08.501 INFO:tasks.workunit.client.0.smithi116.stdout:9/228: readlink d2/d19/d27/d1f/l34 0 2024-03-04T16:04:08.502 INFO:tasks.workunit.client.0.smithi116.stdout:1/218: mkdir d5/d9/d21/d27/d32/d49/d4a 0 2024-03-04T16:04:08.505 INFO:tasks.workunit.client.0.smithi116.stdout:2/252: unlink d7/d2f/l4c 0 2024-03-04T16:04:08.507 INFO:tasks.workunit.client.0.smithi116.stdout:3/348: dwrite dc/d23/d2c/f5e [0,4194304] 0 2024-03-04T16:04:08.507 INFO:tasks.workunit.client.0.smithi116.stdout:2/253: chown d7/d29/f4b 45990973 1 2024-03-04T16:04:08.507 INFO:tasks.workunit.client.0.smithi116.stdout:4/290: dwrite d1/d3/d15/d20/f68 [0,4194304] 0 2024-03-04T16:04:08.509 INFO:tasks.workunit.client.0.smithi116.stdout:2/254: dread - d7/d1b/f39 zero size 2024-03-04T16:04:08.509 INFO:tasks.workunit.client.0.smithi116.stdout:9/229: symlink d2/d33/d40/l45 0 2024-03-04T16:04:08.514 INFO:tasks.workunit.client.0.smithi116.stdout:1/219: creat d5/d9/d21/d27/f4b x:0 0 0 2024-03-04T16:04:08.521 INFO:tasks.workunit.client.0.smithi116.stdout:2/255: chown d7/l1f 53059 1 2024-03-04T16:04:08.524 INFO:tasks.workunit.client.0.smithi116.stdout:1/220: mkdir d5/d9/d21/d27/d32/d49/d4c 0 2024-03-04T16:04:08.527 INFO:tasks.workunit.client.0.smithi116.stdout:9/230: dread d2/d19/d27/d1f/f24 [0,4194304] 0 2024-03-04T16:04:08.529 INFO:tasks.workunit.client.0.smithi116.stdout:2/256: dwrite d7/f4e [8388608,4194304] 0 2024-03-04T16:04:08.536 INFO:tasks.workunit.client.0.smithi116.stdout:9/231: creat d2/d33/f46 x:0 0 0 2024-03-04T16:04:08.536 INFO:tasks.workunit.client.0.smithi116.stdout:9/232: fsync d2/d33/f3a 0 2024-03-04T16:04:08.539 INFO:tasks.workunit.client.0.smithi116.stdout:2/257: symlink d7/d15/d5e/l61 0 2024-03-04T16:04:08.542 INFO:tasks.workunit.client.0.smithi116.stdout:4/291: getdents d1/d3 0 2024-03-04T16:04:08.545 INFO:tasks.workunit.client.0.smithi116.stdout:9/233: creat d2/d26/f47 x:0 0 0 2024-03-04T16:04:08.549 INFO:tasks.workunit.client.0.smithi116.stdout:7/176: truncate d3/f6 64156 0 2024-03-04T16:04:08.549 INFO:tasks.workunit.client.0.smithi116.stdout:1/221: creat d5/f4d x:0 0 0 2024-03-04T16:04:08.550 INFO:tasks.workunit.client.0.smithi116.stdout:7/177: write d3/d4/d8/d1a/f28 [10482,76327] 0 2024-03-04T16:04:08.551 INFO:tasks.workunit.client.0.smithi116.stdout:4/292: write d1/f58 [683177,88418] 0 2024-03-04T16:04:08.552 INFO:tasks.workunit.client.0.smithi116.stdout:4/293: fdatasync d1/d3/de/f13 0 2024-03-04T16:04:08.553 INFO:tasks.workunit.client.0.smithi116.stdout:4/294: chown d1/d3/de/f10 6901205 1 2024-03-04T16:04:08.554 INFO:tasks.workunit.client.0.smithi116.stdout:2/258: unlink d7/d13/d3e/c57 0 2024-03-04T16:04:08.556 INFO:tasks.workunit.client.0.smithi116.stdout:1/222: creat d5/d11/f4e x:0 0 0 2024-03-04T16:04:08.561 INFO:tasks.workunit.client.0.smithi116.stdout:2/259: dwrite d7/d13/f56 [0,4194304] 0 2024-03-04T16:04:08.564 INFO:tasks.workunit.client.0.smithi116.stdout:3/349: write dc/df/d14/f1a [817902,96069] 0 2024-03-04T16:04:08.566 INFO:tasks.workunit.client.0.smithi116.stdout:9/234: rename d2/fc to d2/f48 0 2024-03-04T16:04:08.570 INFO:tasks.workunit.client.0.smithi116.stdout:4/295: getdents d1/d3/d40/d52 0 2024-03-04T16:04:08.572 INFO:tasks.workunit.client.0.smithi116.stdout:2/260: dread d7/d29/d47/f49 [0,4194304] 0 2024-03-04T16:04:08.573 INFO:tasks.workunit.client.0.smithi116.stdout:2/261: chown c2 327231614 1 2024-03-04T16:04:08.582 INFO:tasks.workunit.client.0.smithi116.stdout:4/296: creat d1/d3/dd/d24/f6b x:0 0 0 2024-03-04T16:04:08.588 INFO:tasks.workunit.client.0.smithi116.stdout:4/297: dwrite d1/d3/dd/d45/f4c [0,4194304] 0 2024-03-04T16:04:08.588 INFO:tasks.workunit.client.0.smithi116.stdout:2/262: truncate d7/d15/f20 2074906 0 2024-03-04T16:04:08.592 INFO:tasks.workunit.client.0.smithi116.stdout:9/235: rename d2/d8 to d2/d49 0 2024-03-04T16:04:08.594 INFO:tasks.workunit.client.0.smithi116.stdout:7/178: dwrite d3/fd [0,4194304] 0 2024-03-04T16:04:08.599 INFO:tasks.workunit.client.0.smithi116.stdout:4/298: rename d1/d3/de/d33 to d1/d6c 0 2024-03-04T16:04:08.599 INFO:tasks.workunit.client.0.smithi116.stdout:1/223: write f4 [1917846,94675] 0 2024-03-04T16:04:08.600 INFO:tasks.workunit.client.0.smithi116.stdout:4/299: chown d1/d3/d15/d20/d1c 10853124 1 2024-03-04T16:04:08.601 INFO:tasks.workunit.client.0.smithi116.stdout:4/300: write d1/f58 [54160,7980] 0 2024-03-04T16:04:08.601 INFO:tasks.workunit.client.0.smithi116.stdout:1/224: stat d5/d9/d21/d24/f37 0 2024-03-04T16:04:08.601 INFO:tasks.workunit.client.0.smithi116.stdout:3/350: write dc/d23/d5a/f65 [45208,18937] 0 2024-03-04T16:04:08.604 INFO:tasks.workunit.client.0.smithi116.stdout:4/301: truncate d1/d3/de/f30 768466 0 2024-03-04T16:04:08.605 INFO:tasks.workunit.client.0.smithi116.stdout:4/302: dread - d1/d3/dd/d45/f4b zero size 2024-03-04T16:04:08.605 INFO:tasks.workunit.client.0.smithi116.stdout:2/263: getdents d7/d25 0 2024-03-04T16:04:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:08 smithi152 ceph-mon[36876]: pgmap v20: 65 pgs: 65 active+clean; 357 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 743 KiB/s rd, 27 MiB/s wr, 289 op/s 2024-03-04T16:04:08.611 INFO:tasks.workunit.client.0.smithi116.stdout:9/236: truncate d2/d33/f3a 1499334 0 2024-03-04T16:04:08.612 INFO:tasks.workunit.client.0.smithi116.stdout:1/225: rmdir d5/d9/d21/d27/d32/d49 39 2024-03-04T16:04:08.614 INFO:tasks.workunit.client.0.smithi116.stdout:3/351: creat dc/df/f74 x:0 0 0 2024-03-04T16:04:08.617 INFO:tasks.workunit.client.0.smithi116.stdout:9/237: dwrite d2/d26/f47 [0,4194304] 0 2024-03-04T16:04:08.618 INFO:tasks.workunit.client.0.smithi116.stdout:4/303: mknod d1/d6c/c6d 0 2024-03-04T16:04:08.619 INFO:tasks.workunit.client.0.smithi116.stdout:2/264: creat d7/d25/f62 x:0 0 0 2024-03-04T16:04:08.620 INFO:tasks.workunit.client.0.smithi116.stdout:4/304: write d1/d3/dd/d45/f4b [362588,120791] 0 2024-03-04T16:04:08.621 INFO:tasks.workunit.client.0.smithi116.stdout:2/265: chown l3 35536975 1 2024-03-04T16:04:08.622 INFO:tasks.workunit.client.0.smithi116.stdout:2/266: dread - d7/d25/f34 zero size 2024-03-04T16:04:08.624 INFO:tasks.workunit.client.0.smithi116.stdout:1/226: unlink d5/d9/d21/d27/l40 0 2024-03-04T16:04:08.626 INFO:tasks.workunit.client.0.smithi116.stdout:1/227: write d5/d9/d21/d24/f37 [508117,117542] 0 2024-03-04T16:04:08.628 INFO:tasks.workunit.client.0.smithi116.stdout:9/238: mkdir d2/d19/d27/d3b/d4a 0 2024-03-04T16:04:08.629 INFO:tasks.workunit.client.0.smithi116.stdout:9/239: write d2/d33/f46 [308510,54871] 0 2024-03-04T16:04:08.632 INFO:tasks.workunit.client.0.smithi116.stdout:6/296: sync 2024-03-04T16:04:08.634 INFO:tasks.workunit.client.0.smithi116.stdout:2/267: mkdir d7/d15/d35/d63 0 2024-03-04T16:04:08.634 INFO:tasks.workunit.client.0.smithi116.stdout:7/179: truncate d3/d4/d8/d1a/f20 2376863 0 2024-03-04T16:04:08.635 INFO:tasks.workunit.client.0.smithi116.stdout:2/268: write d7/fb [1372470,35604] 0 2024-03-04T16:04:08.637 INFO:tasks.workunit.client.0.smithi116.stdout:1/228: fdatasync d5/d9/d21/d27/f3c 0 2024-03-04T16:04:08.645 INFO:tasks.workunit.client.0.smithi116.stdout:3/352: rename dc/d23/f6e to dc/d34/f75 0 2024-03-04T16:04:08.645 INFO:tasks.workunit.client.0.smithi116.stdout:6/297: dwrite d6/f14 [0,4194304] 0 2024-03-04T16:04:08.646 INFO:tasks.workunit.client.0.smithi116.stdout:4/305: dread d1/d3/dd/f4a [0,4194304] 0 2024-03-04T16:04:08.647 INFO:tasks.workunit.client.0.smithi116.stdout:6/298: chown d6/d56/l61 284368268 1 2024-03-04T16:04:08.647 INFO:tasks.workunit.client.0.smithi116.stdout:7/180: symlink d3/d4/d8/d1a/l3a 0 2024-03-04T16:04:08.648 INFO:tasks.workunit.client.0.smithi116.stdout:7/181: fsync d3/d4/d8/f13 0 2024-03-04T16:04:08.651 INFO:tasks.workunit.client.0.smithi116.stdout:2/269: mkdir d7/d29/d47/d64 0 2024-03-04T16:04:08.657 INFO:tasks.workunit.client.0.smithi116.stdout:1/229: write d5/f3b [1138110,111513] 0 2024-03-04T16:04:08.662 INFO:tasks.workunit.client.0.smithi116.stdout:3/353: creat dc/d23/d2c/f76 x:0 0 0 2024-03-04T16:04:08.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.662+0000 7fe2fdc13700 1 -- 172.21.15.116:0/1358027220 --> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2d8000bf0 con 0x7fe2e406cee0 2024-03-04T16:04:08.664 INFO:tasks.workunit.client.0.smithi116.stdout:7/182: mknod d3/c3b 0 2024-03-04T16:04:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.664+0000 7fe2f4ff9700 1 -- 172.21.15.116:0/1358027220 <== mgr.24387 v2:172.21.15.152:6828/2881662614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fe2d8000bf0 con 0x7fe2e406cee0 2024-03-04T16:04:08.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 -- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fe2e406cee0 msgr2=0x7fe2e406f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:08.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fe2e406cee0 0x7fe2e406f3a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00059d0 tx=0x7fe2e0005920 comp rx=0 tx=0).stop 2024-03-04T16:04:08.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 -- 172.21.15.116:0/1358027220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8101da0 msgr2=0x7fe2f819ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:08.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8101da0 0x7fe2f819ce80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800cfd0 tx=0x7fe2e800a340 comp rx=0 tx=0).stop 2024-03-04T16:04:08.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 -- 172.21.15.116:0/1358027220 shutdown_connections 2024-03-04T16:04:08.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fe2e406cee0 0x7fe2e406f3a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:08.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2f80ff430 0x7fe2f819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:08.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 --2- 172.21.15.116:0/1358027220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8101da0 0x7fe2f819ce80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:08.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 -- 172.21.15.116:0/1358027220 >> 172.21.15.116:0/1358027220 conn(0x7fe2f80fb030 msgr2=0x7fe2f80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:08.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 -- 172.21.15.116:0/1358027220 shutdown_connections 2024-03-04T16:04:08.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.666+0000 7fe2fdc13700 1 -- 172.21.15.116:0/1358027220 wait complete. 2024-03-04T16:04:08.678 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:04:08.689 INFO:tasks.workunit.client.0.smithi116.stdout:1/230: mknod d5/d9/d21/d27/c4f 0 2024-03-04T16:04:08.690 INFO:tasks.workunit.client.0.smithi116.stdout:9/240: getdents d2/d19/d27 0 2024-03-04T16:04:08.694 INFO:tasks.workunit.client.0.smithi116.stdout:9/241: readlink d2/d19/d27/d1f/l20 0 2024-03-04T16:04:08.699 INFO:tasks.workunit.client.0.smithi116.stdout:7/183: dwrite d3/d4/d11/f38 [0,4194304] 0 2024-03-04T16:04:08.700 INFO:tasks.workunit.client.0.smithi116.stdout:3/354: dwrite dc/f4b [0,4194304] 0 2024-03-04T16:04:08.704 INFO:tasks.workunit.client.0.smithi116.stdout:1/231: getdents d5/d9/d21/d27/d32/d49/d4a 0 2024-03-04T16:04:08.707 INFO:tasks.workunit.client.0.smithi116.stdout:3/355: chown dc/d34/d44/d6a 82 1 2024-03-04T16:04:08.707 INFO:tasks.workunit.client.0.smithi116.stdout:6/299: dread d6/f9 [0,4194304] 0 2024-03-04T16:04:08.709 INFO:tasks.workunit.client.0.smithi116.stdout:7/184: mknod d3/d4/d8/c3c 0 2024-03-04T16:04:08.709 INFO:tasks.workunit.client.0.smithi116.stdout:9/242: link d2/d19/d27/d1f/l22 d2/d26/d28/l4b 0 2024-03-04T16:04:08.711 INFO:tasks.workunit.client.0.smithi116.stdout:9/243: readlink d2/l14 0 2024-03-04T16:04:08.712 INFO:tasks.workunit.client.0.smithi116.stdout:1/232: dwrite d5/d11/f4e [0,4194304] 0 2024-03-04T16:04:08.712 INFO:tasks.workunit.client.0.smithi116.stdout:2/270: truncate d7/d13/f45 3127833 0 2024-03-04T16:04:08.713 INFO:tasks.workunit.client.0.smithi116.stdout:3/356: dread dc/d23/d5a/f65 [0,4194304] 0 2024-03-04T16:04:08.720 INFO:tasks.workunit.client.0.smithi116.stdout:2/271: symlink d7/d1b/l65 0 2024-03-04T16:04:08.721 INFO:tasks.workunit.client.0.smithi116.stdout:6/300: dread d6/f24 [0,4194304] 0 2024-03-04T16:04:08.724 INFO:tasks.workunit.client.0.smithi116.stdout:7/185: dwrite d3/d4/d8/f35 [0,4194304] 0 2024-03-04T16:04:08.725 INFO:tasks.workunit.client.0.smithi116.stdout:9/244: link d2/d49/ff d2/d19/d27/d3b/d4a/f4c 0 2024-03-04T16:04:08.726 INFO:tasks.workunit.client.0.smithi116.stdout:2/272: mknod d7/d15/d35/c66 0 2024-03-04T16:04:08.730 INFO:tasks.workunit.client.0.smithi116.stdout:7/186: dwrite d3/f1f [0,4194304] 0 2024-03-04T16:04:08.739 INFO:tasks.workunit.client.0.smithi116.stdout:3/357: getdents dc/d23 0 2024-03-04T16:04:08.742 INFO:tasks.workunit.client.0.smithi116.stdout:6/301: dwrite d6/de/d25/d4a/d63/f65 [0,4194304] 0 2024-03-04T16:04:08.745 INFO:tasks.workunit.client.0.smithi116.stdout:7/187: read d3/d4/d8/d1a/f20 [1334809,94969] 0 2024-03-04T16:04:08.746 INFO:tasks.workunit.client.0.smithi116.stdout:9/245: truncate d2/f15 6055326 0 2024-03-04T16:04:08.751 INFO:tasks.workunit.client.0.smithi116.stdout:0/256: sync 2024-03-04T16:04:08.754 INFO:tasks.workunit.client.0.smithi116.stdout:2/273: write d7/d29/d47/f49 [1094270,108641] 0 2024-03-04T16:04:08.754 INFO:tasks.workunit.client.0.smithi116.stdout:1/233: truncate d5/d9/d21/d27/d32/f3e 45841 0 2024-03-04T16:04:08.756 INFO:tasks.workunit.client.0.smithi116.stdout:9/246: truncate d2/d19/d27/d1f/f24 3335361 0 2024-03-04T16:04:08.761 INFO:tasks.workunit.client.0.smithi116.stdout:7/188: mknod d3/d4/c3d 0 2024-03-04T16:04:08.761 INFO:tasks.workunit.client.0.smithi116.stdout:7/189: chown d3/d4/d8/f35 310949501 1 2024-03-04T16:04:08.762 INFO:tasks.workunit.client.0.smithi116.stdout:7/190: dread - d3/d4/d8/f25 zero size 2024-03-04T16:04:08.762 INFO:tasks.workunit.client.0.smithi116.stdout:7/191: read - d3/d4/d11/f36 zero size 2024-03-04T16:04:08.763 INFO:tasks.workunit.client.0.smithi116.stdout:2/274: creat d7/d15/d5e/f67 x:0 0 0 2024-03-04T16:04:08.764 INFO:tasks.workunit.client.0.smithi116.stdout:2/275: write d7/d29/f4a [2678687,80994] 0 2024-03-04T16:04:08.768 INFO:tasks.workunit.client.0.smithi116.stdout:3/358: link dc/df/d14/f2b dc/d34/d44/d6a/f77 0 2024-03-04T16:04:08.772 INFO:tasks.workunit.client.0.smithi116.stdout:6/302: write d6/de/d1e/f48 [703889,13638] 0 2024-03-04T16:04:08.772 INFO:tasks.workunit.client.0.smithi116.stdout:6/303: dread - d6/f37 zero size 2024-03-04T16:04:08.772 INFO:tasks.workunit.client.0.smithi116.stdout:1/234: mkdir d5/d9/d46/d50 0 2024-03-04T16:04:08.776 INFO:tasks.workunit.client.0.smithi116.stdout:2/276: creat d7/d25/d41/f68 x:0 0 0 2024-03-04T16:04:08.785 INFO:tasks.workunit.client.0.smithi116.stdout:9/247: rename d2/d33/c36 to d2/d19/c4d 0 2024-03-04T16:04:08.788 INFO:tasks.workunit.client.0.smithi116.stdout:6/304: mknod d6/de/d25/d36/d4e/d50/c68 0 2024-03-04T16:04:08.789 INFO:tasks.workunit.client.0.smithi116.stdout:7/192: creat d3/d4/d8/f3e x:0 0 0 2024-03-04T16:04:08.795 INFO:tasks.workunit.client.0.smithi116.stdout:6/305: rmdir d6/de/d15 39 2024-03-04T16:04:08.796 INFO:tasks.workunit.client.0.smithi116.stdout:7/193: dwrite d3/d4/d8/d1a/f27 [0,4194304] 0 2024-03-04T16:04:08.797 INFO:tasks.workunit.client.0.smithi116.stdout:7/194: chown d3/d4/d8/d1a/f2e 338603 1 2024-03-04T16:04:08.797 INFO:tasks.workunit.client.0.smithi116.stdout:5/276: sync 2024-03-04T16:04:08.797 INFO:tasks.workunit.client.0.smithi116.stdout:8/286: sync 2024-03-04T16:04:08.803 INFO:tasks.workunit.client.0.smithi116.stdout:3/359: getdents dc/d34/d44 0 2024-03-04T16:04:08.806 INFO:tasks.workunit.client.0.smithi116.stdout:7/195: dwrite d3/f1e [4194304,4194304] 0 2024-03-04T16:04:08.808 INFO:tasks.workunit.client.0.smithi116.stdout:9/248: truncate d2/f15 6272738 0 2024-03-04T16:04:08.812 INFO:tasks.workunit.client.0.smithi116.stdout:8/287: chown d2/de/c51 14375 1 2024-03-04T16:04:08.812 INFO:tasks.workunit.client.0.smithi116.stdout:5/277: mkdir dd/d1f/d5c 0 2024-03-04T16:04:08.815 INFO:tasks.workunit.client.0.smithi116.stdout:2/277: link d7/d15/f20 d7/d13/d5f/f69 0 2024-03-04T16:04:08.816 INFO:tasks.workunit.client.0.smithi116.stdout:3/360: rmdir dc/df 39 2024-03-04T16:04:08.817 INFO:tasks.workunit.client.0.smithi116.stdout:3/361: stat dc/d23/l32 0 2024-03-04T16:04:08.818 INFO:tasks.workunit.client.0.smithi116.stdout:7/196: rename d3/d4/d34/c37 to d3/d4/d8/d1a/c3f 0 2024-03-04T16:04:08.821 INFO:tasks.workunit.client.0.smithi116.stdout:5/278: dwrite dd/d1f/f29 [0,4194304] 0 2024-03-04T16:04:08.822 INFO:tasks.workunit.client.0.smithi116.stdout:9/249: unlink d2/d49/ff 0 2024-03-04T16:04:08.824 INFO:tasks.workunit.client.0.smithi116.stdout:6/306: symlink d6/de/d15/l69 0 2024-03-04T16:04:08.826 INFO:tasks.workunit.client.0.smithi116.stdout:1/235: dwrite d5/d9/d21/d27/d32/f3e [0,4194304] 0 2024-03-04T16:04:08.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.827+0000 7fda9f3c9700 1 -- 172.21.15.116:0/2884703868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda98107090 msgr2=0x7fda981074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:08.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.827+0000 7fda9f3c9700 1 --2- 172.21.15.116:0/2884703868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda98107090 0x7fda981074b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fda94009910 tx=0x7fda94009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:08.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.827+0000 7fda9f3c9700 1 -- 172.21.15.116:0/2884703868 shutdown_connections 2024-03-04T16:04:08.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.827+0000 7fda9f3c9700 1 --2- 172.21.15.116:0/2884703868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda98108280 0x7fda981086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:08.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.827+0000 7fda9f3c9700 1 --2- 172.21.15.116:0/2884703868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda98107090 0x7fda981074b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:08.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.827+0000 7fda9f3c9700 1 -- 172.21.15.116:0/2884703868 >> 172.21.15.116:0/2884703868 conn(0x7fda98076080 msgr2=0x7fda980784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:08.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.827+0000 7fda9f3c9700 1 -- 172.21.15.116:0/2884703868 shutdown_connections 2024-03-04T16:04:08.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.827+0000 7fda9f3c9700 1 -- 172.21.15.116:0/2884703868 wait complete. 2024-03-04T16:04:08.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.828+0000 7fda9f3c9700 1 Processor -- start 2024-03-04T16:04:08.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.828+0000 7fda9f3c9700 1 -- start start 2024-03-04T16:04:08.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.828+0000 7fda9f3c9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda98107090 0x7fda9819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:08.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.828+0000 7fda9f3c9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda98108280 0x7fda9819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:08.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.829+0000 7fda9f3c9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9819d320 con 0x7fda98108280 2024-03-04T16:04:08.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.829+0000 7fda9f3c9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9819d460 con 0x7fda98107090 2024-03-04T16:04:08.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.829+0000 7fda9dbc6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda98108280 0x7fda9819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:08.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.829+0000 7fda9dbc6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda98108280 0x7fda9819cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50248/0 (socket says 172.21.15.116:50248) 2024-03-04T16:04:08.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.829+0000 7fda9dbc6700 1 -- 172.21.15.116:0/228036913 learned_addr learned my addr 172.21.15.116:0/228036913 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:08.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.829+0000 7fda9e3c7700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda98107090 0x7fda9819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:08.831 INFO:tasks.workunit.client.0.smithi116.stdout:3/362: mknod dc/d23/d2c/c78 0 2024-03-04T16:04:08.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.830+0000 7fda9dbc6700 1 -- 172.21.15.116:0/228036913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda98107090 msgr2=0x7fda9819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:08.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.830+0000 7fda9dbc6700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda98107090 0x7fda9819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:08.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.830+0000 7fda9dbc6700 1 -- 172.21.15.116:0/228036913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda940095f0 con 0x7fda98108280 2024-03-04T16:04:08.832 INFO:tasks.workunit.client.0.smithi116.stdout:3/363: chown dc/f70 79 1 2024-03-04T16:04:08.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.831+0000 7fda9dbc6700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda98108280 0x7fda9819cd00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fda8800d900 tx=0x7fda8800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:08.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.831+0000 7fda8f7fe700 1 -- 172.21.15.116:0/228036913 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda880041f0 con 0x7fda98108280 2024-03-04T16:04:08.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.831+0000 7fda9f3c9700 1 -- 172.21.15.116:0/228036913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda981a1f10 con 0x7fda98108280 2024-03-04T16:04:08.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.831+0000 7fda8f7fe700 1 -- 172.21.15.116:0/228036913 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fda88004ce0 con 0x7fda98108280 2024-03-04T16:04:08.833 INFO:tasks.workunit.client.0.smithi116.stdout:2/278: rename d7/d15/l28 to d7/d13/l6a 0 2024-03-04T16:04:08.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.831+0000 7fda8f7fe700 1 -- 172.21.15.116:0/228036913 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda8800b750 con 0x7fda98108280 2024-03-04T16:04:08.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.831+0000 7fda9f3c9700 1 -- 172.21.15.116:0/228036913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda981a2430 con 0x7fda98108280 2024-03-04T16:04:08.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.833+0000 7fda9f3c9700 1 -- 172.21.15.116:0/228036913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9810fb70 con 0x7fda98108280 2024-03-04T16:04:08.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.834+0000 7fda8f7fe700 1 -- 172.21.15.116:0/228036913 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) v1 ==== 90928+0+0 (secure 0 0 0) 0x7fda88004e50 con 0x7fda98108280 2024-03-04T16:04:08.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.834+0000 7fda8f7fe700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fda8406ce30 0x7fda8406f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:08.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.835+0000 7fda8f7fe700 1 -- 172.21.15.116:0/228036913 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fda8808c4e0 con 0x7fda98108280 2024-03-04T16:04:08.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.835+0000 7fda9e3c7700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fda8406ce30 0x7fda8406f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:08.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.837+0000 7fda9e3c7700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fda8406ce30 0x7fda8406f2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fda94006010 tx=0x7fda9400b560 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:08.840 INFO:tasks.workunit.client.0.smithi116.stdout:1/236: symlink d5/d9/d21/d27/d32/d35/l51 0 2024-03-04T16:04:08.841 INFO:tasks.workunit.client.0.smithi116.stdout:3/364: write dc/f64 [944580,1404] 0 2024-03-04T16:04:08.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:08.841+0000 7fda8f7fe700 1 -- 172.21.15.116:0/228036913 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fda88059c40 con 0x7fda98108280 2024-03-04T16:04:08.843 INFO:tasks.workunit.client.0.smithi116.stdout:9/250: rename d2/l7 to d2/d33/l4e 0 2024-03-04T16:04:08.843 INFO:tasks.workunit.client.0.smithi116.stdout:3/365: read dc/d23/d2c/f2d [353686,66201] 0 2024-03-04T16:04:08.844 INFO:tasks.workunit.client.0.smithi116.stdout:2/279: write d7/d15/d35/f54 [642924,23220] 0 2024-03-04T16:04:08.845 INFO:tasks.workunit.client.0.smithi116.stdout:8/288: creat d2/d38/f64 x:0 0 0 2024-03-04T16:04:08.846 INFO:tasks.workunit.client.0.smithi116.stdout:1/237: unlink d5/d9/d21/d24/c25 0 2024-03-04T16:04:08.846 INFO:tasks.workunit.client.0.smithi116.stdout:8/289: chown d2/de/d16/f30 7046 1 2024-03-04T16:04:08.850 INFO:tasks.workunit.client.0.smithi116.stdout:2/280: dread d7/d1b/f1c [0,4194304] 0 2024-03-04T16:04:08.852 INFO:tasks.workunit.client.0.smithi116.stdout:3/366: rename dc/d23/f50 to dc/df/f79 0 2024-03-04T16:04:08.852 INFO:tasks.workunit.client.0.smithi116.stdout:1/238: dread d5/d9/d21/d24/f3a [0,4194304] 0 2024-03-04T16:04:08.853 INFO:tasks.workunit.client.0.smithi116.stdout:5/279: dread dd/d1f/f4e [0,4194304] 0 2024-03-04T16:04:08.857 INFO:tasks.workunit.client.0.smithi116.stdout:6/307: link d6/de/c33 d6/de/d29/d3a/c6a 0 2024-03-04T16:04:08.863 INFO:tasks.workunit.client.0.smithi116.stdout:2/281: mkdir d7/d25/d6b 0 2024-03-04T16:04:08.869 INFO:tasks.workunit.client.0.smithi116.stdout:8/290: creat d2/d38/d55/f65 x:0 0 0 2024-03-04T16:04:08.870 INFO:tasks.workunit.client.0.smithi116.stdout:6/308: mknod d6/de/d25/d4a/c6b 0 2024-03-04T16:04:08.877 INFO:tasks.workunit.client.0.smithi116.stdout:2/282: rename d7/d25/d6b to d7/d2f/d6c 0 2024-03-04T16:04:08.877 INFO:tasks.workunit.client.0.smithi116.stdout:9/251: truncate d2/f48 7498621 0 2024-03-04T16:04:08.878 INFO:tasks.workunit.client.0.smithi116.stdout:2/283: read - d7/d15/d5e/f67 zero size 2024-03-04T16:04:08.878 INFO:tasks.workunit.client.0.smithi116.stdout:5/280: chown dd/d14/d2d/c30 86 1 2024-03-04T16:04:08.879 INFO:tasks.workunit.client.0.smithi116.stdout:5/281: dread dd/d14/d2d/f42 [0,4194304] 0 2024-03-04T16:04:08.880 INFO:tasks.workunit.client.0.smithi116.stdout:1/239: link d5/d9/d21/d24/f3a d5/d9/d21/d27/d32/d35/f52 0 2024-03-04T16:04:08.880 INFO:tasks.workunit.client.0.smithi116.stdout:5/282: chown lc 20 1 2024-03-04T16:04:08.881 INFO:tasks.workunit.client.0.smithi116.stdout:1/240: readlink d5/d9/d21/d27/d32/d35/l51 0 2024-03-04T16:04:08.882 INFO:tasks.workunit.client.0.smithi116.stdout:8/291: rmdir d2/d38/d55 39 2024-03-04T16:04:08.884 INFO:tasks.workunit.client.0.smithi116.stdout:3/367: creat dc/d34/f7a x:0 0 0 2024-03-04T16:04:08.890 INFO:tasks.workunit.client.0.smithi116.stdout:9/252: unlink d2/l13 0 2024-03-04T16:04:08.895 INFO:tasks.workunit.client.0.smithi116.stdout:2/284: dread d7/d13/f45 [0,4194304] 0 2024-03-04T16:04:08.901 INFO:tasks.workunit.client.0.smithi116.stdout:8/292: mkdir d2/d13/d3c/d46/d4f/d66 0 2024-03-04T16:04:08.901 INFO:tasks.workunit.client.0.smithi116.stdout:2/285: dwrite d7/f42 [0,4194304] 0 2024-03-04T16:04:08.903 INFO:tasks.workunit.client.0.smithi116.stdout:8/293: truncate d2/de/d3e/f2a 280124 0 2024-03-04T16:04:08.906 INFO:tasks.workunit.client.0.smithi116.stdout:2/286: truncate d7/d25/f34 549091 0 2024-03-04T16:04:08.911 INFO:tasks.workunit.client.0.smithi116.stdout:8/294: dwrite d2/de/d16/f34 [0,4194304] 0 2024-03-04T16:04:08.912 INFO:tasks.workunit.client.0.smithi116.stdout:8/295: readlink d2/de/d3e/l20 0 2024-03-04T16:04:08.913 INFO:tasks.workunit.client.0.smithi116.stdout:5/283: dread dd/f11 [0,4194304] 0 2024-03-04T16:04:08.917 INFO:tasks.workunit.client.0.smithi116.stdout:6/309: getdents d6/de/d29 0 2024-03-04T16:04:08.920 INFO:tasks.workunit.client.0.smithi116.stdout:3/368: symlink dc/d23/d5a/l7b 0 2024-03-04T16:04:08.922 INFO:tasks.workunit.client.0.smithi116.stdout:9/253: dwrite d2/d19/f2a [0,4194304] 0 2024-03-04T16:04:08.924 INFO:tasks.workunit.client.0.smithi116.stdout:5/284: mkdir dd/d14/d5d 0 2024-03-04T16:04:08.924 INFO:tasks.workunit.client.0.smithi116.stdout:4/306: sync 2024-03-04T16:04:08.925 INFO:tasks.workunit.client.0.smithi116.stdout:5/285: write dd/f11 [3376751,14878] 0 2024-03-04T16:04:08.926 INFO:tasks.workunit.client.0.smithi116.stdout:1/241: truncate d5/f1c 2131366 0 2024-03-04T16:04:08.931 INFO:tasks.workunit.client.0.smithi116.stdout:4/307: dwrite d1/f41 [0,4194304] 0 2024-03-04T16:04:08.937 INFO:tasks.workunit.client.0.smithi116.stdout:8/296: rmdir d2/d13/d3c/d46/d4f/d66 0 2024-03-04T16:04:08.937 INFO:tasks.workunit.client.0.smithi116.stdout:8/297: dread - d2/d38/f3b zero size 2024-03-04T16:04:08.938 INFO:tasks.workunit.client.0.smithi116.stdout:8/298: readlink d2/de/l28 0 2024-03-04T16:04:08.939 INFO:tasks.workunit.client.0.smithi116.stdout:8/299: fsync d2/de/d16/d2b/d37/f49 0 2024-03-04T16:04:08.940 INFO:tasks.workunit.client.0.smithi116.stdout:8/300: fdatasync d2/de/d3e/f4e 0 2024-03-04T16:04:08.943 INFO:tasks.workunit.client.0.smithi116.stdout:4/308: mkdir d1/d6e 0 2024-03-04T16:04:08.943 INFO:tasks.workunit.client.0.smithi116.stdout:4/309: chown d1/d3/dd/l5b 2 1 2024-03-04T16:04:08.944 INFO:tasks.workunit.client.0.smithi116.stdout:5/286: mkdir dd/d1f/d5c/d5e 0 2024-03-04T16:04:08.950 INFO:tasks.workunit.client.0.smithi116.stdout:8/301: rmdir d2/d13/d3c 39 2024-03-04T16:04:08.952 INFO:tasks.workunit.client.0.smithi116.stdout:9/254: creat d2/f4f x:0 0 0 2024-03-04T16:04:08.953 INFO:tasks.workunit.client.0.smithi116.stdout:3/369: rename dc/d23/d5a to dc/d23/d2c/d7c 0 2024-03-04T16:04:08.954 INFO:tasks.workunit.client.0.smithi116.stdout:5/287: write dd/d1f/f4e [925107,22873] 0 2024-03-04T16:04:08.955 INFO:tasks.workunit.client.0.smithi116.stdout:5/288: dread - dd/d14/d2d/f50 zero size 2024-03-04T16:04:08.956 INFO:tasks.workunit.client.0.smithi116.stdout:5/289: readlink dd/d14/d2d/l28 0 2024-03-04T16:04:08.956 INFO:tasks.workunit.client.0.smithi116.stdout:4/310: mkdir d1/d3/d15/d20/d1c/d6f 0 2024-03-04T16:04:08.957 INFO:tasks.workunit.client.0.smithi116.stdout:5/290: write dd/d14/d2d/f50 [88118,15927] 0 2024-03-04T16:04:08.960 INFO:tasks.workunit.client.0.smithi116.stdout:6/310: getdents d6/d56/d58 0 2024-03-04T16:04:08.963 INFO:tasks.workunit.client.0.smithi116.stdout:4/311: dwrite d1/d3/de/f30 [0,4194304] 0 2024-03-04T16:04:08.966 INFO:tasks.workunit.client.0.smithi116.stdout:3/370: creat dc/df/d47/f7d x:0 0 0 2024-03-04T16:04:08.968 INFO:tasks.workunit.client.0.smithi116.stdout:3/371: write dc/d23/d2c/f5e [3103669,9811] 0 2024-03-04T16:04:08.968 INFO:tasks.workunit.client.0.smithi116.stdout:6/311: dwrite d6/de/d25/d4a/d63/f65 [0,4194304] 0 2024-03-04T16:04:08.969 INFO:tasks.workunit.client.0.smithi116.stdout:6/312: dread - d6/de/d29/d53/f57 zero size 2024-03-04T16:04:08.971 INFO:tasks.workunit.client.0.smithi116.stdout:6/313: chown d6/de/d1e/f3f 6 1 2024-03-04T16:04:08.971 INFO:tasks.workunit.client.0.smithi116.stdout:6/314: stat d6/de/d1e/f2a 0 2024-03-04T16:04:08.972 INFO:tasks.workunit.client.0.smithi116.stdout:5/291: creat dd/d1f/f5f x:0 0 0 2024-03-04T16:04:08.973 INFO:tasks.workunit.client.0.smithi116.stdout:5/292: fsync fa 0 2024-03-04T16:04:08.973 INFO:tasks.workunit.client.0.smithi116.stdout:6/315: stat d6/de/d25/d36/d4e/d50 0 2024-03-04T16:04:08.980 INFO:tasks.workunit.client.0.smithi116.stdout:8/302: symlink d2/de/d16/l67 0 2024-03-04T16:04:08.981 INFO:tasks.workunit.client.0.smithi116.stdout:1/242: write f2 [695638,40936] 0 2024-03-04T16:04:08.984 INFO:tasks.workunit.client.0.smithi116.stdout:1/243: dread d5/d11/f4e [0,4194304] 0 2024-03-04T16:04:08.990 INFO:tasks.workunit.client.0.smithi116.stdout:6/316: creat d6/d56/f6c x:0 0 0 2024-03-04T16:04:08.990 INFO:tasks.workunit.client.0.smithi116.stdout:6/317: write d6/f45 [311817,90836] 0 2024-03-04T16:04:08.991 INFO:tasks.workunit.client.0.smithi116.stdout:8/303: symlink d2/d13/d1e/l68 0 2024-03-04T16:04:08.991 INFO:tasks.workunit.client.0.smithi116.stdout:6/318: truncate d6/f21 189643 0 2024-03-04T16:04:08.992 INFO:tasks.workunit.client.0.smithi116.stdout:8/304: write d2/d13/d1e/f43 [7980822,103804] 0 2024-03-04T16:04:08.993 INFO:tasks.workunit.client.0.smithi116.stdout:8/305: fsync d2/de/d3e/f2a 0 2024-03-04T16:04:08.994 INFO:tasks.workunit.client.0.smithi116.stdout:9/255: rename d2/d26/d28/c3f to d2/d19/c50 0 2024-03-04T16:04:08.999 INFO:tasks.workunit.client.0.smithi116.stdout:9/256: dwrite d2/d19/f2a [0,4194304] 0 2024-03-04T16:04:08.999 INFO:tasks.workunit.client.0.smithi116.stdout:4/312: truncate d1/f2c 1041663 0 2024-03-04T16:04:09.003 INFO:tasks.workunit.client.0.smithi116.stdout:3/372: link dc/df/d14/f1a dc/df/f7e 0 2024-03-04T16:04:09.021 INFO:tasks.workunit.client.0.smithi116.stdout:9/257: symlink d2/d19/d27/d1f/l51 0 2024-03-04T16:04:09.022 INFO:tasks.workunit.client.0.smithi116.stdout:9/258: chown d2/d19/d27/f38 1033663952 1 2024-03-04T16:04:09.027 INFO:tasks.workunit.client.0.smithi116.stdout:5/293: rmdir dd/d1f/d52 0 2024-03-04T16:04:09.029 INFO:tasks.workunit.client.0.smithi116.stdout:8/306: creat d2/d13/d3c/f69 x:0 0 0 2024-03-04T16:04:09.030 INFO:tasks.workunit.client.0.smithi116.stdout:9/259: rmdir d2/d19/d27/d3b 39 2024-03-04T16:04:09.035 INFO:tasks.workunit.client.0.smithi116.stdout:8/307: dwrite d2/de/d16/f34 [4194304,4194304] 0 2024-03-04T16:04:09.038 INFO:tasks.workunit.client.0.smithi116.stdout:5/294: read dd/f13 [2308404,121239] 0 2024-03-04T16:04:09.043 INFO:tasks.workunit.client.0.smithi116.stdout:3/373: getdents dc/d34/d44 0 2024-03-04T16:04:09.044 INFO:tasks.workunit.client.0.smithi116.stdout:8/308: dread d2/de/fa [0,4194304] 0 2024-03-04T16:04:09.045 INFO:tasks.workunit.client.0.smithi116.stdout:8/309: write d2/d13/d3c/f69 [782926,16570] 0 2024-03-04T16:04:09.049 INFO:tasks.workunit.client.0.smithi116.stdout:5/295: chown dd/f15 5 1 2024-03-04T16:04:09.051 INFO:tasks.workunit.client.0.smithi116.stdout:9/260: fsync d2/d19/d27/d1f/f24 0 2024-03-04T16:04:09.051 INFO:tasks.workunit.client.0.smithi116.stdout:9/261: chown d2/d26 6 1 2024-03-04T16:04:09.052 INFO:tasks.workunit.client.0.smithi116.stdout:1/244: write d5/f1c [718943,22071] 0 2024-03-04T16:04:09.055 INFO:tasks.workunit.client.0.smithi116.stdout:8/310: mknod d2/de/d16/d2b/d37/c6a 0 2024-03-04T16:04:09.063 INFO:tasks.workunit.client.0.smithi116.stdout:1/245: creat d5/d9/d21/d27/d32/d35/f53 x:0 0 0 2024-03-04T16:04:09.065 INFO:tasks.workunit.client.0.smithi116.stdout:3/374: write dc/f70 [3430,8570] 0 2024-03-04T16:04:09.069 INFO:tasks.workunit.client.0.smithi116.stdout:1/246: rename d5/d11/f16 to d5/d11/f54 0 2024-03-04T16:04:09.070 INFO:tasks.workunit.client.0.smithi116.stdout:1/247: chown d5/d9/d21/d24/f3a 14629 1 2024-03-04T16:04:09.070 INFO:tasks.workunit.client.0.smithi116.stdout:8/311: rmdir d2/d38/d41 39 2024-03-04T16:04:09.072 INFO:tasks.workunit.client.0.smithi116.stdout:9/262: dread d2/f15 [0,4194304] 0 2024-03-04T16:04:09.072 INFO:tasks.workunit.client.0.smithi116.stdout:3/375: mkdir dc/d23/d2c/d7f 0 2024-03-04T16:04:09.073 INFO:tasks.workunit.client.0.smithi116.stdout:5/296: creat dd/f60 x:0 0 0 2024-03-04T16:04:09.074 INFO:tasks.workunit.client.0.smithi116.stdout:5/297: write dd/f60 [243776,29293] 0 2024-03-04T16:04:09.075 INFO:tasks.workunit.client.0.smithi116.stdout:5/298: chown dd/d14/d2d/d31/f3c 32745419 1 2024-03-04T16:04:09.078 INFO:tasks.workunit.client.0.smithi116.stdout:9/263: mkdir d2/d49/d52 0 2024-03-04T16:04:09.079 INFO:tasks.workunit.client.0.smithi116.stdout:9/264: rename d2/d49 to d2/d49/d3c/d53 22 2024-03-04T16:04:09.081 INFO:tasks.workunit.client.0.smithi116.stdout:8/312: dwrite d2/de/fa [0,4194304] 0 2024-03-04T16:04:09.083 INFO:tasks.workunit.client.0.smithi116.stdout:8/313: write d2/de/f8 [2001207,84333] 0 2024-03-04T16:04:09.083 INFO:tasks.workunit.client.0.smithi116.stdout:9/265: creat d2/d33/f54 x:0 0 0 2024-03-04T16:04:09.084 INFO:tasks.workunit.client.0.smithi116.stdout:0/257: sync 2024-03-04T16:04:09.084 INFO:tasks.workunit.client.0.smithi116.stdout:7/197: sync 2024-03-04T16:04:09.086 INFO:tasks.workunit.client.0.smithi116.stdout:0/258: write da/ff [3416788,37553] 0 2024-03-04T16:04:09.086 INFO:tasks.workunit.client.0.smithi116.stdout:3/376: getdents dc/d34/d44 0 2024-03-04T16:04:09.088 INFO:tasks.workunit.client.0.smithi116.stdout:5/299: dread dd/d45/f49 [0,4194304] 0 2024-03-04T16:04:09.092 INFO:tasks.workunit.client.0.smithi116.stdout:3/377: creat dc/d23/d2c/d7c/f80 x:0 0 0 2024-03-04T16:04:09.094 INFO:tasks.workunit.client.0.smithi116.stdout:0/259: rename da/d16/d28/d39/f47 to da/d16/d20/f5b 0 2024-03-04T16:04:09.100 INFO:tasks.workunit.client.0.smithi116.stdout:0/260: dwrite da/d16/d28/d40/d48/f52 [0,4194304] 0 2024-03-04T16:04:09.100 INFO:tasks.workunit.client.0.smithi116.stdout:3/378: creat dc/d23/d2c/d7c/f81 x:0 0 0 2024-03-04T16:04:09.106 INFO:tasks.workunit.client.0.smithi116.stdout:3/379: dwrite dc/f5f [0,4194304] 0 2024-03-04T16:04:09.106 INFO:tasks.workunit.client.0.smithi116.stdout:5/300: symlink dd/d14/d2d/d59/l61 0 2024-03-04T16:04:09.111 INFO:tasks.workunit.client.0.smithi116.stdout:0/261: rmdir da/d16/d28/d40 39 2024-03-04T16:04:09.117 INFO:tasks.workunit.client.0.smithi116.stdout:9/266: getdents d2/d19/d27/d1f 0 2024-03-04T16:04:09.121 INFO:tasks.workunit.client.0.smithi116.stdout:3/380: symlink dc/d23/l82 0 2024-03-04T16:04:09.126 INFO:tasks.workunit.client.0.smithi116.stdout:5/301: truncate fa 1228625 0 2024-03-04T16:04:09.127 INFO:tasks.workunit.client.0.smithi116.stdout:9/267: creat d2/d26/d28/f55 x:0 0 0 2024-03-04T16:04:09.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.134+0000 7fda9f3c9700 1 -- 172.21.15.116:0/228036913 --> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda9810cbd0 con 0x7fda8406ce30 2024-03-04T16:04:09.136 INFO:tasks.workunit.client.0.smithi116.stdout:3/381: symlink dc/l83 0 2024-03-04T16:04:09.136 INFO:tasks.workunit.client.0.smithi116.stdout:8/314: write d2/f7 [1033410,44843] 0 2024-03-04T16:04:09.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.136+0000 7fda8f7fe700 1 -- 172.21.15.116:0/228036913 <== mgr.24387 v2:172.21.15.152:6828/2881662614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fda9810cbd0 con 0x7fda8406ce30 2024-03-04T16:04:09.137 INFO:tasks.workunit.client.0.smithi116.stdout:5/302: creat dd/d1f/d5c/d5e/f62 x:0 0 0 2024-03-04T16:04:09.137 INFO:tasks.workunit.client.0.smithi116.stdout:3/382: truncate dc/d23/d2c/f76 42548 0 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.139+0000 7fda9f3c9700 1 -- 172.21.15.116:0/228036913 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fda8406ce30 msgr2=0x7fda8406f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.139+0000 7fda9f3c9700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fda8406ce30 0x7fda8406f2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fda94006010 tx=0x7fda9400b560 comp rx=0 tx=0).stop 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.139+0000 7fda9f3c9700 1 -- 172.21.15.116:0/228036913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda98108280 msgr2=0x7fda9819cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.139+0000 7fda9f3c9700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda98108280 0x7fda9819cd00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fda8800d900 tx=0x7fda8800dc10 comp rx=0 tx=0).stop 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.139+0000 7fda9f3c9700 1 -- 172.21.15.116:0/228036913 shutdown_connections 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.139+0000 7fda9f3c9700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7fda8406ce30 0x7fda8406f2f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.139+0000 7fda9f3c9700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda98107090 0x7fda9819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.139+0000 7fda9f3c9700 1 --2- 172.21.15.116:0/228036913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda98108280 0x7fda9819cd00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.139+0000 7fda9f3c9700 1 -- 172.21.15.116:0/228036913 >> 172.21.15.116:0/228036913 conn(0x7fda98076080 msgr2=0x7fda9810b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.140+0000 7fda9f3c9700 1 -- 172.21.15.116:0/228036913 shutdown_connections 2024-03-04T16:04:09.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.140+0000 7fda9f3c9700 1 -- 172.21.15.116:0/228036913 wait complete. 2024-03-04T16:04:09.141 INFO:tasks.workunit.client.0.smithi116.stdout:7/198: truncate d3/d4/f16 2400304 0 2024-03-04T16:04:09.143 INFO:tasks.workunit.client.0.smithi116.stdout:8/315: dwrite d2/de/d16/f1b [0,4194304] 0 2024-03-04T16:04:09.144 INFO:tasks.workunit.client.0.smithi116.stdout:9/268: dread d2/d26/f44 [0,4194304] 0 2024-03-04T16:04:09.145 INFO:tasks.workunit.client.0.smithi116.stdout:5/303: symlink dd/d1f/d5c/d5e/l63 0 2024-03-04T16:04:09.147 INFO:tasks.workunit.client.0.smithi116.stdout:0/262: rename da/d16/f17 to da/d16/d28/d40/f5c 0 2024-03-04T16:04:09.152 INFO:tasks.workunit.client.0.smithi116.stdout:3/383: unlink dc/d34/c71 0 2024-03-04T16:04:09.163 INFO:tasks.workunit.client.0.smithi116.stdout:9/269: symlink d2/d19/l56 0 2024-03-04T16:04:09.163 INFO:tasks.workunit.client.0.smithi116.stdout:8/316: creat d2/d13/d5e/f6b x:0 0 0 2024-03-04T16:04:09.165 INFO:tasks.workunit.client.0.smithi116.stdout:0/263: mkdir da/d16/d28/d40/d5d 0 2024-03-04T16:04:09.166 INFO:tasks.workunit.client.0.smithi116.stdout:0/264: readlink da/d16/d28/l3a 0 2024-03-04T16:04:09.167 INFO:tasks.workunit.client.0.smithi116.stdout:9/270: write d2/d19/f37 [174668,129032] 0 2024-03-04T16:04:09.169 INFO:tasks.workunit.client.0.smithi116.stdout:9/271: write d2/f4f [782115,115415] 0 2024-03-04T16:04:09.171 INFO:tasks.workunit.client.0.smithi116.stdout:3/384: rename dc/d23/d2c to dc/d23/d84 0 2024-03-04T16:04:09.176 INFO:tasks.workunit.client.0.smithi116.stdout:8/317: creat d2/d38/d41/f6c x:0 0 0 2024-03-04T16:04:09.177 INFO:tasks.workunit.client.0.smithi116.stdout:5/304: getdents dd/d14/d2d/d31 0 2024-03-04T16:04:09.178 INFO:tasks.workunit.client.0.smithi116.stdout:3/385: dwrite dc/d23/d84/d7c/f80 [0,4194304] 0 2024-03-04T16:04:09.180 INFO:tasks.workunit.client.0.smithi116.stdout:0/265: getdents da/d16/d19/d27/d45 0 2024-03-04T16:04:09.181 INFO:tasks.workunit.client.0.smithi116.stdout:0/266: stat da/d16/d19/c29 0 2024-03-04T16:04:09.190 INFO:tasks.workunit.client.0.smithi116.stdout:8/318: creat d2/d13/d3c/d46/f6d x:0 0 0 2024-03-04T16:04:09.192 INFO:tasks.workunit.client.0.smithi116.stdout:5/305: creat dd/d1f/d5c/f64 x:0 0 0 2024-03-04T16:04:09.194 INFO:tasks.workunit.client.0.smithi116.stdout:0/267: chown da/d16/d19/d27/c43 23854834 1 2024-03-04T16:04:09.194 INFO:tasks.workunit.client.0.smithi116.stdout:5/306: dread dd/f60 [0,4194304] 0 2024-03-04T16:04:09.195 INFO:tasks.workunit.client.0.smithi116.stdout:0/268: chown da/d16/d19/l36 15 1 2024-03-04T16:04:09.198 INFO:tasks.workunit.client.0.smithi116.stdout:8/319: unlink d2/d13/l1f 0 2024-03-04T16:04:09.199 INFO:tasks.workunit.client.0.smithi116.stdout:0/269: dread da/fe [0,4194304] 0 2024-03-04T16:04:09.199 INFO:tasks.workunit.client.0.smithi116.stdout:0/270: readlink da/l2a 0 2024-03-04T16:04:09.199 INFO:tasks.workunit.client.0.smithi116.stdout:0/271: readlink da/d16/d28/l3a 0 2024-03-04T16:04:09.205 INFO:tasks.workunit.client.0.smithi116.stdout:9/272: getdents d2/d26 0 2024-03-04T16:04:09.205 INFO:tasks.workunit.client.0.smithi116.stdout:0/272: write da/d16/d28/d3d/d41/f46 [1534734,74232] 0 2024-03-04T16:04:09.205 INFO:tasks.workunit.client.0.smithi116.stdout:7/199: truncate d3/fd 4778256 0 2024-03-04T16:04:09.205 INFO:tasks.workunit.client.0.smithi116.stdout:5/307: creat dd/d14/d2d/d31/f65 x:0 0 0 2024-03-04T16:04:09.205 INFO:tasks.workunit.client.0.smithi116.stdout:2/287: sync 2024-03-04T16:04:09.205 INFO:tasks.workunit.client.0.smithi116.stdout:3/386: creat dc/d23/f85 x:0 0 0 2024-03-04T16:04:09.207 INFO:tasks.workunit.client.0.smithi116.stdout:7/200: dread d3/f1e [0,4194304] 0 2024-03-04T16:04:09.207 INFO:tasks.workunit.client.0.smithi116.stdout:2/288: read d7/fb [244298,82079] 0 2024-03-04T16:04:09.209 INFO:tasks.workunit.client.0.smithi116.stdout:0/273: rmdir da/d16/d28 39 2024-03-04T16:04:09.210 INFO:tasks.workunit.client.0.smithi116.stdout:0/274: read - da/d16/d20/f5b zero size 2024-03-04T16:04:09.210 INFO:tasks.workunit.client.0.smithi116.stdout:0/275: readlink da/l1c 0 2024-03-04T16:04:09.213 INFO:tasks.workunit.client.0.smithi116.stdout:5/308: mknod dd/d14/d2d/d59/c66 0 2024-03-04T16:04:09.217 INFO:tasks.workunit.client.0.smithi116.stdout:2/289: creat d7/d1b/f6d x:0 0 0 2024-03-04T16:04:09.218 INFO:tasks.workunit.client.0.smithi116.stdout:9/273: symlink d2/d19/d27/d3b/l57 0 2024-03-04T16:04:09.219 INFO:tasks.workunit.client.0.smithi116.stdout:2/290: write d7/d25/d41/f5a [484542,51991] 0 2024-03-04T16:04:09.220 INFO:tasks.workunit.client.0.smithi116.stdout:2/291: truncate d7/d15/f38 607913 0 2024-03-04T16:04:09.224 INFO:tasks.workunit.client.0.smithi116.stdout:5/309: dread f4 [0,4194304] 0 2024-03-04T16:04:09.227 INFO:tasks.workunit.client.0.smithi116.stdout:3/387: creat dc/d23/d84/d7f/f86 x:0 0 0 2024-03-04T16:04:09.234 INFO:tasks.workunit.client.0.smithi116.stdout:5/310: truncate dd/f15 3080325 0 2024-03-04T16:04:09.234 INFO:tasks.workunit.client.0.smithi116.stdout:7/201: creat d3/d4/d8/f40 x:0 0 0 2024-03-04T16:04:09.238 INFO:tasks.workunit.client.0.smithi116.stdout:3/388: mknod dc/d34/d44/d6a/c87 0 2024-03-04T16:04:09.240 INFO:tasks.workunit.client.0.smithi116.stdout:9/274: rename d2/d19/c1b to d2/d19/d27/d3b/c58 0 2024-03-04T16:04:09.245 INFO:tasks.workunit.client.0.smithi116.stdout:3/389: dread dc/df/f7e [0,4194304] 0 2024-03-04T16:04:09.252 INFO:tasks.workunit.client.0.smithi116.stdout:0/276: rmdir da/d16/d28/d3d/d41/d49 0 2024-03-04T16:04:09.256 INFO:tasks.workunit.client.0.smithi116.stdout:3/390: rmdir dc/d23 39 2024-03-04T16:04:09.261 INFO:tasks.workunit.client.0.smithi116.stdout:9/275: mknod d2/d19/d27/d3b/d4a/c59 0 2024-03-04T16:04:09.266 INFO:tasks.workunit.client.0.smithi116.stdout:2/292: write d7/f36 [2246079,12273] 0 2024-03-04T16:04:09.267 INFO:tasks.workunit.client.0.smithi116.stdout:9/276: dwrite d2/d19/d27/d3b/f3d [0,4194304] 0 2024-03-04T16:04:09.268 INFO:tasks.workunit.client.0.smithi116.stdout:0/277: link da/d16/d28/d40/d48/l4d da/d16/d28/d40/d5d/l5e 0 2024-03-04T16:04:09.269 INFO:tasks.workunit.client.0.smithi116.stdout:5/311: write dd/f60 [138226,17779] 0 2024-03-04T16:04:09.270 INFO:tasks.workunit.client.0.smithi116.stdout:3/391: creat dc/f88 x:0 0 0 2024-03-04T16:04:09.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.276+0000 7f6e1160d700 1 -- 172.21.15.116:0/4113004048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0c100cf0 msgr2=0x7f6e0c101150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:09.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.276+0000 7f6e1160d700 1 --2- 172.21.15.116:0/4113004048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0c100cf0 0x7f6e0c101150 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009910 tx=0x7f6e00009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:09.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.276+0000 7f6e1160d700 1 -- 172.21.15.116:0/4113004048 shutdown_connections 2024-03-04T16:04:09.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.276+0000 7f6e1160d700 1 --2- 172.21.15.116:0/4113004048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0c100cf0 0x7f6e0c101150 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.276+0000 7f6e1160d700 1 --2- 172.21.15.116:0/4113004048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0ffaf0 0x7f6e0c0fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.276+0000 7f6e1160d700 1 -- 172.21.15.116:0/4113004048 >> 172.21.15.116:0/4113004048 conn(0x7f6e0c0fb070 msgr2=0x7f6e0c0fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:09.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.276+0000 7f6e1160d700 1 -- 172.21.15.116:0/4113004048 shutdown_connections 2024-03-04T16:04:09.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.276+0000 7f6e1160d700 1 -- 172.21.15.116:0/4113004048 wait complete. 2024-03-04T16:04:09.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.277+0000 7f6e1160d700 1 Processor -- start 2024-03-04T16:04:09.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.277+0000 7f6e1160d700 1 -- start start 2024-03-04T16:04:09.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.277+0000 7f6e1160d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0ffaf0 0x7f6e0c196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:09.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.277+0000 7f6e1160d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0c100cf0 0x7f6e0c196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:09.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.277+0000 7f6e1160d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c196f80 con 0x7f6e0c100cf0 2024-03-04T16:04:09.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.277+0000 7f6e1160d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c1970c0 con 0x7f6e0c0ffaf0 2024-03-04T16:04:09.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.277+0000 7f6e0a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0c100cf0 0x7f6e0c196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:09.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.277+0000 7f6e0a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0c100cf0 0x7f6e0c196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50276/0 (socket says 172.21.15.116:50276) 2024-03-04T16:04:09.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.277+0000 7f6e0a7fc700 1 -- 172.21.15.116:0/3563293262 learned_addr learned my addr 172.21.15.116:0/3563293262 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:09.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.277+0000 7f6e0affd700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0ffaf0 0x7f6e0c196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:09.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.278+0000 7f6e0a7fc700 1 -- 172.21.15.116:0/3563293262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0ffaf0 msgr2=0x7f6e0c196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:09.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.278+0000 7f6e0a7fc700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0ffaf0 0x7f6e0c196420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.278+0000 7f6e0a7fc700 1 -- 172.21.15.116:0/3563293262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e000095f0 con 0x7f6e0c100cf0 2024-03-04T16:04:09.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.278+0000 7f6e0a7fc700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0c100cf0 0x7f6e0c196960 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000b5c0 tx=0x7f6e0000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:09.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.278+0000 7f6df3fff700 1 -- 172.21.15.116:0/3563293262 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e0001d070 con 0x7f6e0c100cf0 2024-03-04T16:04:09.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.278+0000 7f6df3fff700 1 -- 172.21.15.116:0/3563293262 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6e00022470 con 0x7f6e0c100cf0 2024-03-04T16:04:09.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.278+0000 7f6e1160d700 1 -- 172.21.15.116:0/3563293262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c19bb10 con 0x7f6e0c100cf0 2024-03-04T16:04:09.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.278+0000 7f6df3fff700 1 -- 172.21.15.116:0/3563293262 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e0000f690 con 0x7f6e0c100cf0 2024-03-04T16:04:09.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.278+0000 7f6e1160d700 1 -- 172.21.15.116:0/3563293262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0c19c080 con 0x7f6e0c100cf0 2024-03-04T16:04:09.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.279+0000 7f6e1160d700 1 -- 172.21.15.116:0/3563293262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c1086f0 con 0x7f6e0c100cf0 2024-03-04T16:04:09.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.280+0000 7f6df3fff700 1 -- 172.21.15.116:0/3563293262 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) v1 ==== 90928+0+0 (secure 0 0 0) 0x7f6e000225e0 con 0x7f6e0c100cf0 2024-03-04T16:04:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.280+0000 7f6df3fff700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6df406ce30 0x7f6df406f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.281+0000 7f6df3fff700 1 -- 172.21.15.116:0/3563293262 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f6e0008de70 con 0x7f6e0c100cf0 2024-03-04T16:04:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.281+0000 7f6e0affd700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6df406ce30 0x7f6df406f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:09.283 INFO:tasks.workunit.client.0.smithi116.stdout:2/293: link d7/d1b/f22 d7/d29/d47/f6e 0 2024-03-04T16:04:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.281+0000 7f6e0affd700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6df406ce30 0x7f6df406f2f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c100b50 tx=0x7f6dfc009190 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:09.284 INFO:tasks.workunit.client.0.smithi116.stdout:2/294: write d7/f4e [8784022,96832] 0 2024-03-04T16:04:09.286 INFO:tasks.workunit.client.0.smithi116.stdout:7/202: dread f1 [0,4194304] 0 2024-03-04T16:04:09.287 INFO:tasks.workunit.client.0.smithi116.stdout:7/203: chown d3/d4/c3d 57 1 2024-03-04T16:04:09.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.286+0000 7f6df3fff700 1 -- 172.21.15.116:0/3563293262 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f6e0005b510 con 0x7f6e0c100cf0 2024-03-04T16:04:09.288 INFO:tasks.workunit.client.0.smithi116.stdout:7/204: chown d3/d4/d11/f2f 8547239 1 2024-03-04T16:04:09.289 INFO:tasks.workunit.client.0.smithi116.stdout:2/295: creat d7/d29/d47/d64/f6f x:0 0 0 2024-03-04T16:04:09.290 INFO:tasks.workunit.client.0.smithi116.stdout:3/392: getdents dc/d34/d44/d6a 0 2024-03-04T16:04:09.291 INFO:tasks.workunit.client.0.smithi116.stdout:3/393: stat dc/df/c10 0 2024-03-04T16:04:09.292 INFO:tasks.workunit.client.0.smithi116.stdout:3/394: chown dc/f70 1 1 2024-03-04T16:04:09.294 INFO:tasks.workunit.client.0.smithi116.stdout:7/205: mknod d3/d4/d8/c41 0 2024-03-04T16:04:09.296 INFO:tasks.workunit.client.0.smithi116.stdout:9/277: dwrite d2/f15 [0,4194304] 0 2024-03-04T16:04:09.297 INFO:tasks.workunit.client.0.smithi116.stdout:7/206: truncate d3/d4/d8/d1a/f28 297182 0 2024-03-04T16:04:09.300 INFO:tasks.workunit.client.0.smithi116.stdout:7/207: stat d3/d4/d11/f2f 0 2024-03-04T16:04:09.303 INFO:tasks.workunit.client.0.smithi116.stdout:7/208: write d3/f18 [981830,52067] 0 2024-03-04T16:04:09.306 INFO:tasks.workunit.client.0.smithi116.stdout:2/296: dread d7/d1b/f26 [0,4194304] 0 2024-03-04T16:04:09.307 INFO:tasks.workunit.client.0.smithi116.stdout:3/395: write dc/d23/d84/d7c/f69 [889657,24478] 0 2024-03-04T16:04:09.308 INFO:tasks.workunit.client.0.smithi116.stdout:2/297: creat d7/d15/d35/f70 x:0 0 0 2024-03-04T16:04:09.312 INFO:tasks.workunit.client.0.smithi116.stdout:7/209: dwrite d3/d4/d8/d1a/f2e [0,4194304] 0 2024-03-04T16:04:09.312 INFO:tasks.workunit.client.0.smithi116.stdout:3/396: creat dc/df/d14/f89 x:0 0 0 2024-03-04T16:04:09.318 INFO:tasks.workunit.client.0.smithi116.stdout:2/298: dwrite d7/d13/f1e [0,4194304] 0 2024-03-04T16:04:09.319 INFO:tasks.workunit.client.0.smithi116.stdout:2/299: write d7/d13/f52 [462125,39432] 0 2024-03-04T16:04:09.322 INFO:tasks.workunit.client.0.smithi116.stdout:3/397: link dc/d34/d44/d6a/f77 dc/d23/d84/d7f/f8a 0 2024-03-04T16:04:09.325 INFO:tasks.workunit.client.0.smithi116.stdout:7/210: write d3/d4/d8/d1a/f20 [137668,62609] 0 2024-03-04T16:04:09.326 INFO:tasks.workunit.client.0.smithi116.stdout:3/398: read - dc/d23/d84/d7c/f62 zero size 2024-03-04T16:04:09.330 INFO:tasks.workunit.client.0.smithi116.stdout:2/300: dwrite d7/d29/d47/d64/f6f [0,4194304] 0 2024-03-04T16:04:09.330 INFO:tasks.workunit.client.0.smithi116.stdout:3/399: mkdir dc/d23/d84/d7f/d8b 0 2024-03-04T16:04:09.333 INFO:tasks.workunit.client.0.smithi116.stdout:7/211: rename d3/d4/d11/f2f to d3/d4/d8/d1a/f42 0 2024-03-04T16:04:09.334 INFO:tasks.workunit.client.0.smithi116.stdout:2/301: mkdir d7/d13/d3e/d71 0 2024-03-04T16:04:09.335 INFO:tasks.workunit.client.0.smithi116.stdout:7/212: chown d3/d4/d8/l2d 26350 1 2024-03-04T16:04:09.337 INFO:tasks.workunit.client.0.smithi116.stdout:3/400: creat dc/df/d14/f8c x:0 0 0 2024-03-04T16:04:09.338 INFO:tasks.workunit.client.0.smithi116.stdout:7/213: symlink d3/d4/d34/l43 0 2024-03-04T16:04:09.341 INFO:tasks.workunit.client.0.smithi116.stdout:2/302: rename d7/d15/d5e/l60 to d7/l72 0 2024-03-04T16:04:09.343 INFO:tasks.workunit.client.0.smithi116.stdout:3/401: rename dc/d34/d44/c67 to dc/d23/d84/d7c/c8d 0 2024-03-04T16:04:09.344 INFO:tasks.workunit.client.0.smithi116.stdout:7/214: dwrite d3/f18 [0,4194304] 0 2024-03-04T16:04:09.345 INFO:tasks.workunit.client.0.smithi116.stdout:7/215: readlink d3/d4/d8/d1a/l3a 0 2024-03-04T16:04:09.345 INFO:tasks.workunit.client.0.smithi116.stdout:2/303: symlink d7/d13/d5f/l73 0 2024-03-04T16:04:09.346 INFO:tasks.workunit.client.0.smithi116.stdout:3/402: creat dc/d23/d84/f8e x:0 0 0 2024-03-04T16:04:09.347 INFO:tasks.workunit.client.0.smithi116.stdout:7/216: creat d3/f44 x:0 0 0 2024-03-04T16:04:09.348 INFO:tasks.workunit.client.0.smithi116.stdout:3/403: write dc/df/d47/f61 [976070,32215] 0 2024-03-04T16:04:09.352 INFO:tasks.workunit.client.0.smithi116.stdout:2/304: symlink d7/d15/d5e/l74 0 2024-03-04T16:04:09.352 INFO:tasks.workunit.client.0.smithi116.stdout:3/404: truncate dc/df/d14/f72 750025 0 2024-03-04T16:04:09.359 INFO:tasks.workunit.client.0.smithi116.stdout:3/405: dwrite dc/d34/d44/d6a/f6b [0,4194304] 0 2024-03-04T16:04:09.359 INFO:tasks.workunit.client.0.smithi116.stdout:7/217: dread d3/d4/d8/d1a/f27 [4194304,4194304] 0 2024-03-04T16:04:09.361 INFO:tasks.workunit.client.0.smithi116.stdout:7/218: write d3/d4/d8/f40 [1018611,35920] 0 2024-03-04T16:04:09.362 INFO:tasks.workunit.client.0.smithi116.stdout:7/219: stat d3/d4/d8/f25 0 2024-03-04T16:04:09.363 INFO:tasks.workunit.client.0.smithi116.stdout:7/220: readlink d3/d4/d8/d1a/l30 0 2024-03-04T16:04:09.363 INFO:tasks.workunit.client.0.smithi116.stdout:3/406: unlink c3 0 2024-03-04T16:04:09.375 INFO:tasks.workunit.client.0.smithi116.stdout:6/319: sync 2024-03-04T16:04:09.378 INFO:tasks.workunit.client.0.smithi116.stdout:6/320: symlink d6/l6d 0 2024-03-04T16:04:09.381 INFO:tasks.workunit.client.0.smithi116.stdout:2/305: truncate d7/d29/d47/f5b 850526 0 2024-03-04T16:04:09.389 INFO:tasks.workunit.client.0.smithi116.stdout:3/407: dwrite dc/df/d14/f2b [0,4194304] 0 2024-03-04T16:04:09.396 INFO:tasks.workunit.client.0.smithi116.stdout:3/408: chown cb 33759957 1 2024-03-04T16:04:09.399 INFO:tasks.workunit.client.0.smithi116.stdout:2/306: dwrite d7/d13/f45 [0,4194304] 0 2024-03-04T16:04:09.401 INFO:tasks.workunit.client.0.smithi116.stdout:3/409: creat dc/f8f x:0 0 0 2024-03-04T16:04:09.410 INFO:tasks.workunit.client.0.smithi116.stdout:2/307: dwrite d7/f36 [0,4194304] 0 2024-03-04T16:04:09.413 INFO:tasks.workunit.client.0.smithi116.stdout:2/308: mkdir d7/d29/d47/d64/d75 0 2024-03-04T16:04:09.414 INFO:tasks.workunit.client.0.smithi116.stdout:2/309: readlink d7/d13/d5f/l73 0 2024-03-04T16:04:09.418 INFO:tasks.workunit.client.0.smithi116.stdout:2/310: mkdir d7/d2f/d76 0 2024-03-04T16:04:09.421 INFO:tasks.workunit.client.0.smithi116.stdout:2/311: mkdir d7/d29/d47/d64/d75/d77 0 2024-03-04T16:04:09.422 INFO:tasks.workunit.client.0.smithi116.stdout:2/312: write d7/f2b [540738,12695] 0 2024-03-04T16:04:09.424 INFO:tasks.workunit.client.0.smithi116.stdout:2/313: mkdir d7/d13/d3e/d78 0 2024-03-04T16:04:09.426 INFO:tasks.workunit.client.0.smithi116.stdout:2/314: creat d7/d1b/f79 x:0 0 0 2024-03-04T16:04:09.427 INFO:tasks.workunit.client.0.smithi116.stdout:2/315: chown d7/d15/d5e/f67 47 1 2024-03-04T16:04:09.428 INFO:tasks.workunit.client.0.smithi116.stdout:2/316: read - d7/d15/f59 zero size 2024-03-04T16:04:09.435 INFO:tasks.workunit.client.0.smithi116.stdout:2/317: dwrite d7/d29/d47/f49 [0,4194304] 0 2024-03-04T16:04:09.435 INFO:tasks.workunit.client.0.smithi116.stdout:3/410: dread dc/d23/f42 [0,4194304] 0 2024-03-04T16:04:09.436 INFO:tasks.workunit.client.0.smithi116.stdout:2/318: fsync d7/d1b/f21 0 2024-03-04T16:04:09.443 INFO:tasks.workunit.client.0.smithi116.stdout:3/411: unlink dc/df/f3c 0 2024-03-04T16:04:09.445 INFO:tasks.workunit.client.0.smithi116.stdout:3/412: truncate dc/d23/d84/d7c/f5c 1006674 0 2024-03-04T16:04:09.452 INFO:tasks.workunit.client.0.smithi116.stdout:2/319: link d7/d13/d5f/f69 d7/d29/d47/f7a 0 2024-03-04T16:04:09.456 INFO:tasks.workunit.client.0.smithi116.stdout:2/320: creat d7/d1b/f7b x:0 0 0 2024-03-04T16:04:09.460 INFO:tasks.workunit.client.0.smithi116.stdout:2/321: mkdir d7/d2f/d76/d7c 0 2024-03-04T16:04:09.462 INFO:tasks.workunit.client.0.smithi116.stdout:2/322: creat d7/d2f/d76/d7c/f7d x:0 0 0 2024-03-04T16:04:09.463 INFO:tasks.workunit.client.0.smithi116.stdout:2/323: chown d7/d1b/c2c 92959 1 2024-03-04T16:04:09.464 INFO:tasks.workunit.client.0.smithi116.stdout:2/324: write d7/d13/f1e [4794336,123177] 0 2024-03-04T16:04:09.468 INFO:tasks.workunit.client.0.smithi116.stdout:2/325: creat d7/d13/d3e/d78/f7e x:0 0 0 2024-03-04T16:04:09.474 INFO:tasks.workunit.client.0.smithi116.stdout:2/326: link c2 d7/d13/d3e/d71/c7f 0 2024-03-04T16:04:09.474 INFO:tasks.workunit.client.0.smithi116.stdout:2/327: readlink d7/d15/d5e/l74 0 2024-03-04T16:04:09.480 INFO:tasks.workunit.client.0.smithi116.stdout:3/413: dwrite dc/df/d14/f49 [4194304,4194304] 0 2024-03-04T16:04:09.484 INFO:tasks.workunit.client.0.smithi116.stdout:3/414: mknod dc/c90 0 2024-03-04T16:04:09.486 INFO:tasks.workunit.client.0.smithi116.stdout:3/415: readlink dc/df/l12 0 2024-03-04T16:04:09.489 INFO:tasks.workunit.client.0.smithi116.stdout:3/416: read dc/df/f7e [117519,2660] 0 2024-03-04T16:04:09.490 INFO:tasks.workunit.client.0.smithi116.stdout:3/417: chown dc/d23/d84/d7f/f8a 377695544 1 2024-03-04T16:04:09.491 INFO:tasks.workunit.client.0.smithi116.stdout:3/418: read - dc/d34/f7a zero size 2024-03-04T16:04:09.494 INFO:tasks.workunit.client.0.smithi116.stdout:2/328: truncate d7/d29/d47/d64/f6f 4166291 0 2024-03-04T16:04:09.506 INFO:tasks.workunit.client.0.smithi116.stdout:2/329: dwrite f6 [0,4194304] 0 2024-03-04T16:04:09.509 INFO:tasks.workunit.client.0.smithi116.stdout:2/330: unlink d7/f42 0 2024-03-04T16:04:09.509 INFO:tasks.workunit.client.0.smithi116.stdout:2/331: chown d7/d15/f59 48 1 2024-03-04T16:04:09.510 INFO:tasks.workunit.client.0.smithi116.stdout:2/332: chown d7/d13/f45 7 1 2024-03-04T16:04:09.517 INFO:tasks.workunit.client.0.smithi116.stdout:1/248: sync 2024-03-04T16:04:09.517 INFO:tasks.workunit.client.0.smithi116.stdout:4/313: sync 2024-03-04T16:04:09.518 INFO:tasks.workunit.client.0.smithi116.stdout:4/314: dread - d1/d3/dd/f5d zero size 2024-03-04T16:04:09.523 INFO:tasks.workunit.client.0.smithi116.stdout:4/315: rename d1/d3/dd/l5b to d1/d3/de/d47/l70 0 2024-03-04T16:04:09.524 INFO:tasks.workunit.client.0.smithi116.stdout:1/249: getdents d5/d11 0 2024-03-04T16:04:09.529 INFO:tasks.workunit.client.0.smithi116.stdout:2/333: dwrite d7/d29/d47/f6e [4194304,4194304] 0 2024-03-04T16:04:09.530 INFO:tasks.workunit.client.0.smithi116.stdout:4/316: getdents d1/d3/dd/d24/d51 0 2024-03-04T16:04:09.535 INFO:tasks.workunit.client.0.smithi116.stdout:4/317: creat d1/d3/dd/d24/d51/f71 x:0 0 0 2024-03-04T16:04:09.536 INFO:tasks.workunit.client.0.smithi116.stdout:2/334: symlink d7/d29/d47/d64/d75/d77/l80 0 2024-03-04T16:04:09.536 INFO:tasks.workunit.client.0.smithi116.stdout:1/250: truncate d5/d9/f41 489500 0 2024-03-04T16:04:09.537 INFO:tasks.workunit.client.0.smithi116.stdout:2/335: chown d7/d15/l1d 26 1 2024-03-04T16:04:09.537 INFO:tasks.workunit.client.0.smithi116.stdout:1/251: read - d5/d11/f20 zero size 2024-03-04T16:04:09.538 INFO:tasks.workunit.client.0.smithi116.stdout:4/318: mkdir d1/d3/d15/d20/d1c/d72 0 2024-03-04T16:04:09.540 INFO:tasks.workunit.client.0.smithi116.stdout:1/252: mkdir d5/d9/d21/d27/d32/d35/d55 0 2024-03-04T16:04:09.542 INFO:tasks.workunit.client.0.smithi116.stdout:1/253: read - d5/d9/d21/d27/f4b zero size 2024-03-04T16:04:09.542 INFO:tasks.workunit.client.0.smithi116.stdout:4/319: symlink d1/d3/de/l73 0 2024-03-04T16:04:09.544 INFO:tasks.workunit.client.0.smithi116.stdout:2/336: dwrite d7/d1b/f7b [0,4194304] 0 2024-03-04T16:04:09.546 INFO:tasks.workunit.client.0.smithi116.stdout:4/320: mkdir d1/d6e/d74 0 2024-03-04T16:04:09.547 INFO:tasks.workunit.client.0.smithi116.stdout:2/337: chown d7/d25/d41/f5a 4 1 2024-03-04T16:04:09.551 INFO:tasks.workunit.client.0.smithi116.stdout:4/321: dread - d1/d3/dd/f3c zero size 2024-03-04T16:04:09.553 INFO:tasks.workunit.client.0.smithi116.stdout:1/254: dwrite d5/f29 [0,4194304] 0 2024-03-04T16:04:09.558 INFO:tasks.workunit.client.0.smithi116.stdout:2/338: rename d7/d1b to d7/d81 0 2024-03-04T16:04:09.559 INFO:tasks.workunit.client.0.smithi116.stdout:2/339: readlink d7/l1f 0 2024-03-04T16:04:09.563 INFO:tasks.workunit.client.0.smithi116.stdout:4/322: link d1/f32 d1/f75 0 2024-03-04T16:04:09.567 INFO:tasks.workunit.client.0.smithi116.stdout:1/255: link d5/l33 d5/d11/d42/l56 0 2024-03-04T16:04:09.570 INFO:tasks.workunit.client.0.smithi116.stdout:2/340: rename d7/cc to d7/d29/d47/d64/d75/c82 0 2024-03-04T16:04:09.570 INFO:tasks.workunit.client.0.smithi116.stdout:2/341: readlink d7/d15/d5e/l74 0 2024-03-04T16:04:09.571 INFO:tasks.workunit.client.0.smithi116.stdout:1/256: mknod d5/d9/d21/d24/c57 0 2024-03-04T16:04:09.572 INFO:tasks.workunit.client.0.smithi116.stdout:1/257: chown f2 294102 1 2024-03-04T16:04:09.573 INFO:tasks.workunit.client.0.smithi116.stdout:4/323: link d1/d3/de/f12 d1/d3/d40/d52/f76 0 2024-03-04T16:04:09.574 INFO:tasks.workunit.client.0.smithi116.stdout:4/324: stat d1/d3/d15/d20/d1c/f6a 0 2024-03-04T16:04:09.582 INFO:tasks.workunit.client.0.smithi116.stdout:1/258: getdents d5/d11/d42 0 2024-03-04T16:04:09.583 INFO:tasks.workunit.client.0.smithi116.stdout:4/325: getdents d1/d3/d40 0 2024-03-04T16:04:09.583 INFO:tasks.workunit.client.0.smithi116.stdout:1/259: write f4 [1001662,11796] 0 2024-03-04T16:04:09.592 INFO:tasks.workunit.client.0.smithi116.stdout:2/342: write d7/d13/d5f/f69 [2228211,76573] 0 2024-03-04T16:04:09.592 INFO:tasks.workunit.client.0.smithi116.stdout:4/326: link d1/d6c/c6d d1/d3/de/c77 0 2024-03-04T16:04:09.596 INFO:tasks.workunit.client.0.smithi116.stdout:4/327: rename d1/d3/dd/f3c to d1/d3/d15/d20/d65/f78 0 2024-03-04T16:04:09.599 INFO:tasks.workunit.client.0.smithi116.stdout:2/343: getdents d7/d29/d47/d64/d75 0 2024-03-04T16:04:09.601 INFO:tasks.workunit.client.0.smithi116.stdout:4/328: dwrite d1/d3/d15/d20/f68 [0,4194304] 0 2024-03-04T16:04:09.603 INFO:tasks.workunit.client.0.smithi116.stdout:4/329: write d1/d3/de/f10 [1953155,7702] 0 2024-03-04T16:04:09.603 INFO:tasks.workunit.client.0.smithi116.stdout:2/344: unlink d7/c37 0 2024-03-04T16:04:09.606 INFO:tasks.workunit.client.0.smithi116.stdout:2/345: truncate d7/d2f/d76/d7c/f7d 703101 0 2024-03-04T16:04:09.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.605+0000 7f6e1160d700 1 -- 172.21.15.116:0/3563293262 --> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e0c069480 con 0x7f6df406ce30 2024-03-04T16:04:09.611 INFO:tasks.workunit.client.0.smithi116.stdout:2/346: dwrite d7/d29/d47/f49 [0,4194304] 0 2024-03-04T16:04:09.613 INFO:tasks.workunit.client.0.smithi116.stdout:4/330: chown d1/d3/de/c77 1630 1 2024-03-04T16:04:09.614 INFO:tasks.workunit.client.0.smithi116.stdout:2/347: mknod d7/d2f/c83 0 2024-03-04T16:04:09.620 INFO:tasks.workunit.client.0.smithi116.stdout:2/348: creat d7/d13/d3e/d78/f84 x:0 0 0 2024-03-04T16:04:09.620 INFO:tasks.workunit.client.0.smithi116.stdout:2/349: dread - d7/d81/f39 zero size 2024-03-04T16:04:09.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.624+0000 7f6df3fff700 1 -- 172.21.15.116:0/3563293262 <== mgr.24387 v2:172.21.15.152:6828/2881662614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6e0c069480 con 0x7f6df406ce30 2024-03-04T16:04:09.625 INFO:tasks.workunit.client.0.smithi116.stdout:2/350: dwrite d7/d15/f20 [0,4194304] 0 2024-03-04T16:04:09.625 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:04:09.625 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (7m) 23s ago 8m 25.2M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:04:09.625 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (8m) 23s ago 8m 7578k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:04:09.625 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (7m) 17s ago 7m 7574k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:04:09.625 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (8m) 23s ago 8m 6665k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (7m) 17s ago 7m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (7m) 23s ago 8m 86.9M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (2m) 23s ago 2m 17.0M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (3m) 23s ago 3m 13.2M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (2m) 17s ago 2m 14.8M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (3m) 17s ago 3m 131M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:9283,8765,8443 running (10m) 23s ago 10m 437M - 18.2.1 5be31c24972a 083879ca4bf4 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (60s) 17s ago 7m 489M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6756dd312323 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (10m) 23s ago 10m 48.7M 2048M 18.2.1 5be31c24972a 60e0bf118c4d 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (7m) 17s ago 7m 41.4M 2048M 18.2.1 5be31c24972a 52450d235516 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (8m) 23s ago 8m 15.5M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:04:09.626 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (7m) 17s ago 7m 16.5M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:04:09.627 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (6m) 23s ago 6m 101M 1216M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:04:09.627 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (6m) 23s ago 6m 104M 1216M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:04:09.627 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (5m) 23s ago 5m 90.0M 1216M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:04:09.627 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (5m) 17s ago 5m 98.4M 2240M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:04:09.627 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (5m) 17s ago 5m 97.4M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:04:09.627 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (4m) 17s ago 4m 89.1M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:04:09.627 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (6m) 23s ago 8m 62.2M - 2.43.0 a07b618ecd1d 31240d908f24 2024-03-04T16:04:09.628 INFO:tasks.workunit.client.0.smithi116.stdout:2/351: mknod d7/d2f/d76/d7c/c85 0 2024-03-04T16:04:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.627+0000 7f6e1160d700 1 -- 172.21.15.116:0/3563293262 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6df406ce30 msgr2=0x7f6df406f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.627+0000 7f6e1160d700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6df406ce30 0x7f6df406f2f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c100b50 tx=0x7f6dfc009190 comp rx=0 tx=0).stop 2024-03-04T16:04:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.627+0000 7f6e1160d700 1 -- 172.21.15.116:0/3563293262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0c100cf0 msgr2=0x7f6e0c196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.627+0000 7f6e1160d700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0c100cf0 0x7f6e0c196960 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000b5c0 tx=0x7f6e0000bd70 comp rx=0 tx=0).stop 2024-03-04T16:04:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.627+0000 7f6e1160d700 1 -- 172.21.15.116:0/3563293262 shutdown_connections 2024-03-04T16:04:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.627+0000 7f6e1160d700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f6df406ce30 0x7f6df406f2f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.627+0000 7f6e1160d700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0ffaf0 0x7f6e0c196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.627+0000 7f6e1160d700 1 --2- 172.21.15.116:0/3563293262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0c100cf0 0x7f6e0c196960 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.627+0000 7f6e1160d700 1 -- 172.21.15.116:0/3563293262 >> 172.21.15.116:0/3563293262 conn(0x7f6e0c0fb070 msgr2=0x7f6e0c068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.628+0000 7f6e1160d700 1 -- 172.21.15.116:0/3563293262 shutdown_connections 2024-03-04T16:04:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.628+0000 7f6e1160d700 1 -- 172.21.15.116:0/3563293262 wait complete. 2024-03-04T16:04:09.630 INFO:tasks.workunit.client.0.smithi116.stdout:2/352: stat d7/d29/f4a 0 2024-03-04T16:04:09.631 INFO:tasks.workunit.client.0.smithi116.stdout:2/353: fdatasync d7/d29/d47/f51 0 2024-03-04T16:04:09.633 INFO:tasks.workunit.client.0.smithi116.stdout:4/331: dwrite d1/d3/dd/d24/f29 [0,4194304] 0 2024-03-04T16:04:09.641 INFO:tasks.workunit.client.0.smithi116.stdout:2/354: truncate d7/d81/f21 2046116 0 2024-03-04T16:04:09.645 INFO:tasks.workunit.client.0.smithi116.stdout:4/332: write d1/d3/d40/d52/f76 [764388,119204] 0 2024-03-04T16:04:09.653 INFO:tasks.workunit.client.0.smithi116.stdout:4/333: dwrite d1/d3/dd/d45/f4c [4194304,4194304] 0 2024-03-04T16:04:09.656 INFO:tasks.workunit.client.0.smithi116.stdout:4/334: mknod d1/d6e/c79 0 2024-03-04T16:04:09.660 INFO:tasks.workunit.client.0.smithi116.stdout:4/335: read d1/d3/f44 [4103882,30012] 0 2024-03-04T16:04:09.663 INFO:tasks.workunit.client.0.smithi116.stdout:4/336: rmdir d1/d3/d15/d20 39 2024-03-04T16:04:09.664 INFO:tasks.workunit.client.0.smithi116.stdout:4/337: readlink d1/d3/d15/d20/d1c/l36 0 2024-03-04T16:04:09.666 INFO:tasks.workunit.client.0.smithi116.stdout:4/338: mknod d1/d3/de/d47/c7a 0 2024-03-04T16:04:09.670 INFO:tasks.workunit.client.0.smithi116.stdout:4/339: creat d1/d6e/d74/f7b x:0 0 0 2024-03-04T16:04:09.678 INFO:tasks.workunit.client.0.smithi116.stdout:4/340: write d1/f2c [919582,57731] 0 2024-03-04T16:04:09.680 INFO:tasks.workunit.client.0.smithi116.stdout:4/341: mknod d1/d6e/d74/c7c 0 2024-03-04T16:04:09.686 INFO:tasks.workunit.client.0.smithi116.stdout:4/342: dwrite d1/d3/de/f23 [4194304,4194304] 0 2024-03-04T16:04:09.689 INFO:tasks.workunit.client.0.smithi116.stdout:4/343: fdatasync d1/d3/d15/d20/f1e 0 2024-03-04T16:04:09.690 INFO:tasks.workunit.client.0.smithi116.stdout:4/344: write d1/d3/d15/d20/f4e [700195,79937] 0 2024-03-04T16:04:09.697 INFO:tasks.workunit.client.0.smithi116.stdout:4/345: dwrite d1/d3/de/f23 [0,4194304] 0 2024-03-04T16:04:09.699 INFO:tasks.workunit.client.0.smithi116.stdout:8/320: sync 2024-03-04T16:04:09.699 INFO:tasks.workunit.client.0.smithi116.stdout:5/312: sync 2024-03-04T16:04:09.699 INFO:tasks.workunit.client.0.smithi116.stdout:8/321: readlink d2/d13/d1e/l68 0 2024-03-04T16:04:09.700 INFO:tasks.workunit.client.0.smithi116.stdout:4/346: rmdir d1/d3/d15/d20/d65 39 2024-03-04T16:04:09.702 INFO:tasks.workunit.client.0.smithi116.stdout:8/322: unlink d2/d13/d1e/d5a/l63 0 2024-03-04T16:04:09.704 INFO:tasks.workunit.client.0.smithi116.stdout:8/323: read d2/de/d16/f34 [1225557,119803] 0 2024-03-04T16:04:09.705 INFO:tasks.workunit.client.0.smithi116.stdout:4/347: write d1/d3/de/d47/f60 [77613,115168] 0 2024-03-04T16:04:09.706 INFO:tasks.workunit.client.0.smithi116.stdout:4/348: write d1/d3/dd/d24/f3d [39616,47638] 0 2024-03-04T16:04:09.707 INFO:tasks.workunit.client.0.smithi116.stdout:4/349: chown d1/f75 24607707 1 2024-03-04T16:04:09.708 INFO:tasks.workunit.client.0.smithi116.stdout:8/324: unlink d2/de/d16/f23 0 2024-03-04T16:04:09.711 INFO:tasks.workunit.client.0.smithi116.stdout:4/350: creat d1/d3/f7d x:0 0 0 2024-03-04T16:04:09.711 INFO:tasks.workunit.client.0.smithi116.stdout:8/325: mkdir d2/d13/d3c/d46/d6e 0 2024-03-04T16:04:09.712 INFO:tasks.workunit.client.0.smithi116.stdout:8/326: stat d2/de/l28 0 2024-03-04T16:04:09.715 INFO:tasks.workunit.client.0.smithi116.stdout:8/327: write d2/d38/d55/f65 [103263,111231] 0 2024-03-04T16:04:09.715 INFO:tasks.workunit.client.0.smithi116.stdout:4/351: dread d1/d3/d15/d20/f56 [0,4194304] 0 2024-03-04T16:04:09.716 INFO:tasks.workunit.client.0.smithi116.stdout:8/328: chown d2/d13/d1e/d42/f5d 16889441 1 2024-03-04T16:04:09.717 INFO:tasks.workunit.client.0.smithi116.stdout:4/352: mkdir d1/d6c/d7e 0 2024-03-04T16:04:09.718 INFO:tasks.workunit.client.0.smithi116.stdout:4/353: readlink d1/d3/dd/d24/d51/l55 0 2024-03-04T16:04:09.719 INFO:tasks.workunit.client.0.smithi116.stdout:8/329: creat d2/d13/d1e/d5a/f6f x:0 0 0 2024-03-04T16:04:09.720 INFO:tasks.workunit.client.0.smithi116.stdout:5/313: truncate dd/f11 3539237 0 2024-03-04T16:04:09.721 INFO:tasks.workunit.client.0.smithi116.stdout:5/314: chown dd/d1f/f29 9864446 1 2024-03-04T16:04:09.723 INFO:tasks.workunit.client.0.smithi116.stdout:8/330: creat d2/d13/d3c/d46/d4f/f70 x:0 0 0 2024-03-04T16:04:09.723 INFO:tasks.workunit.client.0.smithi116.stdout:4/354: dread d1/d3/dd/d24/f29 [0,4194304] 0 2024-03-04T16:04:09.726 INFO:tasks.workunit.client.0.smithi116.stdout:4/355: mknod d1/d6e/c7f 0 2024-03-04T16:04:09.728 INFO:tasks.workunit.client.0.smithi116.stdout:4/356: write d1/d3/f61 [140670,112242] 0 2024-03-04T16:04:09.729 INFO:tasks.workunit.client.0.smithi116.stdout:5/315: dwrite f4 [4194304,4194304] 0 2024-03-04T16:04:09.730 INFO:tasks.workunit.client.0.smithi116.stdout:4/357: write d1/d3/dd/f31 [1020576,117584] 0 2024-03-04T16:04:09.735 INFO:tasks.workunit.client.0.smithi116.stdout:4/358: unlink d1/d3/d15/d20/d1c/l36 0 2024-03-04T16:04:09.742 INFO:tasks.workunit.client.0.smithi116.stdout:5/316: link dd/fe dd/d1f/d5c/d5e/f67 0 2024-03-04T16:04:09.743 INFO:tasks.workunit.client.0.smithi116.stdout:5/317: write dd/d1f/f4d [1490096,99622] 0 2024-03-04T16:04:09.748 INFO:tasks.workunit.client.0.smithi116.stdout:8/331: truncate d2/de/fa 44847 0 2024-03-04T16:04:09.752 INFO:tasks.workunit.client.0.smithi116.stdout:5/318: link dd/c12 dd/d14/d2d/c68 0 2024-03-04T16:04:09.753 INFO:tasks.workunit.client.0.smithi116.stdout:4/359: write d1/d3/dd/f4a [22067,83609] 0 2024-03-04T16:04:09.757 INFO:tasks.workunit.client.0.smithi116.stdout:5/319: creat dd/d1f/f69 x:0 0 0 2024-03-04T16:04:09.758 INFO:tasks.workunit.client.0.smithi116.stdout:5/320: write dd/d14/d2d/d31/f65 [478926,73280] 0 2024-03-04T16:04:09.761 INFO:tasks.workunit.client.0.smithi116.stdout:8/332: getdents d2/de/d3e 0 2024-03-04T16:04:09.762 INFO:tasks.workunit.client.0.smithi116.stdout:8/333: fsync d2/de/d3e/f4e 0 2024-03-04T16:04:09.762 INFO:tasks.workunit.client.0.smithi116.stdout:4/360: getdents d1/d3/d15/d20/d1c/d6f 0 2024-03-04T16:04:09.763 INFO:tasks.workunit.client.0.smithi116.stdout:8/334: write d2/d13/d5e/f6b [296306,101710] 0 2024-03-04T16:04:09.764 INFO:tasks.workunit.client.0.smithi116.stdout:4/361: read d1/d3/dd/d45/f4c [5443218,48910] 0 2024-03-04T16:04:09.768 INFO:tasks.workunit.client.0.smithi116.stdout:5/321: dread dd/d14/d2d/d31/f3c [0,4194304] 0 2024-03-04T16:04:09.769 INFO:tasks.workunit.client.0.smithi116.stdout:8/335: dread d2/d13/d3c/f69 [0,4194304] 0 2024-03-04T16:04:09.773 INFO:tasks.workunit.client.0.smithi116.stdout:5/322: creat dd/d14/d2d/d31/f6a x:0 0 0 2024-03-04T16:04:09.777 INFO:tasks.workunit.client.0.smithi116.stdout:4/362: truncate d1/f41 1155237 0 2024-03-04T16:04:09.778 INFO:tasks.workunit.client.0.smithi116.stdout:4/363: fsync d1/d3/de/f26 0 2024-03-04T16:04:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.777+0000 7f453c510700 1 -- 172.21.15.116:0/2199131764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45340ffab0 msgr2=0x7f45340ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.777+0000 7f453c510700 1 --2- 172.21.15.116:0/2199131764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45340ffab0 0x7f45340ffed0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4530009910 tx=0x7f4530009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.778+0000 7f453c510700 1 -- 172.21.15.116:0/2199131764 shutdown_connections 2024-03-04T16:04:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.778+0000 7f453c510700 1 --2- 172.21.15.116:0/2199131764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4534100cb0 0x7f4534101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.778+0000 7f453c510700 1 --2- 172.21.15.116:0/2199131764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45340ffab0 0x7f45340ffed0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.778+0000 7f453c510700 1 -- 172.21.15.116:0/2199131764 >> 172.21.15.116:0/2199131764 conn(0x7f45340fb050 msgr2=0x7f45340fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.778+0000 7f453c510700 1 -- 172.21.15.116:0/2199131764 shutdown_connections 2024-03-04T16:04:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.778+0000 7f453c510700 1 -- 172.21.15.116:0/2199131764 wait complete. 2024-03-04T16:04:09.780 INFO:tasks.workunit.client.0.smithi116.stdout:4/364: fsync d1/d3/de/f16 0 2024-03-04T16:04:09.780 INFO:tasks.workunit.client.0.smithi116.stdout:5/323: rename dd/d45/l4b to dd/d1f/l6b 0 2024-03-04T16:04:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.778+0000 7f453c510700 1 Processor -- start 2024-03-04T16:04:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.778+0000 7f453c510700 1 -- start start 2024-03-04T16:04:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f453c510700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45340ffab0 0x7f45341915f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f453c510700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4534100cb0 0x7f4534191b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f453c510700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453418e0e0 con 0x7f4534100cb0 2024-03-04T16:04:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f453c510700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453418e250 con 0x7f45340ffab0 2024-03-04T16:04:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f4539aab700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4534100cb0 0x7f4534191b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f4539aab700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4534100cb0 0x7f4534191b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50290/0 (socket says 172.21.15.116:50290) 2024-03-04T16:04:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f4539aab700 1 -- 172.21.15.116:0/2475810645 learned_addr learned my addr 172.21.15.116:0/2475810645 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f453a2ac700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45340ffab0 0x7f45341915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f4539aab700 1 -- 172.21.15.116:0/2475810645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45340ffab0 msgr2=0x7f45341915f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f4539aab700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45340ffab0 0x7f45341915f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f4539aab700 1 -- 172.21.15.116:0/2475810645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45300095f0 con 0x7f4534100cb0 2024-03-04T16:04:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.779+0000 7f4539aab700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4534100cb0 0x7f4534191b30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f452400e920 tx=0x7f452400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.780+0000 7f452b7fe700 1 -- 172.21.15.116:0/2475810645 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f452400c9d0 con 0x7f4534100cb0 2024-03-04T16:04:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.780+0000 7f452b7fe700 1 -- 172.21.15.116:0/2475810645 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4524010430 con 0x7f4534100cb0 2024-03-04T16:04:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.780+0000 7f453c510700 1 -- 172.21.15.116:0/2475810645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453418e530 con 0x7f4534100cb0 2024-03-04T16:04:09.782 INFO:tasks.workunit.client.0.smithi116.stdout:8/336: link d2/d38/f3b d2/d38/f71 0 2024-03-04T16:04:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.780+0000 7f452b7fe700 1 -- 172.21.15.116:0/2475810645 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4524018650 con 0x7f4534100cb0 2024-03-04T16:04:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.780+0000 7f453c510700 1 -- 172.21.15.116:0/2475810645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453418ea80 con 0x7f4534100cb0 2024-03-04T16:04:09.784 INFO:tasks.workunit.client.0.smithi116.stdout:4/365: dread d1/d3/de/d47/f60 [0,4194304] 0 2024-03-04T16:04:09.784 INFO:tasks.workunit.client.0.smithi116.stdout:8/337: stat d2/d13/d5e/f6b 0 2024-03-04T16:04:09.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.781+0000 7f453c510700 1 -- 172.21.15.116:0/2475810645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f453418eca0 con 0x7f4534100cb0 2024-03-04T16:04:09.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.782+0000 7f452b7fe700 1 -- 172.21.15.116:0/2475810645 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) v1 ==== 90928+0+0 (secure 0 0 0) 0x7f45240105a0 con 0x7f4534100cb0 2024-03-04T16:04:09.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.782+0000 7f452b7fe700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f452006ce30 0x7f452006f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:09.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.782+0000 7f452b7fe700 1 -- 172.21.15.116:0/2475810645 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f4524014070 con 0x7f4534100cb0 2024-03-04T16:04:09.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.782+0000 7f453a2ac700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f452006ce30 0x7f452006f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:09.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.783+0000 7f453a2ac700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f452006ce30 0x7f452006f2f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4530009fd0 tx=0x7f4530005a90 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:09.785 INFO:tasks.workunit.client.0.smithi116.stdout:8/338: symlink d2/d13/d3c/l72 0 2024-03-04T16:04:09.786 INFO:tasks.workunit.client.0.smithi116.stdout:8/339: readlink d2/de/l1a 0 2024-03-04T16:04:09.786 INFO:tasks.workunit.client.0.smithi116.stdout:4/366: dread d1/d3/dd/d45/f4b [0,4194304] 0 2024-03-04T16:04:09.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:09.788+0000 7f452b7fe700 1 -- 172.21.15.116:0/2475810645 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f4524052a70 con 0x7f4534100cb0 2024-03-04T16:04:09.796 INFO:tasks.workunit.client.0.smithi116.stdout:4/367: rename d1/d3/dd/f31 to d1/f80 0 2024-03-04T16:04:09.799 INFO:tasks.workunit.client.0.smithi116.stdout:5/324: dread dd/fe [0,4194304] 0 2024-03-04T16:04:09.800 INFO:tasks.workunit.client.0.smithi116.stdout:8/340: dread d2/de/f10 [0,4194304] 0 2024-03-04T16:04:09.800 INFO:tasks.workunit.client.0.smithi116.stdout:9/278: sync 2024-03-04T16:04:09.800 INFO:tasks.workunit.client.0.smithi116.stdout:0/278: sync 2024-03-04T16:04:09.801 INFO:tasks.workunit.client.0.smithi116.stdout:4/368: rename d1/d6c/d7e to d1/d3/d15/d20/d1c/d62/d81 0 2024-03-04T16:04:09.802 INFO:tasks.workunit.client.0.smithi116.stdout:8/341: truncate d2/d13/d3c/d46/f6d 82903 0 2024-03-04T16:04:09.802 INFO:tasks.workunit.client.0.smithi116.stdout:0/279: read da/d16/d19/d27/f34 [1094736,55406] 0 2024-03-04T16:04:09.804 INFO:tasks.workunit.client.0.smithi116.stdout:0/280: write da/d16/d28/d3d/d41/f59 [557331,2182] 0 2024-03-04T16:04:09.806 INFO:tasks.workunit.client.0.smithi116.stdout:9/279: mknod d2/d26/d28/c5a 0 2024-03-04T16:04:09.812 INFO:tasks.workunit.client.0.smithi116.stdout:8/342: creat d2/d13/d5e/f73 x:0 0 0 2024-03-04T16:04:09.813 INFO:tasks.workunit.client.0.smithi116.stdout:8/343: dread - d2/d38/d41/f6c zero size 2024-03-04T16:04:09.813 INFO:tasks.workunit.client.0.smithi116.stdout:9/280: fdatasync d2/d26/f44 0 2024-03-04T16:04:09.819 INFO:tasks.workunit.client.0.smithi116.stdout:8/344: dread d2/de/d16/f2c [0,4194304] 0 2024-03-04T16:04:09.820 INFO:tasks.workunit.client.0.smithi116.stdout:8/345: dread d2/de/d3e/f2a [0,4194304] 0 2024-03-04T16:04:09.820 INFO:tasks.workunit.client.0.smithi116.stdout:4/369: creat d1/d3/d15/d20/f82 x:0 0 0 2024-03-04T16:04:09.825 INFO:tasks.workunit.client.0.smithi116.stdout:9/281: truncate d2/d33/f3a 164942 0 2024-03-04T16:04:09.825 INFO:tasks.workunit.client.0.smithi116.stdout:9/282: read - d2/d49/f42 zero size 2024-03-04T16:04:09.826 INFO:tasks.workunit.client.0.smithi116.stdout:8/346: mknod d2/d13/c74 0 2024-03-04T16:04:09.827 INFO:tasks.workunit.client.0.smithi116.stdout:9/283: dwrite d2/d19/d27/d1f/f2f [0,4194304] 0 2024-03-04T16:04:09.829 INFO:tasks.workunit.client.0.smithi116.stdout:8/347: mkdir d2/d13/d1e/d75 0 2024-03-04T16:04:09.830 INFO:tasks.workunit.client.0.smithi116.stdout:8/348: write d2/de/f12 [3240057,35335] 0 2024-03-04T16:04:09.833 INFO:tasks.workunit.client.0.smithi116.stdout:8/349: truncate d2/d38/d41/f6c 812944 0 2024-03-04T16:04:09.834 INFO:tasks.workunit.client.0.smithi116.stdout:9/284: dread d2/d19/d27/d3b/f3d [0,4194304] 0 2024-03-04T16:04:09.837 INFO:tasks.workunit.client.0.smithi116.stdout:0/281: truncate da/d16/d28/d40/d48/f52 3090949 0 2024-03-04T16:04:09.840 INFO:tasks.workunit.client.0.smithi116.stdout:8/350: dwrite d2/f60 [0,4194304] 0 2024-03-04T16:04:09.844 INFO:tasks.workunit.client.0.smithi116.stdout:0/282: mknod da/d16/d20/c5f 0 2024-03-04T16:04:09.850 INFO:tasks.workunit.client.0.smithi116.stdout:8/351: dwrite d2/de/d16/f1b [0,4194304] 0 2024-03-04T16:04:09.850 INFO:tasks.workunit.client.0.smithi116.stdout:9/285: rename d2/d19/d27/d1f to d2/d19/d27/d3b/d4a/d5b 0 2024-03-04T16:04:09.851 INFO:tasks.workunit.client.0.smithi116.stdout:9/286: fsync d2/d49/f42 0 2024-03-04T16:04:09.855 INFO:tasks.workunit.client.0.smithi116.stdout:0/283: dwrite da/d16/d20/f21 [4194304,4194304] 0 2024-03-04T16:04:09.855 INFO:tasks.workunit.client.0.smithi116.stdout:9/287: dread d2/f15 [4194304,4194304] 0 2024-03-04T16:04:09.857 INFO:tasks.workunit.client.0.smithi116.stdout:0/284: rename da/d16/d19/d27 to da/d16/d19/d27/d60 22 2024-03-04T16:04:09.865 INFO:tasks.workunit.client.0.smithi116.stdout:8/352: truncate d2/de/fa 356835 0 2024-03-04T16:04:09.865 INFO:tasks.workunit.client.0.smithi116.stdout:9/288: chown d2/d19/c50 104278727 1 2024-03-04T16:04:09.868 INFO:tasks.workunit.client.0.smithi116.stdout:8/353: mknod d2/d13/d3c/d46/c76 0 2024-03-04T16:04:09.872 INFO:tasks.workunit.client.0.smithi116.stdout:9/289: dwrite d2/d33/f54 [0,4194304] 0 2024-03-04T16:04:09.897 INFO:tasks.workunit.client.0.smithi116.stdout:0/285: dwrite da/f1a [0,4194304] 0 2024-03-04T16:04:09.901 INFO:tasks.workunit.client.0.smithi116.stdout:8/354: dread d2/de/d16/f18 [0,4194304] 0 2024-03-04T16:04:09.902 INFO:tasks.workunit.client.0.smithi116.stdout:0/286: symlink da/d16/d20/d42/l61 0 2024-03-04T16:04:09.909 INFO:tasks.workunit.client.0.smithi116.stdout:0/287: mknod da/d16/d28/c62 0 2024-03-04T16:04:09.912 INFO:tasks.workunit.client.0.smithi116.stdout:8/355: getdents d2/d38/d41 0 2024-03-04T16:04:09.912 INFO:tasks.workunit.client.0.smithi116.stdout:0/288: rmdir da/d16/d28/d40 39 2024-03-04T16:04:09.914 INFO:tasks.workunit.client.0.smithi116.stdout:9/290: dwrite d2/d33/f3a [0,4194304] 0 2024-03-04T16:04:09.916 INFO:tasks.workunit.client.0.smithi116.stdout:9/291: write d2/d19/d27/f38 [444117,38395] 0 2024-03-04T16:04:09.919 INFO:tasks.workunit.client.0.smithi116.stdout:8/356: dread d2/de/d3e/f50 [0,4194304] 0 2024-03-04T16:04:09.922 INFO:tasks.workunit.client.0.smithi116.stdout:0/289: dread da/d16/d28/d40/f5c [0,4194304] 0 2024-03-04T16:04:09.924 INFO:tasks.workunit.client.0.smithi116.stdout:0/290: write da/d16/d20/f3b [78469,91953] 0 2024-03-04T16:04:09.925 INFO:tasks.workunit.client.0.smithi116.stdout:8/357: rmdir d2/d13/d3c/d46 39 2024-03-04T16:04:09.929 INFO:tasks.workunit.client.0.smithi116.stdout:0/291: dwrite da/f1e [0,4194304] 0 2024-03-04T16:04:09.937 INFO:tasks.workunit.client.0.smithi116.stdout:9/292: symlink d2/d19/d27/d3b/d4a/d5b/l5c 0 2024-03-04T16:04:09.941 INFO:tasks.workunit.client.0.smithi116.stdout:8/358: mkdir d2/d13/d3c/d77 0 2024-03-04T16:04:09.941 INFO:tasks.workunit.client.0.smithi116.stdout:8/359: dread - d2/d38/f3b zero size 2024-03-04T16:04:09.943 INFO:tasks.workunit.client.0.smithi116.stdout:0/292: creat da/d16/d20/d53/f63 x:0 0 0 2024-03-04T16:04:09.943 INFO:tasks.workunit.client.0.smithi116.stdout:0/293: chown da/c14 54645402 1 2024-03-04T16:04:09.947 INFO:tasks.workunit.client.0.smithi116.stdout:0/294: truncate da/fe 1593035 0 2024-03-04T16:04:09.952 INFO:tasks.workunit.client.0.smithi116.stdout:8/360: rename d2/de/d16/d36 to d2/de/d16/d78 0 2024-03-04T16:04:09.954 INFO:tasks.workunit.client.0.smithi116.stdout:6/321: sync 2024-03-04T16:04:09.954 INFO:tasks.workunit.client.0.smithi116.stdout:7/221: sync 2024-03-04T16:04:09.954 INFO:tasks.workunit.client.0.smithi116.stdout:3/419: sync 2024-03-04T16:04:09.954 INFO:tasks.workunit.client.0.smithi116.stdout:2/355: sync 2024-03-04T16:04:09.954 INFO:tasks.workunit.client.0.smithi116.stdout:1/260: sync 2024-03-04T16:04:09.956 INFO:tasks.workunit.client.0.smithi116.stdout:1/261: chown d5/f29 3317 1 2024-03-04T16:04:09.956 INFO:tasks.workunit.client.0.smithi116.stdout:2/356: fsync d7/d15/d35/f70 0 2024-03-04T16:04:09.957 INFO:tasks.workunit.client.0.smithi116.stdout:0/295: rename da/d16/d28/f3c to da/d16/d28/d3d/f64 0 2024-03-04T16:04:09.958 INFO:tasks.workunit.client.0.smithi116.stdout:2/357: write d7/d15/f38 [500577,16261] 0 2024-03-04T16:04:09.961 INFO:tasks.workunit.client.0.smithi116.stdout:2/358: fdatasync d7/d15/d35/f48 0 2024-03-04T16:04:09.962 INFO:tasks.workunit.client.0.smithi116.stdout:8/361: dwrite d2/d13/d1e/f45 [0,4194304] 0 2024-03-04T16:04:09.963 INFO:tasks.workunit.client.0.smithi116.stdout:0/296: write da/d16/d19/d27/f34 [1179946,24067] 0 2024-03-04T16:04:09.963 INFO:tasks.workunit.client.0.smithi116.stdout:6/322: mknod d6/de/d25/c6e 0 2024-03-04T16:04:09.964 INFO:tasks.workunit.client.0.smithi116.stdout:9/293: dread d2/d19/f2a [0,4194304] 0 2024-03-04T16:04:09.966 INFO:tasks.workunit.client.0.smithi116.stdout:2/359: write d7/f36 [3784098,58351] 0 2024-03-04T16:04:09.969 INFO:tasks.workunit.client.0.smithi116.stdout:1/262: dwrite d5/f38 [0,4194304] 0 2024-03-04T16:04:09.969 INFO:tasks.workunit.client.0.smithi116.stdout:6/323: truncate d6/de/f52 246421 0 2024-03-04T16:04:09.970 INFO:tasks.workunit.client.0.smithi116.stdout:1/263: chown d5/d11/f54 91926373 1 2024-03-04T16:04:09.970 INFO:tasks.workunit.client.0.smithi116.stdout:6/324: fsync d6/de/d25/d36/f1a 0 2024-03-04T16:04:09.971 INFO:tasks.workunit.client.0.smithi116.stdout:8/362: creat d2/d13/d1e/d75/f79 x:0 0 0 2024-03-04T16:04:09.978 INFO:tasks.workunit.client.0.smithi116.stdout:2/360: chown d7/d13/d3e/d71/c7f 2 1 2024-03-04T16:04:09.978 INFO:tasks.workunit.client.0.smithi116.stdout:2/361: dread - d7/d25/f62 zero size 2024-03-04T16:04:09.982 INFO:tasks.workunit.client.0.smithi116.stdout:6/325: creat d6/de/d15/f6f x:0 0 0 2024-03-04T16:04:09.984 INFO:tasks.workunit.client.0.smithi116.stdout:6/326: truncate d6/de/f3b 873223 0 2024-03-04T16:04:09.984 INFO:tasks.workunit.client.0.smithi116.stdout:6/327: stat d6/de/c1c 0 2024-03-04T16:04:09.985 INFO:tasks.workunit.client.0.smithi116.stdout:7/222: dread d3/d4/d8/f1c [0,4194304] 0 2024-03-04T16:04:09.985 INFO:tasks.workunit.client.0.smithi116.stdout:6/328: write d6/de/d25/d36/f1a [849483,46709] 0 2024-03-04T16:04:09.987 INFO:tasks.workunit.client.0.smithi116.stdout:6/329: write d6/de/d25/d36/d4e/d50/f5f [988319,116551] 0 2024-03-04T16:04:09.990 INFO:tasks.workunit.client.0.smithi116.stdout:0/297: link da/d16/d28/f2e da/d16/d20/f65 0 2024-03-04T16:04:09.993 INFO:tasks.workunit.client.0.smithi116.stdout:2/362: creat d7/d29/d47/d64/f86 x:0 0 0 2024-03-04T16:04:09.994 INFO:tasks.workunit.client.0.smithi116.stdout:2/363: write d7/d13/d3e/d78/f84 [902619,104863] 0 2024-03-04T16:04:09.995 INFO:tasks.workunit.client.0.smithi116.stdout:1/264: mkdir d5/d9/d21/d27/d32/d35/d55/d58 0 2024-03-04T16:04:09.996 INFO:tasks.workunit.client.0.smithi116.stdout:1/265: chown d5/d11 13726 1 2024-03-04T16:04:09.996 INFO:tasks.workunit.client.0.smithi116.stdout:2/364: write d7/d13/d3e/d78/f7e [1008907,65347] 0 2024-03-04T16:04:10.000 INFO:tasks.workunit.client.0.smithi116.stdout:1/266: dread d5/f38 [0,4194304] 0 2024-03-04T16:04:10.001 INFO:tasks.workunit.client.0.smithi116.stdout:9/294: dread d2/d19/d27/d3b/d4a/f4c [4194304,4194304] 0 2024-03-04T16:04:10.010 INFO:tasks.workunit.client.0.smithi116.stdout:1/267: symlink d5/d9/d21/d27/l59 0 2024-03-04T16:04:10.016 INFO:tasks.workunit.client.0.smithi116.stdout:2/365: dwrite d7/d15/f30 [0,4194304] 0 2024-03-04T16:04:10.016 INFO:tasks.workunit.client.0.smithi116.stdout:9/295: rmdir d2/d19 39 2024-03-04T16:04:10.016 INFO:tasks.workunit.client.0.smithi116.stdout:2/366: write d7/d15/f38 [29698,128717] 0 2024-03-04T16:04:10.017 INFO:tasks.workunit.client.0.smithi116.stdout:8/363: getdents d2/d13/d1e/d5a 0 2024-03-04T16:04:10.020 INFO:tasks.workunit.client.0.smithi116.stdout:7/223: creat d3/d4/d8/f45 x:0 0 0 2024-03-04T16:04:10.020 INFO:tasks.workunit.client.0.smithi116.stdout:9/296: dwrite d2/d33/f46 [0,4194304] 0 2024-03-04T16:04:10.028 INFO:tasks.workunit.client.0.smithi116.stdout:1/268: unlink d5/d11/f20 0 2024-03-04T16:04:10.032 INFO:tasks.workunit.client.0.smithi116.stdout:2/367: rename d7/d13/f45 to d7/d29/d47/d64/f87 0 2024-03-04T16:04:10.035 INFO:tasks.workunit.client.0.smithi116.stdout:8/364: mkdir d2/de/d16/d78/d29/d7a 0 2024-03-04T16:04:10.037 INFO:tasks.workunit.client.0.smithi116.stdout:9/297: write d2/f15 [1535898,18788] 0 2024-03-04T16:04:10.040 INFO:tasks.workunit.client.0.smithi116.stdout:8/365: dread d2/d13/d1e/f45 [0,4194304] 0 2024-03-04T16:04:10.041 INFO:tasks.workunit.client.0.smithi116.stdout:8/366: stat d2/d13/d1e/c58 0 2024-03-04T16:04:10.042 INFO:tasks.workunit.client.0.smithi116.stdout:8/367: readlink d2/de/d3e/l20 0 2024-03-04T16:04:10.043 INFO:tasks.workunit.client.0.smithi116.stdout:0/298: dwrite da/d16/d19/d27/f55 [0,4194304] 0 2024-03-04T16:04:10.044 INFO:tasks.workunit.client.0.smithi116.stdout:0/299: chown da/d16/d28/d40/d5d 1423931003 1 2024-03-04T16:04:10.044 INFO:tasks.workunit.client.0.smithi116.stdout:0/300: stat da/l2a 0 2024-03-04T16:04:10.048 INFO:tasks.workunit.client.0.smithi116.stdout:7/224: link d3/d4/d8/d1a/f32 d3/d4/d8/d1a/f46 0 2024-03-04T16:04:10.054 INFO:tasks.workunit.client.0.smithi116.stdout:7/225: dwrite d3/f18 [0,4194304] 0 2024-03-04T16:04:10.061 INFO:tasks.workunit.client.0.smithi116.stdout:1/269: rename f4 to d5/f5a 0 2024-03-04T16:04:10.064 INFO:tasks.workunit.client.0.smithi116.stdout:7/226: rmdir d3/d4/d34 39 2024-03-04T16:04:10.072 INFO:tasks.workunit.client.0.smithi116.stdout:2/368: rename d7/d81/f6d to d7/d15/d35/d63/f88 0 2024-03-04T16:04:10.072 INFO:tasks.workunit.client.0.smithi116.stdout:0/301: write da/d16/d28/f2e [352938,99504] 0 2024-03-04T16:04:10.072 INFO:tasks.workunit.client.0.smithi116.stdout:2/369: chown c0 22 1 2024-03-04T16:04:10.073 INFO:tasks.workunit.client.0.smithi116.stdout:2/370: chown d7/d2f/f50 277756 1 2024-03-04T16:04:10.074 INFO:tasks.workunit.client.0.smithi116.stdout:1/270: stat d5/d9/d21/l34 0 2024-03-04T16:04:10.075 INFO:tasks.workunit.client.0.smithi116.stdout:2/371: write d7/d25/d41/f53 [165772,11472] 0 2024-03-04T16:04:10.079 INFO:tasks.workunit.client.0.smithi116.stdout:8/368: truncate d2/de/d16/f18 5574677 0 2024-03-04T16:04:10.082 INFO:tasks.workunit.client.0.smithi116.stdout:1/271: readlink d5/d11/d42/l56 0 2024-03-04T16:04:10.084 INFO:tasks.workunit.client.0.smithi116.stdout:2/372: mknod d7/d81/c89 0 2024-03-04T16:04:10.088 INFO:tasks.workunit.client.0.smithi116.stdout:8/369: mkdir d2/de/d3e/d7b 0 2024-03-04T16:04:10.089 INFO:tasks.workunit.client.0.smithi116.stdout:7/227: dwrite d3/fd [0,4194304] 0 2024-03-04T16:04:10.090 INFO:tasks.workunit.client.0.smithi116.stdout:8/370: dread d2/d13/d3c/f69 [0,4194304] 0 2024-03-04T16:04:10.091 INFO:tasks.workunit.client.0.smithi116.stdout:1/272: unlink d5/f1c 0 2024-03-04T16:04:10.091 INFO:tasks.workunit.client.0.smithi116.stdout:2/373: symlink d7/d2f/d6c/l8a 0 2024-03-04T16:04:10.092 INFO:tasks.workunit.client.0.smithi116.stdout:9/298: link d2/d19/d27/d3b/d4a/d5b/l22 d2/d19/l5d 0 2024-03-04T16:04:10.092 INFO:tasks.workunit.client.0.smithi116.stdout:2/374: stat d7/d81/f1c 0 2024-03-04T16:04:10.093 INFO:tasks.workunit.client.0.smithi116.stdout:9/299: write d2/d49/f42 [485223,107122] 0 2024-03-04T16:04:10.095 INFO:tasks.workunit.client.0.smithi116.stdout:0/302: rename da/d16/d28/f2e to da/d16/d28/d3d/f66 0 2024-03-04T16:04:10.095 INFO:tasks.workunit.client.0.smithi116.stdout:2/375: rename d7 to d7/d29/d47/d64/d75/d77/d8b 22 2024-03-04T16:04:10.096 INFO:tasks.workunit.client.0.smithi116.stdout:0/303: chown da/d16/d20/f5b 85 1 2024-03-04T16:04:10.101 INFO:tasks.workunit.client.0.smithi116.stdout:0/304: dwrite da/ff [0,4194304] 0 2024-03-04T16:04:10.104 INFO:tasks.workunit.client.0.smithi116.stdout:8/371: symlink d2/d13/d3c/d46/d4f/l7c 0 2024-03-04T16:04:10.107 INFO:tasks.workunit.client.0.smithi116.stdout:0/305: symlink da/d16/d19/d27/l67 0 2024-03-04T16:04:10.108 INFO:tasks.workunit.client.0.smithi116.stdout:2/376: dwrite d7/d13/f1a [0,4194304] 0 2024-03-04T16:04:10.109 INFO:tasks.workunit.client.0.smithi116.stdout:7/228: link d3/d4/d11/c31 d3/d4/d11/c47 0 2024-03-04T16:04:10.114 INFO:tasks.workunit.client.0.smithi116.stdout:2/377: symlink d7/d15/l8c 0 2024-03-04T16:04:10.115 INFO:tasks.workunit.client.0.smithi116.stdout:7/229: dread d3/d4/d8/f40 [0,4194304] 0 2024-03-04T16:04:10.116 INFO:tasks.workunit.client.0.smithi116.stdout:7/230: chown d3/d4/d8/d1a/f2e 115718 1 2024-03-04T16:04:10.118 INFO:tasks.workunit.client.0.smithi116.stdout:8/372: link d2/de/d16/d78/d29/l52 d2/d13/d3c/d77/l7d 0 2024-03-04T16:04:10.119 INFO:tasks.workunit.client.0.smithi116.stdout:2/378: mknod d7/d2f/d76/d7c/c8d 0 2024-03-04T16:04:10.122 INFO:tasks.workunit.client.0.smithi116.stdout:0/306: truncate da/d16/d28/d40/d48/f52 1008647 0 2024-03-04T16:04:10.124 INFO:tasks.workunit.client.0.smithi116.stdout:8/373: truncate d2/d38/f3b 325398 0 2024-03-04T16:04:10.125 INFO:tasks.workunit.client.0.smithi116.stdout:8/374: dread - d2/d13/d3c/d46/d4f/f70 zero size 2024-03-04T16:04:10.132 INFO:tasks.workunit.client.0.smithi116.stdout:8/375: rename d2/de/c35 to d2/d13/d1e/c7e 0 2024-03-04T16:04:10.140 INFO:tasks.workunit.client.0.smithi116.stdout:8/376: getdents d2 0 2024-03-04T16:04:10.140 INFO:tasks.workunit.client.0.smithi116.stdout:8/377: chown d2/fd 427277 1 2024-03-04T16:04:10.143 INFO:tasks.workunit.client.0.smithi116.stdout:8/378: symlink d2/de/d16/d78/d29/d7a/l7f 0 2024-03-04T16:04:10.155 INFO:tasks.workunit.client.0.smithi116.stdout:7/231: rmdir d3/d4/d11 39 2024-03-04T16:04:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:7/232: write d3/d4/d11/f38 [2855549,95023] 0 2024-03-04T16:04:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:7/233: dwrite d3/d4/d8/d1a/f28 [0,4194304] 0 2024-03-04T16:04:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:7/234: symlink d3/d4/d11/l48 0 2024-03-04T16:04:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:7/235: chown d3/c2a 147 1 2024-03-04T16:04:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:2/379: truncate d7/f24 2026303 0 2024-03-04T16:04:10.179 INFO:tasks.workunit.client.0.smithi116.stdout:0/307: write da/d16/d28/d3d/f64 [3133860,41255] 0 2024-03-04T16:04:10.179 INFO:tasks.workunit.client.0.smithi116.stdout:2/380: dread d7/d13/d3e/d78/f84 [0,4194304] 0 2024-03-04T16:04:10.182 INFO:tasks.workunit.client.0.smithi116.stdout:4/370: sync 2024-03-04T16:04:10.182 INFO:tasks.workunit.client.0.smithi116.stdout:5/325: sync 2024-03-04T16:04:10.183 INFO:tasks.workunit.client.0.smithi116.stdout:2/381: symlink d7/d13/l8e 0 2024-03-04T16:04:10.183 INFO:tasks.workunit.client.0.smithi116.stdout:0/308: dread da/d16/d19/f1d [0,4194304] 0 2024-03-04T16:04:10.184 INFO:tasks.workunit.client.0.smithi116.stdout:2/382: read d7/d13/f1a [3978063,98127] 0 2024-03-04T16:04:10.184 INFO:tasks.workunit.client.0.smithi116.stdout:0/309: chown da/d16/d19/d27 290920901 1 2024-03-04T16:04:10.184 INFO:tasks.workunit.client.0.smithi116.stdout:5/326: truncate dd/d1f/f69 810253 0 2024-03-04T16:04:10.186 INFO:tasks.workunit.client.0.smithi116.stdout:0/310: write da/d16/d28/d3d/f64 [3285916,56115] 0 2024-03-04T16:04:10.188 INFO:tasks.workunit.client.0.smithi116.stdout:8/379: write d2/fd [1134308,127818] 0 2024-03-04T16:04:10.191 INFO:tasks.workunit.client.0.smithi116.stdout:5/327: creat dd/d1f/d55/f6c x:0 0 0 2024-03-04T16:04:10.192 INFO:tasks.workunit.client.0.smithi116.stdout:7/236: dread d3/f1e [0,4194304] 0 2024-03-04T16:04:10.192 INFO:tasks.workunit.client.0.smithi116.stdout:4/371: mknod d1/d3/dd/d45/d50/c83 0 2024-03-04T16:04:10.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.194+0000 7f453c510700 1 -- 172.21.15.116:0/2475810645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f45341a2b40 con 0x7f4534100cb0 2024-03-04T16:04:10.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.194+0000 7f452b7fe700 1 -- 172.21.15.116:0/2475810645 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7f452405a060 con 0x7f4534100cb0 2024-03-04T16:04:10.195 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:04:10.195 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:04:10.195 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-03-04T16:04:10.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:04:10.195 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-03-04T16:04:10.196 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:04:10.197 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:04:10.197 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:04:10.197 INFO:tasks.workunit.client.0.smithi116.stdout:0/311: truncate da/d16/d28/d40/f5c 1126382 0 2024-03-04T16:04:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 -- 172.21.15.116:0/2475810645 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f452006ce30 msgr2=0x7f452006f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f452006ce30 0x7f452006f2f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4530009fd0 tx=0x7f4530005a90 comp rx=0 tx=0).stop 2024-03-04T16:04:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 -- 172.21.15.116:0/2475810645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4534100cb0 msgr2=0x7f4534191b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4534100cb0 0x7f4534191b30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f452400e920 tx=0x7f452400ec30 comp rx=0 tx=0).stop 2024-03-04T16:04:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 -- 172.21.15.116:0/2475810645 shutdown_connections 2024-03-04T16:04:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f452006ce30 0x7f452006f2f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45340ffab0 0x7f45341915f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 --2- 172.21.15.116:0/2475810645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4534100cb0 0x7f4534191b30 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 -- 172.21.15.116:0/2475810645 >> 172.21.15.116:0/2475810645 conn(0x7f45340fb050 msgr2=0x7f4534068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 -- 172.21.15.116:0/2475810645 shutdown_connections 2024-03-04T16:04:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.197+0000 7f453c510700 1 -- 172.21.15.116:0/2475810645 wait complete. 2024-03-04T16:04:10.202 INFO:tasks.workunit.client.0.smithi116.stdout:4/372: fdatasync d1/d3/dd/d45/f4b 0 2024-03-04T16:04:10.204 INFO:tasks.workunit.client.0.smithi116.stdout:0/312: chown da/d16/d28/c37 1515444 1 2024-03-04T16:04:10.208 INFO:tasks.workunit.client.0.smithi116.stdout:5/328: mkdir dd/d6d 0 2024-03-04T16:04:10.213 INFO:tasks.workunit.client.0.smithi116.stdout:2/383: getdents d7/d25 0 2024-03-04T16:04:10.214 INFO:tasks.workunit.client.0.smithi116.stdout:2/384: chown d7/d15/d35/c5d 544896495 1 2024-03-04T16:04:10.215 INFO:tasks.workunit.client.0.smithi116.stdout:2/385: write d7/d29/d47/f51 [1705863,65219] 0 2024-03-04T16:04:10.217 INFO:tasks.workunit.client.0.smithi116.stdout:5/329: rename cb to dd/d1f/d5c/d5e/c6e 0 2024-03-04T16:04:10.218 INFO:tasks.workunit.client.0.smithi116.stdout:5/330: chown dd/d14/d2d/l33 1002421435 1 2024-03-04T16:04:10.219 INFO:tasks.workunit.client.0.smithi116.stdout:4/373: dread d1/d3/dd/f4a [0,4194304] 0 2024-03-04T16:04:10.220 INFO:tasks.workunit.client.0.smithi116.stdout:4/374: write d1/d3/de/f30 [4130787,121928] 0 2024-03-04T16:04:10.221 INFO:tasks.workunit.client.0.smithi116.stdout:2/386: dread d7/f36 [0,4194304] 0 2024-03-04T16:04:10.222 INFO:tasks.workunit.client.0.smithi116.stdout:2/387: write d7/d29/f2d [3121462,118599] 0 2024-03-04T16:04:10.222 INFO:tasks.workunit.client.0.smithi116.stdout:0/313: creat da/d16/d19/f68 x:0 0 0 2024-03-04T16:04:10.223 INFO:tasks.workunit.client.0.smithi116.stdout:4/375: write d1/d3/d40/f4f [109721,77422] 0 2024-03-04T16:04:10.224 INFO:tasks.workunit.client.0.smithi116.stdout:5/331: mknod dd/d14/d2d/c6f 0 2024-03-04T16:04:10.224 INFO:tasks.workunit.client.0.smithi116.stdout:0/314: read - da/d16/d20/f5b zero size 2024-03-04T16:04:10.225 INFO:tasks.workunit.client.0.smithi116.stdout:4/376: write d1/fb [346448,51632] 0 2024-03-04T16:04:10.227 INFO:tasks.workunit.client.0.smithi116.stdout:0/315: fdatasync da/d16/d20/f3b 0 2024-03-04T16:04:10.232 INFO:tasks.workunit.client.0.smithi116.stdout:2/388: unlink d7/d29/d47/d64/d75/d77/l80 0 2024-03-04T16:04:10.233 INFO:tasks.workunit.client.0.smithi116.stdout:0/316: write da/d16/d19/d27/f34 [1167613,8289] 0 2024-03-04T16:04:10.233 INFO:tasks.workunit.client.0.smithi116.stdout:5/332: symlink dd/d1f/d5c/l70 0 2024-03-04T16:04:10.234 INFO:tasks.workunit.client.0.smithi116.stdout:8/380: dwrite d2/d13/d1e/f45 [0,4194304] 0 2024-03-04T16:04:10.234 INFO:tasks.workunit.client.0.smithi116.stdout:7/237: dwrite d3/f1e [4194304,4194304] 0 2024-03-04T16:04:10.236 INFO:tasks.workunit.client.0.smithi116.stdout:8/381: write d2/d38/d41/f6c [1206171,91397] 0 2024-03-04T16:04:10.241 INFO:tasks.workunit.client.0.smithi116.stdout:0/317: readlink da/d16/l23 0 2024-03-04T16:04:10.243 INFO:tasks.workunit.client.0.smithi116.stdout:2/389: creat d7/d15/d35/f8f x:0 0 0 2024-03-04T16:04:10.244 INFO:tasks.workunit.client.0.smithi116.stdout:2/390: write d7/d15/f30 [1774754,279] 0 2024-03-04T16:04:10.244 INFO:tasks.workunit.client.0.smithi116.stdout:5/333: mkdir dd/d14/d2d/d59/d71 0 2024-03-04T16:04:10.254 INFO:tasks.workunit.client.0.smithi116.stdout:5/334: stat dd/f11 0 2024-03-04T16:04:10.256 INFO:tasks.workunit.client.0.smithi116.stdout:0/318: symlink da/d16/d19/l69 0 2024-03-04T16:04:10.257 INFO:tasks.workunit.client.0.smithi116.stdout:5/335: mkdir dd/d1f/d72 0 2024-03-04T16:04:10.259 INFO:tasks.workunit.client.0.smithi116.stdout:7/238: creat d3/d4/f49 x:0 0 0 2024-03-04T16:04:10.260 INFO:tasks.workunit.client.0.smithi116.stdout:0/319: dread da/d16/d28/d3d/d41/f59 [0,4194304] 0 2024-03-04T16:04:10.264 INFO:tasks.workunit.client.0.smithi116.stdout:4/377: getdents d1/d3/d15/d20/d65 0 2024-03-04T16:04:10.264 INFO:tasks.workunit.client.0.smithi116.stdout:4/378: stat d1/d6c/f38 0 2024-03-04T16:04:10.266 INFO:tasks.workunit.client.0.smithi116.stdout:8/382: link d2/de/d16/f2c d2/d13/d3c/f80 0 2024-03-04T16:04:10.267 INFO:tasks.workunit.client.0.smithi116.stdout:8/383: fdatasync d2/fd 0 2024-03-04T16:04:10.269 INFO:tasks.workunit.client.0.smithi116.stdout:0/320: mknod da/d16/d28/d3d/d41/c6a 0 2024-03-04T16:04:10.270 INFO:tasks.workunit.client.0.smithi116.stdout:8/384: dread d2/de/d3e/f4e [0,4194304] 0 2024-03-04T16:04:10.271 INFO:tasks.workunit.client.0.smithi116.stdout:7/239: mkdir d3/d4/d8/d4a 0 2024-03-04T16:04:10.273 INFO:tasks.workunit.client.0.smithi116.stdout:5/336: rename dd/d14 to dd/d1f/d55/d73 0 2024-03-04T16:04:10.274 INFO:tasks.workunit.client.0.smithi116.stdout:8/385: dread d2/d13/d1e/f45 [0,4194304] 0 2024-03-04T16:04:10.277 INFO:tasks.workunit.client.0.smithi116.stdout:4/379: mkdir d1/d3/d40/d52/d84 0 2024-03-04T16:04:10.279 INFO:tasks.workunit.client.0.smithi116.stdout:4/380: chown d1/d3/d15/d20/l53 722650966 1 2024-03-04T16:04:10.280 INFO:tasks.workunit.client.0.smithi116.stdout:4/381: fsync d1/fb 0 2024-03-04T16:04:10.281 INFO:tasks.workunit.client.0.smithi116.stdout:8/386: truncate d2/de/d16/f22 1698104 0 2024-03-04T16:04:10.282 INFO:tasks.workunit.client.0.smithi116.stdout:0/321: dwrite da/d16/d19/d27/f56 [4194304,4194304] 0 2024-03-04T16:04:10.282 INFO:tasks.workunit.client.0.smithi116.stdout:4/382: fdatasync d1/d3/d15/d20/d1c/f6a 0 2024-03-04T16:04:10.283 INFO:tasks.workunit.client.0.smithi116.stdout:7/240: mknod d3/d4/d8/c4b 0 2024-03-04T16:04:10.289 INFO:tasks.workunit.client.0.smithi116.stdout:5/337: dread dd/d1f/d55/d73/f3f [0,4194304] 0 2024-03-04T16:04:10.290 INFO:tasks.workunit.client.0.smithi116.stdout:5/338: chown dd/d1f/d55/d73/d2d/d31/f65 19631 1 2024-03-04T16:04:10.294 INFO:tasks.workunit.client.0.smithi116.stdout:0/322: mknod da/c6b 0 2024-03-04T16:04:10.295 INFO:tasks.workunit.client.0.smithi116.stdout:0/323: stat da/d16/d19/d27/f34 0 2024-03-04T16:04:10.303 INFO:tasks.workunit.client.0.smithi116.stdout:0/324: unlink da/d16/d19/l69 0 2024-03-04T16:04:10.304 INFO:tasks.workunit.client.0.smithi116.stdout:3/420: sync 2024-03-04T16:04:10.304 INFO:tasks.workunit.client.0.smithi116.stdout:6/330: sync 2024-03-04T16:04:10.305 INFO:tasks.workunit.client.0.smithi116.stdout:4/383: getdents d1/d3/d40 0 2024-03-04T16:04:10.307 INFO:tasks.workunit.client.0.smithi116.stdout:8/387: getdents d2/de/d3e 0 2024-03-04T16:04:10.307 INFO:tasks.workunit.client.0.smithi116.stdout:5/339: dread dd/d1f/f41 [0,4194304] 0 2024-03-04T16:04:10.308 INFO:tasks.workunit.client.0.smithi116.stdout:0/325: symlink da/d16/d19/d27/l6c 0 2024-03-04T16:04:10.308 INFO:tasks.workunit.client.0.smithi116.stdout:5/340: fdatasync dd/d1f/f5f 0 2024-03-04T16:04:10.309 INFO:tasks.workunit.client.0.smithi116.stdout:0/326: fsync da/d16/d19/d27/f56 0 2024-03-04T16:04:10.311 INFO:tasks.workunit.client.0.smithi116.stdout:6/331: chown d6/c42 1 1 2024-03-04T16:04:10.312 INFO:tasks.workunit.client.0.smithi116.stdout:4/384: rmdir d1/d3/dd/d24 39 2024-03-04T16:04:10.313 INFO:tasks.workunit.client.0.smithi116.stdout:6/332: fsync d6/de/d15/f6f 0 2024-03-04T16:04:10.314 INFO:tasks.workunit.client.0.smithi116.stdout:3/421: truncate dc/df/d47/f7d 765377 0 2024-03-04T16:04:10.315 INFO:tasks.workunit.client.0.smithi116.stdout:4/385: read d1/d3/dd/f4a [2508032,8622] 0 2024-03-04T16:04:10.315 INFO:tasks.workunit.client.0.smithi116.stdout:8/388: dwrite d2/de/f8 [4194304,4194304] 0 2024-03-04T16:04:10.316 INFO:tasks.workunit.client.0.smithi116.stdout:3/422: chown dc/f70 11 1 2024-03-04T16:04:10.317 INFO:tasks.workunit.client.0.smithi116.stdout:3/423: stat dc/df/d14/l30 0 2024-03-04T16:04:10.317 INFO:tasks.workunit.client.0.smithi116.stdout:8/389: write d2/fd [3710158,54930] 0 2024-03-04T16:04:10.318 INFO:tasks.workunit.client.0.smithi116.stdout:6/333: truncate d6/de/d29/d3a/f46 1301950 0 2024-03-04T16:04:10.318 INFO:tasks.workunit.client.0.smithi116.stdout:3/424: stat dc/df/d14 0 2024-03-04T16:04:10.319 INFO:tasks.workunit.client.0.smithi116.stdout:7/241: getdents d3/d4/d34 0 2024-03-04T16:04:10.322 INFO:tasks.workunit.client.0.smithi116.stdout:4/386: creat d1/f85 x:0 0 0 2024-03-04T16:04:10.323 INFO:tasks.workunit.client.0.smithi116.stdout:0/327: dwrite da/d16/d20/f21 [0,4194304] 0 2024-03-04T16:04:10.323 INFO:tasks.workunit.client.0.smithi116.stdout:5/341: link dd/d45/f49 dd/d1f/d55/d73/d2d/d31/f74 0 2024-03-04T16:04:10.327 INFO:tasks.workunit.client.0.smithi116.stdout:3/425: mkdir dc/df/d47/d91 0 2024-03-04T16:04:10.328 INFO:tasks.workunit.client.0.smithi116.stdout:6/334: symlink d6/de/d25/d4a/d63/l70 0 2024-03-04T16:04:10.329 INFO:tasks.workunit.client.0.smithi116.stdout:7/242: mknod d3/d4/d11/c4c 0 2024-03-04T16:04:10.333 INFO:tasks.workunit.client.0.smithi116.stdout:4/387: rename d1/d3/d15/d20/f68 to d1/d6e/f86 0 2024-03-04T16:04:10.334 INFO:tasks.workunit.client.0.smithi116.stdout:8/390: truncate d2/d38/f71 256508 0 2024-03-04T16:04:10.334 INFO:tasks.workunit.client.0.smithi116.stdout:4/388: stat d1/d3/de 0 2024-03-04T16:04:10.337 INFO:tasks.workunit.client.0.smithi116.stdout:6/335: dwrite d6/de/f1b [0,4194304] 0 2024-03-04T16:04:10.338 INFO:tasks.workunit.client.0.smithi116.stdout:7/243: creat d3/d4/d8/d1a/f4d x:0 0 0 2024-03-04T16:04:10.339 INFO:tasks.workunit.client.0.smithi116.stdout:6/336: chown d6/de/c16 232958 1 2024-03-04T16:04:10.340 INFO:tasks.workunit.client.0.smithi116.stdout:7/244: write d3/d4/d8/f35 [2991838,100880] 0 2024-03-04T16:04:10.341 INFO:tasks.workunit.client.0.smithi116.stdout:0/328: rename da/d16/d28/d3d/d41/f46 to da/d16/d28/d40/d5d/f6d 0 2024-03-04T16:04:10.341 INFO:tasks.workunit.client.0.smithi116.stdout:6/337: truncate d6/d56/f4c 762976 0 2024-03-04T16:04:10.341 INFO:tasks.workunit.client.0.smithi116.stdout:0/329: readlink da/d16/d28/l38 0 2024-03-04T16:04:10.343 INFO:tasks.workunit.client.0.smithi116.stdout:4/389: mknod d1/d3/d15/d20/d65/c87 0 2024-03-04T16:04:10.344 INFO:tasks.workunit.client.0.smithi116.stdout:0/330: write da/d16/d19/d27/f55 [1612599,3804] 0 2024-03-04T16:04:10.346 INFO:tasks.workunit.client.0.smithi116.stdout:8/391: mknod d2/de/d16/d78/d29/d7a/c81 0 2024-03-04T16:04:10.348 INFO:tasks.workunit.client.0.smithi116.stdout:5/342: dread f4 [0,4194304] 0 2024-03-04T16:04:10.351 INFO:tasks.workunit.client.0.smithi116.stdout:3/426: creat dc/df/f92 x:0 0 0 2024-03-04T16:04:10.352 INFO:tasks.workunit.client.0.smithi116.stdout:7/245: creat d3/f4e x:0 0 0 2024-03-04T16:04:10.356 INFO:tasks.workunit.client.0.smithi116.stdout:6/338: creat d6/de/d25/d4a/f71 x:0 0 0 2024-03-04T16:04:10.358 INFO:tasks.workunit.client.0.smithi116.stdout:6/339: write d6/f45 [474780,11977] 0 2024-03-04T16:04:10.359 INFO:tasks.workunit.client.0.smithi116.stdout:7/246: dread f2 [0,4194304] 0 2024-03-04T16:04:10.360 INFO:tasks.workunit.client.0.smithi116.stdout:3/427: dwrite dc/f16 [0,4194304] 0 2024-03-04T16:04:10.361 INFO:tasks.workunit.client.0.smithi116.stdout:8/392: symlink d2/d4a/l82 0 2024-03-04T16:04:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.362+0000 7f3f82d5f700 1 -- 172.21.15.116:0/1196507257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c103f70 msgr2=0x7f3f7c1043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.363+0000 7f3f82d5f700 1 --2- 172.21.15.116:0/1196507257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c103f70 0x7f3f7c1043f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009870 tx=0x7f3f68009b80 comp rx=0 tx=0).stop 2024-03-04T16:04:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.363+0000 7f3f82d5f700 1 -- 172.21.15.116:0/1196507257 shutdown_connections 2024-03-04T16:04:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.363+0000 7f3f82d5f700 1 --2- 172.21.15.116:0/1196507257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c103f70 0x7f3f7c1043f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.363+0000 7f3f82d5f700 1 --2- 172.21.15.116:0/1196507257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f7c102e10 0x7f3f7c103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.363+0000 7f3f82d5f700 1 -- 172.21.15.116:0/1196507257 >> 172.21.15.116:0/1196507257 conn(0x7f3f7c0fe3d0 msgr2=0x7f3f7c1007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:10.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.364+0000 7f3f82d5f700 1 -- 172.21.15.116:0/1196507257 shutdown_connections 2024-03-04T16:04:10.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.364+0000 7f3f82d5f700 1 -- 172.21.15.116:0/1196507257 wait complete. 2024-03-04T16:04:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.365+0000 7f3f82d5f700 1 Processor -- start 2024-03-04T16:04:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.365+0000 7f3f82d5f700 1 -- start start 2024-03-04T16:04:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.365+0000 7f3f82d5f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c102e10 0x7f3f7c19c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.365+0000 7f3f82d5f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f7c103f70 0x7f3f7c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.365+0000 7f3f82d5f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c19d4a0 con 0x7f3f7c103f70 2024-03-04T16:04:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.365+0000 7f3f82d5f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c19d5e0 con 0x7f3f7c102e10 2024-03-04T16:04:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.366+0000 7f3f7bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f7c103f70 0x7f3f7c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.366+0000 7f3f7bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f7c103f70 0x7f3f7c19ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50308/0 (socket says 172.21.15.116:50308) 2024-03-04T16:04:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.366+0000 7f3f7bfff700 1 -- 172.21.15.116:0/3843926793 learned_addr learned my addr 172.21.15.116:0/3843926793 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.366+0000 7f3f80afb700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c102e10 0x7f3f7c19c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.366+0000 7f3f7bfff700 1 -- 172.21.15.116:0/3843926793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c102e10 msgr2=0x7f3f7c19c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:10.370 INFO:tasks.workunit.client.0.smithi116.stdout:4/390: dwrite d1/d3/dd/f4a [4194304,4194304] 0 2024-03-04T16:04:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.366+0000 7f3f7bfff700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c102e10 0x7f3f7c19c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.366+0000 7f3f7bfff700 1 -- 172.21.15.116:0/3843926793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f68009520 con 0x7f3f7c103f70 2024-03-04T16:04:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.366+0000 7f3f7bfff700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f7c103f70 0x7f3f7c19ce80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009f50 tx=0x7f3f6800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.367+0000 7f3f79ffb700 1 -- 172.21.15.116:0/3843926793 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f6801c070 con 0x7f3f7c103f70 2024-03-04T16:04:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.367+0000 7f3f79ffb700 1 -- 172.21.15.116:0/3843926793 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3f6800be80 con 0x7f3f7c103f70 2024-03-04T16:04:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.367+0000 7f3f79ffb700 1 -- 172.21.15.116:0/3843926793 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f680173f0 con 0x7f3f7c103f70 2024-03-04T16:04:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.367+0000 7f3f82d5f700 1 -- 172.21.15.116:0/3843926793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c1a2030 con 0x7f3f7c103f70 2024-03-04T16:04:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.367+0000 7f3f82d5f700 1 -- 172.21.15.116:0/3843926793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c1a25a0 con 0x7f3f7c103f70 2024-03-04T16:04:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.369+0000 7f3f677fe700 1 -- 172.21.15.116:0/3843926793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5c0052f0 con 0x7f3f7c103f70 2024-03-04T16:04:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.369+0000 7f3f79ffb700 1 -- 172.21.15.116:0/3843926793 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) v1 ==== 90928+0+0 (secure 0 0 0) 0x7f3f68020070 con 0x7f3f7c103f70 2024-03-04T16:04:10.372 INFO:tasks.workunit.client.0.smithi116.stdout:6/340: dread d6/f45 [0,4194304] 0 2024-03-04T16:04:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.370+0000 7f3f79ffb700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3f6c06ce70 0x7f3f6c06f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.370+0000 7f3f79ffb700 1 -- 172.21.15.116:0/3843926793 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f3f6808d8b0 con 0x7f3f7c103f70 2024-03-04T16:04:10.373 INFO:tasks.workunit.client.0.smithi116.stdout:6/341: truncate d6/f21 884638 0 2024-03-04T16:04:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.370+0000 7f3f80afb700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3f6c06ce70 0x7f3f6c06f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:10.374 INFO:tasks.workunit.client.0.smithi116.stdout:5/343: truncate dd/d45/f49 2575999 0 2024-03-04T16:04:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.371+0000 7f3f80afb700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3f6c06ce70 0x7f3f6c06f330 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f70005fd0 tx=0x7f3f70009310 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:10.375 INFO:tasks.workunit.client.0.smithi116.stdout:3/428: mkdir dc/d23/d84/d7f/d93 0 2024-03-04T16:04:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.375+0000 7f3f79ffb700 1 -- 172.21.15.116:0/3843926793 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f3f6805b110 con 0x7f3f7c103f70 2024-03-04T16:04:10.379 INFO:tasks.workunit.client.0.smithi116.stdout:1/273: sync 2024-03-04T16:04:10.380 INFO:tasks.workunit.client.0.smithi116.stdout:8/393: unlink d2/d13/d1e/f45 0 2024-03-04T16:04:10.381 INFO:tasks.workunit.client.0.smithi116.stdout:1/274: fdatasync d5/d9/f18 0 2024-03-04T16:04:10.383 INFO:tasks.workunit.client.0.smithi116.stdout:3/429: dwrite dc/d23/d84/d7c/f81 [0,4194304] 0 2024-03-04T16:04:10.384 INFO:tasks.workunit.client.0.smithi116.stdout:0/331: creat da/d16/f6e x:0 0 0 2024-03-04T16:04:10.386 INFO:tasks.workunit.client.0.smithi116.stdout:3/430: write dc/d23/d84/f8e [263140,48247] 0 2024-03-04T16:04:10.386 INFO:tasks.workunit.client.0.smithi116.stdout:4/391: dread - d1/d3/dd/d24/f6b zero size 2024-03-04T16:04:10.386 INFO:tasks.workunit.client.0.smithi116.stdout:6/342: mkdir d6/d56/d58/d72 0 2024-03-04T16:04:10.389 INFO:tasks.workunit.client.0.smithi116.stdout:7/247: symlink d3/d4/d8/d4a/l4f 0 2024-03-04T16:04:10.391 INFO:tasks.workunit.client.0.smithi116.stdout:7/248: write d3/f18 [3403244,109031] 0 2024-03-04T16:04:10.395 INFO:tasks.workunit.client.0.smithi116.stdout:6/343: dwrite d6/de/d1e/f48 [0,4194304] 0 2024-03-04T16:04:10.399 INFO:tasks.workunit.client.0.smithi116.stdout:1/275: rmdir d5/d9/d21/d27 39 2024-03-04T16:04:10.401 INFO:tasks.workunit.client.0.smithi116.stdout:7/249: dwrite d3/f44 [0,4194304] 0 2024-03-04T16:04:10.403 INFO:tasks.workunit.client.0.smithi116.stdout:7/250: dread - d3/d4/d11/f36 zero size 2024-03-04T16:04:10.403 INFO:tasks.workunit.client.0.smithi116.stdout:0/332: read da/fe [808154,61217] 0 2024-03-04T16:04:10.411 INFO:tasks.workunit.client.0.smithi116.stdout:3/431: write dc/df/f53 [164563,57618] 0 2024-03-04T16:04:10.412 INFO:tasks.workunit.client.0.smithi116.stdout:3/432: write dc/d23/d84/f76 [1048140,60506] 0 2024-03-04T16:04:10.419 INFO:tasks.workunit.client.0.smithi116.stdout:3/433: dread dc/df/d47/f61 [0,4194304] 0 2024-03-04T16:04:10.419 INFO:tasks.workunit.client.0.smithi116.stdout:6/344: rename d6/de/d29/d53/l62 to d6/d56/d58/l73 0 2024-03-04T16:04:10.421 INFO:tasks.workunit.client.0.smithi116.stdout:6/345: write d6/de/d1e/f48 [3518865,113101] 0 2024-03-04T16:04:10.423 INFO:tasks.workunit.client.0.smithi116.stdout:7/251: creat d3/f50 x:0 0 0 2024-03-04T16:04:10.424 INFO:tasks.workunit.client.0.smithi116.stdout:7/252: fsync d3/d4/d8/f45 0 2024-03-04T16:04:10.427 INFO:tasks.workunit.client.0.smithi116.stdout:4/392: mkdir d1/d3/d40/d52/d84/d88 0 2024-03-04T16:04:10.429 INFO:tasks.workunit.client.0.smithi116.stdout:4/393: stat d1/d3/d15/d20/d65/c87 0 2024-03-04T16:04:10.430 INFO:tasks.workunit.client.0.smithi116.stdout:1/276: dwrite d5/f38 [0,4194304] 0 2024-03-04T16:04:10.430 INFO:tasks.workunit.client.0.smithi116.stdout:8/394: creat d2/d13/d3c/f83 x:0 0 0 2024-03-04T16:04:10.436 INFO:tasks.workunit.client.0.smithi116.stdout:5/344: getdents dd/d1f/d55/d73/d2d/d31 0 2024-03-04T16:04:10.441 INFO:tasks.workunit.client.0.smithi116.stdout:4/394: symlink d1/d6c/l89 0 2024-03-04T16:04:10.442 INFO:tasks.workunit.client.0.smithi116.stdout:2/391: sync 2024-03-04T16:04:10.442 INFO:tasks.workunit.client.0.smithi116.stdout:9/300: sync 2024-03-04T16:04:10.443 INFO:tasks.workunit.client.0.smithi116.stdout:2/392: fsync d7/d81/f39 0 2024-03-04T16:04:10.444 INFO:tasks.workunit.client.0.smithi116.stdout:3/434: unlink dc/d23/f41 0 2024-03-04T16:04:10.448 INFO:tasks.workunit.client.0.smithi116.stdout:6/346: truncate d6/de/d1e/f51 3013902 0 2024-03-04T16:04:10.450 INFO:tasks.workunit.client.0.smithi116.stdout:3/435: dread dc/df/f7e [0,4194304] 0 2024-03-04T16:04:10.454 INFO:tasks.workunit.client.0.smithi116.stdout:8/395: mknod d2/de/d3e/d7b/c84 0 2024-03-04T16:04:10.456 INFO:tasks.workunit.client.0.smithi116.stdout:2/393: readlink d7/d13/l6a 0 2024-03-04T16:04:10.456 INFO:tasks.workunit.client.0.smithi116.stdout:0/333: link da/d16/d20/d42/c58 da/d16/d20/d42/d4e/c6f 0 2024-03-04T16:04:10.456 INFO:tasks.workunit.client.0.smithi116.stdout:5/345: dwrite dd/fe [0,4194304] 0 2024-03-04T16:04:10.458 INFO:tasks.workunit.client.0.smithi116.stdout:4/395: mkdir d1/d3/dd/d8a 0 2024-03-04T16:04:10.458 INFO:tasks.workunit.client.0.smithi116.stdout:9/301: dread d2/d19/d27/d3b/d4a/d5b/f24 [0,4194304] 0 2024-03-04T16:04:10.461 INFO:tasks.workunit.client.0.smithi116.stdout:5/346: dread - dd/d1f/d55/d73/d2d/d31/f6a zero size 2024-03-04T16:04:10.461 INFO:tasks.workunit.client.0.smithi116.stdout:1/277: unlink d5/d9/d21/d27/f4b 0 2024-03-04T16:04:10.462 INFO:tasks.workunit.client.0.smithi116.stdout:7/253: getdents d3/d4/d8/d4a 0 2024-03-04T16:04:10.463 INFO:tasks.workunit.client.0.smithi116.stdout:5/347: fdatasync dd/f60 0 2024-03-04T16:04:10.465 INFO:tasks.workunit.client.0.smithi116.stdout:0/334: mkdir da/d16/d19/d27/d70 0 2024-03-04T16:04:10.465 INFO:tasks.workunit.client.0.smithi116.stdout:5/348: read - dd/d1f/d5c/f64 zero size 2024-03-04T16:04:10.466 INFO:tasks.workunit.client.0.smithi116.stdout:7/254: write d3/d4/d8/f45 [199038,1252] 0 2024-03-04T16:04:10.469 INFO:tasks.workunit.client.0.smithi116.stdout:2/394: dwrite d7/d15/d35/f70 [0,4194304] 0 2024-03-04T16:04:10.470 INFO:tasks.workunit.client.0.smithi116.stdout:9/302: symlink d2/d19/d27/d3b/d4a/d5b/l5e 0 2024-03-04T16:04:10.470 INFO:tasks.workunit.client.0.smithi116.stdout:5/349: dread - dd/d1f/f5f zero size 2024-03-04T16:04:10.471 INFO:tasks.workunit.client.0.smithi116.stdout:6/347: dread d6/f24 [0,4194304] 0 2024-03-04T16:04:10.474 INFO:tasks.workunit.client.0.smithi116.stdout:3/436: getdents dc/df/d47 0 2024-03-04T16:04:10.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:10 smithi116 ceph-mon[29120]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:10.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:10 smithi116 ceph-mon[29120]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:10.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:10 smithi116 ceph-mon[29120]: pgmap v21: 65 pgs: 65 active+clean; 496 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.1 MiB/s rd, 50 MiB/s wr, 331 op/s 2024-03-04T16:04:10.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:10 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2475810645' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:04:10.478 INFO:tasks.workunit.client.0.smithi116.stdout:6/348: fdatasync d6/de/d1e/f34 0 2024-03-04T16:04:10.478 INFO:tasks.workunit.client.0.smithi116.stdout:1/278: dwrite d5/d9/d21/d27/d32/d35/f53 [0,4194304] 0 2024-03-04T16:04:10.479 INFO:tasks.workunit.client.0.smithi116.stdout:7/255: dread d3/f10 [0,4194304] 0 2024-03-04T16:04:10.487 INFO:tasks.workunit.client.0.smithi116.stdout:5/350: dwrite dd/d1f/d55/d73/f3f [0,4194304] 0 2024-03-04T16:04:10.492 INFO:tasks.workunit.client.0.smithi116.stdout:6/349: creat d6/de/d25/d4a/d63/f74 x:0 0 0 2024-03-04T16:04:10.493 INFO:tasks.workunit.client.0.smithi116.stdout:2/395: link d7/d29/d47/d64/f86 d7/d2f/d6c/f90 0 2024-03-04T16:04:10.495 INFO:tasks.workunit.client.0.smithi116.stdout:2/396: write d7/d29/d47/f51 [2128197,39604] 0 2024-03-04T16:04:10.499 INFO:tasks.workunit.client.0.smithi116.stdout:0/335: link da/d16/d19/c31 da/d16/d28/d40/c71 0 2024-03-04T16:04:10.500 INFO:tasks.workunit.client.0.smithi116.stdout:0/336: write da/d16/d19/d27/f55 [5150181,67340] 0 2024-03-04T16:04:10.502 INFO:tasks.workunit.client.0.smithi116.stdout:5/351: creat dd/d1f/d5c/f75 x:0 0 0 2024-03-04T16:04:10.506 INFO:tasks.workunit.client.0.smithi116.stdout:4/396: write d1/d3/dd/d45/f4b [1383947,29627] 0 2024-03-04T16:04:10.508 INFO:tasks.workunit.client.0.smithi116.stdout:3/437: rename dc/df/f53 to dc/d34/f94 0 2024-03-04T16:04:10.509 INFO:tasks.workunit.client.0.smithi116.stdout:3/438: fsync dc/f70 0 2024-03-04T16:04:10.514 INFO:tasks.workunit.client.0.smithi116.stdout:1/279: mknod d5/d9/d21/d27/d32/d49/d4c/c5b 0 2024-03-04T16:04:10.514 INFO:tasks.workunit.client.0.smithi116.stdout:7/256: dread d3/d4/d8/d1a/f29 [0,4194304] 0 2024-03-04T16:04:10.515 INFO:tasks.workunit.client.0.smithi116.stdout:9/303: write d2/f48 [5864206,121572] 0 2024-03-04T16:04:10.515 INFO:tasks.workunit.client.0.smithi116.stdout:7/257: chown d3/d4/d8/c9 344905391 1 2024-03-04T16:04:10.516 INFO:tasks.workunit.client.0.smithi116.stdout:9/304: chown d2/d33/f46 3266108 1 2024-03-04T16:04:10.517 INFO:tasks.workunit.client.0.smithi116.stdout:7/258: write d3/d4/d8/f35 [3937879,129634] 0 2024-03-04T16:04:10.520 INFO:tasks.workunit.client.0.smithi116.stdout:2/397: dread d7/d15/f20 [0,4194304] 0 2024-03-04T16:04:10.524 INFO:tasks.workunit.client.0.smithi116.stdout:3/439: dread - dc/df/f74 zero size 2024-03-04T16:04:10.525 INFO:tasks.workunit.client.0.smithi116.stdout:3/440: chown dc/f4b 174080 1 2024-03-04T16:04:10.526 INFO:tasks.workunit.client.0.smithi116.stdout:2/398: dwrite d7/d29/f4b [0,4194304] 0 2024-03-04T16:04:10.527 INFO:tasks.workunit.client.0.smithi116.stdout:3/441: write dc/d23/d84/f8e [89570,20619] 0 2024-03-04T16:04:10.528 INFO:tasks.workunit.client.0.smithi116.stdout:1/280: creat d5/d9/d21/d27/d32/d35/f5c x:0 0 0 2024-03-04T16:04:10.528 INFO:tasks.workunit.client.0.smithi116.stdout:6/350: write d6/f37 [923314,76296] 0 2024-03-04T16:04:10.531 INFO:tasks.workunit.client.0.smithi116.stdout:5/352: getdents dd/d1f/d55/d73/d2d/d31 0 2024-03-04T16:04:10.536 INFO:tasks.workunit.client.0.smithi116.stdout:2/399: symlink d7/d15/d35/d63/l91 0 2024-03-04T16:04:10.536 INFO:tasks.workunit.client.0.smithi116.stdout:7/259: dread d3/d4/d8/f35 [0,4194304] 0 2024-03-04T16:04:10.538 INFO:tasks.workunit.client.0.smithi116.stdout:1/281: chown d5/d9/d21/d24 1271671 1 2024-03-04T16:04:10.538 INFO:tasks.workunit.client.0.smithi116.stdout:6/351: dwrite d6/d56/f4c [0,4194304] 0 2024-03-04T16:04:10.543 INFO:tasks.workunit.client.0.smithi116.stdout:4/397: rmdir d1/d3/d15/d20/d1c/d72 0 2024-03-04T16:04:10.543 INFO:tasks.workunit.client.0.smithi116.stdout:1/282: truncate d5/d9/d21/d27/d32/f47 485677 0 2024-03-04T16:04:10.546 INFO:tasks.workunit.client.0.smithi116.stdout:3/442: mknod dc/df/c95 0 2024-03-04T16:04:10.548 INFO:tasks.workunit.client.0.smithi116.stdout:7/260: dwrite d3/f1e [0,4194304] 0 2024-03-04T16:04:10.549 INFO:tasks.workunit.client.0.smithi116.stdout:6/352: symlink d6/de/d15/l75 0 2024-03-04T16:04:10.552 INFO:tasks.workunit.client.0.smithi116.stdout:5/353: dwrite f4 [4194304,4194304] 0 2024-03-04T16:04:10.554 INFO:tasks.workunit.client.0.smithi116.stdout:2/400: mknod d7/d29/c92 0 2024-03-04T16:04:10.557 INFO:tasks.workunit.client.0.smithi116.stdout:1/283: dwrite d5/f29 [0,4194304] 0 2024-03-04T16:04:10.557 INFO:tasks.workunit.client.0.smithi116.stdout:1/284: chown d5/d9/d46/d50 24819332 1 2024-03-04T16:04:10.560 INFO:tasks.workunit.client.0.smithi116.stdout:7/261: rmdir d3 39 2024-03-04T16:04:10.563 INFO:tasks.workunit.client.0.smithi116.stdout:3/443: creat dc/d23/d84/d7c/f96 x:0 0 0 2024-03-04T16:04:10.566 INFO:tasks.workunit.client.0.smithi116.stdout:5/354: dread dd/d1f/d55/d73/d2d/f42 [0,4194304] 0 2024-03-04T16:04:10.567 INFO:tasks.workunit.client.0.smithi116.stdout:6/353: symlink d6/de/d25/d36/d4e/d50/l76 0 2024-03-04T16:04:10.567 INFO:tasks.workunit.client.0.smithi116.stdout:5/355: stat dd/d1f/l44 0 2024-03-04T16:04:10.571 INFO:tasks.workunit.client.0.smithi116.stdout:2/401: unlink d7/d15/c19 0 2024-03-04T16:04:10.572 INFO:tasks.workunit.client.0.smithi116.stdout:9/305: truncate d2/d33/f46 1859643 0 2024-03-04T16:04:10.575 INFO:tasks.workunit.client.0.smithi116.stdout:1/285: rmdir d5/d9/d21/d27/d32/d35 39 2024-03-04T16:04:10.578 INFO:tasks.workunit.client.0.smithi116.stdout:3/444: fdatasync dc/d34/f94 0 2024-03-04T16:04:10.578 INFO:tasks.workunit.client.0.smithi116.stdout:6/354: rmdir d6/de/d25/d36 39 2024-03-04T16:04:10.579 INFO:tasks.workunit.client.0.smithi116.stdout:5/356: symlink dd/d1f/d5c/d5e/l76 0 2024-03-04T16:04:10.581 INFO:tasks.workunit.client.0.smithi116.stdout:2/402: creat d7/d15/f93 x:0 0 0 2024-03-04T16:04:10.584 INFO:tasks.workunit.client.0.smithi116.stdout:3/445: unlink dc/f8f 0 2024-03-04T16:04:10.585 INFO:tasks.workunit.client.0.smithi116.stdout:6/355: creat d6/f77 x:0 0 0 2024-03-04T16:04:10.587 INFO:tasks.workunit.client.0.smithi116.stdout:5/357: creat dd/d1f/d55/f77 x:0 0 0 2024-03-04T16:04:10.588 INFO:tasks.workunit.client.0.smithi116.stdout:5/358: chown dd/d1f/d55/d73/c57 3 1 2024-03-04T16:04:10.588 INFO:tasks.workunit.client.0.smithi116.stdout:9/306: mkdir d2/d19/d5f 0 2024-03-04T16:04:10.589 INFO:tasks.workunit.client.0.smithi116.stdout:5/359: dread - dd/d1f/f54 zero size 2024-03-04T16:04:10.589 INFO:tasks.workunit.client.0.smithi116.stdout:2/403: creat d7/d15/f94 x:0 0 0 2024-03-04T16:04:10.592 INFO:tasks.workunit.client.0.smithi116.stdout:1/286: symlink d5/d9/d21/l5d 0 2024-03-04T16:04:10.592 INFO:tasks.workunit.client.0.smithi116.stdout:4/398: truncate d1/f2c 694378 0 2024-03-04T16:04:10.594 INFO:tasks.workunit.client.0.smithi116.stdout:3/446: fsync dc/f73 0 2024-03-04T16:04:10.595 INFO:tasks.workunit.client.0.smithi116.stdout:3/447: write dc/d23/d84/d7c/f5c [1004446,83774] 0 2024-03-04T16:04:10.600 INFO:tasks.workunit.client.0.smithi116.stdout:2/404: symlink d7/d13/d3e/d78/l95 0 2024-03-04T16:04:10.602 INFO:tasks.workunit.client.0.smithi116.stdout:7/262: dwrite f2 [0,4194304] 0 2024-03-04T16:04:10.607 INFO:tasks.workunit.client.0.smithi116.stdout:2/405: mknod d7/d15/d5e/c96 0 2024-03-04T16:04:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:10 smithi152 ceph-mon[36876]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:10 smithi152 ceph-mon[36876]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:10 smithi152 ceph-mon[36876]: pgmap v21: 65 pgs: 65 active+clean; 496 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.1 MiB/s rd, 50 MiB/s wr, 331 op/s 2024-03-04T16:04:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:10 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2475810645' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:04:10.615 INFO:tasks.workunit.client.0.smithi116.stdout:3/448: symlink dc/d23/d84/d7f/d8b/l97 0 2024-03-04T16:04:10.617 INFO:tasks.workunit.client.0.smithi116.stdout:7/263: creat d3/d4/d11/f51 x:0 0 0 2024-03-04T16:04:10.617 INFO:tasks.workunit.client.0.smithi116.stdout:5/360: write dd/f11 [3347582,14150] 0 2024-03-04T16:04:10.620 INFO:tasks.workunit.client.0.smithi116.stdout:7/264: chown d3/d4/d8/d1a/l26 146935 1 2024-03-04T16:04:10.622 INFO:tasks.workunit.client.0.smithi116.stdout:7/265: write d3/fd [2166345,81355] 0 2024-03-04T16:04:10.622 INFO:tasks.workunit.client.0.smithi116.stdout:2/406: dwrite d7/d13/d5f/f69 [0,4194304] 0 2024-03-04T16:04:10.624 INFO:tasks.workunit.client.0.smithi116.stdout:2/407: readlink d7/d81/l65 0 2024-03-04T16:04:10.625 INFO:tasks.workunit.client.0.smithi116.stdout:4/399: truncate d1/d3/d40/d52/f76 558995 0 2024-03-04T16:04:10.629 INFO:tasks.workunit.client.0.smithi116.stdout:2/408: creat d7/d2f/d6c/f97 x:0 0 0 2024-03-04T16:04:10.634 INFO:tasks.workunit.client.0.smithi116.stdout:7/266: mkdir d3/d4/d8/d52 0 2024-03-04T16:04:10.637 INFO:tasks.workunit.client.0.smithi116.stdout:4/400: dwrite d1/d3/d40/f4f [0,4194304] 0 2024-03-04T16:04:10.637 INFO:tasks.workunit.client.0.smithi116.stdout:9/307: truncate d2/d33/f46 1723273 0 2024-03-04T16:04:10.638 INFO:tasks.workunit.client.0.smithi116.stdout:7/267: write d3/d4/d8/f3e [1004489,129014] 0 2024-03-04T16:04:10.638 INFO:tasks.workunit.client.0.smithi116.stdout:2/409: write d7/d2f/f50 [4241738,100033] 0 2024-03-04T16:04:10.643 INFO:tasks.workunit.client.0.smithi116.stdout:9/308: write d2/f1a [3076506,121012] 0 2024-03-04T16:04:10.643 INFO:tasks.workunit.client.0.smithi116.stdout:5/361: dread dd/f11 [0,4194304] 0 2024-03-04T16:04:10.643 INFO:tasks.workunit.client.0.smithi116.stdout:3/449: dread dc/d23/f42 [4194304,4194304] 0 2024-03-04T16:04:10.644 INFO:tasks.workunit.client.0.smithi116.stdout:5/362: chown dd/d1f/d55/d73/d2d/d59 71 1 2024-03-04T16:04:10.645 INFO:tasks.workunit.client.0.smithi116.stdout:9/309: symlink d2/d26/l60 0 2024-03-04T16:04:10.647 INFO:tasks.workunit.client.0.smithi116.stdout:7/268: creat d3/d4/d8/d52/f53 x:0 0 0 2024-03-04T16:04:10.650 INFO:tasks.workunit.client.0.smithi116.stdout:2/410: dwrite d7/d25/d41/f53 [0,4194304] 0 2024-03-04T16:04:10.650 INFO:tasks.workunit.client.0.smithi116.stdout:4/401: creat d1/d3/d40/d52/d84/d88/f8b x:0 0 0 2024-03-04T16:04:10.652 INFO:tasks.workunit.client.0.smithi116.stdout:3/450: creat dc/df/d47/d91/f98 x:0 0 0 2024-03-04T16:04:10.652 INFO:tasks.workunit.client.0.smithi116.stdout:5/363: mknod dd/d6d/c78 0 2024-03-04T16:04:10.654 INFO:tasks.workunit.client.0.smithi116.stdout:4/402: write d1/d3/f61 [221192,33350] 0 2024-03-04T16:04:10.655 INFO:tasks.workunit.client.0.smithi116.stdout:9/310: creat d2/d19/d5f/f61 x:0 0 0 2024-03-04T16:04:10.656 INFO:tasks.workunit.client.0.smithi116.stdout:9/311: fdatasync d2/f48 0 2024-03-04T16:04:10.657 INFO:tasks.workunit.client.0.smithi116.stdout:2/411: mkdir d7/d13/d3e/d98 0 2024-03-04T16:04:10.658 INFO:tasks.workunit.client.0.smithi116.stdout:2/412: stat d7/l1f 0 2024-03-04T16:04:10.659 INFO:tasks.workunit.client.0.smithi116.stdout:3/451: chown dc/d23/f1b 271240316 1 2024-03-04T16:04:10.659 INFO:tasks.workunit.client.0.smithi116.stdout:5/364: unlink dd/d45/c48 0 2024-03-04T16:04:10.661 INFO:tasks.workunit.client.0.smithi116.stdout:7/269: dwrite d3/d4/d8/d1a/f27 [0,4194304] 0 2024-03-04T16:04:10.661 INFO:tasks.workunit.client.0.smithi116.stdout:4/403: mknod d1/d3/d40/d52/c8c 0 2024-03-04T16:04:10.663 INFO:tasks.workunit.client.0.smithi116.stdout:2/413: mknod d7/d15/d5e/c99 0 2024-03-04T16:04:10.663 INFO:tasks.workunit.client.0.smithi116.stdout:9/312: read d2/d19/d27/d3b/d4a/f4c [1281920,74156] 0 2024-03-04T16:04:10.664 INFO:tasks.workunit.client.0.smithi116.stdout:2/414: readlink d7/d13/l8e 0 2024-03-04T16:04:10.664 INFO:tasks.workunit.client.0.smithi116.stdout:5/365: write dd/d1f/d55/d73/d2d/f50 [298810,35233] 0 2024-03-04T16:04:10.665 INFO:tasks.workunit.client.0.smithi116.stdout:3/452: truncate dc/df/f92 327597 0 2024-03-04T16:04:10.666 INFO:tasks.workunit.client.0.smithi116.stdout:7/270: creat d3/d4/d8/d52/f54 x:0 0 0 2024-03-04T16:04:10.668 INFO:tasks.workunit.client.0.smithi116.stdout:5/366: write dd/d1f/d5c/d5e/f62 [323681,54634] 0 2024-03-04T16:04:10.673 INFO:tasks.workunit.client.0.smithi116.stdout:2/415: creat d7/d2f/d76/f9a x:0 0 0 2024-03-04T16:04:10.673 INFO:tasks.workunit.client.0.smithi116.stdout:7/271: fsync d3/d4/d8/d1a/f29 0 2024-03-04T16:04:10.675 INFO:tasks.workunit.client.0.smithi116.stdout:7/272: readlink d3/d4/d8/d4a/l4f 0 2024-03-04T16:04:10.678 INFO:tasks.workunit.client.0.smithi116.stdout:4/404: getdents d1/d3/dd 0 2024-03-04T16:04:10.678 INFO:tasks.workunit.client.0.smithi116.stdout:2/416: mknod d7/d2f/d76/d7c/c9b 0 2024-03-04T16:04:10.678 INFO:tasks.workunit.client.0.smithi116.stdout:5/367: dwrite dd/d1f/d55/d73/d2d/d31/f6a [0,4194304] 0 2024-03-04T16:04:10.679 INFO:tasks.workunit.client.0.smithi116.stdout:4/405: stat d1/d3/dd/d45/f4b 0 2024-03-04T16:04:10.681 INFO:tasks.workunit.client.0.smithi116.stdout:5/368: dread dd/d1f/d55/d73/d2d/f42 [0,4194304] 0 2024-03-04T16:04:10.681 INFO:tasks.workunit.client.0.smithi116.stdout:9/313: getdents d2/d26/d28 0 2024-03-04T16:04:10.682 INFO:tasks.workunit.client.0.smithi116.stdout:5/369: chown dd/d1f/l3d 31435233 1 2024-03-04T16:04:10.683 INFO:tasks.workunit.client.0.smithi116.stdout:7/273: creat d3/f55 x:0 0 0 2024-03-04T16:04:10.684 INFO:tasks.workunit.client.0.smithi116.stdout:7/274: read - d3/f4e zero size 2024-03-04T16:04:10.686 INFO:tasks.workunit.client.0.smithi116.stdout:2/417: fdatasync d7/d25/f34 0 2024-03-04T16:04:10.689 INFO:tasks.workunit.client.0.smithi116.stdout:4/406: mkdir d1/d3/d40/d52/d84/d88/d8d 0 2024-03-04T16:04:10.690 INFO:tasks.workunit.client.0.smithi116.stdout:9/314: rmdir d2/d49 39 2024-03-04T16:04:10.692 INFO:tasks.workunit.client.0.smithi116.stdout:9/315: truncate d2/d19/d5f/f61 279858 0 2024-03-04T16:04:10.692 INFO:tasks.workunit.client.0.smithi116.stdout:5/370: symlink dd/d45/l79 0 2024-03-04T16:04:10.694 INFO:tasks.workunit.client.0.smithi116.stdout:9/316: write d2/f48 [2211619,109884] 0 2024-03-04T16:04:10.695 INFO:tasks.workunit.client.0.smithi116.stdout:7/275: creat d3/f56 x:0 0 0 2024-03-04T16:04:10.699 INFO:tasks.workunit.client.0.smithi116.stdout:4/407: unlink d1/d3/d15/d20/d65/c87 0 2024-03-04T16:04:10.699 INFO:tasks.workunit.client.0.smithi116.stdout:5/371: dwrite dd/d1f/f4e [0,4194304] 0 2024-03-04T16:04:10.701 INFO:tasks.workunit.client.0.smithi116.stdout:5/372: write dd/d1f/d55/d73/f3f [4154350,116105] 0 2024-03-04T16:04:10.703 INFO:tasks.workunit.client.0.smithi116.stdout:9/317: mknod d2/d19/d27/d3b/c62 0 2024-03-04T16:04:10.703 INFO:tasks.workunit.client.0.smithi116.stdout:4/408: truncate d1/d3/d15/d20/f82 397728 0 2024-03-04T16:04:10.703 INFO:tasks.workunit.client.0.smithi116.stdout:9/318: chown d2/d19 321501367 1 2024-03-04T16:04:10.706 INFO:tasks.workunit.client.0.smithi116.stdout:2/418: rename d7/d2f/c55 to d7/d15/d35/c9c 0 2024-03-04T16:04:10.706 INFO:tasks.workunit.client.0.smithi116.stdout:2/419: chown d7/f2b 865065 1 2024-03-04T16:04:10.711 INFO:tasks.workunit.client.0.smithi116.stdout:5/373: write dd/f11 [2832313,12125] 0 2024-03-04T16:04:10.715 INFO:tasks.workunit.client.0.smithi116.stdout:2/420: mkdir d7/d15/d5e/d9d 0 2024-03-04T16:04:10.715 INFO:tasks.workunit.client.0.smithi116.stdout:9/319: dread d2/d49/f42 [0,4194304] 0 2024-03-04T16:04:10.720 INFO:tasks.workunit.client.0.smithi116.stdout:5/374: rmdir dd/d1f 39 2024-03-04T16:04:10.721 INFO:tasks.workunit.client.0.smithi116.stdout:4/409: symlink d1/d3/d15/d20/l8e 0 2024-03-04T16:04:10.725 INFO:tasks.workunit.client.0.smithi116.stdout:9/320: creat d2/d49/d3c/f63 x:0 0 0 2024-03-04T16:04:10.728 INFO:tasks.workunit.client.0.smithi116.stdout:3/453: write dc/d34/f3e [1117688,130595] 0 2024-03-04T16:04:10.729 INFO:tasks.workunit.client.0.smithi116.stdout:3/454: readlink dc/df/l12 0 2024-03-04T16:04:10.730 INFO:tasks.workunit.client.0.smithi116.stdout:4/410: rename d1/d3/dd/d24/f3d to d1/d3/dd/d24/f8f 0 2024-03-04T16:04:10.731 INFO:tasks.workunit.client.0.smithi116.stdout:7/276: link d3/d4/d11/l48 d3/d4/l57 0 2024-03-04T16:04:10.732 INFO:tasks.workunit.client.0.smithi116.stdout:7/277: chown d3/d4/c3d 6691 1 2024-03-04T16:04:10.733 INFO:tasks.workunit.client.0.smithi116.stdout:7/278: read d3/d4/d8/d1a/f20 [213814,16335] 0 2024-03-04T16:04:10.734 INFO:tasks.workunit.client.0.smithi116.stdout:9/321: mkdir d2/d33/d64 0 2024-03-04T16:04:10.734 INFO:tasks.workunit.client.0.smithi116.stdout:7/279: write f1 [5288626,23175] 0 2024-03-04T16:04:10.736 INFO:tasks.workunit.client.0.smithi116.stdout:9/322: write d2/d19/d27/d3b/d4a/d5b/f43 [609607,8813] 0 2024-03-04T16:04:10.739 INFO:tasks.workunit.client.0.smithi116.stdout:9/323: read d2/f15 [1987743,70781] 0 2024-03-04T16:04:10.740 INFO:tasks.workunit.client.0.smithi116.stdout:3/455: creat dc/d34/d44/f99 x:0 0 0 2024-03-04T16:04:10.741 INFO:tasks.workunit.client.0.smithi116.stdout:4/411: truncate d1/d3/de/d47/f60 593113 0 2024-03-04T16:04:10.741 INFO:tasks.workunit.client.0.smithi116.stdout:3/456: truncate dc/d23/d84/d7c/f62 225751 0 2024-03-04T16:04:10.741 INFO:tasks.workunit.client.0.smithi116.stdout:3/457: chown dc/d23/d84/d7f 795453 1 2024-03-04T16:04:10.742 INFO:tasks.workunit.client.0.smithi116.stdout:3/458: write dc/f70 [731990,98774] 0 2024-03-04T16:04:10.745 INFO:tasks.workunit.client.0.smithi116.stdout:7/280: creat d3/f58 x:0 0 0 2024-03-04T16:04:10.747 INFO:tasks.workunit.client.0.smithi116.stdout:2/421: link d7/d15/d35/c5d d7/d15/d5e/c9e 0 2024-03-04T16:04:10.753 INFO:tasks.workunit.client.0.smithi116.stdout:5/375: creat dd/d1f/d55/d73/d2d/d59/d71/f7a x:0 0 0 2024-03-04T16:04:10.754 INFO:tasks.workunit.client.0.smithi116.stdout:5/376: write dd/f11 [966325,54096] 0 2024-03-04T16:04:10.754 INFO:tasks.workunit.client.0.smithi116.stdout:9/324: mkdir d2/d33/d65 0 2024-03-04T16:04:10.755 INFO:tasks.workunit.client.0.smithi116.stdout:5/377: dread - dd/d1f/d55/f77 zero size 2024-03-04T16:04:10.755 INFO:tasks.workunit.client.0.smithi116.stdout:5/378: readlink dd/d1f/l3d 0 2024-03-04T16:04:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.758+0000 7f3f677fe700 1 -- 172.21.15.116:0/3843926793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3f5c005c90 con 0x7f3f7c103f70 2024-03-04T16:04:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.759+0000 7f3f79ffb700 1 -- 172.21.15.116:0/3843926793 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1831 (secure 0 0 0) 0x7f3f6805aca0 con 0x7f3f7c103f70 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:04:10.760 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:04:10.761 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:04:10.762 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:04:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 -- 172.21.15.116:0/3843926793 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3f6c06ce70 msgr2=0x7f3f6c06f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3f6c06ce70 0x7f3f6c06f330 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f70005fd0 tx=0x7f3f70009310 comp rx=0 tx=0).stop 2024-03-04T16:04:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 -- 172.21.15.116:0/3843926793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f7c103f70 msgr2=0x7f3f7c19ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f7c103f70 0x7f3f7c19ce80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009f50 tx=0x7f3f6800f740 comp rx=0 tx=0).stop 2024-03-04T16:04:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 -- 172.21.15.116:0/3843926793 shutdown_connections 2024-03-04T16:04:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f3f6c06ce70 0x7f3f6c06f330 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c102e10 0x7f3f7c19c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 --2- 172.21.15.116:0/3843926793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f7c103f70 0x7f3f7c19ce80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 -- 172.21.15.116:0/3843926793 >> 172.21.15.116:0/3843926793 conn(0x7f3f7c0fe3d0 msgr2=0x7f3f7c107230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 -- 172.21.15.116:0/3843926793 shutdown_connections 2024-03-04T16:04:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.761+0000 7f3f677fe700 1 -- 172.21.15.116:0/3843926793 wait complete. 2024-03-04T16:04:10.764 INFO:tasks.workunit.client.0.smithi116.stdout:7/281: rename d3/f58 to d3/f59 0 2024-03-04T16:04:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:04:10.765 INFO:tasks.workunit.client.0.smithi116.stdout:7/282: readlink d3/d4/d34/l43 0 2024-03-04T16:04:10.773 INFO:tasks.workunit.client.0.smithi116.stdout:2/422: link d7/f4d d7/d25/d41/f9f 0 2024-03-04T16:04:10.773 INFO:tasks.workunit.client.0.smithi116.stdout:5/379: dwrite dd/f38 [0,4194304] 0 2024-03-04T16:04:10.775 INFO:tasks.workunit.client.0.smithi116.stdout:7/283: mknod d3/d4/d11/c5a 0 2024-03-04T16:04:10.777 INFO:tasks.workunit.client.0.smithi116.stdout:9/325: creat d2/d33/d64/f66 x:0 0 0 2024-03-04T16:04:10.779 INFO:tasks.workunit.client.0.smithi116.stdout:3/459: creat dc/d34/f9a x:0 0 0 2024-03-04T16:04:10.780 INFO:tasks.workunit.client.0.smithi116.stdout:7/284: dread d3/d4/d8/d1a/f27 [4194304,4194304] 0 2024-03-04T16:04:10.781 INFO:tasks.workunit.client.0.smithi116.stdout:2/423: mknod d7/d2f/ca0 0 2024-03-04T16:04:10.782 INFO:tasks.workunit.client.0.smithi116.stdout:2/424: write d7/d15/f94 [858461,2515] 0 2024-03-04T16:04:10.784 INFO:tasks.workunit.client.0.smithi116.stdout:5/380: creat dd/d1f/d55/d73/d2d/d31/f7b x:0 0 0 2024-03-04T16:04:10.790 INFO:tasks.workunit.client.0.smithi116.stdout:4/412: dread d1/d3/de/f12 [0,4194304] 0 2024-03-04T16:04:10.791 INFO:tasks.workunit.client.0.smithi116.stdout:7/285: rmdir d3/d4/d8/d52 39 2024-03-04T16:04:10.791 INFO:tasks.workunit.client.0.smithi116.stdout:4/413: dread - d1/d3/dd/d24/f6b zero size 2024-03-04T16:04:10.792 INFO:tasks.workunit.client.0.smithi116.stdout:2/425: rmdir d7/d13/d5f 39 2024-03-04T16:04:10.796 INFO:tasks.workunit.client.0.smithi116.stdout:5/381: mkdir dd/d1f/d55/d73/d2d/d7c 0 2024-03-04T16:04:10.796 INFO:tasks.workunit.client.0.smithi116.stdout:8/396: sync 2024-03-04T16:04:10.802 INFO:tasks.workunit.client.0.smithi116.stdout:8/397: dwrite d2/de/f15 [0,4194304] 0 2024-03-04T16:04:10.802 INFO:tasks.workunit.client.0.smithi116.stdout:9/326: dread d2/f1a [0,4194304] 0 2024-03-04T16:04:10.803 INFO:tasks.workunit.client.0.smithi116.stdout:7/286: creat d3/d4/d11/f5b x:0 0 0 2024-03-04T16:04:10.804 INFO:tasks.workunit.client.0.smithi116.stdout:7/287: dread - d3/f56 zero size 2024-03-04T16:04:10.806 INFO:tasks.workunit.client.0.smithi116.stdout:8/398: write d2/de/d3e/f4e [595541,104914] 0 2024-03-04T16:04:10.807 INFO:tasks.workunit.client.0.smithi116.stdout:4/414: rmdir d1/d3/d15/d20/d1c/d62 39 2024-03-04T16:04:10.811 INFO:tasks.workunit.client.0.smithi116.stdout:2/426: mkdir d7/d13/d5f/da1 0 2024-03-04T16:04:10.812 INFO:tasks.workunit.client.0.smithi116.stdout:9/327: rename d2/d19/l56 to d2/d19/d5f/l67 0 2024-03-04T16:04:10.814 INFO:tasks.workunit.client.0.smithi116.stdout:8/399: mknod d2/d38/d55/c85 0 2024-03-04T16:04:10.814 INFO:tasks.workunit.client.0.smithi116.stdout:7/288: dwrite d3/f4e [0,4194304] 0 2024-03-04T16:04:10.818 INFO:tasks.workunit.client.0.smithi116.stdout:4/415: dread d1/d3/dd/d45/f4b [0,4194304] 0 2024-03-04T16:04:10.821 INFO:tasks.workunit.client.0.smithi116.stdout:9/328: rename d2/d49/d3c/f63 to d2/d19/d27/d3b/f68 0 2024-03-04T16:04:10.825 INFO:tasks.workunit.client.0.smithi116.stdout:2/427: dwrite d7/d81/f1c [4194304,4194304] 0 2024-03-04T16:04:10.825 INFO:tasks.workunit.client.0.smithi116.stdout:8/400: unlink d2/f7 0 2024-03-04T16:04:10.827 INFO:tasks.workunit.client.0.smithi116.stdout:7/289: creat d3/d4/d8/d1a/f5c x:0 0 0 2024-03-04T16:04:10.829 INFO:tasks.workunit.client.0.smithi116.stdout:4/416: creat d1/d3/dd/d45/f90 x:0 0 0 2024-03-04T16:04:10.835 INFO:tasks.workunit.client.0.smithi116.stdout:2/428: creat d7/d25/fa2 x:0 0 0 2024-03-04T16:04:10.835 INFO:tasks.workunit.client.0.smithi116.stdout:3/460: dwrite dc/d23/d84/f3b [0,4194304] 0 2024-03-04T16:04:10.837 INFO:tasks.workunit.client.0.smithi116.stdout:7/290: mkdir d3/d4/d34/d5d 0 2024-03-04T16:04:10.839 INFO:tasks.workunit.client.0.smithi116.stdout:7/291: chown d3/f56 2859277 1 2024-03-04T16:04:10.842 INFO:tasks.workunit.client.0.smithi116.stdout:3/461: mknod dc/df/d47/d91/c9b 0 2024-03-04T16:04:10.846 INFO:tasks.workunit.client.0.smithi116.stdout:2/429: rename d7/d15/d5e/c96 to d7/d2f/d76/ca3 0 2024-03-04T16:04:10.846 INFO:tasks.workunit.client.0.smithi116.stdout:7/292: rmdir d3/d4 39 2024-03-04T16:04:10.847 INFO:tasks.workunit.client.0.smithi116.stdout:4/417: link d1/d3/dd/d45/f90 d1/d3/f91 0 2024-03-04T16:04:10.849 INFO:tasks.workunit.client.0.smithi116.stdout:3/462: symlink dc/d23/d84/l9c 0 2024-03-04T16:04:10.850 INFO:tasks.workunit.client.0.smithi116.stdout:9/329: getdents d2/d19 0 2024-03-04T16:04:10.850 INFO:tasks.workunit.client.0.smithi116.stdout:7/293: dread f2 [0,4194304] 0 2024-03-04T16:04:10.851 INFO:tasks.workunit.client.0.smithi116.stdout:2/430: unlink d7/d15/f94 0 2024-03-04T16:04:10.851 INFO:tasks.workunit.client.0.smithi116.stdout:9/330: dread - d2/d19/d27/d3b/f68 zero size 2024-03-04T16:04:10.853 INFO:tasks.workunit.client.0.smithi116.stdout:2/431: write d7/d29/f2d [622098,95825] 0 2024-03-04T16:04:10.854 INFO:tasks.workunit.client.0.smithi116.stdout:4/418: symlink d1/d6c/l92 0 2024-03-04T16:04:10.855 INFO:tasks.workunit.client.0.smithi116.stdout:4/419: write d1/d3/dd/f5d [750016,79829] 0 2024-03-04T16:04:10.861 INFO:tasks.workunit.client.0.smithi116.stdout:2/432: dwrite d7/d25/d41/f53 [0,4194304] 0 2024-03-04T16:04:10.878 INFO:tasks.workunit.client.0.smithi116.stdout:7/294: mkdir d3/d4/d8/d52/d5e 0 2024-03-04T16:04:10.883 INFO:tasks.workunit.client.0.smithi116.stdout:7/295: rmdir d3/d4/d8/d52 39 2024-03-04T16:04:10.885 INFO:tasks.workunit.client.0.smithi116.stdout:4/420: link d1/d3/dd/d45/f90 d1/d3/d15/d20/d1c/d62/d81/f93 0 2024-03-04T16:04:10.886 INFO:tasks.workunit.client.0.smithi116.stdout:4/421: stat d1/d3/dd/f43 0 2024-03-04T16:04:10.890 INFO:tasks.workunit.client.0.smithi116.stdout:7/296: truncate d3/d4/d8/d1a/f29 539645 0 2024-03-04T16:04:10.891 INFO:tasks.workunit.client.0.smithi116.stdout:4/422: symlink d1/d3/l94 0 2024-03-04T16:04:10.891 INFO:tasks.workunit.client.0.smithi116.stdout:7/297: readlink d3/d4/d8/d4a/l4f 0 2024-03-04T16:04:10.893 INFO:tasks.workunit.client.0.smithi116.stdout:3/463: truncate dc/df/d14/f49 2054871 0 2024-03-04T16:04:10.894 INFO:tasks.workunit.client.0.smithi116.stdout:4/423: write d1/d3/f61 [468452,36550] 0 2024-03-04T16:04:10.895 INFO:tasks.workunit.client.0.smithi116.stdout:3/464: dread - dc/f88 zero size 2024-03-04T16:04:10.895 INFO:tasks.workunit.client.0.smithi116.stdout:7/298: creat d3/d4/d8/d1a/f5f x:0 0 0 2024-03-04T16:04:10.896 INFO:tasks.workunit.client.0.smithi116.stdout:9/331: dwrite d2/d19/d27/d3b/d4a/d5b/f24 [0,4194304] 0 2024-03-04T16:04:10.897 INFO:tasks.workunit.client.0.smithi116.stdout:3/465: readlink dc/d23/d84/d7c/l7b 0 2024-03-04T16:04:10.906 INFO:tasks.workunit.client.0.smithi116.stdout:4/424: dwrite d1/d3/d15/d20/f5e [0,4194304] 0 2024-03-04T16:04:10.906 INFO:tasks.workunit.client.0.smithi116.stdout:2/433: dwrite d7/fb [0,4194304] 0 2024-03-04T16:04:10.907 INFO:tasks.workunit.client.0.smithi116.stdout:2/434: fdatasync d7/d29/f4a 0 2024-03-04T16:04:10.908 INFO:tasks.workunit.client.0.smithi116.stdout:2/435: stat d7/d13/d5f/l73 0 2024-03-04T16:04:10.909 INFO:tasks.workunit.client.0.smithi116.stdout:2/436: dread - d7/d15/d35/f48 zero size 2024-03-04T16:04:10.920 INFO:tasks.workunit.client.0.smithi116.stdout:4/425: write d1/f80 [2164801,111374] 0 2024-03-04T16:04:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.920+0000 7f530aba2700 1 -- 172.21.15.116:0/9940603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5304074d60 msgr2=0x7f53040731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.920+0000 7f530aba2700 1 --2- 172.21.15.116:0/9940603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5304074d60 0x7f53040731c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009910 tx=0x7f52f8009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.920+0000 7f530aba2700 1 -- 172.21.15.116:0/9940603 shutdown_connections 2024-03-04T16:04:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.920+0000 7f530aba2700 1 --2- 172.21.15.116:0/9940603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5304073790 0x7f5304073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.920+0000 7f530aba2700 1 --2- 172.21.15.116:0/9940603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5304074d60 0x7f53040731c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.920+0000 7f530aba2700 1 -- 172.21.15.116:0/9940603 >> 172.21.15.116:0/9940603 conn(0x7f53040fc110 msgr2=0x7f53040fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.920+0000 7f530aba2700 1 -- 172.21.15.116:0/9940603 shutdown_connections 2024-03-04T16:04:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.920+0000 7f530aba2700 1 -- 172.21.15.116:0/9940603 wait complete. 2024-03-04T16:04:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.921+0000 7f530aba2700 1 Processor -- start 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.921+0000 7f530aba2700 1 -- start start 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.921+0000 7f530aba2700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5304073790 0x7f5304198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.921+0000 7f530aba2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5304074d60 0x7f5304198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.921+0000 7f530aba2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53041990d0 con 0x7f5304074d60 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.921+0000 7f530aba2700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5304199210 con 0x7f5304073790 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.921+0000 7f5303fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5304074d60 0x7f5304198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.921+0000 7f5303fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5304074d60 0x7f5304198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50320/0 (socket says 172.21.15.116:50320) 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.921+0000 7f5303fff700 1 -- 172.21.15.116:0/1524195263 learned_addr learned my addr 172.21.15.116:0/1524195263 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.922+0000 7f530893e700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5304073790 0x7f5304198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.922+0000 7f5303fff700 1 -- 172.21.15.116:0/1524195263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5304073790 msgr2=0x7f5304198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.922+0000 7f5303fff700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5304073790 0x7f5304198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.922+0000 7f5303fff700 1 -- 172.21.15.116:0/1524195263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f80095f0 con 0x7f5304074d60 2024-03-04T16:04:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.922+0000 7f5303fff700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5304074d60 0x7f5304198ab0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52f000d8d0 tx=0x7f52f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.922+0000 7f5301ffb700 1 -- 172.21.15.116:0/1524195263 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52f0009940 con 0x7f5304074d60 2024-03-04T16:04:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.922+0000 7f530aba2700 1 -- 172.21.15.116:0/1524195263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530419dcc0 con 0x7f5304074d60 2024-03-04T16:04:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.922+0000 7f5301ffb700 1 -- 172.21.15.116:0/1524195263 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f52f0010460 con 0x7f5304074d60 2024-03-04T16:04:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.922+0000 7f5301ffb700 1 -- 172.21.15.116:0/1524195263 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52f000f5f0 con 0x7f5304074d60 2024-03-04T16:04:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.922+0000 7f530aba2700 1 -- 172.21.15.116:0/1524195263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530419e210 con 0x7f5304074d60 2024-03-04T16:04:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.924+0000 7f530aba2700 1 -- 172.21.15.116:0/1524195263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530410b900 con 0x7f5304074d60 2024-03-04T16:04:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.924+0000 7f5301ffb700 1 -- 172.21.15.116:0/1524195263 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) v1 ==== 90928+0+0 (secure 0 0 0) 0x7f52f000f750 con 0x7f5304074d60 2024-03-04T16:04:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.924+0000 7f5301ffb700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f52f406ce40 0x7f52f406f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.924+0000 7f5301ffb700 1 -- 172.21.15.116:0/1524195263 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f52f008c6d0 con 0x7f5304074d60 2024-03-04T16:04:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.924+0000 7f530893e700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f52f406ce40 0x7f52f406f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.925+0000 7f530893e700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f52f406ce40 0x7f52f406f300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f52f8006010 tx=0x7f52f800b580 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:10.927 INFO:tasks.workunit.client.0.smithi116.stdout:8/401: dwrite d2/de/f12 [0,4194304] 0 2024-03-04T16:04:10.928 INFO:tasks.workunit.client.0.smithi116.stdout:7/299: link d3/d4/d8/d1a/c33 d3/d4/d11/c60 0 2024-03-04T16:04:10.929 INFO:tasks.workunit.client.0.smithi116.stdout:7/300: dread - d3/d4/f49 zero size 2024-03-04T16:04:10.930 INFO:tasks.workunit.client.0.smithi116.stdout:6/356: sync 2024-03-04T16:04:10.930 INFO:tasks.workunit.client.0.smithi116.stdout:1/287: sync 2024-03-04T16:04:10.930 INFO:tasks.workunit.client.0.smithi116.stdout:0/337: sync 2024-03-04T16:04:10.933 INFO:tasks.workunit.client.0.smithi116.stdout:6/357: dread - d6/f77 zero size 2024-03-04T16:04:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:10.932+0000 7f5301ffb700 1 -- 172.21.15.116:0/1524195263 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f52f005aac0 con 0x7f5304074d60 2024-03-04T16:04:10.934 INFO:tasks.workunit.client.0.smithi116.stdout:3/466: link dc/d23/d84/d7c/f80 dc/d34/f9d 0 2024-03-04T16:04:10.935 INFO:tasks.workunit.client.0.smithi116.stdout:3/467: write dc/d34/f9a [900460,89398] 0 2024-03-04T16:04:10.936 INFO:tasks.workunit.client.0.smithi116.stdout:3/468: fdatasync dc/f70 0 2024-03-04T16:04:10.937 INFO:tasks.workunit.client.0.smithi116.stdout:9/332: rename d2/d19/d27/d3b/d4a/d5b/c31 to d2/d19/d27/d3b/d4a/c69 0 2024-03-04T16:04:10.940 INFO:tasks.workunit.client.0.smithi116.stdout:4/426: fdatasync d1/d3/de/d47/f60 0 2024-03-04T16:04:10.941 INFO:tasks.workunit.client.0.smithi116.stdout:9/333: read d2/d26/f44 [1960932,8947] 0 2024-03-04T16:04:10.941 INFO:tasks.workunit.client.0.smithi116.stdout:6/358: dwrite d6/de/d1e/f48 [4194304,4194304] 0 2024-03-04T16:04:10.941 INFO:tasks.workunit.client.0.smithi116.stdout:8/402: read d2/d13/d3c/f69 [14147,85785] 0 2024-03-04T16:04:10.942 INFO:tasks.workunit.client.0.smithi116.stdout:7/301: mkdir d3/d4/d11/d61 0 2024-03-04T16:04:10.943 INFO:tasks.workunit.client.0.smithi116.stdout:7/302: chown d3/d4/d8 194305 1 2024-03-04T16:04:10.952 INFO:tasks.workunit.client.0.smithi116.stdout:1/288: dwrite d5/d9/d21/d24/f3a [4194304,4194304] 0 2024-03-04T16:04:10.955 INFO:tasks.workunit.client.0.smithi116.stdout:1/289: chown d5/d9/d21/d27/l2d 43272 1 2024-03-04T16:04:10.956 INFO:tasks.workunit.client.0.smithi116.stdout:2/437: rename d7/d13/d5f to d7/d2f/d6c/da4 0 2024-03-04T16:04:10.963 INFO:tasks.workunit.client.0.smithi116.stdout:4/427: mknod d1/d3/d40/d52/d84/d88/c95 0 2024-03-04T16:04:10.969 INFO:tasks.workunit.client.0.smithi116.stdout:7/303: symlink d3/d4/d34/l62 0 2024-03-04T16:04:10.971 INFO:tasks.workunit.client.0.smithi116.stdout:5/382: sync 2024-03-04T16:04:10.975 INFO:tasks.workunit.client.0.smithi116.stdout:2/438: mkdir d7/d29/d47/d64/da5 0 2024-03-04T16:04:10.977 INFO:tasks.workunit.client.0.smithi116.stdout:9/334: mkdir d2/d19/d27/d3b/d4a/d6a 0 2024-03-04T16:04:10.977 INFO:tasks.workunit.client.0.smithi116.stdout:9/335: write d2/d26/f47 [3877742,83241] 0 2024-03-04T16:04:10.984 INFO:tasks.workunit.client.0.smithi116.stdout:7/304: stat d3/d4/d8/d52/f53 0 2024-03-04T16:04:10.986 INFO:tasks.workunit.client.0.smithi116.stdout:1/290: creat d5/d9/d46/d50/f5e x:0 0 0 2024-03-04T16:04:10.992 INFO:tasks.workunit.client.0.smithi116.stdout:4/428: link d1/d3/dd/d24/f6b d1/d3/dd/d45/d50/f96 0 2024-03-04T16:04:10.993 INFO:tasks.workunit.client.0.smithi116.stdout:9/336: dread d2/d26/f44 [0,4194304] 0 2024-03-04T16:04:10.998 INFO:tasks.workunit.client.0.smithi116.stdout:0/338: getdents da/d16/d19/d27 0 2024-03-04T16:04:10.999 INFO:tasks.workunit.client.0.smithi116.stdout:9/337: dwrite d2/d33/f54 [0,4194304] 0 2024-03-04T16:04:10.999 INFO:tasks.workunit.client.0.smithi116.stdout:0/339: chown da/d16/d28/c37 76 1 2024-03-04T16:04:11.000 INFO:tasks.workunit.client.0.smithi116.stdout:5/383: mknod dd/c7d 0 2024-03-04T16:04:11.000 INFO:tasks.workunit.client.0.smithi116.stdout:0/340: stat da/f35 0 2024-03-04T16:04:11.002 INFO:tasks.workunit.client.0.smithi116.stdout:3/469: write dc/df/d14/f49 [880176,39395] 0 2024-03-04T16:04:11.007 INFO:tasks.workunit.client.0.smithi116.stdout:4/429: mkdir d1/d3/d15/d20/d1c/d62/d81/d97 0 2024-03-04T16:04:11.007 INFO:tasks.workunit.client.0.smithi116.stdout:9/338: dwrite d2/d19/d27/d3b/d4a/d5b/f24 [4194304,4194304] 0 2024-03-04T16:04:11.008 INFO:tasks.workunit.client.0.smithi116.stdout:9/339: fsync d2/d19/f37 0 2024-03-04T16:04:11.009 INFO:tasks.workunit.client.0.smithi116.stdout:6/359: link d6/de/d25/d36/d44/l64 d6/d56/l78 0 2024-03-04T16:04:11.010 INFO:tasks.workunit.client.0.smithi116.stdout:8/403: getdents d2/d13/d3c/d46/d4f 0 2024-03-04T16:04:11.011 INFO:tasks.workunit.client.0.smithi116.stdout:9/340: truncate d2/d26/f44 8389752 0 2024-03-04T16:04:11.017 INFO:tasks.workunit.client.0.smithi116.stdout:0/341: creat da/d16/d20/d42/d4e/f72 x:0 0 0 2024-03-04T16:04:11.019 INFO:tasks.workunit.client.0.smithi116.stdout:2/439: link d7/d29/l32 d7/d15/la6 0 2024-03-04T16:04:11.022 INFO:tasks.workunit.client.0.smithi116.stdout:8/404: dwrite d2/d13/d1e/d42/f5d [0,4194304] 0 2024-03-04T16:04:11.023 INFO:tasks.workunit.client.0.smithi116.stdout:6/360: dwrite d6/d56/f4c [0,4194304] 0 2024-03-04T16:04:11.023 INFO:tasks.workunit.client.0.smithi116.stdout:2/440: chown d7/d15/d35/f54 0 1 2024-03-04T16:04:11.026 INFO:tasks.workunit.client.0.smithi116.stdout:9/341: mknod d2/d33/d40/c6b 0 2024-03-04T16:04:11.033 INFO:tasks.workunit.client.0.smithi116.stdout:6/361: dwrite d6/de/f4f [0,4194304] 0 2024-03-04T16:04:11.034 INFO:tasks.workunit.client.0.smithi116.stdout:6/362: write d6/de/d29/d53/f57 [263680,78534] 0 2024-03-04T16:04:11.038 INFO:tasks.workunit.client.0.smithi116.stdout:3/470: getdents dc/d34/d44/d6a 0 2024-03-04T16:04:11.041 INFO:tasks.workunit.client.0.smithi116.stdout:0/342: readlink da/d16/d28/d40/d48/l4d 0 2024-03-04T16:04:11.041 INFO:tasks.workunit.client.0.smithi116.stdout:3/471: truncate dc/df/f74 398380 0 2024-03-04T16:04:11.043 INFO:tasks.workunit.client.0.smithi116.stdout:4/430: truncate d1/d3/de/f16 1681742 0 2024-03-04T16:04:11.045 INFO:tasks.workunit.client.0.smithi116.stdout:7/305: dwrite d3/f6 [0,4194304] 0 2024-03-04T16:04:11.046 INFO:tasks.workunit.client.0.smithi116.stdout:4/431: stat d1/d3/d15/d20/d1c/d6f 0 2024-03-04T16:04:11.049 INFO:tasks.workunit.client.0.smithi116.stdout:9/342: rename d2/d33/d64 to d2/d33/d65/d6c 0 2024-03-04T16:04:11.052 INFO:tasks.workunit.client.0.smithi116.stdout:0/343: dwrite f7 [0,4194304] 0 2024-03-04T16:04:11.062 INFO:tasks.workunit.client.0.smithi116.stdout:6/363: dread d6/f9 [0,4194304] 0 2024-03-04T16:04:11.062 INFO:tasks.workunit.client.0.smithi116.stdout:0/344: dwrite da/ff [0,4194304] 0 2024-03-04T16:04:11.063 INFO:tasks.workunit.client.0.smithi116.stdout:0/345: fsync da/d16/d20/f21 0 2024-03-04T16:04:11.068 INFO:tasks.workunit.client.0.smithi116.stdout:5/384: getdents dd/d1f 0 2024-03-04T16:04:11.069 INFO:tasks.workunit.client.0.smithi116.stdout:0/346: dread da/d16/d19/f1d [0,4194304] 0 2024-03-04T16:04:11.070 INFO:tasks.workunit.client.0.smithi116.stdout:0/347: write da/d16/d20/f3b [448880,14316] 0 2024-03-04T16:04:11.072 INFO:tasks.workunit.client.0.smithi116.stdout:4/432: rename d1/d3/dd/f43 to d1/d3/dd/f98 0 2024-03-04T16:04:11.073 INFO:tasks.workunit.client.0.smithi116.stdout:5/385: dread dd/d1f/d55/d73/d2d/d31/f6a [0,4194304] 0 2024-03-04T16:04:11.075 INFO:tasks.workunit.client.0.smithi116.stdout:7/306: truncate d3/d4/d8/d1a/f20 1998036 0 2024-03-04T16:04:11.081 INFO:tasks.workunit.client.0.smithi116.stdout:3/472: dread dc/f33 [0,4194304] 0 2024-03-04T16:04:11.082 INFO:tasks.workunit.client.0.smithi116.stdout:3/473: write dc/f64 [463132,7060] 0 2024-03-04T16:04:11.084 INFO:tasks.workunit.client.0.smithi116.stdout:9/343: mknod d2/c6d 0 2024-03-04T16:04:11.090 INFO:tasks.workunit.client.0.smithi116.stdout:5/386: rename dd/d1f/d55/d73/c1d to dd/d1f/d55/d73/d2d/d59/c7e 0 2024-03-04T16:04:11.094 INFO:tasks.workunit.client.0.smithi116.stdout:6/364: truncate d6/de/d25/f40 2624842 0 2024-03-04T16:04:11.094 INFO:tasks.workunit.client.0.smithi116.stdout:6/365: chown d6/de/f3b 1676412052 1 2024-03-04T16:04:11.095 INFO:tasks.workunit.client.0.smithi116.stdout:2/441: truncate d7/d15/d35/f3b 714687 0 2024-03-04T16:04:11.100 INFO:tasks.workunit.client.0.smithi116.stdout:5/387: chown dd/l43 707 1 2024-03-04T16:04:11.104 INFO:tasks.workunit.client.0.smithi116.stdout:9/344: rename d2/d19/d27/d3b/d4a/d5b/l51 to d2/d33/d65/l6e 0 2024-03-04T16:04:11.106 INFO:tasks.workunit.client.0.smithi116.stdout:5/388: dwrite dd/d1f/f69 [0,4194304] 0 2024-03-04T16:04:11.106 INFO:tasks.workunit.client.0.smithi116.stdout:8/405: write d2/d38/f71 [556409,54773] 0 2024-03-04T16:04:11.122 INFO:tasks.workunit.client.0.smithi116.stdout:7/307: link d3/d4/d8/l2d d3/d4/d8/d1a/l63 0 2024-03-04T16:04:11.125 INFO:tasks.workunit.client.0.smithi116.stdout:6/366: creat d6/d56/d58/d72/f79 x:0 0 0 2024-03-04T16:04:11.128 INFO:tasks.workunit.client.0.smithi116.stdout:6/367: chown d6/de/d25/d4a/c6b 305801018 1 2024-03-04T16:04:11.129 INFO:tasks.workunit.client.0.smithi116.stdout:7/308: dwrite d3/f18 [0,4194304] 0 2024-03-04T16:04:11.130 INFO:tasks.workunit.client.0.smithi116.stdout:6/368: write d6/de/d25/d36/d44/f4b [4901588,87923] 0 2024-03-04T16:04:11.130 INFO:tasks.workunit.client.0.smithi116.stdout:7/309: readlink d3/d4/d8/d1a/l2c 0 2024-03-04T16:04:11.133 INFO:tasks.workunit.client.0.smithi116.stdout:3/474: link dc/f73 dc/f9e 0 2024-03-04T16:04:11.134 INFO:tasks.workunit.client.0.smithi116.stdout:9/345: truncate d2/d33/f46 1986406 0 2024-03-04T16:04:11.137 INFO:tasks.workunit.client.0.smithi116.stdout:5/389: getdents dd/d1f/d72 0 2024-03-04T16:04:11.138 INFO:tasks.workunit.client.0.smithi116.stdout:6/369: dwrite d6/de/d25/d36/d4e/d50/f5f [0,4194304] 0 2024-03-04T16:04:11.140 INFO:tasks.workunit.client.0.smithi116.stdout:9/346: read d2/d19/d27/d3b/f3d [3316026,63088] 0 2024-03-04T16:04:11.147 INFO:tasks.workunit.client.0.smithi116.stdout:7/310: mkdir d3/d4/d8/d4a/d64 0 2024-03-04T16:04:11.148 INFO:tasks.workunit.client.0.smithi116.stdout:2/442: dwrite d7/d25/d41/f9f [0,4194304] 0 2024-03-04T16:04:11.149 INFO:tasks.workunit.client.0.smithi116.stdout:7/311: truncate d3/d4/d11/f36 567775 0 2024-03-04T16:04:11.150 INFO:tasks.workunit.client.0.smithi116.stdout:7/312: chown d3/d4/d8/d52/d5e 1785 1 2024-03-04T16:04:11.151 INFO:tasks.workunit.client.0.smithi116.stdout:3/475: creat dc/df/d47/d91/f9f x:0 0 0 2024-03-04T16:04:11.152 INFO:tasks.workunit.client.0.smithi116.stdout:3/476: chown dc/d23/d84/d7c/f96 6303 1 2024-03-04T16:04:11.153 INFO:tasks.workunit.client.0.smithi116.stdout:6/370: symlink d6/d56/d58/d72/l7a 0 2024-03-04T16:04:11.157 INFO:tasks.workunit.client.0.smithi116.stdout:8/406: rename d2/de/l9 to d2/d13/d3c/d46/d6e/l86 0 2024-03-04T16:04:11.161 INFO:tasks.workunit.client.0.smithi116.stdout:9/347: dwrite d2/d19/f2a [0,4194304] 0 2024-03-04T16:04:11.161 INFO:tasks.workunit.client.0.smithi116.stdout:6/371: dwrite d6/f37 [0,4194304] 0 2024-03-04T16:04:11.162 INFO:tasks.workunit.client.0.smithi116.stdout:2/443: unlink d7/d29/d47/f6e 0 2024-03-04T16:04:11.163 INFO:tasks.workunit.client.0.smithi116.stdout:9/348: fsync d2/d26/f39 0 2024-03-04T16:04:11.163 INFO:tasks.workunit.client.0.smithi116.stdout:7/313: mkdir d3/d4/d8/d4a/d65 0 2024-03-04T16:04:11.165 INFO:tasks.workunit.client.0.smithi116.stdout:5/390: symlink dd/l7f 0 2024-03-04T16:04:11.165 INFO:tasks.workunit.client.0.smithi116.stdout:7/314: chown d3/d4/d8/c2b 267402 1 2024-03-04T16:04:11.167 INFO:tasks.workunit.client.0.smithi116.stdout:7/315: readlink d3/d4/d8/d4a/l4f 0 2024-03-04T16:04:11.167 INFO:tasks.workunit.client.0.smithi116.stdout:5/391: dread dd/d1f/d55/d73/d2d/f42 [0,4194304] 0 2024-03-04T16:04:11.168 INFO:tasks.workunit.client.0.smithi116.stdout:3/477: creat dc/d34/d44/d6a/fa0 x:0 0 0 2024-03-04T16:04:11.174 INFO:tasks.workunit.client.0.smithi116.stdout:8/407: creat d2/d13/d5e/f87 x:0 0 0 2024-03-04T16:04:11.180 INFO:tasks.workunit.client.0.smithi116.stdout:6/372: mkdir d6/d56/d58/d72/d7b 0 2024-03-04T16:04:11.182 INFO:tasks.workunit.client.0.smithi116.stdout:9/349: creat d2/d49/f6f x:0 0 0 2024-03-04T16:04:11.186 INFO:tasks.workunit.client.0.smithi116.stdout:8/408: write d2/de/d16/d78/d29/f39 [3598819,97204] 0 2024-03-04T16:04:11.189 INFO:tasks.workunit.client.0.smithi116.stdout:1/291: sync 2024-03-04T16:04:11.200 INFO:tasks.workunit.client.0.smithi116.stdout:6/373: mkdir d6/de/d1e/d7c 0 2024-03-04T16:04:11.200 INFO:tasks.workunit.client.0.smithi116.stdout:9/350: truncate d2/d19/d27/d3b/d4a/f4c 2159199 0 2024-03-04T16:04:11.200 INFO:tasks.workunit.client.0.smithi116.stdout:2/444: link d7/d29/d47/f7a d7/d29/d47/d64/fa7 0 2024-03-04T16:04:11.201 INFO:tasks.workunit.client.0.smithi116.stdout:3/478: rmdir dc/d34/d44/d6a 39 2024-03-04T16:04:11.201 INFO:tasks.workunit.client.0.smithi116.stdout:2/445: fsync d7/d15/f93 0 2024-03-04T16:04:11.202 INFO:tasks.workunit.client.0.smithi116.stdout:1/292: symlink d5/d9/d21/d27/d32/d49/d4c/l5f 0 2024-03-04T16:04:11.202 INFO:tasks.workunit.client.0.smithi116.stdout:1/293: stat d5/d11/c12 0 2024-03-04T16:04:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.229+0000 7f530aba2700 1 -- 172.21.15.116:0/1524195263 --> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5304108110 con 0x7f52f406ce40 2024-03-04T16:04:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.231+0000 7f5301ffb700 1 -- 172.21.15.116:0/1524195263 <== mgr.24387 v2:172.21.15.152:6828/2881662614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f5304108110 con 0x7f52f406ce40 2024-03-04T16:04:11.232 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:04:11.232 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:04:11.232 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:04:11.232 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:04:11.232 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2024-03-04T16:04:11.232 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "1/23 daemons upgraded", 2024-03-04T16:04:11.232 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2024-03-04T16:04:11.232 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:04:11.233 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:04:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 -- 172.21.15.116:0/1524195263 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f52f406ce40 msgr2=0x7f52f406f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f52f406ce40 0x7f52f406f300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f52f8006010 tx=0x7f52f800b580 comp rx=0 tx=0).stop 2024-03-04T16:04:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 -- 172.21.15.116:0/1524195263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5304074d60 msgr2=0x7f5304198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5304074d60 0x7f5304198ab0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52f000d8d0 tx=0x7f52f000dc90 comp rx=0 tx=0).stop 2024-03-04T16:04:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 -- 172.21.15.116:0/1524195263 shutdown_connections 2024-03-04T16:04:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f52f406ce40 0x7f52f406f300 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5304073790 0x7f5304198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 --2- 172.21.15.116:0/1524195263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5304074d60 0x7f5304198ab0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 -- 172.21.15.116:0/1524195263 >> 172.21.15.116:0/1524195263 conn(0x7f53040fc110 msgr2=0x7f53041069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 -- 172.21.15.116:0/1524195263 shutdown_connections 2024-03-04T16:04:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.233+0000 7f530aba2700 1 -- 172.21.15.116:0/1524195263 wait complete. 2024-03-04T16:04:11.263 INFO:tasks.workunit.client.0.smithi116.stdout:6/374: symlink d6/de/d29/d53/l7d 0 2024-03-04T16:04:11.263 INFO:tasks.workunit.client.0.smithi116.stdout:3/479: creat dc/d23/d84/d7c/fa1 x:0 0 0 2024-03-04T16:04:11.264 INFO:tasks.workunit.client.0.smithi116.stdout:1/294: dread - d5/d9/d21/d27/d32/d35/f43 zero size 2024-03-04T16:04:11.265 INFO:tasks.workunit.client.0.smithi116.stdout:9/351: link d2/d33/f54 d2/d19/d27/d3b/f70 0 2024-03-04T16:04:11.268 INFO:tasks.workunit.client.0.smithi116.stdout:6/375: mknod d6/de/d25/d4a/d63/c7e 0 2024-03-04T16:04:11.270 INFO:tasks.workunit.client.0.smithi116.stdout:8/409: dwrite d2/de/d3e/f50 [0,4194304] 0 2024-03-04T16:04:11.272 INFO:tasks.workunit.client.0.smithi116.stdout:6/376: write d6/de/d15/f6f [607896,92806] 0 2024-03-04T16:04:11.272 INFO:tasks.workunit.client.0.smithi116.stdout:9/352: fsync d2/f15 0 2024-03-04T16:04:11.273 INFO:tasks.workunit.client.0.smithi116.stdout:6/377: write d6/de/f4d [639667,114566] 0 2024-03-04T16:04:11.273 INFO:tasks.workunit.client.0.smithi116.stdout:1/295: mknod d5/d9/d46/c60 0 2024-03-04T16:04:11.276 INFO:tasks.workunit.client.0.smithi116.stdout:3/480: symlink dc/d34/d44/d6a/la2 0 2024-03-04T16:04:11.276 INFO:tasks.workunit.client.0.smithi116.stdout:6/378: write d6/de/d25/d36/d44/f4b [9187731,115299] 0 2024-03-04T16:04:11.277 INFO:tasks.workunit.client.0.smithi116.stdout:9/353: write d2/f4f [121375,42149] 0 2024-03-04T16:04:11.281 INFO:tasks.workunit.client.0.smithi116.stdout:8/410: mknod d2/de/d16/d78/d29/d7a/c88 0 2024-03-04T16:04:11.284 INFO:tasks.workunit.client.0.smithi116.stdout:3/481: dread dc/d34/f9a [0,4194304] 0 2024-03-04T16:04:11.285 INFO:tasks.workunit.client.0.smithi116.stdout:3/482: dread dc/d34/f9a [0,4194304] 0 2024-03-04T16:04:11.286 INFO:tasks.workunit.client.0.smithi116.stdout:9/354: rmdir d2/d33/d65 39 2024-03-04T16:04:11.287 INFO:tasks.workunit.client.0.smithi116.stdout:9/355: write d2/f48 [2672589,43467] 0 2024-03-04T16:04:11.288 INFO:tasks.workunit.client.0.smithi116.stdout:9/356: write d2/d19/d27/d3b/d4a/d5b/f24 [206347,56508] 0 2024-03-04T16:04:11.288 INFO:tasks.workunit.client.0.smithi116.stdout:0/348: sync 2024-03-04T16:04:11.288 INFO:tasks.workunit.client.0.smithi116.stdout:4/433: sync 2024-03-04T16:04:11.291 INFO:tasks.workunit.client.0.smithi116.stdout:8/411: symlink d2/d13/d1e/d42/l89 0 2024-03-04T16:04:11.293 INFO:tasks.workunit.client.0.smithi116.stdout:1/296: unlink d5/d9/d21/f36 0 2024-03-04T16:04:11.297 INFO:tasks.workunit.client.0.smithi116.stdout:9/357: readlink d2/d19/l5d 0 2024-03-04T16:04:11.300 INFO:tasks.workunit.client.0.smithi116.stdout:6/379: mknod d6/d56/d58/d72/d7b/c7f 0 2024-03-04T16:04:11.301 INFO:tasks.workunit.client.0.smithi116.stdout:0/349: creat da/d16/d28/d3d/d41/f73 x:0 0 0 2024-03-04T16:04:11.306 INFO:tasks.workunit.client.0.smithi116.stdout:9/358: creat d2/d26/f71 x:0 0 0 2024-03-04T16:04:11.307 INFO:tasks.workunit.client.0.smithi116.stdout:4/434: fsync d1/d3/dd/d45/f4b 0 2024-03-04T16:04:11.310 INFO:tasks.workunit.client.0.smithi116.stdout:6/380: rename d6/de/d29/d53 to d6/de/d1e/d80 0 2024-03-04T16:04:11.313 INFO:tasks.workunit.client.0.smithi116.stdout:2/446: truncate d7/f36 3645291 0 2024-03-04T16:04:11.314 INFO:tasks.workunit.client.0.smithi116.stdout:8/412: mkdir d2/d4a/d8a 0 2024-03-04T16:04:11.317 INFO:tasks.workunit.client.0.smithi116.stdout:8/413: dread d2/d38/f3b [0,4194304] 0 2024-03-04T16:04:11.318 INFO:tasks.workunit.client.0.smithi116.stdout:4/435: readlink d1/d3/dd/l17 0 2024-03-04T16:04:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:6/381: symlink d6/de/d29/l81 0 2024-03-04T16:04:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:2/447: rename d7/d15/l1d to d7/d2f/d76/d7c/la8 0 2024-03-04T16:04:11.327 INFO:tasks.workunit.client.0.smithi116.stdout:9/359: symlink d2/d19/d27/d3b/d4a/l72 0 2024-03-04T16:04:11.329 INFO:tasks.workunit.client.0.smithi116.stdout:8/414: chown d2/d13/d3c/f80 6 1 2024-03-04T16:04:11.330 INFO:tasks.workunit.client.0.smithi116.stdout:8/415: write d2/d13/d1e/d75/f79 [473284,31536] 0 2024-03-04T16:04:11.333 INFO:tasks.workunit.client.0.smithi116.stdout:4/436: rename d1/d3/d15/d20/d65/f78 to d1/d3/d15/d20/d65/f99 0 2024-03-04T16:04:11.334 INFO:tasks.workunit.client.0.smithi116.stdout:2/448: symlink d7/d2f/d76/d7c/la9 0 2024-03-04T16:04:11.335 INFO:tasks.workunit.client.0.smithi116.stdout:2/449: truncate d7/d81/f1c 8581756 0 2024-03-04T16:04:11.338 INFO:tasks.workunit.client.0.smithi116.stdout:9/360: fdatasync d2/d19/d27/d3b/d4a/d5b/f2f 0 2024-03-04T16:04:11.339 INFO:tasks.workunit.client.0.smithi116.stdout:9/361: chown d2/d19/d27/d3b/d4a/d5b/f43 601177010 1 2024-03-04T16:04:11.339 INFO:tasks.workunit.client.0.smithi116.stdout:8/416: mknod d2/d4a/c8b 0 2024-03-04T16:04:11.341 INFO:tasks.workunit.client.0.smithi116.stdout:9/362: dread d2/d19/d27/d3b/d4a/d5b/f25 [0,4194304] 0 2024-03-04T16:04:11.344 INFO:tasks.workunit.client.0.smithi116.stdout:8/417: unlink d2/de/d3e/l20 0 2024-03-04T16:04:11.349 INFO:tasks.workunit.client.0.smithi116.stdout:1/297: truncate d5/d9/d21/d27/d32/d35/f52 2878446 0 2024-03-04T16:04:11.353 INFO:tasks.workunit.client.0.smithi116.stdout:0/350: truncate da/ff 1582139 0 2024-03-04T16:04:11.353 INFO:tasks.workunit.client.0.smithi116.stdout:9/363: dwrite d2/d33/d65/d6c/f66 [0,4194304] 0 2024-03-04T16:04:11.355 INFO:tasks.workunit.client.0.smithi116.stdout:8/418: symlink d2/de/d16/d78/l8c 0 2024-03-04T16:04:11.358 INFO:tasks.workunit.client.0.smithi116.stdout:1/298: mkdir d5/d11/d61 0 2024-03-04T16:04:11.359 INFO:tasks.workunit.client.0.smithi116.stdout:1/299: truncate f2 1669784 0 2024-03-04T16:04:11.360 INFO:tasks.workunit.client.0.smithi116.stdout:9/364: dwrite d2/f1a [0,4194304] 0 2024-03-04T16:04:11.361 INFO:tasks.workunit.client.0.smithi116.stdout:9/365: write d2/d26/f71 [7128,72484] 0 2024-03-04T16:04:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.365+0000 7f8e092dd700 1 -- 172.21.15.116:0/3481351252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e04100cb0 msgr2=0x7f8e04101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.365+0000 7f8e092dd700 1 --2- 172.21.15.116:0/3481351252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e04100cb0 0x7f8e04101110 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8df8009910 tx=0x7f8df8009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.365+0000 7f8e092dd700 1 -- 172.21.15.116:0/3481351252 shutdown_connections 2024-03-04T16:04:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.365+0000 7f8e092dd700 1 --2- 172.21.15.116:0/3481351252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e04100cb0 0x7f8e04101110 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.365+0000 7f8e092dd700 1 --2- 172.21.15.116:0/3481351252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e040ffab0 0x7f8e040ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.365+0000 7f8e092dd700 1 -- 172.21.15.116:0/3481351252 >> 172.21.15.116:0/3481351252 conn(0x7f8e040fb050 msgr2=0x7f8e040fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.365+0000 7f8e092dd700 1 -- 172.21.15.116:0/3481351252 shutdown_connections 2024-03-04T16:04:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.365+0000 7f8e092dd700 1 -- 172.21.15.116:0/3481351252 wait complete. 2024-03-04T16:04:11.367 INFO:tasks.workunit.client.0.smithi116.stdout:4/437: link d1/d3/l94 d1/d3/d15/l9a 0 2024-03-04T16:04:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.366+0000 7f8e092dd700 1 Processor -- start 2024-03-04T16:04:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e092dd700 1 -- start start 2024-03-04T16:04:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e092dd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e040ffab0 0x7f8e04103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e092dd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e04100cb0 0x7f8e04102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e092dd700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e041025d0 con 0x7f8e040ffab0 2024-03-04T16:04:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e092dd700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e04102710 con 0x7f8e04100cb0 2024-03-04T16:04:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e02ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e040ffab0 0x7f8e04103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e02ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e040ffab0 0x7f8e04103a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50334/0 (socket says 172.21.15.116:50334) 2024-03-04T16:04:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e02ffd700 1 -- 172.21.15.116:0/3427877196 learned_addr learned my addr 172.21.15.116:0/3427877196 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e02ffd700 1 -- 172.21.15.116:0/3427877196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e04100cb0 msgr2=0x7f8e04102090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e02ffd700 1 --2- 172.21.15.116:0/3427877196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e04100cb0 0x7f8e04102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e02ffd700 1 -- 172.21.15.116:0/3427877196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df80095f0 con 0x7f8e040ffab0 2024-03-04T16:04:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.367+0000 7f8e02ffd700 1 --2- 172.21.15.116:0/3427877196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e040ffab0 0x7f8e04103a40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8df400b480 tx=0x7f8df400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.368+0000 7f8debfff700 1 -- 172.21.15.116:0/3427877196 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8df40118c0 con 0x7f8e040ffab0 2024-03-04T16:04:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.368+0000 7f8e092dd700 1 -- 172.21.15.116:0/3427877196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e041029f0 con 0x7f8e040ffab0 2024-03-04T16:04:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.368+0000 7f8debfff700 1 -- 172.21.15.116:0/3427877196 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8df4010380 con 0x7f8e040ffab0 2024-03-04T16:04:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.368+0000 7f8debfff700 1 -- 172.21.15.116:0/3427877196 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8df400f4f0 con 0x7f8e040ffab0 2024-03-04T16:04:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.368+0000 7f8e092dd700 1 -- 172.21.15.116:0/3427877196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0419e460 con 0x7f8e040ffab0 2024-03-04T16:04:11.371 INFO:tasks.workunit.client.0.smithi116.stdout:1/300: creat d5/d11/d42/f62 x:0 0 0 2024-03-04T16:04:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.369+0000 7f8e092dd700 1 -- 172.21.15.116:0/3427877196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e041086b0 con 0x7f8e040ffab0 2024-03-04T16:04:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.370+0000 7f8debfff700 1 -- 172.21.15.116:0/3427877196 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) v1 ==== 90928+0+0 (secure 0 0 0) 0x7f8df40104f0 con 0x7f8e040ffab0 2024-03-04T16:04:11.371 INFO:tasks.workunit.client.0.smithi116.stdout:1/301: stat d5/d9/d21/d27/d32/d35/f53 0 2024-03-04T16:04:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.370+0000 7f8debfff700 1 --2- 172.21.15.116:0/3427877196 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8dec06ce90 0x7f8dec06f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.371+0000 7f8debfff700 1 -- 172.21.15.116:0/3427877196 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f8df408c260 con 0x7f8e040ffab0 2024-03-04T16:04:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.371+0000 7f8e027fc700 1 --2- 172.21.15.116:0/3427877196 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8dec06ce90 0x7f8dec06f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.372+0000 7f8e027fc700 1 --2- 172.21.15.116:0/3427877196 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8dec06ce90 0x7f8dec06f350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8df800b5c0 tx=0x7f8df8005960 comp rx=0 tx=0).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:11.373 INFO:tasks.workunit.client.0.smithi116.stdout:6/382: write d6/de/d25/f40 [3634595,57407] 0 2024-03-04T16:04:11.374 INFO:tasks.workunit.client.0.smithi116.stdout:6/383: fdatasync d6/de/d25/d4a/d63/f74 0 2024-03-04T16:04:11.375 INFO:tasks.workunit.client.0.smithi116.stdout:5/392: sync 2024-03-04T16:04:11.375 INFO:tasks.workunit.client.0.smithi116.stdout:7/316: sync 2024-03-04T16:04:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.376+0000 7f8debfff700 1 -- 172.21.15.116:0/3427877196 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f8df4059b30 con 0x7f8e040ffab0 2024-03-04T16:04:11.380 INFO:tasks.workunit.client.0.smithi116.stdout:1/302: creat d5/d9/d46/f63 x:0 0 0 2024-03-04T16:04:11.381 INFO:tasks.workunit.client.0.smithi116.stdout:2/450: truncate d7/d13/f52 68693 0 2024-03-04T16:04:11.386 INFO:tasks.workunit.client.0.smithi116.stdout:6/384: unlink d6/c3e 0 2024-03-04T16:04:11.387 INFO:tasks.workunit.client.0.smithi116.stdout:4/438: symlink d1/d3/dd/d24/l9b 0 2024-03-04T16:04:11.393 INFO:tasks.workunit.client.0.smithi116.stdout:7/317: rename d3/d4/d8/d4a/d64 to d3/d4/d34/d5d/d66 0 2024-03-04T16:04:11.393 INFO:tasks.workunit.client.0.smithi116.stdout:7/318: write d3/f50 [693799,72771] 0 2024-03-04T16:04:11.397 INFO:tasks.workunit.client.0.smithi116.stdout:0/351: getdents da/d16/d28/d3d 0 2024-03-04T16:04:11.398 INFO:tasks.workunit.client.0.smithi116.stdout:1/303: symlink d5/d9/d21/d24/l64 0 2024-03-04T16:04:11.400 INFO:tasks.workunit.client.0.smithi116.stdout:8/419: link d2/de/l5 d2/d13/d5e/l8d 0 2024-03-04T16:04:11.401 INFO:tasks.workunit.client.0.smithi116.stdout:8/420: chown d2/d13/d3c/d46/c76 546348 1 2024-03-04T16:04:11.402 INFO:tasks.workunit.client.0.smithi116.stdout:6/385: rmdir d6/de/d25 39 2024-03-04T16:04:11.402 INFO:tasks.workunit.client.0.smithi116.stdout:8/421: write d2/d13/d5e/f87 [82416,15669] 0 2024-03-04T16:04:11.404 INFO:tasks.workunit.client.0.smithi116.stdout:4/439: chown d1/d3/l94 6122351 1 2024-03-04T16:04:11.406 INFO:tasks.workunit.client.0.smithi116.stdout:9/366: getdents d2/d33 0 2024-03-04T16:04:11.413 INFO:tasks.workunit.client.0.smithi116.stdout:5/393: write dd/f13 [609932,76821] 0 2024-03-04T16:04:11.417 INFO:tasks.workunit.client.0.smithi116.stdout:2/451: write d7/d2f/d6c/da4/f69 [1486847,109055] 0 2024-03-04T16:04:11.421 INFO:tasks.workunit.client.0.smithi116.stdout:5/394: dwrite dd/d1f/f4d [0,4194304] 0 2024-03-04T16:04:11.422 INFO:tasks.workunit.client.0.smithi116.stdout:5/395: chown dd/d1f/l44 11869242 1 2024-03-04T16:04:11.423 INFO:tasks.workunit.client.0.smithi116.stdout:7/319: write d3/f10 [844819,17749] 0 2024-03-04T16:04:11.423 INFO:tasks.workunit.client.0.smithi116.stdout:8/422: dwrite d2/d13/d1e/f44 [0,4194304] 0 2024-03-04T16:04:11.425 INFO:tasks.workunit.client.0.smithi116.stdout:1/304: mknod d5/c65 0 2024-03-04T16:04:11.431 INFO:tasks.workunit.client.0.smithi116.stdout:6/386: symlink d6/de/d25/d4a/l82 0 2024-03-04T16:04:11.431 INFO:tasks.workunit.client.0.smithi116.stdout:4/440: mknod d1/d3/d15/c9c 0 2024-03-04T16:04:11.433 INFO:tasks.workunit.client.0.smithi116.stdout:4/441: readlink d1/d6c/l92 0 2024-03-04T16:04:11.433 INFO:tasks.workunit.client.0.smithi116.stdout:9/367: dwrite d2/d33/f46 [0,4194304] 0 2024-03-04T16:04:11.433 INFO:tasks.workunit.client.0.smithi116.stdout:2/452: creat d7/d2f/d6c/da4/faa x:0 0 0 2024-03-04T16:04:11.434 INFO:tasks.workunit.client.0.smithi116.stdout:2/453: fsync d7/d15/f30 0 2024-03-04T16:04:11.436 INFO:tasks.workunit.client.0.smithi116.stdout:5/396: creat dd/d6d/f80 x:0 0 0 2024-03-04T16:04:11.436 INFO:tasks.workunit.client.0.smithi116.stdout:9/368: dread d2/d19/d5f/f61 [0,4194304] 0 2024-03-04T16:04:11.437 INFO:tasks.workunit.client.0.smithi116.stdout:5/397: read - dd/d6d/f80 zero size 2024-03-04T16:04:11.438 INFO:tasks.workunit.client.0.smithi116.stdout:7/320: mkdir d3/d4/d11/d67 0 2024-03-04T16:04:11.440 INFO:tasks.workunit.client.0.smithi116.stdout:8/423: rename d2/d38/f71 to d2/de/d3e/d7b/f8e 0 2024-03-04T16:04:11.442 INFO:tasks.workunit.client.0.smithi116.stdout:1/305: creat d5/d9/d21/d27/d32/d49/f66 x:0 0 0 2024-03-04T16:04:11.444 INFO:tasks.workunit.client.0.smithi116.stdout:6/387: mknod d6/c83 0 2024-03-04T16:04:11.449 INFO:tasks.workunit.client.0.smithi116.stdout:2/454: symlink d7/d29/d47/d64/d75/lab 0 2024-03-04T16:04:11.449 INFO:tasks.workunit.client.0.smithi116.stdout:2/455: chown d7/d13/f1a 4069 1 2024-03-04T16:04:11.451 INFO:tasks.workunit.client.0.smithi116.stdout:2/456: write d7/d15/f59 [136737,118156] 0 2024-03-04T16:04:11.451 INFO:tasks.workunit.client.0.smithi116.stdout:5/398: creat dd/d1f/d55/d73/d2d/d59/f81 x:0 0 0 2024-03-04T16:04:11.453 INFO:tasks.workunit.client.0.smithi116.stdout:8/424: mknod d2/de/d16/d78/c8f 0 2024-03-04T16:04:11.454 INFO:tasks.workunit.client.0.smithi116.stdout:4/442: symlink d1/d3/d15/d20/d1c/l9d 0 2024-03-04T16:04:11.456 INFO:tasks.workunit.client.0.smithi116.stdout:7/321: getdents d3/d4/d11/d67 0 2024-03-04T16:04:11.456 INFO:tasks.workunit.client.0.smithi116.stdout:7/322: fsync d3/d4/d11/f5b 0 2024-03-04T16:04:11.457 INFO:tasks.workunit.client.0.smithi116.stdout:2/457: mknod d7/d13/d3e/cac 0 2024-03-04T16:04:11.458 INFO:tasks.workunit.client.0.smithi116.stdout:5/399: mknod dd/d1f/d55/d73/d2d/c82 0 2024-03-04T16:04:11.460 INFO:tasks.workunit.client.0.smithi116.stdout:6/388: mkdir d6/de/d84 0 2024-03-04T16:04:11.462 INFO:tasks.workunit.client.0.smithi116.stdout:7/323: dread d3/f4e [0,4194304] 0 2024-03-04T16:04:11.465 INFO:tasks.workunit.client.0.smithi116.stdout:2/458: mkdir d7/d13/d3e/d71/dad 0 2024-03-04T16:04:11.466 INFO:tasks.workunit.client.0.smithi116.stdout:1/306: dread d5/d11/f4e [0,4194304] 0 2024-03-04T16:04:11.466 INFO:tasks.workunit.client.0.smithi116.stdout:8/425: creat d2/d13/f90 x:0 0 0 2024-03-04T16:04:11.469 INFO:tasks.workunit.client.0.smithi116.stdout:7/324: unlink d3/d4/d8/d1a/f2e 0 2024-03-04T16:04:11.470 INFO:tasks.workunit.client.0.smithi116.stdout:5/400: mknod dd/d1f/d51/c83 0 2024-03-04T16:04:11.474 INFO:tasks.workunit.client.0.smithi116.stdout:8/426: creat d2/d13/d1e/d5a/f91 x:0 0 0 2024-03-04T16:04:11.475 INFO:tasks.workunit.client.0.smithi116.stdout:1/307: mkdir d5/d9/d21/d27/d32/d49/d4c/d67 0 2024-03-04T16:04:11.482 INFO:tasks.workunit.client.0.smithi116.stdout:6/389: creat d6/de/d25/f85 x:0 0 0 2024-03-04T16:04:11.484 INFO:tasks.workunit.client.0.smithi116.stdout:4/443: dread d1/d3/d15/d20/f56 [4194304,4194304] 0 2024-03-04T16:04:11.484 INFO:tasks.workunit.client.0.smithi116.stdout:2/459: symlink d7/d13/d3e/d71/dad/lae 0 2024-03-04T16:04:11.485 INFO:tasks.workunit.client.0.smithi116.stdout:8/427: read d2/de/d3e/d7b/f8e [430576,81151] 0 2024-03-04T16:04:11.486 INFO:tasks.workunit.client.0.smithi116.stdout:4/444: read d1/f75 [3284266,80389] 0 2024-03-04T16:04:11.489 INFO:tasks.workunit.client.0.smithi116.stdout:5/401: unlink dd/d1f/d55/d73/d2d/l32 0 2024-03-04T16:04:11.490 INFO:tasks.workunit.client.0.smithi116.stdout:6/390: symlink d6/de/d25/d4a/d63/l86 0 2024-03-04T16:04:11.492 INFO:tasks.workunit.client.0.smithi116.stdout:5/402: write dd/d6d/f80 [941133,64771] 0 2024-03-04T16:04:11.493 INFO:tasks.workunit.client.0.smithi116.stdout:6/391: chown d6/de/d25/f85 14166669 1 2024-03-04T16:04:11.495 INFO:tasks.workunit.client.0.smithi116.stdout:8/428: dwrite d2/d13/d5e/f73 [0,4194304] 0 2024-03-04T16:04:11.498 INFO:tasks.workunit.client.0.smithi116.stdout:9/369: dwrite d2/d33/f54 [4194304,4194304] 0 2024-03-04T16:04:11.499 INFO:tasks.workunit.client.0.smithi116.stdout:7/325: creat d3/f68 x:0 0 0 2024-03-04T16:04:11.500 INFO:tasks.workunit.client.0.smithi116.stdout:4/445: chown d1/d3/d15/d20/l64 173 1 2024-03-04T16:04:11.504 INFO:tasks.workunit.client.0.smithi116.stdout:9/370: dread d2/d19/d27/d3b/d4a/d5b/f24 [4194304,4194304] 0 2024-03-04T16:04:11.504 INFO:tasks.workunit.client.0.smithi116.stdout:2/460: mknod d7/d13/d3e/d98/caf 0 2024-03-04T16:04:11.508 INFO:tasks.workunit.client.0.smithi116.stdout:6/392: mknod d6/de/d25/d36/d4e/c87 0 2024-03-04T16:04:11.510 INFO:tasks.workunit.client.0.smithi116.stdout:5/403: rename dd/l7f to dd/d1f/d55/d73/d2d/d31/l84 0 2024-03-04T16:04:11.511 INFO:tasks.workunit.client.0.smithi116.stdout:5/404: dread - dd/d1f/d55/d73/d2d/d59/f81 zero size 2024-03-04T16:04:11.515 INFO:tasks.workunit.client.0.smithi116.stdout:3/483: sync 2024-03-04T16:04:11.516 INFO:tasks.workunit.client.0.smithi116.stdout:7/326: rmdir d3/d4/d8/d1a 39 2024-03-04T16:04:11.518 INFO:tasks.workunit.client.0.smithi116.stdout:4/446: mknod d1/d3/c9e 0 2024-03-04T16:04:11.518 INFO:tasks.workunit.client.0.smithi116.stdout:8/429: dwrite d2/d13/d3c/f69 [0,4194304] 0 2024-03-04T16:04:11.523 INFO:tasks.workunit.client.0.smithi116.stdout:6/393: creat d6/de/d1e/f88 x:0 0 0 2024-03-04T16:04:11.527 INFO:tasks.workunit.client.0.smithi116.stdout:6/394: chown d6/de/d25/d36/l32 1030484 1 2024-03-04T16:04:11.531 INFO:tasks.workunit.client.0.smithi116.stdout:3/484: dwrite dc/d23/d84/d7c/f62 [0,4194304] 0 2024-03-04T16:04:11.534 INFO:tasks.workunit.client.0.smithi116.stdout:5/405: unlink dd/d1f/d55/d73/d2d/c5a 0 2024-03-04T16:04:11.535 INFO:tasks.workunit.client.0.smithi116.stdout:8/430: dwrite d2/d13/d3c/d46/d4f/f5b [0,4194304] 0 2024-03-04T16:04:11.539 INFO:tasks.workunit.client.0.smithi116.stdout:5/406: chown dd/d1f/d55/d73/f3f 29 1 2024-03-04T16:04:11.540 INFO:tasks.workunit.client.0.smithi116.stdout:1/308: dwrite d5/d9/d21/d27/d32/f3e [0,4194304] 0 2024-03-04T16:04:11.543 INFO:tasks.workunit.client.0.smithi116.stdout:9/371: creat d2/d19/d27/d3b/d4a/f73 x:0 0 0 2024-03-04T16:04:11.544 INFO:tasks.workunit.client.0.smithi116.stdout:1/309: write d5/d9/f3f [130748,91663] 0 2024-03-04T16:04:11.546 INFO:tasks.workunit.client.0.smithi116.stdout:5/407: read dd/fe [1502491,49156] 0 2024-03-04T16:04:11.547 INFO:tasks.workunit.client.0.smithi116.stdout:2/461: dread d7/f24 [0,4194304] 0 2024-03-04T16:04:11.547 INFO:tasks.workunit.client.0.smithi116.stdout:6/395: creat d6/de/d1e/d80/f89 x:0 0 0 2024-03-04T16:04:11.547 INFO:tasks.workunit.client.0.smithi116.stdout:3/485: mkdir dc/d34/d44/d6a/da3 0 2024-03-04T16:04:11.548 INFO:tasks.workunit.client.0.smithi116.stdout:8/431: symlink d2/d13/d1e/d42/l92 0 2024-03-04T16:04:11.548 INFO:tasks.workunit.client.0.smithi116.stdout:5/408: fdatasync dd/d1f/d55/f77 0 2024-03-04T16:04:11.549 INFO:tasks.workunit.client.0.smithi116.stdout:3/486: stat dc/d23/d84/f76 0 2024-03-04T16:04:11.552 INFO:tasks.workunit.client.0.smithi116.stdout:9/372: rmdir d2/d19/d5f 39 2024-03-04T16:04:11.553 INFO:tasks.workunit.client.0.smithi116.stdout:2/462: dread d7/fb [0,4194304] 0 2024-03-04T16:04:11.554 INFO:tasks.workunit.client.0.smithi116.stdout:2/463: stat d7/d13/f1a 0 2024-03-04T16:04:11.554 INFO:tasks.workunit.client.0.smithi116.stdout:1/310: fsync d5/d11/f4e 0 2024-03-04T16:04:11.558 INFO:tasks.workunit.client.0.smithi116.stdout:5/409: dwrite dd/d1f/d55/d73/d2d/d31/f65 [0,4194304] 0 2024-03-04T16:04:11.558 INFO:tasks.workunit.client.0.smithi116.stdout:6/396: rmdir d6/de/d25/d4a/d63 39 2024-03-04T16:04:11.561 INFO:tasks.workunit.client.0.smithi116.stdout:5/410: write dd/d1f/d5c/d5e/f62 [1026157,109277] 0 2024-03-04T16:04:11.568 INFO:tasks.workunit.client.0.smithi116.stdout:7/327: dwrite d3/d4/d8/d1a/f46 [0,4194304] 0 2024-03-04T16:04:11.569 INFO:tasks.workunit.client.0.smithi116.stdout:1/311: stat d5/d9/f41 0 2024-03-04T16:04:11.570 INFO:tasks.workunit.client.0.smithi116.stdout:1/312: chown d5/d9/d21/d27/d32/d49/d4c/d67 142090 1 2024-03-04T16:04:11.572 INFO:tasks.workunit.client.0.smithi116.stdout:1/313: read d5/d9/f41 [71767,14813] 0 2024-03-04T16:04:11.574 INFO:tasks.workunit.client.0.smithi116.stdout:6/397: truncate d6/de/d1e/f2a 620951 0 2024-03-04T16:04:11.579 INFO:tasks.workunit.client.0.smithi116.stdout:5/411: rename dd/d1f/d55/d73/d2d/l2a to dd/d1f/d55/d73/l85 0 2024-03-04T16:04:11.579 INFO:tasks.workunit.client.0.smithi116.stdout:3/487: dwrite dc/d23/d84/f2d [0,4194304] 0 2024-03-04T16:04:11.580 INFO:tasks.workunit.client.0.smithi116.stdout:8/432: link d2/d13/d1e/d75/f79 d2/d13/d1e/f93 0 2024-03-04T16:04:11.582 INFO:tasks.workunit.client.0.smithi116.stdout:8/433: chown d2/de/d16/d2b/c2d 405617 1 2024-03-04T16:04:11.582 INFO:tasks.workunit.client.0.smithi116.stdout:2/464: mknod d7/d2f/d6c/da4/da1/cb0 0 2024-03-04T16:04:11.584 INFO:tasks.workunit.client.0.smithi116.stdout:5/412: chown dd/d45/f49 626 1 2024-03-04T16:04:11.587 INFO:tasks.workunit.client.0.smithi116.stdout:1/314: symlink d5/d9/d21/d27/d32/d35/d55/d58/l68 0 2024-03-04T16:04:11.589 INFO:tasks.workunit.client.0.smithi116.stdout:2/465: creat d7/d13/d3e/d78/fb1 x:0 0 0 2024-03-04T16:04:11.589 INFO:tasks.workunit.client.0.smithi116.stdout:2/466: chown d7/d2f 1612656 1 2024-03-04T16:04:11.590 INFO:tasks.workunit.client.0.smithi116.stdout:2/467: fdatasync d7/d2f/f50 0 2024-03-04T16:04:11.592 INFO:tasks.workunit.client.0.smithi116.stdout:3/488: mknod dc/d23/ca4 0 2024-03-04T16:04:11.595 INFO:tasks.workunit.client.0.smithi116.stdout:1/315: creat d5/d11/f69 x:0 0 0 2024-03-04T16:04:11.597 INFO:tasks.workunit.client.0.smithi116.stdout:1/316: write d5/d9/d21/d27/d32/f47 [1253384,94857] 0 2024-03-04T16:04:11.599 INFO:tasks.workunit.client.0.smithi116.stdout:6/398: dread d6/de/d1e/f2a [0,4194304] 0 2024-03-04T16:04:11.600 INFO:tasks.workunit.client.0.smithi116.stdout:3/489: rename dc/c90 to dc/d23/d84/d7f/d8b/ca5 0 2024-03-04T16:04:11.605 INFO:tasks.workunit.client.0.smithi116.stdout:1/317: creat d5/d11/f6a x:0 0 0 2024-03-04T16:04:11.606 INFO:tasks.workunit.client.0.smithi116.stdout:1/318: write d5/f4d [137061,16500] 0 2024-03-04T16:04:11.607 INFO:tasks.workunit.client.0.smithi116.stdout:8/434: dread d2/de/f10 [0,4194304] 0 2024-03-04T16:04:11.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:11 smithi152 ceph-mon[36876]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:11.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:11 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3843926793' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:04:11.608 INFO:tasks.workunit.client.0.smithi116.stdout:3/490: mknod dc/df/d14/ca6 0 2024-03-04T16:04:11.609 INFO:tasks.workunit.client.0.smithi116.stdout:3/491: chown dc/f33 1636971 1 2024-03-04T16:04:11.610 INFO:tasks.workunit.client.0.smithi116.stdout:1/319: mknod d5/d9/d46/d50/c6b 0 2024-03-04T16:04:11.610 INFO:tasks.workunit.client.0.smithi116.stdout:3/492: write dc/d23/d84/d7f/f86 [7175,127001] 0 2024-03-04T16:04:11.611 INFO:tasks.workunit.client.0.smithi116.stdout:8/435: creat d2/d13/d1e/f94 x:0 0 0 2024-03-04T16:04:11.614 INFO:tasks.workunit.client.0.smithi116.stdout:7/328: dwrite d3/d4/f16 [0,4194304] 0 2024-03-04T16:04:11.616 INFO:tasks.workunit.client.0.smithi116.stdout:3/493: symlink dc/df/d47/d91/la7 0 2024-03-04T16:04:11.617 INFO:tasks.workunit.client.0.smithi116.stdout:8/436: mkdir d2/de/d16/d78/d29/d95 0 2024-03-04T16:04:11.621 INFO:tasks.workunit.client.0.smithi116.stdout:8/437: dread d2/de/d3e/f2a [0,4194304] 0 2024-03-04T16:04:11.621 INFO:tasks.workunit.client.0.smithi116.stdout:1/320: mknod d5/d9/d21/d27/d32/d49/d4c/d67/c6c 0 2024-03-04T16:04:11.622 INFO:tasks.workunit.client.0.smithi116.stdout:7/329: dwrite d3/f6 [0,4194304] 0 2024-03-04T16:04:11.622 INFO:tasks.workunit.client.0.smithi116.stdout:3/494: chown dc/df/f79 636 1 2024-03-04T16:04:11.625 INFO:tasks.workunit.client.0.smithi116.stdout:7/330: write d3/d4/d8/f13 [47623,11623] 0 2024-03-04T16:04:11.626 INFO:tasks.workunit.client.0.smithi116.stdout:1/321: write d5/d9/d21/d24/f3a [3317599,19984] 0 2024-03-04T16:04:11.627 INFO:tasks.workunit.client.0.smithi116.stdout:2/468: dwrite f4 [0,4194304] 0 2024-03-04T16:04:11.630 INFO:tasks.workunit.client.0.smithi116.stdout:3/495: write dc/d34/d44/d6a/f6b [4018701,97246] 0 2024-03-04T16:04:11.635 INFO:tasks.workunit.client.0.smithi116.stdout:7/331: mkdir d3/d4/d8/d52/d69 0 2024-03-04T16:04:11.636 INFO:tasks.workunit.client.0.smithi116.stdout:7/332: chown d3/d4/d34/l62 12004 1 2024-03-04T16:04:11.637 INFO:tasks.workunit.client.0.smithi116.stdout:7/333: fsync d3/f18 0 2024-03-04T16:04:11.637 INFO:tasks.workunit.client.0.smithi116.stdout:1/322: creat d5/d9/d21/d27/d32/d49/f6d x:0 0 0 2024-03-04T16:04:11.646 INFO:tasks.workunit.client.0.smithi116.stdout:7/334: mknod d3/d4/d8/c6a 0 2024-03-04T16:04:11.647 INFO:tasks.workunit.client.0.smithi116.stdout:7/335: readlink d3/d4/d8/d1a/l3a 0 2024-03-04T16:04:11.648 INFO:tasks.workunit.client.0.smithi116.stdout:7/336: write d3/f56 [927679,109564] 0 2024-03-04T16:04:11.649 INFO:tasks.workunit.client.0.smithi116.stdout:1/323: link d5/d11/l31 d5/d9/d21/d27/d32/d49/d4a/l6e 0 2024-03-04T16:04:11.650 INFO:tasks.workunit.client.0.smithi116.stdout:3/496: write dc/df/f7e [16357,7081] 0 2024-03-04T16:04:11.650 INFO:tasks.workunit.client.0.smithi116.stdout:3/497: fsync dc/f64 0 2024-03-04T16:04:11.651 INFO:tasks.workunit.client.0.smithi116.stdout:1/324: write d5/d9/d21/d27/d32/f47 [933859,84030] 0 2024-03-04T16:04:11.654 INFO:tasks.workunit.client.0.smithi116.stdout:2/469: truncate d7/d2f/d6c/da4/f69 2319302 0 2024-03-04T16:04:11.654 INFO:tasks.workunit.client.0.smithi116.stdout:2/470: chown d7/d2f/l3d 72902 1 2024-03-04T16:04:11.655 INFO:tasks.workunit.client.0.smithi116.stdout:2/471: fsync d7/d2f/d76/f9a 0 2024-03-04T16:04:11.656 INFO:tasks.workunit.client.0.smithi116.stdout:7/337: link d3/d4/d8/f13 d3/d4/d8/d4a/f6b 0 2024-03-04T16:04:11.657 INFO:tasks.workunit.client.0.smithi116.stdout:3/498: rmdir dc/df/d14 39 2024-03-04T16:04:11.661 INFO:tasks.workunit.client.0.smithi116.stdout:2/472: dwrite d7/d2f/f44 [0,4194304] 0 2024-03-04T16:04:11.663 INFO:tasks.workunit.client.0.smithi116.stdout:3/499: dread dc/f70 [0,4194304] 0 2024-03-04T16:04:11.663 INFO:tasks.workunit.client.0.smithi116.stdout:7/338: mknod d3/d4/c6c 0 2024-03-04T16:04:11.665 INFO:tasks.workunit.client.0.smithi116.stdout:2/473: symlink d7/d13/d3e/d71/lb2 0 2024-03-04T16:04:11.672 INFO:tasks.workunit.client.0.smithi116.stdout:3/500: dwrite dc/df/f92 [0,4194304] 0 2024-03-04T16:04:11.675 INFO:tasks.workunit.client.0.smithi116.stdout:2/474: unlink d7/d2f/d76/d7c/la8 0 2024-03-04T16:04:11.677 INFO:tasks.workunit.client.0.smithi116.stdout:7/339: dwrite d3/d4/d8/d1a/f4d [0,4194304] 0 2024-03-04T16:04:11.679 INFO:tasks.workunit.client.0.smithi116.stdout:1/325: dread d5/d9/fb [0,4194304] 0 2024-03-04T16:04:11.681 INFO:tasks.workunit.client.0.smithi116.stdout:2/475: read d7/d15/d35/f70 [3540679,114471] 0 2024-03-04T16:04:11.683 INFO:tasks.workunit.client.0.smithi116.stdout:7/340: dread d3/d4/d8/d1a/f4d [0,4194304] 0 2024-03-04T16:04:11.683 INFO:tasks.workunit.client.0.smithi116.stdout:2/476: read - d7/d15/d5e/f67 zero size 2024-03-04T16:04:11.690 INFO:tasks.workunit.client.0.smithi116.stdout:3/501: rename dc/df/d47/l63 to dc/d34/la8 0 2024-03-04T16:04:11.691 INFO:tasks.workunit.client.0.smithi116.stdout:3/502: stat dc/d34 0 2024-03-04T16:04:11.694 INFO:tasks.workunit.client.0.smithi116.stdout:7/341: mkdir d3/d4/d8/d4a/d6d 0 2024-03-04T16:04:11.696 INFO:tasks.workunit.client.0.smithi116.stdout:2/477: rename d7/d25/f34 to d7/d25/d41/fb3 0 2024-03-04T16:04:11.701 INFO:tasks.workunit.client.0.smithi116.stdout:2/478: creat d7/d15/d35/d63/fb4 x:0 0 0 2024-03-04T16:04:11.702 INFO:tasks.workunit.client.0.smithi116.stdout:7/342: creat d3/d4/d8/d52/d5e/f6e x:0 0 0 2024-03-04T16:04:11.705 INFO:tasks.workunit.client.0.smithi116.stdout:7/343: creat d3/d4/d34/d5d/d66/f6f x:0 0 0 2024-03-04T16:04:11.705 INFO:tasks.workunit.client.0.smithi116.stdout:3/503: dread dc/d23/d84/d7f/f8a [0,4194304] 0 2024-03-04T16:04:11.708 INFO:tasks.workunit.client.0.smithi116.stdout:3/504: symlink dc/d34/d44/la9 0 2024-03-04T16:04:11.710 INFO:tasks.workunit.client.0.smithi116.stdout:7/344: symlink d3/d4/d8/d4a/d65/l70 0 2024-03-04T16:04:11.714 INFO:tasks.workunit.client.0.smithi116.stdout:3/505: mknod dc/d23/caa 0 2024-03-04T16:04:11.716 INFO:tasks.workunit.client.0.smithi116.stdout:2/479: dread d7/d2f/d6c/da4/f69 [0,4194304] 0 2024-03-04T16:04:11.716 INFO:tasks.workunit.client.0.smithi116.stdout:7/345: creat d3/d4/d8/f71 x:0 0 0 2024-03-04T16:04:11.717 INFO:tasks.workunit.client.0.smithi116.stdout:2/480: write d7/d15/d5e/f67 [269595,68431] 0 2024-03-04T16:04:11.718 INFO:tasks.workunit.client.0.smithi116.stdout:7/346: chown d3/d4/d8/d4a/l4f 89609068 1 2024-03-04T16:04:11.721 INFO:tasks.workunit.client.0.smithi116.stdout:2/481: creat d7/fb5 x:0 0 0 2024-03-04T16:04:11.723 INFO:tasks.workunit.client.0.smithi116.stdout:7/347: rmdir d3/d4/d8/d4a/d6d 0 2024-03-04T16:04:11.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:11 smithi116 ceph-mon[29120]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:11.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:11 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3843926793' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:04:11.729 INFO:tasks.workunit.client.0.smithi116.stdout:2/482: dwrite d7/d25/d41/fb3 [0,4194304] 0 2024-03-04T16:04:11.733 INFO:tasks.workunit.client.0.smithi116.stdout:2/483: creat d7/d29/d47/d64/d75/fb6 x:0 0 0 2024-03-04T16:04:11.737 INFO:tasks.workunit.client.0.smithi116.stdout:2/484: creat d7/d29/d47/d64/d75/fb7 x:0 0 0 2024-03-04T16:04:11.737 INFO:tasks.workunit.client.0.smithi116.stdout:2/485: chown d7/d29 430491 1 2024-03-04T16:04:11.737 INFO:tasks.workunit.client.0.smithi116.stdout:7/348: dread d3/d4/d8/f1c [0,4194304] 0 2024-03-04T16:04:11.741 INFO:tasks.workunit.client.0.smithi116.stdout:2/486: mknod d7/d15/cb8 0 2024-03-04T16:04:11.746 INFO:tasks.workunit.client.0.smithi116.stdout:2/487: symlink d7/lb9 0 2024-03-04T16:04:11.748 INFO:tasks.workunit.client.0.smithi116.stdout:7/349: getdents d3/d4/d8/d1a 0 2024-03-04T16:04:11.749 INFO:tasks.workunit.client.0.smithi116.stdout:7/350: truncate d3/d4/d8/f25 268563 0 2024-03-04T16:04:11.751 INFO:tasks.workunit.client.0.smithi116.stdout:2/488: rmdir d7/d2f 39 2024-03-04T16:04:11.755 INFO:tasks.workunit.client.0.smithi116.stdout:2/489: creat d7/d13/d3e/d98/fba x:0 0 0 2024-03-04T16:04:11.757 INFO:tasks.workunit.client.0.smithi116.stdout:7/351: link d3/f1f d3/d4/d8/d4a/d65/f72 0 2024-03-04T16:04:11.762 INFO:tasks.workunit.client.0.smithi116.stdout:2/490: symlink d7/d2f/lbb 0 2024-03-04T16:04:11.763 INFO:tasks.workunit.client.0.smithi116.stdout:2/491: chown d7/d2f/d6c/da4/l73 3346 1 2024-03-04T16:04:11.764 INFO:tasks.workunit.client.0.smithi116.stdout:2/492: write d7/d2f/d6c/f97 [162165,118788] 0 2024-03-04T16:04:11.764 INFO:tasks.workunit.client.0.smithi116.stdout:2/493: chown d7/d13/d3e/d98/fba 4 1 2024-03-04T16:04:11.765 INFO:tasks.workunit.client.0.smithi116.stdout:7/352: rmdir d3/d4/d11/d67 0 2024-03-04T16:04:11.765 INFO:tasks.workunit.client.0.smithi116.stdout:2/494: fsync d7/d29/d47/d64/d75/fb7 0 2024-03-04T16:04:11.766 INFO:tasks.workunit.client.0.smithi116.stdout:7/353: write d3/d4/d8/d52/f54 [416967,95096] 0 2024-03-04T16:04:11.768 INFO:tasks.workunit.client.0.smithi116.stdout:2/495: mknod d7/d15/d5e/d9d/cbc 0 2024-03-04T16:04:11.769 INFO:tasks.workunit.client.0.smithi116.stdout:2/496: write d7/d29/d47/f51 [1609529,41753] 0 2024-03-04T16:04:11.770 INFO:tasks.workunit.client.0.smithi116.stdout:7/354: mknod d3/d4/d11/d61/c73 0 2024-03-04T16:04:11.777 INFO:tasks.workunit.client.0.smithi116.stdout:7/355: dwrite d3/f18 [0,4194304] 0 2024-03-04T16:04:11.778 INFO:tasks.workunit.client.0.smithi116.stdout:2/497: dwrite d7/d29/f4a [4194304,4194304] 0 2024-03-04T16:04:11.780 INFO:tasks.workunit.client.0.smithi116.stdout:7/356: creat d3/d4/d34/f74 x:0 0 0 2024-03-04T16:04:11.782 INFO:tasks.workunit.client.0.smithi116.stdout:7/357: readlink d3/d4/d8/d4a/l4f 0 2024-03-04T16:04:11.784 INFO:tasks.workunit.client.0.smithi116.stdout:2/498: dread d7/d25/d41/f9f [0,4194304] 0 2024-03-04T16:04:11.784 INFO:tasks.workunit.client.0.smithi116.stdout:2/499: readlink d7/d13/l23 0 2024-03-04T16:04:11.794 INFO:tasks.workunit.client.0.smithi116.stdout:2/500: rename d7/d13/f52 to d7/d25/fbd 0 2024-03-04T16:04:11.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.796+0000 7f8e092dd700 1 -- 172.21.15.116:0/3427877196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8e041088a0 con 0x7f8e040ffab0 2024-03-04T16:04:11.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.796+0000 7f8debfff700 1 -- 172.21.15.116:0/3427877196 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f8df40596c0 con 0x7f8e040ffab0 2024-03-04T16:04:11.797 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:04:11.798 INFO:tasks.workunit.client.0.smithi116.stdout:2/501: rmdir d7/d15/d5e 39 2024-03-04T16:04:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.799+0000 7f8e092dd700 1 -- 172.21.15.116:0/3427877196 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8dec06ce90 msgr2=0x7f8dec06f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.799+0000 7f8e092dd700 1 --2- 172.21.15.116:0/3427877196 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8dec06ce90 0x7f8dec06f350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8df800b5c0 tx=0x7f8df8005960 comp rx=0 tx=0).stop 2024-03-04T16:04:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.799+0000 7f8e092dd700 1 -- 172.21.15.116:0/3427877196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e040ffab0 msgr2=0x7f8e04103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.799+0000 7f8e092dd700 1 --2- 172.21.15.116:0/3427877196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e040ffab0 0x7f8e04103a40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8df400b480 tx=0x7f8df400b840 comp rx=0 tx=0).stop 2024-03-04T16:04:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.799+0000 7f8e092dd700 1 -- 172.21.15.116:0/3427877196 shutdown_connections 2024-03-04T16:04:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.799+0000 7f8e092dd700 1 --2- 172.21.15.116:0/3427877196 >> [v2:172.21.15.152:6828/2881662614,v1:172.21.15.152:6829/2881662614] conn(0x7f8dec06ce90 0x7f8dec06f350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.799+0000 7f8e092dd700 1 --2- 172.21.15.116:0/3427877196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e040ffab0 0x7f8e04103a40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.799+0000 7f8e092dd700 1 --2- 172.21.15.116:0/3427877196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e04100cb0 0x7f8e04102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.799+0000 7f8e092dd700 1 -- 172.21.15.116:0/3427877196 >> 172.21.15.116:0/3427877196 conn(0x7f8e040fb050 msgr2=0x7f8e04068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:11.801 INFO:tasks.workunit.client.0.smithi116.stdout:2/502: symlink d7/d13/d3e/d98/lbe 0 2024-03-04T16:04:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.800+0000 7f8e092dd700 1 -- 172.21.15.116:0/3427877196 shutdown_connections 2024-03-04T16:04:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:11.800+0000 7f8e092dd700 1 -- 172.21.15.116:0/3427877196 wait complete. 2024-03-04T16:04:11.807 INFO:tasks.workunit.client.0.smithi116.stdout:2/503: creat d7/d13/fbf x:0 0 0 2024-03-04T16:04:11.812 INFO:tasks.workunit.client.0.smithi116.stdout:2/504: rename d7/d13/d3e/cac to d7/d29/d47/d64/cc0 0 2024-03-04T16:04:11.814 INFO:tasks.workunit.client.0.smithi116.stdout:7/358: dwrite d3/f1e [0,4194304] 0 2024-03-04T16:04:11.815 INFO:tasks.workunit.client.0.smithi116.stdout:2/505: truncate d7/d15/d35/d63/fb4 955974 0 2024-03-04T16:04:11.821 INFO:tasks.workunit.client.0.smithi116.stdout:2/506: link d7/f4e d7/d2f/d76/d7c/fc1 0 2024-03-04T16:04:11.822 INFO:tasks.workunit.client.0.smithi116.stdout:7/359: dwrite d3/d4/d8/d52/d5e/f6e [0,4194304] 0 2024-03-04T16:04:11.829 INFO:tasks.workunit.client.0.smithi116.stdout:2/507: dwrite d7/d15/f93 [0,4194304] 0 2024-03-04T16:04:11.831 INFO:tasks.workunit.client.0.smithi116.stdout:7/360: dread d3/d4/d8/d1a/f46 [0,4194304] 0 2024-03-04T16:04:11.834 INFO:tasks.workunit.client.0.smithi116.stdout:2/508: mkdir d7/d81/dc2 0 2024-03-04T16:04:11.837 INFO:tasks.workunit.client.0.smithi116.stdout:7/361: mknod d3/d4/d8/d1a/c75 0 2024-03-04T16:04:11.840 INFO:tasks.workunit.client.0.smithi116.stdout:2/509: creat d7/d2f/d6c/da4/fc3 x:0 0 0 2024-03-04T16:04:11.841 INFO:tasks.workunit.client.0.smithi116.stdout:7/362: symlink d3/d4/d34/d5d/d66/l76 0 2024-03-04T16:04:11.842 INFO:tasks.workunit.client.0.smithi116.stdout:7/363: chown d3/d4/d8/c4b 26097 1 2024-03-04T16:04:11.844 INFO:tasks.workunit.client.0.smithi116.stdout:0/352: sync 2024-03-04T16:04:11.845 INFO:tasks.workunit.client.0.smithi116.stdout:7/364: rename d3/d4/d8/d1a/l2c to d3/d4/d8/d1a/l77 0 2024-03-04T16:04:11.847 INFO:tasks.workunit.client.0.smithi116.stdout:0/353: creat da/d16/d19/d27/f74 x:0 0 0 2024-03-04T16:04:11.854 INFO:tasks.workunit.client.0.smithi116.stdout:0/354: truncate da/d16/d20/f65 857163 0 2024-03-04T16:04:11.857 INFO:tasks.workunit.client.0.smithi116.stdout:0/355: dread da/fe [0,4194304] 0 2024-03-04T16:04:11.861 INFO:tasks.workunit.client.0.smithi116.stdout:7/365: truncate d3/f6 2092200 0 2024-03-04T16:04:11.863 INFO:tasks.workunit.client.0.smithi116.stdout:0/356: link da/f1e da/d16/d19/d27/f75 0 2024-03-04T16:04:11.863 INFO:tasks.workunit.client.0.smithi116.stdout:0/357: fsync da/d16/d19/d27/f56 0 2024-03-04T16:04:11.865 INFO:tasks.workunit.client.0.smithi116.stdout:0/358: truncate da/d16/d19/f1d 995791 0 2024-03-04T16:04:11.866 INFO:tasks.workunit.client.0.smithi116.stdout:7/366: write d3/d4/d8/f35 [4472460,60142] 0 2024-03-04T16:04:11.867 INFO:tasks.workunit.client.0.smithi116.stdout:7/367: write d3/d4/f16 [622107,79230] 0 2024-03-04T16:04:11.868 INFO:tasks.workunit.client.0.smithi116.stdout:7/368: dread - d3/d4/d34/f74 zero size 2024-03-04T16:04:11.870 INFO:tasks.workunit.client.0.smithi116.stdout:7/369: write d3/f4e [2323147,39142] 0 2024-03-04T16:04:11.875 INFO:tasks.workunit.client.0.smithi116.stdout:7/370: rename d3/d4/d8/ce to d3/d4/d8/d4a/c78 0 2024-03-04T16:04:11.878 INFO:tasks.workunit.client.0.smithi116.stdout:7/371: rmdir d3/d4/d8/d4a/d65 39 2024-03-04T16:04:11.879 INFO:tasks.workunit.client.0.smithi116.stdout:7/372: fsync d3/d4/d34/f74 0 2024-03-04T16:04:11.884 INFO:tasks.workunit.client.0.smithi116.stdout:7/373: dwrite d3/f50 [0,4194304] 0 2024-03-04T16:04:11.889 INFO:tasks.workunit.client.0.smithi116.stdout:0/359: dwrite da/d16/d28/d3d/f66 [0,4194304] 0 2024-03-04T16:04:11.894 INFO:tasks.workunit.client.0.smithi116.stdout:7/374: rmdir d3/d4/d8/d52/d69 0 2024-03-04T16:04:11.895 INFO:tasks.workunit.client.0.smithi116.stdout:7/375: fdatasync d3/d4/d8/f25 0 2024-03-04T16:04:11.897 INFO:tasks.workunit.client.0.smithi116.stdout:0/360: rename da/d16/d19/l26 to da/d16/d19/d27/d45/l76 0 2024-03-04T16:04:11.898 INFO:tasks.workunit.client.0.smithi116.stdout:0/361: truncate da/d16/d20/d53/f63 64696 0 2024-03-04T16:04:11.900 INFO:tasks.workunit.client.0.smithi116.stdout:0/362: dread - da/d16/d19/d27/f74 zero size 2024-03-04T16:04:11.902 INFO:tasks.workunit.client.0.smithi116.stdout:7/376: rename d3/d4/d8/c41 to d3/d4/d8/d4a/c79 0 2024-03-04T16:04:11.903 INFO:tasks.workunit.client.0.smithi116.stdout:0/363: mkdir da/d16/d28/d40/d48/d77 0 2024-03-04T16:04:11.905 INFO:tasks.workunit.client.0.smithi116.stdout:7/377: readlink d3/d4/d8/d4a/d65/l70 0 2024-03-04T16:04:11.907 INFO:tasks.workunit.client.0.smithi116.stdout:7/378: symlink d3/d4/d8/d4a/l7a 0 2024-03-04T16:04:11.908 INFO:tasks.workunit.client.0.smithi116.stdout:7/379: chown d3/d4/d8/d1a/f42 117565770 1 2024-03-04T16:04:11.912 INFO:tasks.workunit.client.0.smithi116.stdout:4/447: sync 2024-03-04T16:04:11.913 INFO:tasks.workunit.client.0.smithi116.stdout:7/380: dwrite d3/f4e [0,4194304] 0 2024-03-04T16:04:11.920 INFO:tasks.workunit.client.0.smithi116.stdout:7/381: dwrite d3/f18 [0,4194304] 0 2024-03-04T16:04:11.920 INFO:tasks.workunit.client.0.smithi116.stdout:0/364: dread da/d16/d19/d27/f56 [0,4194304] 0 2024-03-04T16:04:11.925 INFO:tasks.workunit.client.0.smithi116.stdout:7/382: creat d3/d4/d8/d4a/d65/f7b x:0 0 0 2024-03-04T16:04:11.926 INFO:tasks.workunit.client.0.smithi116.stdout:0/365: creat da/d16/d19/d27/f78 x:0 0 0 2024-03-04T16:04:11.928 INFO:tasks.workunit.client.0.smithi116.stdout:7/383: creat d3/d4/d11/d61/f7c x:0 0 0 2024-03-04T16:04:11.932 INFO:tasks.workunit.client.0.smithi116.stdout:7/384: stat d3/d4/l57 0 2024-03-04T16:04:11.934 INFO:tasks.workunit.client.0.smithi116.stdout:4/448: dread d1/d3/f44 [0,4194304] 0 2024-03-04T16:04:11.942 INFO:tasks.workunit.client.0.smithi116.stdout:4/449: symlink d1/d3/de/l9f 0 2024-03-04T16:04:11.946 INFO:tasks.workunit.client.0.smithi116.stdout:4/450: creat d1/d3/d40/d52/d84/fa0 x:0 0 0 2024-03-04T16:04:11.963 INFO:tasks.workunit.client.0.smithi116.stdout:0/366: write da/d16/d28/d3d/f4a [397684,37675] 0 2024-03-04T16:04:11.967 INFO:tasks.workunit.client.0.smithi116.stdout:7/385: write d3/d4/d8/d4a/f6b [935059,113356] 0 2024-03-04T16:04:11.975 INFO:tasks.workunit.client.0.smithi116.stdout:4/451: write d1/f2c [1320166,43189] 0 2024-03-04T16:04:11.982 INFO:tasks.workunit.client.0.smithi116.stdout:4/452: getdents d1/d3/dd/d8a 0 2024-03-04T16:04:11.982 INFO:tasks.workunit.client.0.smithi116.stdout:4/453: read - d1/f85 zero size 2024-03-04T16:04:11.992 INFO:tasks.workunit.client.0.smithi116.stdout:0/367: write da/ff [1055008,31216] 0 2024-03-04T16:04:12.005 INFO:tasks.workunit.client.0.smithi116.stdout:4/454: write d1/d3/d15/d20/f1e [1056054,110755] 0 2024-03-04T16:04:12.010 INFO:tasks.workunit.client.0.smithi116.stdout:4/455: read d1/d3/dd/f9 [547344,87566] 0 2024-03-04T16:04:12.011 INFO:tasks.workunit.client.0.smithi116.stdout:4/456: write d1/d3/d15/d20/f5e [586877,78534] 0 2024-03-04T16:04:12.014 INFO:tasks.workunit.client.0.smithi116.stdout:0/368: write da/fe [2129553,57888] 0 2024-03-04T16:04:12.015 INFO:tasks.workunit.client.0.smithi116.stdout:0/369: dread - da/d16/d19/d27/f74 zero size 2024-03-04T16:04:12.016 INFO:tasks.workunit.client.0.smithi116.stdout:0/370: dread - da/d16/f6e zero size 2024-03-04T16:04:12.016 INFO:tasks.workunit.client.0.smithi116.stdout:4/457: symlink d1/d3/d40/la1 0 2024-03-04T16:04:12.018 INFO:tasks.workunit.client.0.smithi116.stdout:0/371: mkdir da/d16/d20/d79 0 2024-03-04T16:04:12.019 INFO:tasks.workunit.client.0.smithi116.stdout:0/372: truncate da/d16/d19/f1d 1878175 0 2024-03-04T16:04:12.020 INFO:tasks.workunit.client.0.smithi116.stdout:4/458: mknod d1/d6e/d74/ca2 0 2024-03-04T16:04:12.025 INFO:tasks.workunit.client.0.smithi116.stdout:0/373: symlink da/d16/d19/d27/d70/l7a 0 2024-03-04T16:04:12.026 INFO:tasks.workunit.client.0.smithi116.stdout:0/374: read da/d16/d20/f21 [1117253,47916] 0 2024-03-04T16:04:12.029 INFO:tasks.workunit.client.0.smithi116.stdout:4/459: creat d1/d3/d15/d20/fa3 x:0 0 0 2024-03-04T16:04:12.030 INFO:tasks.workunit.client.0.smithi116.stdout:4/460: dread - d1/d3/d15/d20/fa3 zero size 2024-03-04T16:04:12.030 INFO:tasks.workunit.client.0.smithi116.stdout:4/461: dread - d1/d3/dd/f98 zero size 2024-03-04T16:04:12.033 INFO:tasks.workunit.client.0.smithi116.stdout:0/375: mkdir da/d16/d28/d7b 0 2024-03-04T16:04:12.034 INFO:tasks.workunit.client.0.smithi116.stdout:4/462: mknod d1/d3/dd/d24/d51/ca4 0 2024-03-04T16:04:12.036 INFO:tasks.workunit.client.0.smithi116.stdout:0/376: creat da/d16/d20/d79/f7c x:0 0 0 2024-03-04T16:04:12.039 INFO:tasks.workunit.client.0.smithi116.stdout:4/463: dwrite d1/d3/f39 [0,4194304] 0 2024-03-04T16:04:12.047 INFO:tasks.workunit.client.0.smithi116.stdout:4/464: rename d1/d3/d15/d20/d1c/l49 to d1/d3/d15/d20/d1c/d62/la5 0 2024-03-04T16:04:12.048 INFO:tasks.workunit.client.0.smithi116.stdout:4/465: write d1/f80 [2218227,130378] 0 2024-03-04T16:04:12.050 INFO:tasks.workunit.client.0.smithi116.stdout:4/466: mkdir d1/d3/d15/d20/d1c/d62/d81/da6 0 2024-03-04T16:04:12.054 INFO:tasks.workunit.client.0.smithi116.stdout:0/377: dread da/d16/d19/d27/f75 [0,4194304] 0 2024-03-04T16:04:12.054 INFO:tasks.workunit.client.0.smithi116.stdout:4/467: dread d1/d3/f39 [0,4194304] 0 2024-03-04T16:04:12.058 INFO:tasks.workunit.client.0.smithi116.stdout:4/468: fdatasync d1/d3/d15/d20/d65/f99 0 2024-03-04T16:04:12.059 INFO:tasks.workunit.client.0.smithi116.stdout:0/378: getdents da/d16/d28/d40/d48/d77 0 2024-03-04T16:04:12.063 INFO:tasks.workunit.client.0.smithi116.stdout:0/379: creat da/d16/d28/d3d/d41/f7d x:0 0 0 2024-03-04T16:04:12.065 INFO:tasks.workunit.client.0.smithi116.stdout:4/469: link d1/d3/d15/d20/fa3 d1/d3/d15/d20/d1c/d62/d81/da6/fa7 0 2024-03-04T16:04:12.069 INFO:tasks.workunit.client.0.smithi116.stdout:9/373: sync 2024-03-04T16:04:12.069 INFO:tasks.workunit.client.0.smithi116.stdout:5/413: sync 2024-03-04T16:04:12.069 INFO:tasks.workunit.client.0.smithi116.stdout:6/399: sync 2024-03-04T16:04:12.071 INFO:tasks.workunit.client.0.smithi116.stdout:4/470: rename d1/d3/dd/f9 to d1/d3/d15/fa8 0 2024-03-04T16:04:12.072 INFO:tasks.workunit.client.0.smithi116.stdout:9/374: rmdir d2/d19/d27/d3b 39 2024-03-04T16:04:12.076 INFO:tasks.workunit.client.0.smithi116.stdout:6/400: link d6/de/d29/d3a/f46 d6/de/d1e/f8a 0 2024-03-04T16:04:12.081 INFO:tasks.workunit.client.0.smithi116.stdout:0/380: dread da/f1a [0,4194304] 0 2024-03-04T16:04:12.085 INFO:tasks.workunit.client.0.smithi116.stdout:9/375: dwrite d2/d19/d27/d3b/f68 [0,4194304] 0 2024-03-04T16:04:12.085 INFO:tasks.workunit.client.0.smithi116.stdout:9/376: readlink d2/l14 0 2024-03-04T16:04:12.086 INFO:tasks.workunit.client.0.smithi116.stdout:0/381: dread da/d16/d19/f1d [0,4194304] 0 2024-03-04T16:04:12.087 INFO:tasks.workunit.client.0.smithi116.stdout:9/377: fsync d2/d33/f54 0 2024-03-04T16:04:12.088 INFO:tasks.workunit.client.0.smithi116.stdout:0/382: write da/d16/d28/d3d/f64 [2077889,77068] 0 2024-03-04T16:04:12.089 INFO:tasks.workunit.client.0.smithi116.stdout:4/471: getdents d1/d3/dd/d45/d50 0 2024-03-04T16:04:12.097 INFO:tasks.workunit.client.0.smithi116.stdout:6/401: dread d6/de/d1e/f35 [0,4194304] 0 2024-03-04T16:04:12.098 INFO:tasks.workunit.client.0.smithi116.stdout:4/472: symlink d1/d3/d40/d52/d84/d88/la9 0 2024-03-04T16:04:12.101 INFO:tasks.workunit.client.0.smithi116.stdout:5/414: truncate dd/d1f/d55/d73/d2d/d31/f65 3089134 0 2024-03-04T16:04:12.101 INFO:tasks.workunit.client.0.smithi116.stdout:5/415: readlink lc 0 2024-03-04T16:04:12.102 INFO:tasks.workunit.client.0.smithi116.stdout:6/402: creat d6/d56/f8b x:0 0 0 2024-03-04T16:04:12.103 INFO:tasks.workunit.client.0.smithi116.stdout:0/383: dwrite da/d16/d20/f5b [0,4194304] 0 2024-03-04T16:04:12.105 INFO:tasks.workunit.client.0.smithi116.stdout:9/378: link d2/d19/c4d d2/d49/c74 0 2024-03-04T16:04:12.108 INFO:tasks.workunit.client.0.smithi116.stdout:0/384: dread da/f1a [0,4194304] 0 2024-03-04T16:04:12.110 INFO:tasks.workunit.client.0.smithi116.stdout:4/473: unlink d1/d3/de/f16 0 2024-03-04T16:04:12.112 INFO:tasks.workunit.client.0.smithi116.stdout:5/416: rename dd/f60 to dd/d1f/d55/d73/d2d/d7c/f86 0 2024-03-04T16:04:12.116 INFO:tasks.workunit.client.0.smithi116.stdout:6/403: mkdir d6/de/d25/d4a/d63/d8c 0 2024-03-04T16:04:12.117 INFO:tasks.workunit.client.0.smithi116.stdout:0/385: unlink da/d16/d19/d27/f56 0 2024-03-04T16:04:12.124 INFO:tasks.workunit.client.0.smithi116.stdout:1/326: sync 2024-03-04T16:04:12.125 INFO:tasks.workunit.client.0.smithi116.stdout:8/438: sync 2024-03-04T16:04:12.125 INFO:tasks.workunit.client.0.smithi116.stdout:5/417: creat dd/d1f/d5c/f87 x:0 0 0 2024-03-04T16:04:12.126 INFO:tasks.workunit.client.0.smithi116.stdout:1/327: stat d5/d9/d21/d27/d32/d49/f66 0 2024-03-04T16:04:12.130 INFO:tasks.workunit.client.0.smithi116.stdout:6/404: rmdir d6/d56/d58/d72 39 2024-03-04T16:04:12.131 INFO:tasks.workunit.client.0.smithi116.stdout:6/405: chown d6/de/d25 78625 1 2024-03-04T16:04:12.131 INFO:tasks.workunit.client.0.smithi116.stdout:5/418: dwrite dd/d1f/d5c/d5e/f62 [0,4194304] 0 2024-03-04T16:04:12.132 INFO:tasks.workunit.client.0.smithi116.stdout:0/386: mkdir da/d16/d28/d40/d5d/d7e 0 2024-03-04T16:04:12.133 INFO:tasks.workunit.client.0.smithi116.stdout:0/387: write da/d16/d20/f5b [3815865,51063] 0 2024-03-04T16:04:12.135 INFO:tasks.workunit.client.0.smithi116.stdout:8/439: mkdir d2/d13/d3c/d46/d96 0 2024-03-04T16:04:12.135 INFO:tasks.workunit.client.0.smithi116.stdout:0/388: readlink l3 0 2024-03-04T16:04:12.135 INFO:tasks.workunit.client.0.smithi116.stdout:4/474: write d1/d3/dd/d24/f6b [886468,101974] 0 2024-03-04T16:04:12.136 INFO:tasks.workunit.client.0.smithi116.stdout:0/389: chown da/c14 337 1 2024-03-04T16:04:12.138 INFO:tasks.workunit.client.0.smithi116.stdout:5/419: creat dd/d6d/f88 x:0 0 0 2024-03-04T16:04:12.142 INFO:tasks.workunit.client.0.smithi116.stdout:4/475: dwrite d1/d3/de/f10 [4194304,4194304] 0 2024-03-04T16:04:12.144 INFO:tasks.workunit.client.0.smithi116.stdout:4/476: fsync d1/d3/d15/d20/f5e 0 2024-03-04T16:04:12.199 INFO:tasks.workunit.client.0.smithi116.stdout:9/379: getdents d2/d33/d65/d6c 0 2024-03-04T16:04:12.201 INFO:tasks.workunit.client.0.smithi116.stdout:9/380: dread d2/d26/f71 [0,4194304] 0 2024-03-04T16:04:12.202 INFO:tasks.workunit.client.0.smithi116.stdout:0/390: truncate da/d16/d20/f21 1400992 0 2024-03-04T16:04:12.205 INFO:tasks.workunit.client.0.smithi116.stdout:5/420: mknod dd/d45/c89 0 2024-03-04T16:04:12.205 INFO:tasks.workunit.client.0.smithi116.stdout:8/440: dwrite d2/de/f10 [0,4194304] 0 2024-03-04T16:04:12.229 INFO:tasks.workunit.client.0.smithi116.stdout:9/381: creat d2/d19/d27/d3b/d4a/d6a/f75 x:0 0 0 2024-03-04T16:04:12.235 INFO:tasks.workunit.client.0.smithi116.stdout:4/477: link d1/d3/dd/d24/d51/ca4 d1/d3/dd/d24/d51/caa 0 2024-03-04T16:04:12.235 INFO:tasks.workunit.client.0.smithi116.stdout:4/478: stat d1/d3/dd 0 2024-03-04T16:04:12.244 INFO:tasks.workunit.client.0.smithi116.stdout:7/386: sync 2024-03-04T16:04:12.244 INFO:tasks.workunit.client.0.smithi116.stdout:3/506: sync 2024-03-04T16:04:12.244 INFO:tasks.workunit.client.0.smithi116.stdout:2/510: sync 2024-03-04T16:04:12.245 INFO:tasks.workunit.client.0.smithi116.stdout:5/421: link dd/l17 dd/d1f/d55/d73/d2d/d59/l8a 0 2024-03-04T16:04:12.245 INFO:tasks.workunit.client.0.smithi116.stdout:9/382: dread d2/d19/d27/d3b/d4a/f4c [0,4194304] 0 2024-03-04T16:04:12.246 INFO:tasks.workunit.client.0.smithi116.stdout:9/383: chown d2/d49/l18 3962705 1 2024-03-04T16:04:12.253 INFO:tasks.workunit.client.0.smithi116.stdout:3/507: dread dc/d34/d44/d6a/f77 [0,4194304] 0 2024-03-04T16:04:12.264 INFO:tasks.workunit.client.0.smithi116.stdout:3/508: truncate dc/f33 1806134 0 2024-03-04T16:04:12.271 INFO:tasks.workunit.client.0.smithi116.stdout:1/328: sync 2024-03-04T16:04:12.272 INFO:tasks.workunit.client.0.smithi116.stdout:8/441: rename d2/d13/d1e to d2/d13/d3c/d77/d97 0 2024-03-04T16:04:12.277 INFO:tasks.workunit.client.0.smithi116.stdout:3/509: chown dc/df/c59 53 1 2024-03-04T16:04:12.279 INFO:tasks.workunit.client.0.smithi116.stdout:8/442: mknod d2/de/d16/d78/d29/d7a/c98 0 2024-03-04T16:04:12.287 INFO:tasks.workunit.client.0.smithi116.stdout:3/510: mkdir dc/d34/d44/dab 0 2024-03-04T16:04:12.290 INFO:tasks.workunit.client.0.smithi116.stdout:3/511: chown dc/df/c10 2256 1 2024-03-04T16:04:12.291 INFO:tasks.workunit.client.0.smithi116.stdout:5/422: getdents dd/d1f 0 2024-03-04T16:04:12.298 INFO:tasks.workunit.client.0.smithi116.stdout:5/423: fsync dd/f15 0 2024-03-04T16:04:12.300 INFO:tasks.workunit.client.0.smithi116.stdout:3/512: creat dc/d34/d44/fac x:0 0 0 2024-03-04T16:04:12.300 INFO:tasks.workunit.client.0.smithi116.stdout:3/513: write dc/d34/d44/fac [805671,22698] 0 2024-03-04T16:04:12.305 INFO:tasks.workunit.client.0.smithi116.stdout:5/424: rename dd/d1f/d51/c83 to dd/d1f/d5c/c8b 0 2024-03-04T16:04:12.307 INFO:tasks.workunit.client.0.smithi116.stdout:5/425: dread - dd/d1f/d55/d73/d2d/d59/d71/f7a zero size 2024-03-04T16:04:12.307 INFO:tasks.workunit.client.0.smithi116.stdout:3/514: dwrite dc/f70 [0,4194304] 0 2024-03-04T16:04:12.309 INFO:tasks.workunit.client.0.smithi116.stdout:6/406: truncate d6/de/d1e/f43 1534550 0 2024-03-04T16:04:12.313 INFO:tasks.workunit.client.0.smithi116.stdout:3/515: dread dc/df/d14/f72 [0,4194304] 0 2024-03-04T16:04:12.316 INFO:tasks.workunit.client.0.smithi116.stdout:6/407: unlink d6/de/d25/d4a/d63/f65 0 2024-03-04T16:04:12.317 INFO:tasks.workunit.client.0.smithi116.stdout:6/408: write d6/d56/f8b [405537,116441] 0 2024-03-04T16:04:12.323 INFO:tasks.workunit.client.0.smithi116.stdout:6/409: rmdir d6/de/d25/d36/d4e/d50 39 2024-03-04T16:04:12.323 INFO:tasks.workunit.client.0.smithi116.stdout:6/410: chown d6/c83 255120915 1 2024-03-04T16:04:12.324 INFO:tasks.workunit.client.0.smithi116.stdout:3/516: mknod dc/d34/d44/d6a/da3/cad 0 2024-03-04T16:04:12.325 INFO:tasks.workunit.client.0.smithi116.stdout:3/517: truncate dc/df/d14/f8c 251392 0 2024-03-04T16:04:12.335 INFO:tasks.workunit.client.0.smithi116.stdout:3/518: dwrite dc/d23/d84/d7c/f69 [4194304,4194304] 0 2024-03-04T16:04:12.336 INFO:tasks.workunit.client.0.smithi116.stdout:5/426: dread dd/f22 [0,4194304] 0 2024-03-04T16:04:12.342 INFO:tasks.workunit.client.0.smithi116.stdout:9/384: write d2/d49/f21 [707755,65733] 0 2024-03-04T16:04:12.342 INFO:tasks.workunit.client.0.smithi116.stdout:9/385: chown d2/d19/d27/d3b/d4a/d5b/f24 107582 1 2024-03-04T16:04:12.345 INFO:tasks.workunit.client.0.smithi116.stdout:7/387: unlink d3/f10 0 2024-03-04T16:04:12.347 INFO:tasks.workunit.client.0.smithi116.stdout:2/511: dwrite d7/d13/f1a [0,4194304] 0 2024-03-04T16:04:12.349 INFO:tasks.workunit.client.0.smithi116.stdout:9/386: truncate d2/d19/d27/d3b/d4a/d5b/f25 1286615 0 2024-03-04T16:04:12.354 INFO:tasks.workunit.client.0.smithi116.stdout:7/388: symlink d3/d4/d8/l7d 0 2024-03-04T16:04:12.355 INFO:tasks.workunit.client.0.smithi116.stdout:4/479: truncate d1/d3/dd/f4a 4144459 0 2024-03-04T16:04:12.355 INFO:tasks.workunit.client.0.smithi116.stdout:2/512: dwrite d7/d81/f79 [0,4194304] 0 2024-03-04T16:04:12.355 INFO:tasks.workunit.client.0.smithi116.stdout:7/389: write d3/d4/f16 [4916162,13159] 0 2024-03-04T16:04:12.364 INFO:tasks.workunit.client.0.smithi116.stdout:7/390: dwrite d3/d4/d8/f13 [0,4194304] 0 2024-03-04T16:04:12.367 INFO:tasks.workunit.client.0.smithi116.stdout:2/513: creat d7/d13/d3e/d71/dad/fc4 x:0 0 0 2024-03-04T16:04:12.369 INFO:tasks.workunit.client.0.smithi116.stdout:7/391: dread d3/d4/d8/d1a/f4d [0,4194304] 0 2024-03-04T16:04:12.372 INFO:tasks.workunit.client.0.smithi116.stdout:4/480: mknod d1/d3/d15/d20/d1c/d62/d81/d97/cab 0 2024-03-04T16:04:12.375 INFO:tasks.workunit.client.0.smithi116.stdout:2/514: rename d7/d13/l8e to d7/d15/d35/lc5 0 2024-03-04T16:04:12.376 INFO:tasks.workunit.client.0.smithi116.stdout:8/443: write d2/de/d16/f34 [8474155,86330] 0 2024-03-04T16:04:12.379 INFO:tasks.workunit.client.0.smithi116.stdout:4/481: symlink d1/lac 0 2024-03-04T16:04:12.383 INFO:tasks.workunit.client.0.smithi116.stdout:5/427: unlink dd/f11 0 2024-03-04T16:04:12.384 INFO:tasks.workunit.client.0.smithi116.stdout:5/428: truncate dd/d1f/d55/f77 546473 0 2024-03-04T16:04:12.386 INFO:tasks.workunit.client.0.smithi116.stdout:8/444: creat d2/d13/d3c/d46/f99 x:0 0 0 2024-03-04T16:04:12.387 INFO:tasks.workunit.client.0.smithi116.stdout:4/482: mkdir d1/d3/dd/d45/d50/dad 0 2024-03-04T16:04:12.388 INFO:tasks.workunit.client.0.smithi116.stdout:4/483: fsync d1/d3/de/f26 0 2024-03-04T16:04:12.390 INFO:tasks.workunit.client.0.smithi116.stdout:4/484: dread - d1/d6c/f38 zero size 2024-03-04T16:04:12.391 INFO:tasks.workunit.client.0.smithi116.stdout:5/429: symlink dd/d1f/d55/l8c 0 2024-03-04T16:04:12.391 INFO:tasks.workunit.client.0.smithi116.stdout:0/391: dwrite da/d16/d20/f65 [0,4194304] 0 2024-03-04T16:04:12.392 INFO:tasks.workunit.client.0.smithi116.stdout:7/392: getdents d3/d4/d8/d52/d5e 0 2024-03-04T16:04:12.395 INFO:tasks.workunit.client.0.smithi116.stdout:0/392: truncate da/d16/d19/d27/f78 634631 0 2024-03-04T16:04:12.400 INFO:tasks.workunit.client.0.smithi116.stdout:4/485: dwrite d1/d3/de/f30 [0,4194304] 0 2024-03-04T16:04:12.402 INFO:tasks.workunit.client.0.smithi116.stdout:5/430: mkdir dd/d1f/d55/d73/d2d/d8d 0 2024-03-04T16:04:12.402 INFO:tasks.workunit.client.0.smithi116.stdout:4/486: chown d1/d3/d40/d52/d84/fa0 0 1 2024-03-04T16:04:12.403 INFO:tasks.workunit.client.0.smithi116.stdout:8/445: dread d2/de/d16/f30 [0,4194304] 0 2024-03-04T16:04:12.404 INFO:tasks.workunit.client.0.smithi116.stdout:7/393: mkdir d3/d4/d7e 0 2024-03-04T16:04:12.405 INFO:tasks.workunit.client.0.smithi116.stdout:7/394: write d3/d4/d8/d52/f53 [194963,123601] 0 2024-03-04T16:04:12.410 INFO:tasks.workunit.client.0.smithi116.stdout:4/487: write d1/d3/f44 [1624633,106766] 0 2024-03-04T16:04:12.411 INFO:tasks.workunit.client.0.smithi116.stdout:4/488: fsync d1/d3/d40/d52/d84/fa0 0 2024-03-04T16:04:12.417 INFO:tasks.workunit.client.0.smithi116.stdout:7/395: creat d3/d4/d8/d52/f7f x:0 0 0 2024-03-04T16:04:12.417 INFO:tasks.workunit.client.0.smithi116.stdout:0/393: link da/d16/d28/d3d/d41/f73 da/d16/d28/d40/d48/d57/f7f 0 2024-03-04T16:04:12.417 INFO:tasks.workunit.client.0.smithi116.stdout:6/411: write d6/d56/f60 [412203,49493] 0 2024-03-04T16:04:12.418 INFO:tasks.workunit.client.0.smithi116.stdout:3/519: write dc/d34/f9d [3247561,34513] 0 2024-03-04T16:04:12.419 INFO:tasks.workunit.client.0.smithi116.stdout:7/396: write d3/d4/d8/d1a/f28 [2394752,25883] 0 2024-03-04T16:04:12.420 INFO:tasks.workunit.client.0.smithi116.stdout:4/489: creat d1/fae x:0 0 0 2024-03-04T16:04:12.425 INFO:tasks.workunit.client.0.smithi116.stdout:0/394: dwrite da/ff [0,4194304] 0 2024-03-04T16:04:12.434 INFO:tasks.workunit.client.0.smithi116.stdout:9/387: write d2/d19/d27/d3b/d4a/f4c [2038250,23190] 0 2024-03-04T16:04:12.435 INFO:tasks.workunit.client.0.smithi116.stdout:9/388: write d2/d26/f44 [5752711,118997] 0 2024-03-04T16:04:12.435 INFO:tasks.workunit.client.0.smithi116.stdout:5/431: rmdir dd/d1f/d55/d73/d5d 0 2024-03-04T16:04:12.436 INFO:tasks.workunit.client.0.smithi116.stdout:9/389: fsync d2/d19/d27/d3b/f70 0 2024-03-04T16:04:12.443 INFO:tasks.workunit.client.0.smithi116.stdout:4/490: mkdir d1/d3/d15/d20/d1c/d62/d81/daf 0 2024-03-04T16:04:12.447 INFO:tasks.workunit.client.0.smithi116.stdout:5/432: rmdir dd/d1f/d55 39 2024-03-04T16:04:12.448 INFO:tasks.workunit.client.0.smithi116.stdout:9/390: creat d2/d49/d3c/f76 x:0 0 0 2024-03-04T16:04:12.451 INFO:tasks.workunit.client.0.smithi116.stdout:6/412: getdents d6/de/d25/d4a/d63/d8c 0 2024-03-04T16:04:12.453 INFO:tasks.workunit.client.0.smithi116.stdout:4/491: creat d1/d3/d15/d20/d1c/d62/d81/fb0 x:0 0 0 2024-03-04T16:04:12.454 INFO:tasks.workunit.client.0.smithi116.stdout:2/515: dwrite d7/d2f/d6c/f90 [0,4194304] 0 2024-03-04T16:04:12.456 INFO:tasks.workunit.client.0.smithi116.stdout:4/492: dread - d1/d3/dd/f98 zero size 2024-03-04T16:04:12.458 INFO:tasks.workunit.client.0.smithi116.stdout:9/391: dwrite d2/d19/d27/f38 [0,4194304] 0 2024-03-04T16:04:12.458 INFO:tasks.workunit.client.0.smithi116.stdout:4/493: write d1/d3/d40/f4f [2517608,55051] 0 2024-03-04T16:04:12.464 INFO:tasks.workunit.client.0.smithi116.stdout:3/520: rename dc/l2f to dc/d23/lae 0 2024-03-04T16:04:12.465 INFO:tasks.workunit.client.0.smithi116.stdout:2/516: dread d7/d15/f38 [0,4194304] 0 2024-03-04T16:04:12.466 INFO:tasks.workunit.client.0.smithi116.stdout:6/413: getdents d6/de/d84 0 2024-03-04T16:04:12.467 INFO:tasks.workunit.client.0.smithi116.stdout:4/494: dwrite d1/d3/d15/d20/d1c/f6a [0,4194304] 0 2024-03-04T16:04:12.474 INFO:tasks.workunit.client.0.smithi116.stdout:9/392: fdatasync d2/d49/f42 0 2024-03-04T16:04:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:12 smithi116 ceph-mon[29120]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:12.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:12 smithi116 ceph-mon[29120]: pgmap v22: 65 pgs: 65 active+clean; 820 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 15 MiB/s rd, 103 MiB/s wr, 406 op/s 2024-03-04T16:04:12.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:12 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3427877196' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:04:12.484 INFO:tasks.workunit.client.0.smithi116.stdout:4/495: rmdir d1/d3/dd/d24 39 2024-03-04T16:04:12.488 INFO:tasks.workunit.client.0.smithi116.stdout:9/393: creat d2/d33/f77 x:0 0 0 2024-03-04T16:04:12.490 INFO:tasks.workunit.client.0.smithi116.stdout:5/433: truncate dd/d1f/d55/d73/d2d/f3b 2878665 0 2024-03-04T16:04:12.492 INFO:tasks.workunit.client.0.smithi116.stdout:7/397: write f2 [3829206,103736] 0 2024-03-04T16:04:12.496 INFO:tasks.workunit.client.0.smithi116.stdout:9/394: unlink d2/f35 0 2024-03-04T16:04:12.496 INFO:tasks.workunit.client.0.smithi116.stdout:9/395: stat d2/c6d 0 2024-03-04T16:04:12.497 INFO:tasks.workunit.client.0.smithi116.stdout:9/396: stat d2/d49/f6f 0 2024-03-04T16:04:12.497 INFO:tasks.workunit.client.0.smithi116.stdout:9/397: chown d2/d49/f42 108713 1 2024-03-04T16:04:12.498 INFO:tasks.workunit.client.0.smithi116.stdout:9/398: write d2/d19/d27/d3b/d4a/f4c [1711755,70356] 0 2024-03-04T16:04:12.499 INFO:tasks.workunit.client.0.smithi116.stdout:3/521: creat dc/df/faf x:0 0 0 2024-03-04T16:04:12.504 INFO:tasks.workunit.client.0.smithi116.stdout:7/398: rename d3/f50 to d3/d4/d8/d4a/d65/f80 0 2024-03-04T16:04:12.504 INFO:tasks.workunit.client.0.smithi116.stdout:7/399: chown d3/fd 404457 1 2024-03-04T16:04:12.505 INFO:tasks.workunit.client.0.smithi116.stdout:7/400: write d3/d4/d8/f35 [1144431,118829] 0 2024-03-04T16:04:12.505 INFO:tasks.workunit.client.0.smithi116.stdout:9/399: symlink d2/d26/l78 0 2024-03-04T16:04:12.507 INFO:tasks.workunit.client.0.smithi116.stdout:7/401: read d3/d4/d8/f25 [226518,120931] 0 2024-03-04T16:04:12.507 INFO:tasks.workunit.client.0.smithi116.stdout:5/434: mkdir dd/d1f/d72/d8e 0 2024-03-04T16:04:12.509 INFO:tasks.workunit.client.0.smithi116.stdout:3/522: rmdir dc/df/d47/d91 39 2024-03-04T16:04:12.514 INFO:tasks.workunit.client.0.smithi116.stdout:9/400: creat d2/d49/d3c/f79 x:0 0 0 2024-03-04T16:04:12.516 INFO:tasks.workunit.client.0.smithi116.stdout:9/401: truncate d2/d19/d27/d3b/d4a/f73 321506 0 2024-03-04T16:04:12.516 INFO:tasks.workunit.client.0.smithi116.stdout:7/402: creat d3/d4/d11/d61/f81 x:0 0 0 2024-03-04T16:04:12.517 INFO:tasks.workunit.client.0.smithi116.stdout:7/403: chown d3/f18 53068 1 2024-03-04T16:04:12.517 INFO:tasks.workunit.client.0.smithi116.stdout:5/435: mknod dd/d1f/d5c/d5e/c8f 0 2024-03-04T16:04:12.518 INFO:tasks.workunit.client.0.smithi116.stdout:9/402: write d2/d19/d27/d3b/d4a/f4c [2787520,2635] 0 2024-03-04T16:04:12.518 INFO:tasks.workunit.client.0.smithi116.stdout:7/404: chown d3/d4/d8/c2b 329259 1 2024-03-04T16:04:12.518 INFO:tasks.workunit.client.0.smithi116.stdout:3/523: mknod dc/d23/d84/cb0 0 2024-03-04T16:04:12.519 INFO:tasks.workunit.client.0.smithi116.stdout:2/517: write d7/f24 [2336886,123416] 0 2024-03-04T16:04:12.519 INFO:tasks.workunit.client.0.smithi116.stdout:6/414: write f2 [4795417,11764] 0 2024-03-04T16:04:12.525 INFO:tasks.workunit.client.0.smithi116.stdout:7/405: mknod d3/d4/d7e/c82 0 2024-03-04T16:04:12.527 INFO:tasks.workunit.client.0.smithi116.stdout:5/436: dwrite dd/d1f/f4d [4194304,4194304] 0 2024-03-04T16:04:12.529 INFO:tasks.workunit.client.0.smithi116.stdout:2/518: dread d7/d15/d35/f3b [0,4194304] 0 2024-03-04T16:04:12.529 INFO:tasks.workunit.client.0.smithi116.stdout:5/437: fdatasync dd/d1f/d55/d73/d2d/f50 0 2024-03-04T16:04:12.530 INFO:tasks.workunit.client.0.smithi116.stdout:3/524: creat dc/d23/d84/d7f/d93/fb1 x:0 0 0 2024-03-04T16:04:12.533 INFO:tasks.workunit.client.0.smithi116.stdout:3/525: read - dc/d23/d84/d7c/fa1 zero size 2024-03-04T16:04:12.537 INFO:tasks.workunit.client.0.smithi116.stdout:4/496: dwrite d1/f41 [0,4194304] 0 2024-03-04T16:04:12.539 INFO:tasks.workunit.client.0.smithi116.stdout:4/497: read d1/d3/d40/f4f [1600244,84985] 0 2024-03-04T16:04:12.539 INFO:tasks.workunit.client.0.smithi116.stdout:5/438: symlink dd/d1f/d55/d73/d2d/d31/l90 0 2024-03-04T16:04:12.542 INFO:tasks.workunit.client.0.smithi116.stdout:2/519: rename d7/d13/d3e/d71/dad/lae to d7/d2f/d6c/lc6 0 2024-03-04T16:04:12.543 INFO:tasks.workunit.client.0.smithi116.stdout:2/520: chown d7/f4e 33 1 2024-03-04T16:04:12.547 INFO:tasks.workunit.client.0.smithi116.stdout:2/521: readlink d7/d13/d3e/d98/lbe 0 2024-03-04T16:04:12.551 INFO:tasks.workunit.client.0.smithi116.stdout:4/498: creat d1/d3/d40/d52/d84/d88/fb1 x:0 0 0 2024-03-04T16:04:12.552 INFO:tasks.workunit.client.0.smithi116.stdout:5/439: dwrite dd/d1f/d55/d73/d2d/d31/f7b [0,4194304] 0 2024-03-04T16:04:12.553 INFO:tasks.workunit.client.0.smithi116.stdout:1/329: sync 2024-03-04T16:04:12.560 INFO:tasks.workunit.client.0.smithi116.stdout:6/415: truncate d6/f21 20252 0 2024-03-04T16:04:12.561 INFO:tasks.workunit.client.0.smithi116.stdout:6/416: write f2 [4805244,67167] 0 2024-03-04T16:04:12.562 INFO:tasks.workunit.client.0.smithi116.stdout:2/522: symlink d7/d29/d47/d64/lc7 0 2024-03-04T16:04:12.564 INFO:tasks.workunit.client.0.smithi116.stdout:7/406: getdents d3/d4/d34 0 2024-03-04T16:04:12.565 INFO:tasks.workunit.client.0.smithi116.stdout:4/499: write d1/d3/de/f23 [8341686,96551] 0 2024-03-04T16:04:12.565 INFO:tasks.workunit.client.0.smithi116.stdout:7/407: write d3/d4/d8/d52/f7f [433220,15115] 0 2024-03-04T16:04:12.567 INFO:tasks.workunit.client.0.smithi116.stdout:7/408: write d3/d4/d34/f74 [989971,1787] 0 2024-03-04T16:04:12.568 INFO:tasks.workunit.client.0.smithi116.stdout:3/526: link dc/df/d14/f72 dc/d34/d44/d6a/da3/fb2 0 2024-03-04T16:04:12.569 INFO:tasks.workunit.client.0.smithi116.stdout:7/409: chown d3/d4/d8/d52 54 1 2024-03-04T16:04:12.570 INFO:tasks.workunit.client.0.smithi116.stdout:7/410: fsync d3/d4/f16 0 2024-03-04T16:04:12.570 INFO:tasks.workunit.client.0.smithi116.stdout:7/411: chown d3/d4/l57 10072522 1 2024-03-04T16:04:12.573 INFO:tasks.workunit.client.0.smithi116.stdout:2/523: unlink d7/d25/d41/f5a 0 2024-03-04T16:04:12.578 INFO:tasks.workunit.client.0.smithi116.stdout:5/440: dread dd/d1f/f29 [0,4194304] 0 2024-03-04T16:04:12.582 INFO:tasks.workunit.client.0.smithi116.stdout:6/417: link d6/de/d25/d4a/f71 d6/de/d29/f8d 0 2024-03-04T16:04:12.584 INFO:tasks.workunit.client.0.smithi116.stdout:4/500: fsync d1/d3/d15/d20/d1c/d62/d81/f93 0 2024-03-04T16:04:12.585 INFO:tasks.workunit.client.0.smithi116.stdout:4/501: dread - d1/d3/d40/d52/d84/d88/fb1 zero size 2024-03-04T16:04:12.587 INFO:tasks.workunit.client.0.smithi116.stdout:1/330: creat d5/f6f x:0 0 0 2024-03-04T16:04:12.589 INFO:tasks.workunit.client.0.smithi116.stdout:5/441: mkdir dd/d1f/d5c/d91 0 2024-03-04T16:04:12.593 INFO:tasks.workunit.client.0.smithi116.stdout:3/527: creat dc/df/d47/fb3 x:0 0 0 2024-03-04T16:04:12.594 INFO:tasks.workunit.client.0.smithi116.stdout:3/528: write dc/df/d14/f2b [8584738,83566] 0 2024-03-04T16:04:12.595 INFO:tasks.workunit.client.0.smithi116.stdout:1/331: chown d5/d9/d21/l34 362167191 1 2024-03-04T16:04:12.595 INFO:tasks.workunit.client.0.smithi116.stdout:3/529: dread - dc/d34/d44/f99 zero size 2024-03-04T16:04:12.598 INFO:tasks.workunit.client.0.smithi116.stdout:1/332: write d5/d9/f18 [211364,117308] 0 2024-03-04T16:04:12.602 INFO:tasks.workunit.client.0.smithi116.stdout:7/412: dwrite d3/f1f [0,4194304] 0 2024-03-04T16:04:12.606 INFO:tasks.workunit.client.0.smithi116.stdout:3/530: creat dc/d34/d44/fb4 x:0 0 0 2024-03-04T16:04:12.608 INFO:tasks.workunit.client.0.smithi116.stdout:7/413: dread d3/d4/d8/d1a/f27 [0,4194304] 0 2024-03-04T16:04:12.609 INFO:tasks.workunit.client.0.smithi116.stdout:3/531: truncate dc/df/d14/f5d 4822935 0 2024-03-04T16:04:12.609 INFO:tasks.workunit.client.0.smithi116.stdout:1/333: link d5/d9/d21/d27/d32/d49/f66 d5/d9/d46/f70 0 2024-03-04T16:04:12.610 INFO:tasks.workunit.client.0.smithi116.stdout:5/442: dread dd/d1f/d55/d73/f3f [0,4194304] 0 2024-03-04T16:04:12.611 INFO:tasks.workunit.client.0.smithi116.stdout:1/334: truncate d5/f6f 867247 0 2024-03-04T16:04:12.611 INFO:tasks.workunit.client.0.smithi116.stdout:5/443: write dd/d6d/f80 [1482312,4783] 0 2024-03-04T16:04:12.612 INFO:tasks.workunit.client.0.smithi116.stdout:5/444: readlink dd/d1f/d5c/l70 0 2024-03-04T16:04:12.613 INFO:tasks.workunit.client.0.smithi116.stdout:3/532: unlink dc/df/d14/f89 0 2024-03-04T16:04:12.613 INFO:tasks.workunit.client.0.smithi116.stdout:7/414: write d3/d4/d8/d1a/f32 [1239629,84881] 0 2024-03-04T16:04:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:7/415: dread d3/d4/d8/d52/f7f [0,4194304] 0 2024-03-04T16:04:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:7/416: dread - d3/d4/d8/d1a/f5f zero size 2024-03-04T16:04:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:5/445: read dd/d1f/d55/d73/d2d/d31/f65 [1553439,2337] 0 2024-03-04T16:04:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:1/335: link d5/f4d d5/d9/d46/d50/f71 0 2024-03-04T16:04:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:1/336: fdatasync d5/f29 0 2024-03-04T16:04:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:5/446: truncate dd/d1f/d5c/f64 562846 0 2024-03-04T16:04:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:1/337: chown d5/d9/d21/d27/d32/d49/d4a/l6e 104876 1 2024-03-04T16:04:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:5/447: dread dd/d1f/d55/d73/f3f [0,4194304] 0 2024-03-04T16:04:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:7/417: rename d3/d4/d11/c60 to d3/d4/d8/d4a/d65/c83 0 2024-03-04T16:04:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:1/338: unlink d5/d11/l31 0 2024-03-04T16:04:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:5/448: fdatasync dd/d1f/d55/d73/d2d/d7c/f86 0 2024-03-04T16:04:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:5/449: fdatasync dd/d1f/d55/f6c 0 2024-03-04T16:04:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:1/339: fsync d5/f1d 0 2024-03-04T16:04:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:5/450: chown dd/d1f/d55/d73/d2d/d59/d71/f7a 60185 1 2024-03-04T16:04:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:5/451: readlink dd/d1f/d55/l8c 0 2024-03-04T16:04:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:1/340: dread - d5/d9/d21/d27/d32/d49/f6d zero size 2024-03-04T16:04:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:1/341: chown d5/d9/d21/d24/c28 45889 1 2024-03-04T16:04:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:7/418: creat d3/d4/d8/d4a/f84 x:0 0 0 2024-03-04T16:04:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:1/342: mkdir d5/d9/d21/d27/d72 0 2024-03-04T16:04:12.638 INFO:tasks.workunit.client.0.smithi116.stdout:7/419: mkdir d3/d4/d8/d1a/d85 0 2024-03-04T16:04:12.639 INFO:tasks.workunit.client.0.smithi116.stdout:7/420: write d3/d4/d8/f13 [3330291,91997] 0 2024-03-04T16:04:12.646 INFO:tasks.workunit.client.0.smithi116.stdout:1/343: fsync d5/f48 0 2024-03-04T16:04:12.649 INFO:tasks.workunit.client.0.smithi116.stdout:5/452: getdents dd/d1f/d55/d73/d2d/d7c 0 2024-03-04T16:04:12.653 INFO:tasks.workunit.client.0.smithi116.stdout:1/344: creat d5/d9/d21/d27/d32/d49/d4c/d67/f73 x:0 0 0 2024-03-04T16:04:12.654 INFO:tasks.workunit.client.0.smithi116.stdout:7/421: mkdir d3/d4/d8/d4a/d86 0 2024-03-04T16:04:12.654 INFO:tasks.workunit.client.0.smithi116.stdout:5/453: dwrite dd/d1f/d5c/d5e/f62 [0,4194304] 0 2024-03-04T16:04:12.660 INFO:tasks.workunit.client.0.smithi116.stdout:1/345: dwrite d5/f3b [0,4194304] 0 2024-03-04T16:04:12.663 INFO:tasks.workunit.client.0.smithi116.stdout:5/454: fdatasync dd/d1f/d55/d73/d2d/d31/f3c 0 2024-03-04T16:04:12.667 INFO:tasks.workunit.client.0.smithi116.stdout:7/422: mkdir d3/d4/d8/d4a/d87 0 2024-03-04T16:04:12.673 INFO:tasks.workunit.client.0.smithi116.stdout:5/455: mknod dd/d1f/d51/c92 0 2024-03-04T16:04:12.682 INFO:tasks.workunit.client.0.smithi116.stdout:7/423: mkdir d3/d4/d8/d4a/d88 0 2024-03-04T16:04:12.685 INFO:tasks.workunit.client.0.smithi116.stdout:7/424: creat d3/d4/d8/d1a/d85/f89 x:0 0 0 2024-03-04T16:04:12.688 INFO:tasks.workunit.client.0.smithi116.stdout:7/425: unlink d3/d4/d8/cf 0 2024-03-04T16:04:12.690 INFO:tasks.workunit.client.0.smithi116.stdout:1/346: write d5/f4d [776946,3608] 0 2024-03-04T16:04:12.691 INFO:tasks.workunit.client.0.smithi116.stdout:7/426: fsync d3/d4/d8/d1a/f27 0 2024-03-04T16:04:12.692 INFO:tasks.workunit.client.0.smithi116.stdout:7/427: write d3/d4/d8/d1a/f46 [1750167,24938] 0 2024-03-04T16:04:12.693 INFO:tasks.workunit.client.0.smithi116.stdout:1/347: write d5/d11/f54 [2156418,23058] 0 2024-03-04T16:04:12.694 INFO:tasks.workunit.client.0.smithi116.stdout:1/348: chown d5/d9/f18 12633 1 2024-03-04T16:04:12.697 INFO:tasks.workunit.client.0.smithi116.stdout:5/456: dread dd/d1f/d55/d73/d2d/d31/f6a [0,4194304] 0 2024-03-04T16:04:12.701 INFO:tasks.workunit.client.0.smithi116.stdout:7/428: truncate d3/d4/f16 4740387 0 2024-03-04T16:04:12.701 INFO:tasks.workunit.client.0.smithi116.stdout:7/429: chown d3/d4/d8/d4a/f6b 1589 1 2024-03-04T16:04:12.705 INFO:tasks.workunit.client.0.smithi116.stdout:7/430: rename d3/d4/d8/d1a/f4d to d3/d4/d8/d4a/d87/f8a 0 2024-03-04T16:04:12.705 INFO:tasks.workunit.client.0.smithi116.stdout:7/431: chown d3/d4/d7e 6524156 1 2024-03-04T16:04:12.707 INFO:tasks.workunit.client.0.smithi116.stdout:5/457: getdents dd/d1f/d51 0 2024-03-04T16:04:12.708 INFO:tasks.workunit.client.0.smithi116.stdout:7/432: chown d3/f6 294381 1 2024-03-04T16:04:12.711 INFO:tasks.workunit.client.0.smithi116.stdout:7/433: creat d3/d4/d34/d5d/f8b x:0 0 0 2024-03-04T16:04:12.714 INFO:tasks.workunit.client.0.smithi116.stdout:5/458: dwrite dd/d1f/f5f [0,4194304] 0 2024-03-04T16:04:12.718 INFO:tasks.workunit.client.0.smithi116.stdout:7/434: dwrite d3/d4/d8/d52/d5e/f6e [0,4194304] 0 2024-03-04T16:04:12.719 INFO:tasks.workunit.client.0.smithi116.stdout:5/459: rmdir dd/d1f/d55 39 2024-03-04T16:04:12.723 INFO:tasks.workunit.client.0.smithi116.stdout:7/435: symlink d3/d4/d8/d52/l8c 0 2024-03-04T16:04:12.724 INFO:tasks.workunit.client.0.smithi116.stdout:7/436: dread - d3/d4/d8/d4a/d65/f7b zero size 2024-03-04T16:04:12.724 INFO:tasks.workunit.client.0.smithi116.stdout:7/437: chown d3/d4/l57 472146519 1 2024-03-04T16:04:12.725 INFO:tasks.workunit.client.0.smithi116.stdout:7/438: read - d3/f59 zero size 2024-03-04T16:04:12.725 INFO:tasks.workunit.client.0.smithi116.stdout:5/460: fdatasync dd/d1f/d55/d73/d2d/d31/f7b 0 2024-03-04T16:04:12.728 INFO:tasks.workunit.client.0.smithi116.stdout:5/461: write dd/d1f/d55/d73/d2d/d59/d71/f7a [625759,68379] 0 2024-03-04T16:04:12.730 INFO:tasks.workunit.client.0.smithi116.stdout:7/439: dread d3/d4/d8/f13 [0,4194304] 0 2024-03-04T16:04:12.732 INFO:tasks.workunit.client.0.smithi116.stdout:5/462: readlink dd/d1f/d55/d73/d2d/d31/l84 0 2024-03-04T16:04:12.734 INFO:tasks.workunit.client.0.smithi116.stdout:5/463: creat dd/d1f/d51/f93 x:0 0 0 2024-03-04T16:04:12.738 INFO:tasks.workunit.client.0.smithi116.stdout:7/440: dwrite d3/d4/d11/f36 [0,4194304] 0 2024-03-04T16:04:12.739 INFO:tasks.workunit.client.0.smithi116.stdout:7/441: fsync d3/d4/d8/d1a/f5c 0 2024-03-04T16:04:12.743 INFO:tasks.workunit.client.0.smithi116.stdout:7/442: creat d3/d4/d8/d4a/d86/f8d x:0 0 0 2024-03-04T16:04:12.746 INFO:tasks.workunit.client.0.smithi116.stdout:7/443: creat d3/d4/d34/d5d/d66/f8e x:0 0 0 2024-03-04T16:04:12.751 INFO:tasks.workunit.client.0.smithi116.stdout:5/464: truncate dd/d1f/f69 3068245 0 2024-03-04T16:04:12.752 INFO:tasks.workunit.client.0.smithi116.stdout:7/444: dread d3/d4/d8/d1a/f29 [0,4194304] 0 2024-03-04T16:04:12.753 INFO:tasks.workunit.client.0.smithi116.stdout:5/465: creat dd/d1f/d55/d73/f94 x:0 0 0 2024-03-04T16:04:12.756 INFO:tasks.workunit.client.0.smithi116.stdout:5/466: truncate dd/f22 3070997 0 2024-03-04T16:04:12.759 INFO:tasks.workunit.client.0.smithi116.stdout:7/445: getdents d3/d4/d34/d5d 0 2024-03-04T16:04:12.760 INFO:tasks.workunit.client.0.smithi116.stdout:5/467: truncate dd/d1f/d55/d73/d2d/d31/f3c 3174422 0 2024-03-04T16:04:12.762 INFO:tasks.workunit.client.0.smithi116.stdout:5/468: write dd/d1f/d55/d73/d2d/d31/f7b [2522198,84880] 0 2024-03-04T16:04:12.765 INFO:tasks.workunit.client.0.smithi116.stdout:5/469: creat dd/d45/f95 x:0 0 0 2024-03-04T16:04:12.766 INFO:tasks.workunit.client.0.smithi116.stdout:7/446: dwrite d3/f4e [4194304,4194304] 0 2024-03-04T16:04:12.767 INFO:tasks.workunit.client.0.smithi116.stdout:7/447: write f2 [1959999,3536] 0 2024-03-04T16:04:12.768 INFO:tasks.workunit.client.0.smithi116.stdout:7/448: write d3/f4e [2410917,80851] 0 2024-03-04T16:04:12.780 INFO:tasks.workunit.client.0.smithi116.stdout:7/449: mknod d3/d4/d8/c8f 0 2024-03-04T16:04:12.783 INFO:tasks.workunit.client.0.smithi116.stdout:7/450: dread d3/d4/d8/d4a/d65/f72 [0,4194304] 0 2024-03-04T16:04:12.784 INFO:tasks.workunit.client.0.smithi116.stdout:7/451: dread - d3/d4/d8/d1a/f5c zero size 2024-03-04T16:04:12.785 INFO:tasks.workunit.client.0.smithi116.stdout:7/452: dread - d3/d4/d8/d4a/d65/f7b zero size 2024-03-04T16:04:12.786 INFO:tasks.workunit.client.0.smithi116.stdout:7/453: stat d3/d4/d11/f5b 0 2024-03-04T16:04:12.792 INFO:tasks.workunit.client.0.smithi116.stdout:5/470: dwrite dd/d1f/d55/d73/d2d/f3b [0,4194304] 0 2024-03-04T16:04:12.795 INFO:tasks.workunit.client.0.smithi116.stdout:5/471: symlink dd/d1f/d5c/l96 0 2024-03-04T16:04:12.804 INFO:tasks.workunit.client.0.smithi116.stdout:5/472: truncate dd/f22 2397337 0 2024-03-04T16:04:12.804 INFO:tasks.workunit.client.0.smithi116.stdout:7/454: dwrite d3/d4/f16 [0,4194304] 0 2024-03-04T16:04:12.808 INFO:tasks.workunit.client.0.smithi116.stdout:5/473: unlink dd/d1f/d55/d73/d2d/c68 0 2024-03-04T16:04:12.818 INFO:tasks.workunit.client.0.smithi116.stdout:8/446: sync 2024-03-04T16:04:12.818 INFO:tasks.workunit.client.0.smithi116.stdout:9/403: sync 2024-03-04T16:04:12.819 INFO:tasks.workunit.client.0.smithi116.stdout:7/455: creat d3/f90 x:0 0 0 2024-03-04T16:04:12.820 INFO:tasks.workunit.client.0.smithi116.stdout:9/404: chown d2/d49/f6f 93 1 2024-03-04T16:04:12.820 INFO:tasks.workunit.client.0.smithi116.stdout:5/474: getdents dd/d1f/d72/d8e 0 2024-03-04T16:04:12.821 INFO:tasks.workunit.client.0.smithi116.stdout:9/405: fsync d2/d26/f44 0 2024-03-04T16:04:12.821 INFO:tasks.workunit.client.0.smithi116.stdout:7/456: write d3/d4/d8/f13 [454821,1265] 0 2024-03-04T16:04:12.823 INFO:tasks.workunit.client.0.smithi116.stdout:5/475: dread dd/d1f/d55/f77 [0,4194304] 0 2024-03-04T16:04:12.824 INFO:tasks.workunit.client.0.smithi116.stdout:9/406: symlink d2/d49/d3c/l7a 0 2024-03-04T16:04:12.826 INFO:tasks.workunit.client.0.smithi116.stdout:8/447: getdents d2/d13/d3c/d77/d97/d75 0 2024-03-04T16:04:12.827 INFO:tasks.workunit.client.0.smithi116.stdout:7/457: write d3/d4/d8/d4a/d65/f72 [1008239,43062] 0 2024-03-04T16:04:12.828 INFO:tasks.workunit.client.0.smithi116.stdout:8/448: rmdir d2/de/d16/d2b 39 2024-03-04T16:04:12.830 INFO:tasks.workunit.client.0.smithi116.stdout:9/407: creat d2/d19/d27/d3b/d4a/f7b x:0 0 0 2024-03-04T16:04:12.832 INFO:tasks.workunit.client.0.smithi116.stdout:7/458: write d3/d4/d8/d4a/d65/f80 [1018689,16287] 0 2024-03-04T16:04:12.832 INFO:tasks.workunit.client.0.smithi116.stdout:5/476: dwrite dd/d1f/d5c/d5e/f62 [0,4194304] 0 2024-03-04T16:04:12.836 INFO:tasks.workunit.client.0.smithi116.stdout:9/408: creat d2/d19/d27/d3b/d4a/d6a/f7c x:0 0 0 2024-03-04T16:04:12.837 INFO:tasks.workunit.client.0.smithi116.stdout:8/449: getdents d2/d13/d3c/d46 0 2024-03-04T16:04:12.842 INFO:tasks.workunit.client.0.smithi116.stdout:8/450: symlink d2/d13/d3c/d77/d97/d42/l9a 0 2024-03-04T16:04:12.844 INFO:tasks.workunit.client.0.smithi116.stdout:8/451: dread d2/d13/d3c/d46/f6d [0,4194304] 0 2024-03-04T16:04:12.848 INFO:tasks.workunit.client.0.smithi116.stdout:9/409: rename d2/d19/d27/d3b/d4a/d5b/l5c to d2/d33/d65/d6c/l7d 0 2024-03-04T16:04:12.854 INFO:tasks.workunit.client.0.smithi116.stdout:9/410: unlink d2/d19/d27/d3b/f70 0 2024-03-04T16:04:12.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:12 smithi152 ceph-mon[36876]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:12.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:12 smithi152 ceph-mon[36876]: pgmap v22: 65 pgs: 65 active+clean; 820 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 15 MiB/s rd, 103 MiB/s wr, 406 op/s 2024-03-04T16:04:12.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:12 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3427877196' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:04:12.865 INFO:tasks.workunit.client.0.smithi116.stdout:9/411: dread d2/d26/f47 [0,4194304] 0 2024-03-04T16:04:12.869 INFO:tasks.workunit.client.0.smithi116.stdout:9/412: mkdir d2/d33/d65/d7e 0 2024-03-04T16:04:12.870 INFO:tasks.workunit.client.0.smithi116.stdout:9/413: write d2/f4f [168443,120136] 0 2024-03-04T16:04:12.872 INFO:tasks.workunit.client.0.smithi116.stdout:8/452: write d2/de/d16/f18 [4472192,28119] 0 2024-03-04T16:04:12.876 INFO:tasks.workunit.client.0.smithi116.stdout:9/414: dwrite d2/d19/f37 [0,4194304] 0 2024-03-04T16:04:12.877 INFO:tasks.workunit.client.0.smithi116.stdout:9/415: dread - d2/d33/f77 zero size 2024-03-04T16:04:12.878 INFO:tasks.workunit.client.0.smithi116.stdout:8/453: dread d2/de/d16/f30 [0,4194304] 0 2024-03-04T16:04:12.882 INFO:tasks.workunit.client.0.smithi116.stdout:8/454: fdatasync d2/de/f8 0 2024-03-04T16:04:12.883 INFO:tasks.workunit.client.0.smithi116.stdout:8/455: chown d2/d13/d3c/d77/d97/l68 1918925 1 2024-03-04T16:04:12.885 INFO:tasks.workunit.client.0.smithi116.stdout:9/416: dwrite d2/d33/f3a [0,4194304] 0 2024-03-04T16:04:12.886 INFO:tasks.workunit.client.0.smithi116.stdout:8/456: fdatasync d2/de/d3e/d7b/f8e 0 2024-03-04T16:04:12.896 INFO:tasks.workunit.client.0.smithi116.stdout:9/417: dwrite d2/d19/d27/d3b/d4a/f73 [0,4194304] 0 2024-03-04T16:04:12.897 INFO:tasks.workunit.client.0.smithi116.stdout:8/457: dwrite d2/de/d16/f18 [4194304,4194304] 0 2024-03-04T16:04:12.901 INFO:tasks.workunit.client.0.smithi116.stdout:9/418: truncate d2/d26/d28/f55 6412 0 2024-03-04T16:04:12.901 INFO:tasks.workunit.client.0.smithi116.stdout:8/458: creat d2/d38/d41/f9b x:0 0 0 2024-03-04T16:04:12.905 INFO:tasks.workunit.client.0.smithi116.stdout:9/419: rmdir d2/d19/d27/d3b/d4a 39 2024-03-04T16:04:12.907 INFO:tasks.workunit.client.0.smithi116.stdout:8/459: mknod d2/d13/d3c/d46/d4f/c9c 0 2024-03-04T16:04:12.908 INFO:tasks.workunit.client.0.smithi116.stdout:8/460: chown d2/d38/d41/f9b 124333 1 2024-03-04T16:04:12.908 INFO:tasks.workunit.client.0.smithi116.stdout:9/420: mknod d2/d33/d65/d7e/c7f 0 2024-03-04T16:04:12.909 INFO:tasks.workunit.client.0.smithi116.stdout:8/461: truncate d2/de/f10 5066912 0 2024-03-04T16:04:12.910 INFO:tasks.workunit.client.0.smithi116.stdout:8/462: chown d2/d4a/c62 1480423 1 2024-03-04T16:04:12.912 INFO:tasks.workunit.client.0.smithi116.stdout:8/463: write d2/f32 [477320,97387] 0 2024-03-04T16:04:12.913 INFO:tasks.workunit.client.0.smithi116.stdout:9/421: dread d2/d19/d27/d3b/f68 [0,4194304] 0 2024-03-04T16:04:12.914 INFO:tasks.workunit.client.0.smithi116.stdout:8/464: fsync d2/d38/d55/f65 0 2024-03-04T16:04:12.918 INFO:tasks.workunit.client.0.smithi116.stdout:9/422: dread d2/d19/d27/d3b/f68 [0,4194304] 0 2024-03-04T16:04:12.920 INFO:tasks.workunit.client.0.smithi116.stdout:8/465: dwrite d2/de/d16/f18 [0,4194304] 0 2024-03-04T16:04:12.921 INFO:tasks.workunit.client.0.smithi116.stdout:8/466: chown d2/d4a/l82 26456 1 2024-03-04T16:04:12.924 INFO:tasks.workunit.client.0.smithi116.stdout:9/423: unlink d2/d19/d27/f38 0 2024-03-04T16:04:12.928 INFO:tasks.workunit.client.0.smithi116.stdout:8/467: dwrite d2/de/f10 [4194304,4194304] 0 2024-03-04T16:04:12.929 INFO:tasks.workunit.client.0.smithi116.stdout:8/468: chown d2/de/d3e/d7b/c84 14438505 1 2024-03-04T16:04:12.935 INFO:tasks.workunit.client.0.smithi116.stdout:8/469: creat d2/d13/d3c/d77/d97/d5a/f9d x:0 0 0 2024-03-04T16:04:12.941 INFO:tasks.workunit.client.0.smithi116.stdout:9/424: getdents d2/d33/d65/d6c 0 2024-03-04T16:04:12.943 INFO:tasks.workunit.client.0.smithi116.stdout:8/470: rmdir d2/de/d16/d78/d29/d95 0 2024-03-04T16:04:12.945 INFO:tasks.workunit.client.0.smithi116.stdout:9/425: mknod d2/d19/d27/d3b/d4a/d6a/c80 0 2024-03-04T16:04:12.946 INFO:tasks.workunit.client.0.smithi116.stdout:8/471: mknod d2/d4a/d8a/c9e 0 2024-03-04T16:04:12.948 INFO:tasks.workunit.client.0.smithi116.stdout:8/472: write d2/d13/d5e/f6b [1430542,79927] 0 2024-03-04T16:04:12.952 INFO:tasks.workunit.client.0.smithi116.stdout:9/426: dwrite d2/d19/d27/d3b/d4a/f7b [0,4194304] 0 2024-03-04T16:04:12.953 INFO:tasks.workunit.client.0.smithi116.stdout:9/427: write d2/d49/f6f [901440,101803] 0 2024-03-04T16:04:12.959 INFO:tasks.workunit.client.0.smithi116.stdout:9/428: write d2/d19/d27/d3b/f3d [459266,13947] 0 2024-03-04T16:04:12.960 INFO:tasks.workunit.client.0.smithi116.stdout:9/429: chown d2/d49/l30 114472287 1 2024-03-04T16:04:12.973 INFO:tasks.workunit.client.0.smithi116.stdout:8/473: dread d2/de/d16/d78/d29/f39 [0,4194304] 0 2024-03-04T16:04:12.974 INFO:tasks.workunit.client.0.smithi116.stdout:8/474: write d2/de/d3e/f50 [1049120,129751] 0 2024-03-04T16:04:12.980 INFO:tasks.workunit.client.0.smithi116.stdout:8/475: truncate d2/de/fc 2997530 0 2024-03-04T16:04:12.981 INFO:tasks.workunit.client.0.smithi116.stdout:8/476: truncate d2/de/f10 8402281 0 2024-03-04T16:04:12.988 INFO:tasks.workunit.client.0.smithi116.stdout:8/477: rename d2/d13/d3c/l72 to d2/de/d16/l9f 0 2024-03-04T16:04:12.995 INFO:tasks.workunit.client.0.smithi116.stdout:9/430: write d2/d26/f71 [108480,40423] 0 2024-03-04T16:04:12.999 INFO:tasks.workunit.client.0.smithi116.stdout:9/431: rename d2/d19/d27/d3b/d4a/f4c to d2/d26/d28/f81 0 2024-03-04T16:04:13.001 INFO:tasks.workunit.client.0.smithi116.stdout:8/478: dread d2/de/d16/f34 [4194304,4194304] 0 2024-03-04T16:04:13.006 INFO:tasks.workunit.client.0.smithi116.stdout:0/395: sync 2024-03-04T16:04:13.006 INFO:tasks.workunit.client.0.smithi116.stdout:2/524: sync 2024-03-04T16:04:13.012 INFO:tasks.workunit.client.0.smithi116.stdout:8/479: dread d2/d13/d3c/d46/d4f/f5b [0,4194304] 0 2024-03-04T16:04:13.012 INFO:tasks.workunit.client.0.smithi116.stdout:0/396: symlink da/d16/d20/d79/l80 0 2024-03-04T16:04:13.015 INFO:tasks.workunit.client.0.smithi116.stdout:2/525: mknod d7/d29/d47/d64/d75/d77/cc8 0 2024-03-04T16:04:13.017 INFO:tasks.workunit.client.0.smithi116.stdout:8/480: chown d2/d13/d3c/d77/d97/c7e 6834 1 2024-03-04T16:04:13.020 INFO:tasks.workunit.client.0.smithi116.stdout:0/397: rename da/l2a to da/d16/d20/d53/l81 0 2024-03-04T16:04:13.021 INFO:tasks.workunit.client.0.smithi116.stdout:0/398: chown da/d16/d19/f68 714 1 2024-03-04T16:04:13.026 INFO:tasks.workunit.client.0.smithi116.stdout:0/399: readlink da/d16/d19/d27/d45/l76 0 2024-03-04T16:04:13.030 INFO:tasks.workunit.client.0.smithi116.stdout:8/481: creat d2/de/d16/d2b/fa0 x:0 0 0 2024-03-04T16:04:13.032 INFO:tasks.workunit.client.0.smithi116.stdout:0/400: dwrite da/d16/d20/f65 [4194304,4194304] 0 2024-03-04T16:04:13.036 INFO:tasks.workunit.client.0.smithi116.stdout:8/482: rename d2/de/d3e/l21 to d2/d13/d3c/d77/la1 0 2024-03-04T16:04:13.038 INFO:tasks.workunit.client.0.smithi116.stdout:0/401: rmdir da/d16 39 2024-03-04T16:04:13.042 INFO:tasks.workunit.client.0.smithi116.stdout:9/432: truncate d2/f1a 4087794 0 2024-03-04T16:04:13.043 INFO:tasks.workunit.client.0.smithi116.stdout:8/483: dwrite d2/de/d16/d2b/fa0 [0,4194304] 0 2024-03-04T16:04:13.047 INFO:tasks.workunit.client.0.smithi116.stdout:0/402: mkdir da/d16/d19/d27/d82 0 2024-03-04T16:04:13.049 INFO:tasks.workunit.client.0.smithi116.stdout:8/484: dread d2/d38/f3b [0,4194304] 0 2024-03-04T16:04:13.051 INFO:tasks.workunit.client.0.smithi116.stdout:0/403: rmdir da 39 2024-03-04T16:04:13.054 INFO:tasks.workunit.client.0.smithi116.stdout:9/433: creat d2/d19/d27/d3b/f82 x:0 0 0 2024-03-04T16:04:13.058 INFO:tasks.workunit.client.0.smithi116.stdout:8/485: symlink d2/d13/d3c/d77/d97/d42/la2 0 2024-03-04T16:04:13.059 INFO:tasks.workunit.client.0.smithi116.stdout:0/404: stat da/d16/d28/d40/d5d/d7e 0 2024-03-04T16:04:13.062 INFO:tasks.workunit.client.0.smithi116.stdout:9/434: symlink d2/d19/d5f/l83 0 2024-03-04T16:04:13.064 INFO:tasks.workunit.client.0.smithi116.stdout:2/526: dwrite d7/d2f/d6c/da4/f69 [0,4194304] 0 2024-03-04T16:04:13.064 INFO:tasks.workunit.client.0.smithi116.stdout:4/502: sync 2024-03-04T16:04:13.064 INFO:tasks.workunit.client.0.smithi116.stdout:3/533: sync 2024-03-04T16:04:13.064 INFO:tasks.workunit.client.0.smithi116.stdout:1/349: sync 2024-03-04T16:04:13.064 INFO:tasks.workunit.client.0.smithi116.stdout:5/477: sync 2024-03-04T16:04:13.064 INFO:tasks.workunit.client.0.smithi116.stdout:6/418: sync 2024-03-04T16:04:13.064 INFO:tasks.workunit.client.0.smithi116.stdout:7/459: sync 2024-03-04T16:04:13.068 INFO:tasks.workunit.client.0.smithi116.stdout:3/534: write dc/d23/d84/d7c/fa1 [226233,86313] 0 2024-03-04T16:04:13.068 INFO:tasks.workunit.client.0.smithi116.stdout:0/405: creat da/d16/d28/d40/d5d/d7e/f83 x:0 0 0 2024-03-04T16:04:13.070 INFO:tasks.workunit.client.0.smithi116.stdout:0/406: chown da/d16/d28/d40/d48/d57/f7f 7305755 1 2024-03-04T16:04:13.072 INFO:tasks.workunit.client.0.smithi116.stdout:0/407: write da/d16/f4c [408822,54696] 0 2024-03-04T16:04:13.078 INFO:tasks.workunit.client.0.smithi116.stdout:3/535: dwrite dc/df/d14/f8c [0,4194304] 0 2024-03-04T16:04:13.080 INFO:tasks.workunit.client.0.smithi116.stdout:6/419: rename d6/de/d1e/d80/f54 to d6/de/d5b/f8e 0 2024-03-04T16:04:13.081 INFO:tasks.workunit.client.0.smithi116.stdout:2/527: dread d7/d2f/f44 [0,4194304] 0 2024-03-04T16:04:13.086 INFO:tasks.workunit.client.0.smithi116.stdout:8/486: getdents d2/d13/d5e 0 2024-03-04T16:04:13.087 INFO:tasks.workunit.client.0.smithi116.stdout:6/420: dwrite d6/de/f1b [0,4194304] 0 2024-03-04T16:04:13.088 INFO:tasks.workunit.client.0.smithi116.stdout:1/350: getdents d5/d11/d61 0 2024-03-04T16:04:13.089 INFO:tasks.workunit.client.0.smithi116.stdout:8/487: write d2/de/d16/f18 [8833194,126367] 0 2024-03-04T16:04:13.091 INFO:tasks.workunit.client.0.smithi116.stdout:1/351: fdatasync d5/d9/f18 0 2024-03-04T16:04:13.092 INFO:tasks.workunit.client.0.smithi116.stdout:2/528: dwrite d7/d2f/d76/d7c/f7d [0,4194304] 0 2024-03-04T16:04:13.094 INFO:tasks.workunit.client.0.smithi116.stdout:9/435: rmdir d2/d49/d52 0 2024-03-04T16:04:13.094 INFO:tasks.workunit.client.0.smithi116.stdout:1/352: write d5/d11/f69 [360220,8270] 0 2024-03-04T16:04:13.095 INFO:tasks.workunit.client.0.smithi116.stdout:2/529: write d7/d15/f59 [299332,36122] 0 2024-03-04T16:04:13.095 INFO:tasks.workunit.client.0.smithi116.stdout:4/503: mknod d1/d3/d15/d20/d1c/d6f/cb2 0 2024-03-04T16:04:13.096 INFO:tasks.workunit.client.0.smithi116.stdout:4/504: dread - d1/d3/d15/d20/d65/f99 zero size 2024-03-04T16:04:13.101 INFO:tasks.workunit.client.0.smithi116.stdout:3/536: mkdir dc/d34/d44/d6a/db5 0 2024-03-04T16:04:13.103 INFO:tasks.workunit.client.0.smithi116.stdout:7/460: rename d3/d4/d8/d52/l8c to d3/d4/d8/d4a/d65/l91 0 2024-03-04T16:04:13.104 INFO:tasks.workunit.client.0.smithi116.stdout:3/537: dread dc/df/d47/f7d [0,4194304] 0 2024-03-04T16:04:13.111 INFO:tasks.workunit.client.0.smithi116.stdout:5/478: dwrite dd/d1f/d55/d73/d2d/d31/f74 [0,4194304] 0 2024-03-04T16:04:13.117 INFO:tasks.workunit.client.0.smithi116.stdout:8/488: fsync d2/d13/d3c/d77/d97/d75/f79 0 2024-03-04T16:04:13.128 INFO:tasks.workunit.client.0.smithi116.stdout:7/461: unlink d3/d4/d8/d52/f7f 0 2024-03-04T16:04:13.129 INFO:tasks.workunit.client.0.smithi116.stdout:7/462: fdatasync d3/d4/d8/d52/f54 0 2024-03-04T16:04:13.129 INFO:tasks.workunit.client.0.smithi116.stdout:7/463: dread - d3/d4/d11/f5b zero size 2024-03-04T16:04:13.130 INFO:tasks.workunit.client.0.smithi116.stdout:7/464: chown d3/d4/d11/d61/c73 26 1 2024-03-04T16:04:13.131 INFO:tasks.workunit.client.0.smithi116.stdout:3/538: mkdir dc/d23/d84/d7f/d93/db6 0 2024-03-04T16:04:13.131 INFO:tasks.workunit.client.0.smithi116.stdout:0/408: creat da/d16/f84 x:0 0 0 2024-03-04T16:04:13.132 INFO:tasks.workunit.client.0.smithi116.stdout:5/479: rmdir dd/d1f/d5c 39 2024-03-04T16:04:13.135 INFO:tasks.workunit.client.0.smithi116.stdout:6/421: creat d6/de/d84/f8f x:0 0 0 2024-03-04T16:04:13.135 INFO:tasks.workunit.client.0.smithi116.stdout:2/530: write d7/f4d [953815,21209] 0 2024-03-04T16:04:13.136 INFO:tasks.workunit.client.0.smithi116.stdout:4/505: write d1/d3/de/d47/f60 [941771,100342] 0 2024-03-04T16:04:13.142 INFO:tasks.workunit.client.0.smithi116.stdout:7/465: chown d3/d4/d8/d1a/l77 1322065 1 2024-03-04T16:04:13.144 INFO:tasks.workunit.client.0.smithi116.stdout:0/409: mknod da/d16/d19/d27/c85 0 2024-03-04T16:04:13.145 INFO:tasks.workunit.client.0.smithi116.stdout:6/422: dwrite d6/de/f1b [0,4194304] 0 2024-03-04T16:04:13.145 INFO:tasks.workunit.client.0.smithi116.stdout:5/480: readlink dd/l23 0 2024-03-04T16:04:13.148 INFO:tasks.workunit.client.0.smithi116.stdout:0/410: fdatasync da/d16/d20/d42/d4e/f72 0 2024-03-04T16:04:13.148 INFO:tasks.workunit.client.0.smithi116.stdout:6/423: write d6/f13 [887086,112999] 0 2024-03-04T16:04:13.149 INFO:tasks.workunit.client.0.smithi116.stdout:9/436: creat d2/d19/f84 x:0 0 0 2024-03-04T16:04:13.151 INFO:tasks.workunit.client.0.smithi116.stdout:6/424: readlink d6/de/d29/l81 0 2024-03-04T16:04:13.151 INFO:tasks.workunit.client.0.smithi116.stdout:9/437: dread - d2/d49/d3c/f79 zero size 2024-03-04T16:04:13.152 INFO:tasks.workunit.client.0.smithi116.stdout:8/489: dread d2/de/f12 [0,4194304] 0 2024-03-04T16:04:13.155 INFO:tasks.workunit.client.0.smithi116.stdout:4/506: dread d1/d3/f44 [0,4194304] 0 2024-03-04T16:04:13.156 INFO:tasks.workunit.client.0.smithi116.stdout:3/539: rename dc/f56 to dc/df/d14/fb7 0 2024-03-04T16:04:13.159 INFO:tasks.workunit.client.0.smithi116.stdout:0/411: dread da/fe [0,4194304] 0 2024-03-04T16:04:13.161 INFO:tasks.workunit.client.0.smithi116.stdout:5/481: creat dd/d1f/d55/f97 x:0 0 0 2024-03-04T16:04:13.162 INFO:tasks.workunit.client.0.smithi116.stdout:9/438: dread d2/d19/f37 [0,4194304] 0 2024-03-04T16:04:13.164 INFO:tasks.workunit.client.0.smithi116.stdout:6/425: dwrite d6/de/d25/d4a/d63/f74 [0,4194304] 0 2024-03-04T16:04:13.169 INFO:tasks.workunit.client.0.smithi116.stdout:8/490: readlink d2/d13/d3c/d77/la1 0 2024-03-04T16:04:13.169 INFO:tasks.workunit.client.0.smithi116.stdout:7/466: dread d3/d4/d8/f35 [0,4194304] 0 2024-03-04T16:04:13.175 INFO:tasks.workunit.client.0.smithi116.stdout:2/531: write d7/d81/f21 [2708367,76242] 0 2024-03-04T16:04:13.175 INFO:tasks.workunit.client.0.smithi116.stdout:8/491: dwrite d2/d13/d5e/f87 [0,4194304] 0 2024-03-04T16:04:13.183 INFO:tasks.workunit.client.0.smithi116.stdout:6/426: rename d6/de/c1c to d6/de/d5b/c90 0 2024-03-04T16:04:13.188 INFO:tasks.workunit.client.0.smithi116.stdout:3/540: mkdir dc/df/db8 0 2024-03-04T16:04:13.190 INFO:tasks.workunit.client.0.smithi116.stdout:4/507: creat d1/d3/d40/d52/d84/d88/d8d/fb3 x:0 0 0 2024-03-04T16:04:13.192 INFO:tasks.workunit.client.0.smithi116.stdout:4/508: stat d1/d3/f7d 0 2024-03-04T16:04:13.196 INFO:tasks.workunit.client.0.smithi116.stdout:9/439: dwrite d2/d49/f42 [0,4194304] 0 2024-03-04T16:04:13.196 INFO:tasks.workunit.client.0.smithi116.stdout:7/467: dwrite d3/d4/d8/f1c [0,4194304] 0 2024-03-04T16:04:13.196 INFO:tasks.workunit.client.0.smithi116.stdout:3/541: dread dc/f73 [0,4194304] 0 2024-03-04T16:04:13.199 INFO:tasks.workunit.client.0.smithi116.stdout:4/509: chown d1/d3/d15/d20/d1c/d62/d81/da6/fa7 2771 1 2024-03-04T16:04:13.201 INFO:tasks.workunit.client.0.smithi116.stdout:5/482: write dd/d1f/d55/d73/d2d/d31/f3c [1742808,13550] 0 2024-03-04T16:04:13.202 INFO:tasks.workunit.client.0.smithi116.stdout:0/412: truncate da/d16/d19/d27/f34 1470370 0 2024-03-04T16:04:13.205 INFO:tasks.workunit.client.0.smithi116.stdout:5/483: read dd/d1f/f4e [2568979,55713] 0 2024-03-04T16:04:13.208 INFO:tasks.workunit.client.0.smithi116.stdout:8/492: creat d2/d13/d3c/fa3 x:0 0 0 2024-03-04T16:04:13.210 INFO:tasks.workunit.client.0.smithi116.stdout:2/532: dread d7/f4e [4194304,4194304] 0 2024-03-04T16:04:13.213 INFO:tasks.workunit.client.0.smithi116.stdout:9/440: symlink d2/d19/l85 0 2024-03-04T16:04:13.214 INFO:tasks.workunit.client.0.smithi116.stdout:4/510: mkdir d1/d3/dd/d24/db4 0 2024-03-04T16:04:13.215 INFO:tasks.workunit.client.0.smithi116.stdout:6/427: dwrite d6/f17 [0,4194304] 0 2024-03-04T16:04:13.215 INFO:tasks.workunit.client.0.smithi116.stdout:2/533: rename d7/d2f/d6c to d7/d2f/d6c/da4/da1/dc9 22 2024-03-04T16:04:13.215 INFO:tasks.workunit.client.0.smithi116.stdout:0/413: dwrite da/d16/d20/f65 [0,4194304] 0 2024-03-04T16:04:13.221 INFO:tasks.workunit.client.0.smithi116.stdout:1/353: sync 2024-03-04T16:04:13.222 INFO:tasks.workunit.client.0.smithi116.stdout:8/493: mkdir d2/de/d16/d2b/da4 0 2024-03-04T16:04:13.223 INFO:tasks.workunit.client.0.smithi116.stdout:5/484: creat dd/d1f/d72/d8e/f98 x:0 0 0 2024-03-04T16:04:13.223 INFO:tasks.workunit.client.0.smithi116.stdout:7/468: creat d3/d4/f92 x:0 0 0 2024-03-04T16:04:13.225 INFO:tasks.workunit.client.0.smithi116.stdout:8/494: write d2/de/d16/f18 [597879,3500] 0 2024-03-04T16:04:13.226 INFO:tasks.workunit.client.0.smithi116.stdout:7/469: write d3/d4/d8/d52/f54 [631387,26438] 0 2024-03-04T16:04:13.227 INFO:tasks.workunit.client.0.smithi116.stdout:4/511: rename d1/c3b to d1/d3/d40/d52/d84/d88/cb5 0 2024-03-04T16:04:13.228 INFO:tasks.workunit.client.0.smithi116.stdout:8/495: readlink d2/d13/d3c/d77/d97/d42/l89 0 2024-03-04T16:04:13.228 INFO:tasks.workunit.client.0.smithi116.stdout:4/512: read - d1/d3/dd/d45/f90 zero size 2024-03-04T16:04:13.231 INFO:tasks.workunit.client.0.smithi116.stdout:0/414: creat da/d16/d20/d42/f86 x:0 0 0 2024-03-04T16:04:13.237 INFO:tasks.workunit.client.0.smithi116.stdout:2/534: symlink d7/d29/lca 0 2024-03-04T16:04:13.240 INFO:tasks.workunit.client.0.smithi116.stdout:1/354: creat d5/d9/d21/d27/d32/d35/f74 x:0 0 0 2024-03-04T16:04:13.241 INFO:tasks.workunit.client.0.smithi116.stdout:5/485: dwrite dd/d1f/f29 [0,4194304] 0 2024-03-04T16:04:13.245 INFO:tasks.workunit.client.0.smithi116.stdout:3/542: dread dc/d23/d84/d7c/f65 [0,4194304] 0 2024-03-04T16:04:13.250 INFO:tasks.workunit.client.0.smithi116.stdout:1/355: dwrite d5/f4d [0,4194304] 0 2024-03-04T16:04:13.252 INFO:tasks.workunit.client.0.smithi116.stdout:8/496: dwrite d2/de/d16/d78/d29/f39 [4194304,4194304] 0 2024-03-04T16:04:13.253 INFO:tasks.workunit.client.0.smithi116.stdout:8/497: dread - d2/d13/d3c/fa3 zero size 2024-03-04T16:04:13.254 INFO:tasks.workunit.client.0.smithi116.stdout:7/470: creat d3/d4/d8/d4a/d88/f93 x:0 0 0 2024-03-04T16:04:13.259 INFO:tasks.workunit.client.0.smithi116.stdout:5/486: rename dd/d1f/f4e to dd/d45/f99 0 2024-03-04T16:04:13.260 INFO:tasks.workunit.client.0.smithi116.stdout:6/428: dread d6/de/d25/f40 [0,4194304] 0 2024-03-04T16:04:13.263 INFO:tasks.workunit.client.0.smithi116.stdout:0/415: unlink da/d16/d20/f21 0 2024-03-04T16:04:13.264 INFO:tasks.workunit.client.0.smithi116.stdout:1/356: fsync d5/d9/d21/d27/d32/d35/f74 0 2024-03-04T16:04:13.264 INFO:tasks.workunit.client.0.smithi116.stdout:8/498: dwrite d2/de/d3e/f2a [0,4194304] 0 2024-03-04T16:04:13.265 INFO:tasks.workunit.client.0.smithi116.stdout:9/441: getdents d2/d19/d27 0 2024-03-04T16:04:13.268 INFO:tasks.workunit.client.0.smithi116.stdout:2/535: symlink d7/d15/d5e/lcb 0 2024-03-04T16:04:13.269 INFO:tasks.workunit.client.0.smithi116.stdout:4/513: creat d1/d3/d15/fb6 x:0 0 0 2024-03-04T16:04:13.272 INFO:tasks.workunit.client.0.smithi116.stdout:4/514: truncate d1/d3/d40/d52/d84/fa0 16621 0 2024-03-04T16:04:13.273 INFO:tasks.workunit.client.0.smithi116.stdout:6/429: symlink d6/de/d29/l91 0 2024-03-04T16:04:13.275 INFO:tasks.workunit.client.0.smithi116.stdout:3/543: mknod dc/d34/d44/cb9 0 2024-03-04T16:04:13.277 INFO:tasks.workunit.client.0.smithi116.stdout:9/442: symlink d2/d33/d65/l86 0 2024-03-04T16:04:13.277 INFO:tasks.workunit.client.0.smithi116.stdout:9/443: stat d2 0 2024-03-04T16:04:13.280 INFO:tasks.workunit.client.0.smithi116.stdout:8/499: truncate d2/d13/d3c/d77/d97/d75/f79 1027076 0 2024-03-04T16:04:13.282 INFO:tasks.workunit.client.0.smithi116.stdout:7/471: rename d3/d4/d8/d1a/c33 to d3/d4/d34/c94 0 2024-03-04T16:04:13.282 INFO:tasks.workunit.client.0.smithi116.stdout:8/500: read - d2/d13/d3c/d77/d97/f94 zero size 2024-03-04T16:04:13.283 INFO:tasks.workunit.client.0.smithi116.stdout:5/487: rename dd/d1f/d55/d73/d2d to dd/d1f/d55/d73/d2d/d9a 22 2024-03-04T16:04:13.285 INFO:tasks.workunit.client.0.smithi116.stdout:9/444: dwrite d2/d49/d3c/f79 [0,4194304] 0 2024-03-04T16:04:13.290 INFO:tasks.workunit.client.0.smithi116.stdout:3/544: symlink dc/d23/d84/d7f/d93/lba 0 2024-03-04T16:04:13.291 INFO:tasks.workunit.client.0.smithi116.stdout:3/545: stat dc/d23/d84 0 2024-03-04T16:04:13.292 INFO:tasks.workunit.client.0.smithi116.stdout:2/536: symlink d7/d29/d47/d64/da5/lcc 0 2024-03-04T16:04:13.292 INFO:tasks.workunit.client.0.smithi116.stdout:9/445: dwrite d2/d19/d27/d3b/d4a/d6a/f7c [0,4194304] 0 2024-03-04T16:04:13.295 INFO:tasks.workunit.client.0.smithi116.stdout:0/416: write da/f35 [394921,37717] 0 2024-03-04T16:04:13.298 INFO:tasks.workunit.client.0.smithi116.stdout:0/417: write da/d16/d20/d42/f86 [822241,16209] 0 2024-03-04T16:04:13.301 INFO:tasks.workunit.client.0.smithi116.stdout:1/357: rename d5/d9/d21/d27/d72 to d5/d9/d21/d27/d32/d35/d55/d75 0 2024-03-04T16:04:13.303 INFO:tasks.workunit.client.0.smithi116.stdout:1/358: write d5/f4d [351187,97605] 0 2024-03-04T16:04:13.312 INFO:tasks.workunit.client.0.smithi116.stdout:6/430: dwrite d6/de/d29/f8d [0,4194304] 0 2024-03-04T16:04:13.313 INFO:tasks.workunit.client.0.smithi116.stdout:4/515: truncate d1/d3/d15/d20/f56 5825634 0 2024-03-04T16:04:13.314 INFO:tasks.workunit.client.0.smithi116.stdout:5/488: write dd/fe [5000931,1282] 0 2024-03-04T16:04:13.320 INFO:tasks.workunit.client.0.smithi116.stdout:2/537: dread d7/d29/d47/d64/f87 [0,4194304] 0 2024-03-04T16:04:13.322 INFO:tasks.workunit.client.0.smithi116.stdout:3/546: dread dc/df/d14/f5d [0,4194304] 0 2024-03-04T16:04:13.323 INFO:tasks.workunit.client.0.smithi116.stdout:3/547: fdatasync dc/d23/d84/d7f/f8a 0 2024-03-04T16:04:13.324 INFO:tasks.workunit.client.0.smithi116.stdout:3/548: truncate dc/d23/d84/d7c/f96 497682 0 2024-03-04T16:04:13.328 INFO:tasks.workunit.client.0.smithi116.stdout:0/418: write da/f1e [2052386,61708] 0 2024-03-04T16:04:13.329 INFO:tasks.workunit.client.0.smithi116.stdout:0/419: write da/d16/d20/d53/f63 [543318,78693] 0 2024-03-04T16:04:13.332 INFO:tasks.workunit.client.0.smithi116.stdout:4/516: creat d1/d6e/fb7 x:0 0 0 2024-03-04T16:04:13.333 INFO:tasks.workunit.client.0.smithi116.stdout:5/489: stat dd/d1f/d55/d73/l56 0 2024-03-04T16:04:13.338 INFO:tasks.workunit.client.0.smithi116.stdout:1/359: link d5/f4d d5/d9/d21/d27/d32/d35/f76 0 2024-03-04T16:04:13.340 INFO:tasks.workunit.client.0.smithi116.stdout:1/360: write d5/d9/d21/d27/d32/d35/f43 [168799,50804] 0 2024-03-04T16:04:13.341 INFO:tasks.workunit.client.0.smithi116.stdout:3/549: dread dc/df/f7e [0,4194304] 0 2024-03-04T16:04:13.345 INFO:tasks.workunit.client.0.smithi116.stdout:6/431: mkdir d6/de/d25/d36/d4e/d50/d92 0 2024-03-04T16:04:13.346 INFO:tasks.workunit.client.0.smithi116.stdout:3/550: dread dc/df/d14/f8c [0,4194304] 0 2024-03-04T16:04:13.347 INFO:tasks.workunit.client.0.smithi116.stdout:0/420: dread da/d16/d28/d40/d48/f52 [0,4194304] 0 2024-03-04T16:04:13.347 INFO:tasks.workunit.client.0.smithi116.stdout:3/551: dread - dc/f88 zero size 2024-03-04T16:04:13.348 INFO:tasks.workunit.client.0.smithi116.stdout:9/446: creat d2/f87 x:0 0 0 2024-03-04T16:04:13.348 INFO:tasks.workunit.client.0.smithi116.stdout:0/421: truncate da/f35 445599 0 2024-03-04T16:04:13.349 INFO:tasks.workunit.client.0.smithi116.stdout:4/517: unlink d1/d3/d15/d20/d1c/d62/d81/f93 0 2024-03-04T16:04:13.350 INFO:tasks.workunit.client.0.smithi116.stdout:4/518: readlink d1/d3/de/l73 0 2024-03-04T16:04:13.351 INFO:tasks.workunit.client.0.smithi116.stdout:5/490: symlink dd/d1f/d55/d73/d2d/d59/l9b 0 2024-03-04T16:04:13.353 INFO:tasks.workunit.client.0.smithi116.stdout:8/501: rename d2/de/d16/f34 to d2/de/fa5 0 2024-03-04T16:04:13.354 INFO:tasks.workunit.client.0.smithi116.stdout:8/502: write d2/d13/d3c/d77/d97/d5a/f6f [504170,57617] 0 2024-03-04T16:04:13.356 INFO:tasks.workunit.client.0.smithi116.stdout:1/361: symlink d5/d9/d21/d27/d32/d35/l77 0 2024-03-04T16:04:13.357 INFO:tasks.workunit.client.0.smithi116.stdout:6/432: rmdir d6/de/d25/d4a 39 2024-03-04T16:04:13.361 INFO:tasks.workunit.client.0.smithi116.stdout:0/422: symlink da/d16/d19/d27/l87 0 2024-03-04T16:04:13.364 INFO:tasks.workunit.client.0.smithi116.stdout:4/519: unlink d1/d3/dd/d24/f69 0 2024-03-04T16:04:13.366 INFO:tasks.workunit.client.0.smithi116.stdout:5/491: rmdir dd/d1f 39 2024-03-04T16:04:13.373 INFO:tasks.workunit.client.0.smithi116.stdout:6/433: mknod d6/de/d25/d36/d4e/d50/c93 0 2024-03-04T16:04:13.376 INFO:tasks.workunit.client.0.smithi116.stdout:3/552: creat dc/d23/d84/d7f/d93/db6/fbb x:0 0 0 2024-03-04T16:04:13.377 INFO:tasks.workunit.client.0.smithi116.stdout:3/553: readlink dc/l2a 0 2024-03-04T16:04:13.379 INFO:tasks.workunit.client.0.smithi116.stdout:4/520: write d1/d6e/f86 [2044096,69373] 0 2024-03-04T16:04:13.382 INFO:tasks.workunit.client.0.smithi116.stdout:4/521: write d1/d3/d40/d52/d84/fa0 [160501,32785] 0 2024-03-04T16:04:13.382 INFO:tasks.workunit.client.0.smithi116.stdout:6/434: unlink d6/de/d15/l28 0 2024-03-04T16:04:13.384 INFO:tasks.workunit.client.0.smithi116.stdout:3/554: dwrite dc/df/d14/f58 [0,4194304] 0 2024-03-04T16:04:13.384 INFO:tasks.workunit.client.0.smithi116.stdout:9/447: creat d2/d33/f88 x:0 0 0 2024-03-04T16:04:13.387 INFO:tasks.workunit.client.0.smithi116.stdout:7/472: rename d3/d4/d34/d5d/d66 to d3/d95 0 2024-03-04T16:04:13.390 INFO:tasks.workunit.client.0.smithi116.stdout:5/492: dwrite dd/d1f/d55/d73/d2d/d31/f7b [0,4194304] 0 2024-03-04T16:04:13.391 INFO:tasks.workunit.client.0.smithi116.stdout:5/493: readlink dd/d1f/d55/d73/d2d/d31/l84 0 2024-03-04T16:04:13.402 INFO:tasks.workunit.client.0.smithi116.stdout:9/448: rmdir d2/d19/d27/d3b 39 2024-03-04T16:04:13.405 INFO:tasks.workunit.client.0.smithi116.stdout:1/362: creat d5/d9/d21/d27/d32/d49/d4a/f78 x:0 0 0 2024-03-04T16:04:13.405 INFO:tasks.workunit.client.0.smithi116.stdout:1/363: stat d5/d11/f69 0 2024-03-04T16:04:13.407 INFO:tasks.workunit.client.0.smithi116.stdout:2/538: rename d7/d15/d5e/lcb to d7/d2f/d6c/da4/lcd 0 2024-03-04T16:04:13.412 INFO:tasks.workunit.client.0.smithi116.stdout:3/555: mkdir dc/df/dbc 0 2024-03-04T16:04:13.413 INFO:tasks.workunit.client.0.smithi116.stdout:3/556: chown dc/d34/d44/d6a/l6d 9716 1 2024-03-04T16:04:13.414 INFO:tasks.workunit.client.0.smithi116.stdout:9/449: creat d2/d49/f89 x:0 0 0 2024-03-04T16:04:13.414 INFO:tasks.workunit.client.0.smithi116.stdout:9/450: fdatasync d2/d49/f21 0 2024-03-04T16:04:13.415 INFO:tasks.workunit.client.0.smithi116.stdout:4/522: write d1/d3/de/f12 [1071090,20125] 0 2024-03-04T16:04:13.422 INFO:tasks.workunit.client.0.smithi116.stdout:8/503: rename d2/de/f12 to d2/de/d16/d2b/fa6 0 2024-03-04T16:04:13.422 INFO:tasks.workunit.client.0.smithi116.stdout:7/473: unlink d3/d4/d8/l2d 0 2024-03-04T16:04:13.423 INFO:tasks.workunit.client.0.smithi116.stdout:6/435: symlink d6/de/d25/d4a/d63/d8c/l94 0 2024-03-04T16:04:13.424 INFO:tasks.workunit.client.0.smithi116.stdout:7/474: chown d3/d4/d8/f25 31984 1 2024-03-04T16:04:13.427 INFO:tasks.workunit.client.0.smithi116.stdout:1/364: write d5/f48 [220245,58709] 0 2024-03-04T16:04:13.430 INFO:tasks.workunit.client.0.smithi116.stdout:8/504: dwrite d2/d13/d3c/d77/d97/d5a/f9d [0,4194304] 0 2024-03-04T16:04:13.432 INFO:tasks.workunit.client.0.smithi116.stdout:8/505: write d2/d13/d3c/d46/d4f/f70 [12255,69415] 0 2024-03-04T16:04:13.435 INFO:tasks.workunit.client.0.smithi116.stdout:4/523: dwrite d1/d3/dd/d24/f29 [0,4194304] 0 2024-03-04T16:04:13.445 INFO:tasks.workunit.client.0.smithi116.stdout:5/494: rename dd/d1f/d55/d73/d2d/c82 to dd/d1f/d55/d73/c9c 0 2024-03-04T16:04:13.452 INFO:tasks.workunit.client.0.smithi116.stdout:3/557: creat dc/d34/d44/fbd x:0 0 0 2024-03-04T16:04:13.452 INFO:tasks.workunit.client.0.smithi116.stdout:5/495: dread dd/d1f/d55/d73/d2d/d31/f7b [0,4194304] 0 2024-03-04T16:04:13.452 INFO:tasks.workunit.client.0.smithi116.stdout:4/524: fsync d1/d3/dd/f5d 0 2024-03-04T16:04:13.454 INFO:tasks.workunit.client.0.smithi116.stdout:2/539: rename d7/d29 to d7/d13/d3e/d98/dce 0 2024-03-04T16:04:13.455 INFO:tasks.workunit.client.0.smithi116.stdout:7/475: symlink d3/d4/d8/l96 0 2024-03-04T16:04:13.458 INFO:tasks.workunit.client.0.smithi116.stdout:8/506: dread d2/d13/d3c/d77/d97/d75/f79 [0,4194304] 0 2024-03-04T16:04:13.459 INFO:tasks.workunit.client.0.smithi116.stdout:1/365: dwrite d5/d11/f45 [0,4194304] 0 2024-03-04T16:04:13.466 INFO:tasks.workunit.client.0.smithi116.stdout:5/496: creat dd/d1f/d55/d73/d2d/d7c/f9d x:0 0 0 2024-03-04T16:04:13.470 INFO:tasks.workunit.client.0.smithi116.stdout:7/476: symlink d3/d4/d7e/l97 0 2024-03-04T16:04:13.476 INFO:tasks.workunit.client.0.smithi116.stdout:8/507: mkdir d2/d13/d5e/da7 0 2024-03-04T16:04:13.477 INFO:tasks.workunit.client.0.smithi116.stdout:1/366: mknod d5/d11/d42/c79 0 2024-03-04T16:04:13.485 INFO:tasks.workunit.client.0.smithi116.stdout:9/451: getdents d2/d33/d65 0 2024-03-04T16:04:13.486 INFO:tasks.workunit.client.0.smithi116.stdout:7/477: mknod d3/d4/d11/c98 0 2024-03-04T16:04:13.487 INFO:tasks.workunit.client.0.smithi116.stdout:8/508: truncate d2/d13/d3c/d77/d97/d75/f79 751633 0 2024-03-04T16:04:13.493 INFO:tasks.workunit.client.0.smithi116.stdout:1/367: dread d5/d9/d21/d24/f3a [0,4194304] 0 2024-03-04T16:04:13.493 INFO:tasks.workunit.client.0.smithi116.stdout:7/478: dwrite d3/d4/d8/d52/f53 [0,4194304] 0 2024-03-04T16:04:13.495 INFO:tasks.workunit.client.0.smithi116.stdout:5/497: creat dd/d1f/d5c/f9e x:0 0 0 2024-03-04T16:04:13.496 INFO:tasks.workunit.client.0.smithi116.stdout:5/498: chown dd/d1f 1471058379 1 2024-03-04T16:04:13.501 INFO:tasks.workunit.client.0.smithi116.stdout:8/509: fdatasync d2/d13/d3c/d46/f6d 0 2024-03-04T16:04:13.501 INFO:tasks.workunit.client.0.smithi116.stdout:5/499: dread dd/d1f/d55/d73/d2d/f50 [0,4194304] 0 2024-03-04T16:04:13.508 INFO:tasks.workunit.client.0.smithi116.stdout:2/540: getdents d7 0 2024-03-04T16:04:13.508 INFO:tasks.workunit.client.0.smithi116.stdout:7/479: mknod d3/d4/d8/d4a/d88/c99 0 2024-03-04T16:04:13.509 INFO:tasks.workunit.client.0.smithi116.stdout:2/541: dread - d7/d25/d41/f68 zero size 2024-03-04T16:04:13.509 INFO:tasks.workunit.client.0.smithi116.stdout:7/480: fdatasync d3/f4e 0 2024-03-04T16:04:13.511 INFO:tasks.workunit.client.0.smithi116.stdout:1/368: rename d5/d9/d21/d27/f3c to d5/d9/d46/f7a 0 2024-03-04T16:04:13.513 INFO:tasks.workunit.client.0.smithi116.stdout:4/525: truncate d1/d3/de/f30 540658 0 2024-03-04T16:04:13.515 INFO:tasks.workunit.client.0.smithi116.stdout:9/452: write d2/d19/d27/d3b/d4a/f73 [3011714,111962] 0 2024-03-04T16:04:13.516 INFO:tasks.workunit.client.0.smithi116.stdout:9/453: chown d2 5930517 1 2024-03-04T16:04:13.516 INFO:tasks.workunit.client.0.smithi116.stdout:9/454: dread - d2/d49/d3c/f76 zero size 2024-03-04T16:04:13.516 INFO:tasks.workunit.client.0.smithi116.stdout:8/510: creat d2/de/d3e/fa8 x:0 0 0 2024-03-04T16:04:13.525 INFO:tasks.workunit.client.0.smithi116.stdout:5/500: symlink dd/d1f/d5c/l9f 0 2024-03-04T16:04:13.525 INFO:tasks.workunit.client.0.smithi116.stdout:3/558: getdents dc/df/d47/d91 0 2024-03-04T16:04:13.525 INFO:tasks.workunit.client.0.smithi116.stdout:8/511: dwrite d2/d13/d3c/fa3 [0,4194304] 0 2024-03-04T16:04:13.525 INFO:tasks.workunit.client.0.smithi116.stdout:2/542: dread d7/d25/fbd [0,4194304] 0 2024-03-04T16:04:13.526 INFO:tasks.workunit.client.0.smithi116.stdout:7/481: mkdir d3/d4/d8/d52/d5e/d9a 0 2024-03-04T16:04:13.527 INFO:tasks.workunit.client.0.smithi116.stdout:2/543: write d7/d81/f7b [624837,13606] 0 2024-03-04T16:04:13.530 INFO:tasks.workunit.client.0.smithi116.stdout:4/526: dread d1/d3/f39 [0,4194304] 0 2024-03-04T16:04:13.530 INFO:tasks.workunit.client.0.smithi116.stdout:2/544: chown d7/d15/d35/c66 79 1 2024-03-04T16:04:13.537 INFO:tasks.workunit.client.0.smithi116.stdout:5/501: creat dd/d1f/d72/fa0 x:0 0 0 2024-03-04T16:04:13.537 INFO:tasks.workunit.client.0.smithi116.stdout:5/502: stat dd/d1f/d55/d73/d2d/d59/d71 0 2024-03-04T16:04:13.544 INFO:tasks.workunit.client.0.smithi116.stdout:1/369: mknod d5/d11/d61/c7b 0 2024-03-04T16:04:13.545 INFO:tasks.workunit.client.0.smithi116.stdout:3/559: dread dc/d23/d84/f5e [0,4194304] 0 2024-03-04T16:04:13.547 INFO:tasks.workunit.client.0.smithi116.stdout:8/512: dwrite d2/fd [0,4194304] 0 2024-03-04T16:04:13.547 INFO:tasks.workunit.client.0.smithi116.stdout:4/527: symlink d1/d3/dd/d45/d50/lb8 0 2024-03-04T16:04:13.548 INFO:tasks.workunit.client.0.smithi116.stdout:8/513: dread - d2/d38/d41/f9b zero size 2024-03-04T16:04:13.549 INFO:tasks.workunit.client.0.smithi116.stdout:2/545: creat d7/d25/d41/fcf x:0 0 0 2024-03-04T16:04:13.549 INFO:tasks.workunit.client.0.smithi116.stdout:4/528: chown d1/d3/d15/d20/d65/f99 7901 1 2024-03-04T16:04:13.549 INFO:tasks.workunit.client.0.smithi116.stdout:2/546: stat d7/d15/f3f 0 2024-03-04T16:04:13.551 INFO:tasks.workunit.client.0.smithi116.stdout:9/455: mknod d2/c8a 0 2024-03-04T16:04:13.552 INFO:tasks.workunit.client.0.smithi116.stdout:8/514: chown d2/de/d16/d78/d29/d7a/c98 125436 1 2024-03-04T16:04:13.565 INFO:tasks.workunit.client.0.smithi116.stdout:4/529: rmdir d1/d3/d15/d20/d1c/d62/d81/d97 39 2024-03-04T16:04:13.566 INFO:tasks.workunit.client.0.smithi116.stdout:9/456: dwrite d2/d19/d27/d3b/d4a/f7b [0,4194304] 0 2024-03-04T16:04:13.566 INFO:tasks.workunit.client.0.smithi116.stdout:3/560: dread dc/f5f [0,4194304] 0 2024-03-04T16:04:13.567 INFO:tasks.workunit.client.0.smithi116.stdout:7/482: dread - d3/d4/d8/d1a/d85/f89 zero size 2024-03-04T16:04:13.567 INFO:tasks.workunit.client.0.smithi116.stdout:8/515: dwrite d2/d38/d41/f9b [0,4194304] 0 2024-03-04T16:04:13.568 INFO:tasks.workunit.client.0.smithi116.stdout:5/503: rename dd/d1f/d55/d73/c9c to dd/d1f/d55/d73/d2d/d59/ca1 0 2024-03-04T16:04:13.576 INFO:tasks.workunit.client.0.smithi116.stdout:3/561: mkdir dc/d23/d84/d7f/d8b/dbe 0 2024-03-04T16:04:13.577 INFO:tasks.workunit.client.0.smithi116.stdout:7/483: write d3/d4/d8/d52/d5e/f6e [2210460,45497] 0 2024-03-04T16:04:13.577 INFO:tasks.workunit.client.0.smithi116.stdout:2/547: dread d7/d81/f7b [0,4194304] 0 2024-03-04T16:04:13.578 INFO:tasks.workunit.client.0.smithi116.stdout:1/370: creat d5/f7c x:0 0 0 2024-03-04T16:04:13.579 INFO:tasks.workunit.client.0.smithi116.stdout:8/516: symlink d2/d4a/la9 0 2024-03-04T16:04:13.580 INFO:tasks.workunit.client.0.smithi116.stdout:1/371: fdatasync d5/d9/f18 0 2024-03-04T16:04:13.583 INFO:tasks.workunit.client.0.smithi116.stdout:9/457: dwrite d2/f4f [0,4194304] 0 2024-03-04T16:04:13.584 INFO:tasks.workunit.client.0.smithi116.stdout:8/517: read d2/d13/d3c/fa3 [4069418,20350] 0 2024-03-04T16:04:13.585 INFO:tasks.workunit.client.0.smithi116.stdout:2/548: chown d7/d13/d3e/d98/dce/d47/d64/d75/d77/cc8 161953566 1 2024-03-04T16:04:13.586 INFO:tasks.workunit.client.0.smithi116.stdout:3/562: dread dc/d23/d84/f76 [0,4194304] 0 2024-03-04T16:04:13.587 INFO:tasks.workunit.client.0.smithi116.stdout:5/504: dwrite dd/d1f/d55/d73/f94 [0,4194304] 0 2024-03-04T16:04:13.588 INFO:tasks.workunit.client.0.smithi116.stdout:3/563: fdatasync dc/d34/f3e 0 2024-03-04T16:04:13.593 INFO:tasks.workunit.client.0.smithi116.stdout:4/530: creat d1/d3/d15/d20/d1c/d62/d81/d97/fb9 x:0 0 0 2024-03-04T16:04:13.594 INFO:tasks.workunit.client.0.smithi116.stdout:1/372: symlink d5/d9/d21/d27/d32/d49/d4c/l7d 0 2024-03-04T16:04:13.595 INFO:tasks.workunit.client.0.smithi116.stdout:8/518: symlink d2/d13/d3c/d77/laa 0 2024-03-04T16:04:13.597 INFO:tasks.workunit.client.0.smithi116.stdout:2/549: rmdir d7/d13/d3e/d98 39 2024-03-04T16:04:13.599 INFO:tasks.workunit.client.0.smithi116.stdout:2/550: write d7/d15/d5e/f67 [559981,84734] 0 2024-03-04T16:04:13.601 INFO:tasks.workunit.client.0.smithi116.stdout:1/373: dread d5/f48 [0,4194304] 0 2024-03-04T16:04:13.602 INFO:tasks.workunit.client.0.smithi116.stdout:4/531: dwrite d1/fae [0,4194304] 0 2024-03-04T16:04:13.602 INFO:tasks.workunit.client.0.smithi116.stdout:7/484: rename d3/d4/d8/d1a/f27 to d3/d4/d7e/f9b 0 2024-03-04T16:04:13.602 INFO:tasks.workunit.client.0.smithi116.stdout:2/551: chown d7/d15/l4f 307101484 1 2024-03-04T16:04:13.605 INFO:tasks.workunit.client.0.smithi116.stdout:7/485: fsync d3/d4/d8/f25 0 2024-03-04T16:04:13.605 INFO:tasks.workunit.client.0.smithi116.stdout:9/458: creat d2/d33/d40/f8b x:0 0 0 2024-03-04T16:04:13.606 INFO:tasks.workunit.client.0.smithi116.stdout:8/519: mkdir d2/d38/d41/dab 0 2024-03-04T16:04:13.613 INFO:tasks.workunit.client.0.smithi116.stdout:1/374: write d5/d9/d21/d27/d32/d49/d4a/f78 [622858,103387] 0 2024-03-04T16:04:13.615 INFO:tasks.workunit.client.0.smithi116.stdout:1/375: chown d5/d9/d21/d27/c4f 470 1 2024-03-04T16:04:13.615 INFO:tasks.workunit.client.0.smithi116.stdout:1/376: chown d5/d11/f69 10760 1 2024-03-04T16:04:13.620 INFO:tasks.workunit.client.0.smithi116.stdout:7/486: stat d3/d4/d34/c94 0 2024-03-04T16:04:13.622 INFO:tasks.workunit.client.0.smithi116.stdout:8/520: mknod d2/d13/d3c/d46/d4f/cac 0 2024-03-04T16:04:13.623 INFO:tasks.workunit.client.0.smithi116.stdout:2/552: creat d7/d81/dc2/fd0 x:0 0 0 2024-03-04T16:04:13.629 INFO:tasks.workunit.client.0.smithi116.stdout:1/377: rename d5/d11 to d5/d9/d21/d27/d32/d35/d55/d58/d7e 0 2024-03-04T16:04:13.631 INFO:tasks.workunit.client.0.smithi116.stdout:4/532: symlink d1/d3/d15/d20/d1c/d62/d81/da6/lba 0 2024-03-04T16:04:13.633 INFO:tasks.workunit.client.0.smithi116.stdout:4/533: write d1/d3/d15/d20/d1c/f6a [2151988,90159] 0 2024-03-04T16:04:13.633 INFO:tasks.workunit.client.0.smithi116.stdout:7/487: chown d3/d4/d8/d4a/d65/l91 10 1 2024-03-04T16:04:13.634 INFO:tasks.workunit.client.0.smithi116.stdout:9/459: symlink d2/d19/d27/d3b/d4a/l8c 0 2024-03-04T16:04:13.635 INFO:tasks.workunit.client.0.smithi116.stdout:9/460: read d2/d49/f21 [163376,81183] 0 2024-03-04T16:04:13.637 INFO:tasks.workunit.client.0.smithi116.stdout:3/564: getdents dc 0 2024-03-04T16:04:13.638 INFO:tasks.workunit.client.0.smithi116.stdout:9/461: write d2/d49/d3c/f76 [92845,112963] 0 2024-03-04T16:04:13.638 INFO:tasks.workunit.client.0.smithi116.stdout:8/521: creat d2/de/d16/d2b/d37/fad x:0 0 0 2024-03-04T16:04:13.639 INFO:tasks.workunit.client.0.smithi116.stdout:2/553: rmdir d7 39 2024-03-04T16:04:13.639 INFO:tasks.workunit.client.0.smithi116.stdout:5/505: link dd/d1f/d55/d73/l85 dd/d1f/d55/d73/d2d/d59/d71/la2 0 2024-03-04T16:04:13.640 INFO:tasks.workunit.client.0.smithi116.stdout:5/506: readlink dd/d1f/d5c/d5e/l76 0 2024-03-04T16:04:13.643 INFO:tasks.workunit.client.0.smithi116.stdout:4/534: mknod d1/d3/d40/d52/cbb 0 2024-03-04T16:04:13.643 INFO:tasks.workunit.client.0.smithi116.stdout:8/522: dwrite d2/de/d16/f18 [4194304,4194304] 0 2024-03-04T16:04:13.644 INFO:tasks.workunit.client.0.smithi116.stdout:4/535: fsync d1/d3/dd/d45/d50/f96 0 2024-03-04T16:04:13.644 INFO:tasks.workunit.client.0.smithi116.stdout:8/523: dread - d2/de/d16/d2b/d37/fad zero size 2024-03-04T16:04:13.651 INFO:tasks.workunit.client.0.smithi116.stdout:9/462: rmdir d2/d19/d27/d3b/d4a 39 2024-03-04T16:04:13.656 INFO:tasks.workunit.client.0.smithi116.stdout:1/378: truncate d5/d9/d21/d27/d32/d35/f76 3248724 0 2024-03-04T16:04:13.659 INFO:tasks.workunit.client.0.smithi116.stdout:4/536: creat d1/d3/d40/d52/fbc x:0 0 0 2024-03-04T16:04:13.662 INFO:tasks.workunit.client.0.smithi116.stdout:9/463: symlink d2/d33/d65/l8d 0 2024-03-04T16:04:13.662 INFO:tasks.workunit.client.0.smithi116.stdout:1/379: chown d5/d9/d21/d27/d32/d35/d55/d58/d7e/f69 430948 1 2024-03-04T16:04:13.662 INFO:tasks.workunit.client.0.smithi116.stdout:7/488: link d3/d4/d8/c6a d3/d4/d8/d1a/d85/c9c 0 2024-03-04T16:04:13.663 INFO:tasks.workunit.client.0.smithi116.stdout:1/380: chown d5/d9/d21 5630681 1 2024-03-04T16:04:13.663 INFO:tasks.workunit.client.0.smithi116.stdout:3/565: dwrite dc/df/d47/f61 [0,4194304] 0 2024-03-04T16:04:13.664 INFO:tasks.workunit.client.0.smithi116.stdout:6/436: sync 2024-03-04T16:04:13.665 INFO:tasks.workunit.client.0.smithi116.stdout:0/423: sync 2024-03-04T16:04:13.665 INFO:tasks.workunit.client.0.smithi116.stdout:1/381: write d5/f7c [60215,21939] 0 2024-03-04T16:04:13.667 INFO:tasks.workunit.client.0.smithi116.stdout:9/464: stat d2/d19/c2d 0 2024-03-04T16:04:13.669 INFO:tasks.workunit.client.0.smithi116.stdout:3/566: dread dc/d23/d84/d7c/fa1 [0,4194304] 0 2024-03-04T16:04:13.669 INFO:tasks.workunit.client.0.smithi116.stdout:5/507: dread fa [0,4194304] 0 2024-03-04T16:04:13.670 INFO:tasks.workunit.client.0.smithi116.stdout:5/508: chown dd/d1f/d5c/d5e/l63 3431877 1 2024-03-04T16:04:13.671 INFO:tasks.workunit.client.0.smithi116.stdout:8/524: link d2/de/d3e/f2a d2/d13/d3c/fae 0 2024-03-04T16:04:13.677 INFO:tasks.workunit.client.0.smithi116.stdout:4/537: rename d1/f41 to d1/d3/d15/d20/d1c/d62/d81/da6/fbd 0 2024-03-04T16:04:13.678 INFO:tasks.workunit.client.0.smithi116.stdout:4/538: fdatasync d1/d3/f61 0 2024-03-04T16:04:13.680 INFO:tasks.workunit.client.0.smithi116.stdout:6/437: rmdir d6/de/d15 39 2024-03-04T16:04:13.682 INFO:tasks.workunit.client.0.smithi116.stdout:2/554: fsync d7/d13/d3e/d98/dce/d47/d64/f6f 0 2024-03-04T16:04:13.683 INFO:tasks.workunit.client.0.smithi116.stdout:2/555: read d7/d13/d3e/d98/dce/d47/d64/f87 [3975931,13605] 0 2024-03-04T16:04:13.684 INFO:tasks.workunit.client.0.smithi116.stdout:0/424: creat da/d16/d20/d42/f88 x:0 0 0 2024-03-04T16:04:13.686 INFO:tasks.workunit.client.0.smithi116.stdout:9/465: mknod d2/d33/d65/c8e 0 2024-03-04T16:04:13.686 INFO:tasks.workunit.client.0.smithi116.stdout:7/489: dread d3/d4/f16 [0,4194304] 0 2024-03-04T16:04:13.688 INFO:tasks.workunit.client.0.smithi116.stdout:5/509: unlink dd/d1f/d5c/f87 0 2024-03-04T16:04:13.688 INFO:tasks.workunit.client.0.smithi116.stdout:7/490: dread - d3/d4/d8/d4a/d65/f7b zero size 2024-03-04T16:04:13.690 INFO:tasks.workunit.client.0.smithi116.stdout:8/525: unlink d2/de/d16/d78/d29/f39 0 2024-03-04T16:04:13.695 INFO:tasks.workunit.client.0.smithi116.stdout:1/382: rename d5/f7c to d5/d9/d46/f7f 0 2024-03-04T16:04:13.698 INFO:tasks.workunit.client.0.smithi116.stdout:2/556: dread d7/d13/d3e/d98/dce/d47/f51 [0,4194304] 0 2024-03-04T16:04:13.699 INFO:tasks.workunit.client.0.smithi116.stdout:4/539: unlink d1/d3/dd/d24/f8f 0 2024-03-04T16:04:13.701 INFO:tasks.workunit.client.0.smithi116.stdout:6/438: readlink d6/de/d15/l59 0 2024-03-04T16:04:13.708 INFO:tasks.workunit.client.0.smithi116.stdout:9/466: creat d2/d49/d3c/f8f x:0 0 0 2024-03-04T16:04:13.708 INFO:tasks.workunit.client.0.smithi116.stdout:5/510: write dd/d1f/f41 [5163286,67779] 0 2024-03-04T16:04:13.709 INFO:tasks.workunit.client.0.smithi116.stdout:7/491: rmdir d3/d4/d11 39 2024-03-04T16:04:13.710 INFO:tasks.workunit.client.0.smithi116.stdout:2/557: dread f4 [0,4194304] 0 2024-03-04T16:04:13.710 INFO:tasks.workunit.client.0.smithi116.stdout:5/511: write dd/d1f/f29 [840434,115954] 0 2024-03-04T16:04:13.711 INFO:tasks.workunit.client.0.smithi116.stdout:7/492: chown d3/d4/d34/d5d/f8b 8044 1 2024-03-04T16:04:13.711 INFO:tasks.workunit.client.0.smithi116.stdout:5/512: fdatasync dd/f38 0 2024-03-04T16:04:13.712 INFO:tasks.workunit.client.0.smithi116.stdout:8/526: chown d2/d13/d5e/l8d 2342339 1 2024-03-04T16:04:13.717 INFO:tasks.workunit.client.0.smithi116.stdout:5/513: dwrite dd/d1f/d5c/d5e/f67 [0,4194304] 0 2024-03-04T16:04:13.718 INFO:tasks.workunit.client.0.smithi116.stdout:3/567: rename dc/d23/d84/d7c/l68 to dc/d34/d44/dab/lbf 0 2024-03-04T16:04:13.722 INFO:tasks.workunit.client.0.smithi116.stdout:4/540: unlink d1/d3/d15/d20/d1c/d62/d81/da6/fa7 0 2024-03-04T16:04:13.729 INFO:tasks.workunit.client.0.smithi116.stdout:1/383: stat d5/f4d 0 2024-03-04T16:04:13.729 INFO:tasks.workunit.client.0.smithi116.stdout:0/425: symlink da/d16/d28/d40/d48/d77/l89 0 2024-03-04T16:04:13.730 INFO:tasks.workunit.client.0.smithi116.stdout:0/426: stat da/d16/d28/d40/d48/d77/l89 0 2024-03-04T16:04:13.730 INFO:tasks.workunit.client.0.smithi116.stdout:1/384: dread d5/d9/d21/d27/d32/d35/d55/d58/d7e/f69 [0,4194304] 0 2024-03-04T16:04:13.732 INFO:tasks.workunit.client.0.smithi116.stdout:9/467: chown d2/d19/d27/d3b/d4a/d5b/l34 803695 1 2024-03-04T16:04:13.734 INFO:tasks.workunit.client.0.smithi116.stdout:9/468: fdatasync d2/d19/d27/d3b/f3d 0 2024-03-04T16:04:13.736 INFO:tasks.workunit.client.0.smithi116.stdout:9/469: chown d2/d33/d65/d6c 3 1 2024-03-04T16:04:13.737 INFO:tasks.workunit.client.0.smithi116.stdout:5/514: symlink dd/d1f/d55/d73/d2d/d31/la3 0 2024-03-04T16:04:13.738 INFO:tasks.workunit.client.0.smithi116.stdout:6/439: dread d6/de/d1e/f20 [4194304,4194304] 0 2024-03-04T16:04:13.739 INFO:tasks.workunit.client.0.smithi116.stdout:1/385: dwrite d5/d9/d21/d27/d32/d49/f6d [0,4194304] 0 2024-03-04T16:04:13.739 INFO:tasks.workunit.client.0.smithi116.stdout:4/541: mknod d1/d6e/cbe 0 2024-03-04T16:04:13.745 INFO:tasks.workunit.client.0.smithi116.stdout:2/558: symlink d7/d13/d3e/d98/dce/ld1 0 2024-03-04T16:04:13.758 INFO:tasks.workunit.client.0.smithi116.stdout:9/470: dread d2/d19/d27/d3b/d4a/d5b/f25 [0,4194304] 0 2024-03-04T16:04:13.759 INFO:tasks.workunit.client.0.smithi116.stdout:0/427: mkdir da/d16/d28/d40/d48/d57/d8a 0 2024-03-04T16:04:13.760 INFO:tasks.workunit.client.0.smithi116.stdout:6/440: mkdir d6/de/d25/d4a/d63/d95 0 2024-03-04T16:04:13.761 INFO:tasks.workunit.client.0.smithi116.stdout:9/471: dread d2/d19/d27/d3b/d4a/d5b/f25 [0,4194304] 0 2024-03-04T16:04:13.766 INFO:tasks.workunit.client.0.smithi116.stdout:3/568: truncate dc/f5f 3729500 0 2024-03-04T16:04:13.767 INFO:tasks.workunit.client.0.smithi116.stdout:5/515: write dd/d1f/d55/f58 [685180,32288] 0 2024-03-04T16:04:13.770 INFO:tasks.workunit.client.0.smithi116.stdout:6/441: unlink d6/de/d1e/f20 0 2024-03-04T16:04:13.774 INFO:tasks.workunit.client.0.smithi116.stdout:8/527: dwrite d2/de/fa [0,4194304] 0 2024-03-04T16:04:13.777 INFO:tasks.workunit.client.0.smithi116.stdout:2/559: dwrite d7/d13/d3e/d78/f84 [0,4194304] 0 2024-03-04T16:04:13.777 INFO:tasks.workunit.client.0.smithi116.stdout:3/569: dread dc/df/d14/f1a [0,4194304] 0 2024-03-04T16:04:13.777 INFO:tasks.workunit.client.0.smithi116.stdout:7/493: getdents d3/d4/d8/d4a/d87 0 2024-03-04T16:04:13.777 INFO:tasks.workunit.client.0.smithi116.stdout:8/528: fdatasync d2/d38/d41/f9b 0 2024-03-04T16:04:13.778 INFO:tasks.workunit.client.0.smithi116.stdout:9/472: mkdir d2/d33/d90 0 2024-03-04T16:04:13.782 INFO:tasks.workunit.client.0.smithi116.stdout:7/494: readlink d3/d4/d8/d4a/l7a 0 2024-03-04T16:04:13.782 INFO:tasks.workunit.client.0.smithi116.stdout:1/386: rename d5/d9/d21/d24/l64 to d5/d9/d21/d27/d32/l80 0 2024-03-04T16:04:13.784 INFO:tasks.workunit.client.0.smithi116.stdout:3/570: read dc/d23/d84/d7c/f80 [1359641,101057] 0 2024-03-04T16:04:13.786 INFO:tasks.workunit.client.0.smithi116.stdout:6/442: dread d6/de/d25/f40 [0,4194304] 0 2024-03-04T16:04:13.787 INFO:tasks.workunit.client.0.smithi116.stdout:5/516: dwrite dd/d1f/d72/d8e/f98 [0,4194304] 0 2024-03-04T16:04:13.787 INFO:tasks.workunit.client.0.smithi116.stdout:2/560: fdatasync d7/f4e 0 2024-03-04T16:04:13.789 INFO:tasks.workunit.client.0.smithi116.stdout:0/428: rename da/d16/d28/l38 to da/d16/d28/l8b 0 2024-03-04T16:04:13.792 INFO:tasks.workunit.client.0.smithi116.stdout:1/387: stat d5/d9/d21/d27/d32/d49/d4c/l7d 0 2024-03-04T16:04:13.792 INFO:tasks.workunit.client.0.smithi116.stdout:2/561: write d7/d2f/d76/d7c/f7d [4828416,12511] 0 2024-03-04T16:04:13.794 INFO:tasks.workunit.client.0.smithi116.stdout:5/517: write dd/d6d/f80 [291575,47726] 0 2024-03-04T16:04:13.795 INFO:tasks.workunit.client.0.smithi116.stdout:4/542: write d1/d3/dd/d45/f90 [1035747,18484] 0 2024-03-04T16:04:13.797 INFO:tasks.workunit.client.0.smithi116.stdout:8/529: unlink d2/d13/d3c/d77/d97/c61 0 2024-03-04T16:04:13.798 INFO:tasks.workunit.client.0.smithi116.stdout:7/495: chown d3/d4/d11/d61/f81 10722 1 2024-03-04T16:04:13.799 INFO:tasks.workunit.client.0.smithi116.stdout:3/571: fdatasync dc/df/f79 0 2024-03-04T16:04:13.801 INFO:tasks.workunit.client.0.smithi116.stdout:9/473: dwrite d2/d33/d40/f8b [0,4194304] 0 2024-03-04T16:04:13.805 INFO:tasks.workunit.client.0.smithi116.stdout:3/572: fsync dc/d23/f85 0 2024-03-04T16:04:13.809 INFO:tasks.workunit.client.0.smithi116.stdout:1/388: dwrite d5/d9/d21/d27/d32/d49/d4a/f78 [0,4194304] 0 2024-03-04T16:04:13.810 INFO:tasks.workunit.client.0.smithi116.stdout:2/562: mknod d7/d15/d35/cd2 0 2024-03-04T16:04:13.813 INFO:tasks.workunit.client.0.smithi116.stdout:1/389: rename d5/d9 to d5/d9/d21/d27/d81 22 2024-03-04T16:04:13.813 INFO:tasks.workunit.client.0.smithi116.stdout:2/563: read - d7/d2f/d6c/da4/faa zero size 2024-03-04T16:04:13.816 INFO:tasks.workunit.client.0.smithi116.stdout:2/564: dread - d7/d15/d35/f48 zero size 2024-03-04T16:04:13.817 INFO:tasks.workunit.client.0.smithi116.stdout:0/429: dwrite da/f1a [0,4194304] 0 2024-03-04T16:04:13.818 INFO:tasks.workunit.client.0.smithi116.stdout:8/530: write d2/de/d16/d2b/fa6 [46767,42593] 0 2024-03-04T16:04:13.828 INFO:tasks.workunit.client.0.smithi116.stdout:3/573: creat dc/df/d14/fc0 x:0 0 0 2024-03-04T16:04:13.833 INFO:tasks.workunit.client.0.smithi116.stdout:2/565: mkdir d7/dd3 0 2024-03-04T16:04:13.833 INFO:tasks.workunit.client.0.smithi116.stdout:1/390: dread d5/d9/d21/f2f [0,4194304] 0 2024-03-04T16:04:13.835 INFO:tasks.workunit.client.0.smithi116.stdout:0/430: rmdir da/d16/d20/d42/d4e 39 2024-03-04T16:04:13.836 INFO:tasks.workunit.client.0.smithi116.stdout:7/496: write d3/f6 [1806743,44845] 0 2024-03-04T16:04:13.837 INFO:tasks.workunit.client.0.smithi116.stdout:9/474: truncate d2/d33/f54 1630310 0 2024-03-04T16:04:13.842 INFO:tasks.workunit.client.0.smithi116.stdout:3/574: chown dc/df/c3f 378800629 1 2024-03-04T16:04:13.846 INFO:tasks.workunit.client.0.smithi116.stdout:0/431: unlink da/d16/d20/d42/l61 0 2024-03-04T16:04:13.847 INFO:tasks.workunit.client.0.smithi116.stdout:7/497: unlink d3/d4/d8/d4a/d87/f8a 0 2024-03-04T16:04:13.849 INFO:tasks.workunit.client.0.smithi116.stdout:0/432: write f7 [1356966,83012] 0 2024-03-04T16:04:13.852 INFO:tasks.workunit.client.0.smithi116.stdout:8/531: link d2/fd d2/d13/d3c/d46/d6e/faf 0 2024-03-04T16:04:13.856 INFO:tasks.workunit.client.0.smithi116.stdout:2/566: dwrite f4 [0,4194304] 0 2024-03-04T16:04:13.857 INFO:tasks.workunit.client.0.smithi116.stdout:5/518: dwrite dd/d45/f99 [0,4194304] 0 2024-03-04T16:04:13.860 INFO:tasks.workunit.client.0.smithi116.stdout:1/391: dwrite d5/d9/d46/f7a [0,4194304] 0 2024-03-04T16:04:13.860 INFO:tasks.workunit.client.0.smithi116.stdout:4/543: truncate d1/d3/de/f10 7805284 0 2024-03-04T16:04:13.864 INFO:tasks.workunit.client.0.smithi116.stdout:6/443: getdents d6/d56/d58 0 2024-03-04T16:04:13.865 INFO:tasks.workunit.client.0.smithi116.stdout:9/475: link d2/d49/f89 d2/d19/d5f/f91 0 2024-03-04T16:04:13.866 INFO:tasks.workunit.client.0.smithi116.stdout:9/476: readlink d2/d26/d28/l4b 0 2024-03-04T16:04:13.869 INFO:tasks.workunit.client.0.smithi116.stdout:0/433: creat da/d16/d28/d40/d48/d57/f8c x:0 0 0 2024-03-04T16:04:13.869 INFO:tasks.workunit.client.0.smithi116.stdout:0/434: stat da/d16 0 2024-03-04T16:04:13.871 INFO:tasks.workunit.client.0.smithi116.stdout:4/544: mkdir d1/d6c/dbf 0 2024-03-04T16:04:13.877 INFO:tasks.workunit.client.0.smithi116.stdout:8/532: rename d2/d13/d3c/d77/d97/f94 to d2/d13/d3c/d46/fb0 0 2024-03-04T16:04:13.877 INFO:tasks.workunit.client.0.smithi116.stdout:7/498: write d3/d4/d8/f40 [958200,89391] 0 2024-03-04T16:04:13.877 INFO:tasks.workunit.client.0.smithi116.stdout:3/575: mkdir dc/d34/dc1 0 2024-03-04T16:04:13.878 INFO:tasks.workunit.client.0.smithi116.stdout:8/533: read - d2/d13/f90 zero size 2024-03-04T16:04:13.882 INFO:tasks.workunit.client.0.smithi116.stdout:1/392: creat d5/d9/d21/d27/d32/d35/d55/f82 x:0 0 0 2024-03-04T16:04:13.884 INFO:tasks.workunit.client.0.smithi116.stdout:1/393: read - d5/d9/d21/d27/d32/d35/f74 zero size 2024-03-04T16:04:13.886 INFO:tasks.workunit.client.0.smithi116.stdout:0/435: unlink da/d16/d19/d27/f75 0 2024-03-04T16:04:13.888 INFO:tasks.workunit.client.0.smithi116.stdout:0/436: truncate da/d16/d19/d27/f78 1312058 0 2024-03-04T16:04:13.892 INFO:tasks.workunit.client.0.smithi116.stdout:8/534: readlink d2/de/d16/d78/l54 0 2024-03-04T16:04:13.895 INFO:tasks.workunit.client.0.smithi116.stdout:3/576: rename dc/d23/d84/d7f/d8b/dbe to dc/d23/d84/d7f/d93/db6/dc2 0 2024-03-04T16:04:13.895 INFO:tasks.workunit.client.0.smithi116.stdout:3/577: chown dc/d23/caa 6 1 2024-03-04T16:04:13.898 INFO:tasks.workunit.client.0.smithi116.stdout:9/477: write d2/d19/d5f/f61 [1185960,69491] 0 2024-03-04T16:04:13.899 INFO:tasks.workunit.client.0.smithi116.stdout:2/567: chown d7/d13/d3e/d98/dce/l32 0 1 2024-03-04T16:04:13.900 INFO:tasks.workunit.client.0.smithi116.stdout:2/568: chown d7/d13/d3e/d98/dce/d47/d64/f6f 472579442 1 2024-03-04T16:04:13.901 INFO:tasks.workunit.client.0.smithi116.stdout:4/545: creat d1/d3/dd/d24/db4/fc0 x:0 0 0 2024-03-04T16:04:13.902 INFO:tasks.workunit.client.0.smithi116.stdout:0/437: creat da/d16/d28/d40/d48/d57/f8d x:0 0 0 2024-03-04T16:04:13.903 INFO:tasks.workunit.client.0.smithi116.stdout:4/546: chown d1/d3/d40/d52/d84/d88 0 1 2024-03-04T16:04:13.906 INFO:tasks.workunit.client.0.smithi116.stdout:6/444: creat d6/de/d25/f96 x:0 0 0 2024-03-04T16:04:13.906 INFO:tasks.workunit.client.0.smithi116.stdout:8/535: mkdir d2/de/d16/d78/d29/d7a/db1 0 2024-03-04T16:04:13.908 INFO:tasks.workunit.client.0.smithi116.stdout:7/499: rename d3/d4/d8/d4a/d65/f7b to d3/d4/d8/d4a/d87/f9d 0 2024-03-04T16:04:13.908 INFO:tasks.workunit.client.0.smithi116.stdout:8/536: fsync d2/d38/d41/f6c 0 2024-03-04T16:04:13.913 INFO:tasks.workunit.client.0.smithi116.stdout:9/478: write d2/d33/f3a [4155394,59773] 0 2024-03-04T16:04:13.913 INFO:tasks.workunit.client.0.smithi116.stdout:5/519: link dd/d1f/d55/d73/d2d/d59/c66 dd/ca4 0 2024-03-04T16:04:13.913 INFO:tasks.workunit.client.0.smithi116.stdout:0/438: write da/fe [2725444,49125] 0 2024-03-04T16:04:13.914 INFO:tasks.workunit.client.0.smithi116.stdout:9/479: stat d2/l14 0 2024-03-04T16:04:13.914 INFO:tasks.workunit.client.0.smithi116.stdout:3/578: dread dc/d34/d44/d6a/f77 [4194304,4194304] 0 2024-03-04T16:04:13.916 INFO:tasks.workunit.client.0.smithi116.stdout:6/445: symlink d6/de/d25/d36/d4e/d50/l97 0 2024-03-04T16:04:13.917 INFO:tasks.workunit.client.0.smithi116.stdout:1/394: creat d5/d9/d21/d27/d32/f83 x:0 0 0 2024-03-04T16:04:13.918 INFO:tasks.workunit.client.0.smithi116.stdout:0/439: write da/f35 [333078,39745] 0 2024-03-04T16:04:13.918 INFO:tasks.workunit.client.0.smithi116.stdout:2/569: mkdir d7/d15/d5e/d9d/dd4 0 2024-03-04T16:04:13.919 INFO:tasks.workunit.client.0.smithi116.stdout:9/480: readlink d2/d33/d65/l8d 0 2024-03-04T16:04:13.919 INFO:tasks.workunit.client.0.smithi116.stdout:8/537: creat d2/d13/d3c/d77/fb2 x:0 0 0 2024-03-04T16:04:13.922 INFO:tasks.workunit.client.0.smithi116.stdout:6/446: chown d6/de/d15/l49 2 1 2024-03-04T16:04:13.922 INFO:tasks.workunit.client.0.smithi116.stdout:5/520: truncate dd/d1f/d55/d73/d2d/d7c/f9d 1976 0 2024-03-04T16:04:13.922 INFO:tasks.workunit.client.0.smithi116.stdout:7/500: truncate d3/d4/d8/d1a/f42 1455533 0 2024-03-04T16:04:13.926 INFO:tasks.workunit.client.0.smithi116.stdout:1/395: mkdir d5/d9/d21/d84 0 2024-03-04T16:04:13.927 INFO:tasks.workunit.client.0.smithi116.stdout:8/538: fsync d2/d13/d3c/d46/d4f/f5b 0 2024-03-04T16:04:13.931 INFO:tasks.workunit.client.0.smithi116.stdout:1/396: chown d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/l56 44 1 2024-03-04T16:04:13.936 INFO:tasks.workunit.client.0.smithi116.stdout:4/547: dread d1/f5 [0,4194304] 0 2024-03-04T16:04:13.936 INFO:tasks.workunit.client.0.smithi116.stdout:5/521: truncate dd/f15 2770656 0 2024-03-04T16:04:13.937 INFO:tasks.workunit.client.0.smithi116.stdout:0/440: dread f7 [0,4194304] 0 2024-03-04T16:04:13.937 INFO:tasks.workunit.client.0.smithi116.stdout:5/522: chown dd/d1f/d72 249 1 2024-03-04T16:04:13.939 INFO:tasks.workunit.client.0.smithi116.stdout:3/579: link dc/df/f7e dc/d34/d44/fc3 0 2024-03-04T16:04:13.939 INFO:tasks.workunit.client.0.smithi116.stdout:8/539: dwrite d2/d38/f64 [0,4194304] 0 2024-03-04T16:04:13.941 INFO:tasks.workunit.client.0.smithi116.stdout:3/580: fsync dc/d23/d84/f3b 0 2024-03-04T16:04:13.944 INFO:tasks.workunit.client.0.smithi116.stdout:7/501: dread d3/f1f [0,4194304] 0 2024-03-04T16:04:13.945 INFO:tasks.workunit.client.0.smithi116.stdout:0/441: dread da/d16/d20/d42/f86 [0,4194304] 0 2024-03-04T16:04:13.946 INFO:tasks.workunit.client.0.smithi116.stdout:0/442: chown da/d16/d19/d27/d45/l76 5 1 2024-03-04T16:04:13.947 INFO:tasks.workunit.client.0.smithi116.stdout:8/540: read d2/d13/d3c/f80 [12265,123948] 0 2024-03-04T16:04:13.951 INFO:tasks.workunit.client.0.smithi116.stdout:7/502: dwrite f2 [0,4194304] 0 2024-03-04T16:04:13.953 INFO:tasks.workunit.client.0.smithi116.stdout:4/548: mkdir d1/d3/d40/d52/d84/dc1 0 2024-03-04T16:04:13.956 INFO:tasks.workunit.client.0.smithi116.stdout:8/541: dwrite d2/d13/f90 [0,4194304] 0 2024-03-04T16:04:13.959 INFO:tasks.workunit.client.0.smithi116.stdout:5/523: symlink dd/d1f/d5c/la5 0 2024-03-04T16:04:13.964 INFO:tasks.workunit.client.0.smithi116.stdout:2/570: write d7/d13/d3e/d98/dce/d47/f5b [1632394,64251] 0 2024-03-04T16:04:13.971 INFO:tasks.workunit.client.0.smithi116.stdout:9/481: getdents d2/d33/d40 0 2024-03-04T16:04:13.974 INFO:tasks.workunit.client.0.smithi116.stdout:9/482: chown d2/d19/d5f/l83 14399211 1 2024-03-04T16:04:13.974 INFO:tasks.workunit.client.0.smithi116.stdout:2/571: dwrite d7/d15/d35/f8f [0,4194304] 0 2024-03-04T16:04:13.978 INFO:tasks.workunit.client.0.smithi116.stdout:0/443: creat da/d16/d19/d27/d70/f8e x:0 0 0 2024-03-04T16:04:13.979 INFO:tasks.workunit.client.0.smithi116.stdout:0/444: dread - da/d16/d28/d40/d5d/d7e/f83 zero size 2024-03-04T16:04:13.982 INFO:tasks.workunit.client.0.smithi116.stdout:4/549: mkdir d1/d6e/d74/dc2 0 2024-03-04T16:04:13.983 INFO:tasks.workunit.client.0.smithi116.stdout:6/447: truncate d6/de/d25/d36/f1a 659641 0 2024-03-04T16:04:13.984 INFO:tasks.workunit.client.0.smithi116.stdout:6/448: fdatasync d6/de/d29/f8d 0 2024-03-04T16:04:13.990 INFO:tasks.workunit.client.0.smithi116.stdout:1/397: creat d5/d9/d21/f85 x:0 0 0 2024-03-04T16:04:13.990 INFO:tasks.workunit.client.0.smithi116.stdout:1/398: stat d5/f48 0 2024-03-04T16:04:13.990 INFO:tasks.workunit.client.0.smithi116.stdout:1/399: chown d5/d9/d21/l5d 491 1 2024-03-04T16:04:13.991 INFO:tasks.workunit.client.0.smithi116.stdout:1/400: write d5/d9/d21/d27/d32/d35/f74 [344945,22082] 0 2024-03-04T16:04:13.995 INFO:tasks.workunit.client.0.smithi116.stdout:3/581: dwrite dc/df/f7e [0,4194304] 0 2024-03-04T16:04:13.995 INFO:tasks.workunit.client.0.smithi116.stdout:9/483: dread d2/d19/f37 [0,4194304] 0 2024-03-04T16:04:13.998 INFO:tasks.workunit.client.0.smithi116.stdout:2/572: mknod d7/d15/d5e/d9d/cd5 0 2024-03-04T16:04:14.000 INFO:tasks.workunit.client.0.smithi116.stdout:3/582: dread dc/df/f7e [0,4194304] 0 2024-03-04T16:04:14.000 INFO:tasks.workunit.client.0.smithi116.stdout:7/503: mkdir d3/d4/d9e 0 2024-03-04T16:04:14.004 INFO:tasks.workunit.client.0.smithi116.stdout:0/445: rmdir da/d16/d19/d27/d70 39 2024-03-04T16:04:14.005 INFO:tasks.workunit.client.0.smithi116.stdout:4/550: mknod d1/d3/d15/d20/d1c/d6f/cc3 0 2024-03-04T16:04:14.011 INFO:tasks.workunit.client.0.smithi116.stdout:8/542: truncate d2/de/d16/f22 25912 0 2024-03-04T16:04:14.011 INFO:tasks.workunit.client.0.smithi116.stdout:8/543: stat d2/de/d16/d2b 0 2024-03-04T16:04:14.013 INFO:tasks.workunit.client.0.smithi116.stdout:5/524: creat dd/d1f/d55/d73/d2d/d8d/fa6 x:0 0 0 2024-03-04T16:04:14.015 INFO:tasks.workunit.client.0.smithi116.stdout:1/401: rmdir d5/d9/d21/d27/d32/d35/d55 39 2024-03-04T16:04:14.015 INFO:tasks.workunit.client.0.smithi116.stdout:8/544: read d2/de/f8 [4704309,29263] 0 2024-03-04T16:04:14.017 INFO:tasks.workunit.client.0.smithi116.stdout:1/402: chown d5/d9/d21/d27/l59 6692 1 2024-03-04T16:04:14.017 INFO:tasks.workunit.client.0.smithi116.stdout:8/545: stat d2/d13/d5e/f6b 0 2024-03-04T16:04:14.019 INFO:tasks.workunit.client.0.smithi116.stdout:9/484: creat d2/d33/d65/f92 x:0 0 0 2024-03-04T16:04:14.020 INFO:tasks.workunit.client.0.smithi116.stdout:5/525: dwrite dd/d1f/d55/d73/d2d/d7c/f86 [0,4194304] 0 2024-03-04T16:04:14.023 INFO:tasks.workunit.client.0.smithi116.stdout:8/546: dread d2/d13/d3c/d46/d6e/faf [0,4194304] 0 2024-03-04T16:04:14.023 INFO:tasks.workunit.client.0.smithi116.stdout:9/485: read d2/d19/d27/d3b/f68 [769855,97324] 0 2024-03-04T16:04:14.024 INFO:tasks.workunit.client.0.smithi116.stdout:7/504: rmdir d3/d4/d8/d52 39 2024-03-04T16:04:14.032 INFO:tasks.workunit.client.0.smithi116.stdout:4/551: rename d1/d3/de/f1b to d1/d3/d40/d52/d84/d88/fc4 0 2024-03-04T16:04:14.032 INFO:tasks.workunit.client.0.smithi116.stdout:7/505: rename d3/d4/d8/d4a to d3/d4/d8/d4a/d65/d9f 22 2024-03-04T16:04:14.038 INFO:tasks.workunit.client.0.smithi116.stdout:4/552: dwrite d1/d6e/d74/f7b [0,4194304] 0 2024-03-04T16:04:14.044 INFO:tasks.workunit.client.0.smithi116.stdout:5/526: fsync dd/d1f/d55/f77 0 2024-03-04T16:04:14.050 INFO:tasks.workunit.client.0.smithi116.stdout:8/547: unlink d2/d38/d41/f9b 0 2024-03-04T16:04:14.053 INFO:tasks.workunit.client.0.smithi116.stdout:0/446: creat da/d16/d20/d42/d4e/f8f x:0 0 0 2024-03-04T16:04:14.055 INFO:tasks.workunit.client.0.smithi116.stdout:7/506: mknod d3/d4/d34/ca0 0 2024-03-04T16:04:14.059 INFO:tasks.workunit.client.0.smithi116.stdout:6/449: dwrite d6/de/d1e/f43 [0,4194304] 0 2024-03-04T16:04:14.061 INFO:tasks.workunit.client.0.smithi116.stdout:1/403: mkdir d5/d86 0 2024-03-04T16:04:14.062 INFO:tasks.workunit.client.0.smithi116.stdout:4/553: mkdir d1/d3/dc5 0 2024-03-04T16:04:14.064 INFO:tasks.workunit.client.0.smithi116.stdout:4/554: write d1/d3/d40/d52/d84/d88/d8d/fb3 [807442,19597] 0 2024-03-04T16:04:14.077 INFO:tasks.workunit.client.0.smithi116.stdout:7/507: truncate d3/d4/d8/d1a/f5c 1033295 0 2024-03-04T16:04:14.078 INFO:tasks.workunit.client.0.smithi116.stdout:7/508: dread - d3/d4/d8/d1a/d85/f89 zero size 2024-03-04T16:04:14.079 INFO:tasks.workunit.client.0.smithi116.stdout:7/509: truncate d3/d4/d8/f40 1170234 0 2024-03-04T16:04:14.079 INFO:tasks.workunit.client.0.smithi116.stdout:7/510: stat d3/d95/f8e 0 2024-03-04T16:04:14.080 INFO:tasks.workunit.client.0.smithi116.stdout:7/511: dread - d3/d4/d8/d4a/d86/f8d zero size 2024-03-04T16:04:14.081 INFO:tasks.workunit.client.0.smithi116.stdout:6/450: write d6/de/d5b/f8e [4257892,91870] 0 2024-03-04T16:04:14.083 INFO:tasks.workunit.client.0.smithi116.stdout:1/404: truncate d5/d9/d21/d27/d32/d35/f76 1720513 0 2024-03-04T16:04:14.085 INFO:tasks.workunit.client.0.smithi116.stdout:6/451: dread d6/d56/f8b [0,4194304] 0 2024-03-04T16:04:14.085 INFO:tasks.workunit.client.0.smithi116.stdout:9/486: truncate d2/d19/d27/d3b/f68 167635 0 2024-03-04T16:04:14.086 INFO:tasks.workunit.client.0.smithi116.stdout:9/487: write d2/d49/d3c/f79 [4480706,111076] 0 2024-03-04T16:04:14.087 INFO:tasks.workunit.client.0.smithi116.stdout:4/555: rmdir d1/d6e/d74 39 2024-03-04T16:04:14.089 INFO:tasks.workunit.client.0.smithi116.stdout:9/488: truncate d2/d19/d27/d3b/f3d 4524224 0 2024-03-04T16:04:14.090 INFO:tasks.workunit.client.0.smithi116.stdout:6/452: dread d6/de/d1e/f8a [0,4194304] 0 2024-03-04T16:04:14.090 INFO:tasks.workunit.client.0.smithi116.stdout:3/583: link dc/l52 dc/d23/lc4 0 2024-03-04T16:04:14.093 INFO:tasks.workunit.client.0.smithi116.stdout:8/548: link d2/de/d3e/fa8 d2/d13/d3c/d77/fb3 0 2024-03-04T16:04:14.095 INFO:tasks.workunit.client.0.smithi116.stdout:9/489: dwrite d2/d33/f88 [0,4194304] 0 2024-03-04T16:04:14.099 INFO:tasks.workunit.client.0.smithi116.stdout:7/512: rmdir d3/d4/d8 39 2024-03-04T16:04:14.102 INFO:tasks.workunit.client.0.smithi116.stdout:6/453: dread d6/de/d1e/f35 [0,4194304] 0 2024-03-04T16:04:14.111 INFO:tasks.workunit.client.0.smithi116.stdout:4/556: mkdir d1/d3/d15/d20/d1c/d62/d81/d97/dc6 0 2024-03-04T16:04:14.113 INFO:tasks.workunit.client.0.smithi116.stdout:5/527: rename dd/d1f/d55/d73/d2d/c6f to dd/ca7 0 2024-03-04T16:04:14.115 INFO:tasks.workunit.client.0.smithi116.stdout:4/557: read d1/d3/d15/d20/d1c/d62/d81/da6/fbd [3061524,92082] 0 2024-03-04T16:04:14.116 INFO:tasks.workunit.client.0.smithi116.stdout:3/584: mkdir dc/d23/d84/d7f/d93/db6/dc5 0 2024-03-04T16:04:14.119 INFO:tasks.workunit.client.0.smithi116.stdout:0/447: link da/d16/d19/d27/c85 da/d16/d28/d39/c90 0 2024-03-04T16:04:14.120 INFO:tasks.workunit.client.0.smithi116.stdout:9/490: rmdir d2/d33/d65/d6c 39 2024-03-04T16:04:14.122 INFO:tasks.workunit.client.0.smithi116.stdout:7/513: write d3/f1f [4020652,67537] 0 2024-03-04T16:04:14.123 INFO:tasks.workunit.client.0.smithi116.stdout:9/491: dread d2/d49/f21 [0,4194304] 0 2024-03-04T16:04:14.128 INFO:tasks.workunit.client.0.smithi116.stdout:5/528: symlink dd/d1f/d55/d73/la8 0 2024-03-04T16:04:14.132 INFO:tasks.workunit.client.0.smithi116.stdout:4/558: unlink d1/d3/f7d 0 2024-03-04T16:04:14.132 INFO:tasks.workunit.client.0.smithi116.stdout:0/448: mknod da/d16/d20/d42/d4e/c91 0 2024-03-04T16:04:14.133 INFO:tasks.workunit.client.0.smithi116.stdout:0/449: dread - da/d16/d28/d40/d48/d57/f7f zero size 2024-03-04T16:04:14.136 INFO:tasks.workunit.client.0.smithi116.stdout:1/405: creat d5/d9/d21/d27/d32/d35/f87 x:0 0 0 2024-03-04T16:04:14.138 INFO:tasks.workunit.client.0.smithi116.stdout:4/559: truncate d1/d6c/f42 2442560 0 2024-03-04T16:04:14.139 INFO:tasks.workunit.client.0.smithi116.stdout:3/585: truncate dc/f5f 628537 0 2024-03-04T16:04:14.141 INFO:tasks.workunit.client.0.smithi116.stdout:1/406: dwrite d5/d9/d21/d27/d32/d49/f6d [0,4194304] 0 2024-03-04T16:04:14.142 INFO:tasks.workunit.client.0.smithi116.stdout:1/407: stat d5/d9/d21/d27/d32/d35 0 2024-03-04T16:04:14.142 INFO:tasks.workunit.client.0.smithi116.stdout:7/514: symlink d3/d4/d8/d4a/d87/la1 0 2024-03-04T16:04:14.149 INFO:tasks.workunit.client.0.smithi116.stdout:5/529: mknod dd/d1f/ca9 0 2024-03-04T16:04:14.152 INFO:tasks.workunit.client.0.smithi116.stdout:4/560: creat d1/d3/d15/d20/d65/fc7 x:0 0 0 2024-03-04T16:04:14.158 INFO:tasks.workunit.client.0.smithi116.stdout:3/586: dread dc/d23/d84/d7c/f62 [0,4194304] 0 2024-03-04T16:04:14.160 INFO:tasks.workunit.client.0.smithi116.stdout:0/450: mkdir da/d16/d19/d27/d70/d92 0 2024-03-04T16:04:14.162 INFO:tasks.workunit.client.0.smithi116.stdout:6/454: getdents d6/de 0 2024-03-04T16:04:14.163 INFO:tasks.workunit.client.0.smithi116.stdout:4/561: dread d1/f80 [0,4194304] 0 2024-03-04T16:04:14.172 INFO:tasks.workunit.client.0.smithi116.stdout:1/408: creat d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/f88 x:0 0 0 2024-03-04T16:04:14.176 INFO:tasks.workunit.client.0.smithi116.stdout:9/492: getdents d2/d19/d5f 0 2024-03-04T16:04:14.176 INFO:tasks.workunit.client.0.smithi116.stdout:9/493: readlink d2/d19/d27/d3b/l41 0 2024-03-04T16:04:14.177 INFO:tasks.workunit.client.0.smithi116.stdout:6/455: creat d6/d56/d58/f98 x:0 0 0 2024-03-04T16:04:14.178 INFO:tasks.workunit.client.0.smithi116.stdout:6/456: chown d6/de/d25/d4a/f71 49108156 1 2024-03-04T16:04:14.182 INFO:tasks.workunit.client.0.smithi116.stdout:4/562: unlink d1/d3/d40/d52/d84/d88/la9 0 2024-03-04T16:04:14.191 INFO:tasks.workunit.client.0.smithi116.stdout:3/587: truncate dc/d23/d84/d7f/f8a 724598 0 2024-03-04T16:04:14.192 INFO:tasks.workunit.client.0.smithi116.stdout:9/494: creat d2/d49/d3c/f93 x:0 0 0 2024-03-04T16:04:14.193 INFO:tasks.workunit.client.0.smithi116.stdout:9/495: fsync d2/d49/f42 0 2024-03-04T16:04:14.195 INFO:tasks.workunit.client.0.smithi116.stdout:6/457: rename d6/de/d1e/f8a to d6/de/d25/d36/f99 0 2024-03-04T16:04:14.195 INFO:tasks.workunit.client.0.smithi116.stdout:4/563: readlink d1/d3/d15/d20/d1c/d62/la5 0 2024-03-04T16:04:14.197 INFO:tasks.workunit.client.0.smithi116.stdout:4/564: chown d1/d3/d15/d20/d1c/d62/d81/d97/fb9 224582942 1 2024-03-04T16:04:14.197 INFO:tasks.workunit.client.0.smithi116.stdout:7/515: link d3/d4/d8/d1a/d85/c9c d3/d4/d8/d1a/d85/ca2 0 2024-03-04T16:04:14.197 INFO:tasks.workunit.client.0.smithi116.stdout:6/458: read d6/de/d25/d36/d4e/d50/f5f [2180824,23372] 0 2024-03-04T16:04:14.199 INFO:tasks.workunit.client.0.smithi116.stdout:1/409: creat d5/d9/d21/d84/f89 x:0 0 0 2024-03-04T16:04:14.199 INFO:tasks.workunit.client.0.smithi116.stdout:6/459: fsync d6/d56/f60 0 2024-03-04T16:04:14.199 INFO:tasks.workunit.client.0.smithi116.stdout:0/451: creat da/d16/f93 x:0 0 0 2024-03-04T16:04:14.200 INFO:tasks.workunit.client.0.smithi116.stdout:6/460: fsync d6/de/d5b/f8e 0 2024-03-04T16:04:14.201 INFO:tasks.workunit.client.0.smithi116.stdout:1/410: readlink d5/d9/d21/d27/d32/d35/d55/d58/l68 0 2024-03-04T16:04:14.201 INFO:tasks.workunit.client.0.smithi116.stdout:0/452: dread - da/d16/d20/d79/f7c zero size 2024-03-04T16:04:14.205 INFO:tasks.workunit.client.0.smithi116.stdout:3/588: rename dc/l2a to dc/d23/d84/d7f/d93/db6/dc5/lc6 0 2024-03-04T16:04:14.206 INFO:tasks.workunit.client.0.smithi116.stdout:5/530: truncate dd/d1f/d72/d8e/f98 3997294 0 2024-03-04T16:04:14.208 INFO:tasks.workunit.client.0.smithi116.stdout:4/565: creat d1/d3/d40/d52/d84/d88/d8d/fc8 x:0 0 0 2024-03-04T16:04:14.213 INFO:tasks.workunit.client.0.smithi116.stdout:6/461: rmdir d6/de/d25/d36 39 2024-03-04T16:04:14.217 INFO:tasks.workunit.client.0.smithi116.stdout:1/411: symlink d5/d9/d21/d24/l8a 0 2024-03-04T16:04:14.221 INFO:tasks.workunit.client.0.smithi116.stdout:5/531: creat dd/d1f/d5c/d5e/faa x:0 0 0 2024-03-04T16:04:14.234 INFO:tasks.workunit.client.0.smithi116.stdout:1/412: mkdir d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/d8b 0 2024-03-04T16:04:14.235 INFO:tasks.workunit.client.0.smithi116.stdout:1/413: dread - d5/d9/d21/d27/d32/d35/d55/f82 zero size 2024-03-04T16:04:14.236 INFO:tasks.workunit.client.0.smithi116.stdout:1/414: chown d5/d9/d21/d24/f3a 1543812 1 2024-03-04T16:04:14.237 INFO:tasks.workunit.client.0.smithi116.stdout:5/532: symlink dd/d1f/d55/lab 0 2024-03-04T16:04:14.239 INFO:tasks.workunit.client.0.smithi116.stdout:4/566: rename d1/f32 to d1/d6e/d74/dc2/fc9 0 2024-03-04T16:04:14.241 INFO:tasks.workunit.client.0.smithi116.stdout:0/453: creat da/d16/d28/d3d/d41/f94 x:0 0 0 2024-03-04T16:04:14.244 INFO:tasks.workunit.client.0.smithi116.stdout:4/567: dread d1/d3/de/f12 [0,4194304] 0 2024-03-04T16:04:14.244 INFO:tasks.workunit.client.0.smithi116.stdout:1/415: mkdir d5/d9/d21/d27/d8c 0 2024-03-04T16:04:14.245 INFO:tasks.workunit.client.0.smithi116.stdout:7/516: truncate d3/d4/d8/d4a/d65/f72 4132160 0 2024-03-04T16:04:14.245 INFO:tasks.workunit.client.0.smithi116.stdout:6/462: write d6/de/d15/f26 [4580531,45302] 0 2024-03-04T16:04:14.246 INFO:tasks.workunit.client.0.smithi116.stdout:7/517: dread - d3/d4/d34/d5d/f8b zero size 2024-03-04T16:04:14.253 INFO:tasks.workunit.client.0.smithi116.stdout:7/518: symlink d3/d4/d8/d4a/d87/la3 0 2024-03-04T16:04:14.253 INFO:tasks.workunit.client.0.smithi116.stdout:3/589: write dc/df/d14/f72 [601167,37621] 0 2024-03-04T16:04:14.253 INFO:tasks.workunit.client.0.smithi116.stdout:7/519: stat d3/f18 0 2024-03-04T16:04:14.259 INFO:tasks.workunit.client.0.smithi116.stdout:3/590: creat dc/d23/d84/d7f/d8b/fc7 x:0 0 0 2024-03-04T16:04:14.259 INFO:tasks.workunit.client.0.smithi116.stdout:1/416: dread d5/d9/f18 [0,4194304] 0 2024-03-04T16:04:14.264 INFO:tasks.workunit.client.0.smithi116.stdout:5/533: getdents dd/d1f/d5c/d5e 0 2024-03-04T16:04:14.265 INFO:tasks.workunit.client.0.smithi116.stdout:1/417: dwrite d5/d9/d46/f7a [0,4194304] 0 2024-03-04T16:04:14.270 INFO:tasks.workunit.client.0.smithi116.stdout:7/520: rename d3/d4/d11/f5b to d3/d4/d8/d52/d5e/fa4 0 2024-03-04T16:04:14.271 INFO:tasks.workunit.client.0.smithi116.stdout:7/521: fsync d3/d4/d8/f3e 0 2024-03-04T16:04:14.272 INFO:tasks.workunit.client.0.smithi116.stdout:5/534: rmdir dd/d1f/d5c 39 2024-03-04T16:04:14.275 INFO:tasks.workunit.client.0.smithi116.stdout:1/418: write d5/d9/d46/f7f [500277,90416] 0 2024-03-04T16:04:14.277 INFO:tasks.workunit.client.0.smithi116.stdout:4/568: write d1/d3/de/f30 [229673,119179] 0 2024-03-04T16:04:14.279 INFO:tasks.workunit.client.0.smithi116.stdout:4/569: chown d1/d3/d15/d20/d1c/d62 716843 1 2024-03-04T16:04:14.282 INFO:tasks.workunit.client.0.smithi116.stdout:1/419: dwrite d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/f62 [0,4194304] 0 2024-03-04T16:04:14.282 INFO:tasks.workunit.client.0.smithi116.stdout:5/535: creat dd/d1f/d55/d73/d2d/d59/fac x:0 0 0 2024-03-04T16:04:14.284 INFO:tasks.workunit.client.0.smithi116.stdout:7/522: symlink d3/la5 0 2024-03-04T16:04:14.290 INFO:tasks.workunit.client.0.smithi116.stdout:3/591: getdents dc/d23 0 2024-03-04T16:04:14.294 INFO:tasks.workunit.client.0.smithi116.stdout:5/536: unlink dd/d1f/d55/d73/c57 0 2024-03-04T16:04:14.294 INFO:tasks.workunit.client.0.smithi116.stdout:7/523: mknod d3/d4/d8/d52/ca6 0 2024-03-04T16:04:14.295 INFO:tasks.workunit.client.0.smithi116.stdout:5/537: chown dd/d1f/d51/f93 38386721 1 2024-03-04T16:04:14.295 INFO:tasks.workunit.client.0.smithi116.stdout:1/420: dread d5/d9/fb [0,4194304] 0 2024-03-04T16:04:14.296 INFO:tasks.workunit.client.0.smithi116.stdout:5/538: dread - dd/d1f/d51/f93 zero size 2024-03-04T16:04:14.299 INFO:tasks.workunit.client.0.smithi116.stdout:3/592: rename dc/df/d14/c3d to dc/d34/d44/d6a/da3/cc8 0 2024-03-04T16:04:14.300 INFO:tasks.workunit.client.0.smithi116.stdout:7/524: creat d3/d4/d34/d5d/fa7 x:0 0 0 2024-03-04T16:04:14.301 INFO:tasks.workunit.client.0.smithi116.stdout:7/525: write d3/d4/d8/d52/d5e/f6e [4163984,87673] 0 2024-03-04T16:04:14.303 INFO:tasks.workunit.client.0.smithi116.stdout:7/526: truncate d3/d4/d8/d4a/f6b 4510324 0 2024-03-04T16:04:14.304 INFO:tasks.workunit.client.0.smithi116.stdout:1/421: symlink d5/d9/d21/d27/d32/d49/d4c/l8d 0 2024-03-04T16:04:14.304 INFO:tasks.workunit.client.0.smithi116.stdout:7/527: fsync d3/d4/f92 0 2024-03-04T16:04:14.305 INFO:tasks.workunit.client.0.smithi116.stdout:3/593: creat dc/d23/d84/d7f/d8b/fc9 x:0 0 0 2024-03-04T16:04:14.310 INFO:tasks.workunit.client.0.smithi116.stdout:3/594: dread dc/df/f74 [0,4194304] 0 2024-03-04T16:04:14.312 INFO:tasks.workunit.client.0.smithi116.stdout:5/539: dread dd/f13 [0,4194304] 0 2024-03-04T16:04:14.313 INFO:tasks.workunit.client.0.smithi116.stdout:3/595: chown dc/d23/d84/l9c 0 1 2024-03-04T16:04:14.313 INFO:tasks.workunit.client.0.smithi116.stdout:7/528: dwrite d3/f4e [4194304,4194304] 0 2024-03-04T16:04:14.313 INFO:tasks.workunit.client.0.smithi116.stdout:4/570: dwrite d1/d3/de/f10 [0,4194304] 0 2024-03-04T16:04:14.314 INFO:tasks.workunit.client.0.smithi116.stdout:7/529: readlink d3/d4/l21 0 2024-03-04T16:04:14.315 INFO:tasks.workunit.client.0.smithi116.stdout:5/540: readlink dd/d1f/d55/d73/d2d/d31/la3 0 2024-03-04T16:04:14.316 INFO:tasks.workunit.client.0.smithi116.stdout:3/596: write dc/d34/d44/d6a/da3/fb2 [917074,54883] 0 2024-03-04T16:04:14.316 INFO:tasks.workunit.client.0.smithi116.stdout:7/530: readlink d3/d4/d8/d4a/l7a 0 2024-03-04T16:04:14.320 INFO:tasks.workunit.client.0.smithi116.stdout:5/541: read dd/d1f/d55/d73/f3f [483116,28829] 0 2024-03-04T16:04:14.321 INFO:tasks.workunit.client.0.smithi116.stdout:4/571: creat d1/d3/d15/d20/d1c/d62/d81/daf/fca x:0 0 0 2024-03-04T16:04:14.322 INFO:tasks.workunit.client.0.smithi116.stdout:3/597: mknod dc/d23/d84/d7f/d93/db6/dc5/cca 0 2024-03-04T16:04:14.323 INFO:tasks.workunit.client.0.smithi116.stdout:1/422: dwrite d5/d9/fb [4194304,4194304] 0 2024-03-04T16:04:14.324 INFO:tasks.workunit.client.0.smithi116.stdout:7/531: symlink d3/d4/d8/d4a/d88/la8 0 2024-03-04T16:04:14.329 INFO:tasks.workunit.client.0.smithi116.stdout:1/423: rename d5/d9/d21/d84/f89 to d5/d9/d46/d50/f8e 0 2024-03-04T16:04:14.329 INFO:tasks.workunit.client.0.smithi116.stdout:4/572: symlink d1/d3/d15/d20/lcb 0 2024-03-04T16:04:14.330 INFO:tasks.workunit.client.0.smithi116.stdout:1/424: write d5/d9/d21/d24/f37 [4283961,94257] 0 2024-03-04T16:04:14.333 INFO:tasks.workunit.client.0.smithi116.stdout:8/549: sync 2024-03-04T16:04:14.334 INFO:tasks.workunit.client.0.smithi116.stdout:2/573: sync 2024-03-04T16:04:14.336 INFO:tasks.workunit.client.0.smithi116.stdout:5/542: rename dd/d1f/d5c/d5e/l76 to dd/d1f/d55/d73/d2d/lad 0 2024-03-04T16:04:14.336 INFO:tasks.workunit.client.0.smithi116.stdout:4/573: symlink d1/d3/dd/d24/db4/lcc 0 2024-03-04T16:04:14.337 INFO:tasks.workunit.client.0.smithi116.stdout:2/574: readlink d7/d13/l6a 0 2024-03-04T16:04:14.340 INFO:tasks.workunit.client.0.smithi116.stdout:5/543: mkdir dd/d1f/d55/d73/d2d/dae 0 2024-03-04T16:04:14.344 INFO:tasks.workunit.client.0.smithi116.stdout:4/574: creat d1/d3/d15/d20/d1c/d6f/fcd x:0 0 0 2024-03-04T16:04:14.344 INFO:tasks.workunit.client.0.smithi116.stdout:8/550: fsync d2/d13/d3c/d77/d97/d75/f79 0 2024-03-04T16:04:14.345 INFO:tasks.workunit.client.0.smithi116.stdout:1/425: truncate d5/d9/d21/d27/d32/d35/d55/d58/d7e/f4e 1490151 0 2024-03-04T16:04:14.346 INFO:tasks.workunit.client.0.smithi116.stdout:5/544: symlink dd/d1f/d72/d8e/laf 0 2024-03-04T16:04:14.347 INFO:tasks.workunit.client.0.smithi116.stdout:1/426: write d5/d9/d21/d27/d32/d35/f87 [400924,101815] 0 2024-03-04T16:04:14.348 INFO:tasks.workunit.client.0.smithi116.stdout:5/545: dread - dd/d1f/d55/f6c zero size 2024-03-04T16:04:14.348 INFO:tasks.workunit.client.0.smithi116.stdout:2/575: symlink d7/d15/d5e/d9d/dd4/ld6 0 2024-03-04T16:04:14.349 INFO:tasks.workunit.client.0.smithi116.stdout:2/576: write f4 [3773064,48152] 0 2024-03-04T16:04:14.352 INFO:tasks.workunit.client.0.smithi116.stdout:8/551: fdatasync d2/de/d3e/fa8 0 2024-03-04T16:04:14.354 INFO:tasks.workunit.client.0.smithi116.stdout:5/546: dwrite dd/d1f/d5c/f64 [0,4194304] 0 2024-03-04T16:04:14.366 INFO:tasks.workunit.client.0.smithi116.stdout:8/552: mknod d2/d13/d3c/d77/d97/cb4 0 2024-03-04T16:04:14.367 INFO:tasks.workunit.client.0.smithi116.stdout:8/553: write d2/d13/d3c/f83 [869868,91742] 0 2024-03-04T16:04:14.368 INFO:tasks.workunit.client.0.smithi116.stdout:5/547: rmdir dd/d6d 39 2024-03-04T16:04:14.368 INFO:tasks.workunit.client.0.smithi116.stdout:5/548: truncate dd/d1f/f54 70877 0 2024-03-04T16:04:14.373 INFO:tasks.workunit.client.0.smithi116.stdout:8/554: dwrite d2/de/d3e/f50 [0,4194304] 0 2024-03-04T16:04:14.375 INFO:tasks.workunit.client.0.smithi116.stdout:4/575: creat d1/d3/d15/d20/d1c/fce x:0 0 0 2024-03-04T16:04:14.379 INFO:tasks.workunit.client.0.smithi116.stdout:5/549: mkdir dd/d1f/d55/d73/d2d/d59/db0 0 2024-03-04T16:04:14.382 INFO:tasks.workunit.client.0.smithi116.stdout:2/577: rename d7/d15/d35/d63 to d7/d15/dd7 0 2024-03-04T16:04:14.386 INFO:tasks.workunit.client.0.smithi116.stdout:4/576: rmdir d1/d3/d40/d52/d84/d88/d8d 39 2024-03-04T16:04:14.387 INFO:tasks.workunit.client.0.smithi116.stdout:1/427: getdents d5/d9/d21/d27/d32/d35/d55/d58 0 2024-03-04T16:04:14.387 INFO:tasks.workunit.client.0.smithi116.stdout:2/578: dread d7/d15/d5e/f67 [0,4194304] 0 2024-03-04T16:04:14.389 INFO:tasks.workunit.client.0.smithi116.stdout:5/550: chown dd/d1f/d55/d73/d2d/d59/c7e 3 1 2024-03-04T16:04:14.392 INFO:tasks.workunit.client.0.smithi116.stdout:8/555: rename d2/d13/d3c/d46/f99 to d2/d38/d55/fb5 0 2024-03-04T16:04:14.396 INFO:tasks.workunit.client.0.smithi116.stdout:4/577: truncate d1/d3/d15/d20/fa3 354631 0 2024-03-04T16:04:14.396 INFO:tasks.workunit.client.0.smithi116.stdout:4/578: chown d1/f58 510105056 1 2024-03-04T16:04:14.397 INFO:tasks.workunit.client.0.smithi116.stdout:1/428: mknod d5/d9/d46/d50/c8f 0 2024-03-04T16:04:14.402 INFO:tasks.workunit.client.0.smithi116.stdout:2/579: dwrite d7/f4d [0,4194304] 0 2024-03-04T16:04:14.402 INFO:tasks.workunit.client.0.smithi116.stdout:5/551: dread f4 [0,4194304] 0 2024-03-04T16:04:14.409 INFO:tasks.workunit.client.0.smithi116.stdout:1/429: creat d5/d9/d21/d84/f90 x:0 0 0 2024-03-04T16:04:14.409 INFO:tasks.workunit.client.0.smithi116.stdout:8/556: mknod d2/d38/cb6 0 2024-03-04T16:04:14.410 INFO:tasks.workunit.client.0.smithi116.stdout:5/552: symlink dd/d1f/d55/d73/d2d/d59/lb1 0 2024-03-04T16:04:14.411 INFO:tasks.workunit.client.0.smithi116.stdout:2/580: creat d7/d13/d3e/d98/dce/d47/d64/d75/fd8 x:0 0 0 2024-03-04T16:04:14.415 INFO:tasks.workunit.client.0.smithi116.stdout:1/430: readlink d5/d9/d21/d27/d32/l80 0 2024-03-04T16:04:14.415 INFO:tasks.workunit.client.0.smithi116.stdout:1/431: stat d5/d9/d21/d27/d32/d35/d55 0 2024-03-04T16:04:14.416 INFO:tasks.workunit.client.0.smithi116.stdout:2/581: mkdir d7/d13/d3e/d98/dce/d47/dd9 0 2024-03-04T16:04:14.416 INFO:tasks.workunit.client.0.smithi116.stdout:1/432: write d5/d9/d21/d84/f90 [838391,44795] 0 2024-03-04T16:04:14.418 INFO:tasks.workunit.client.0.smithi116.stdout:8/557: unlink d2/de/l19 0 2024-03-04T16:04:14.419 INFO:tasks.workunit.client.0.smithi116.stdout:8/558: dread - d2/d13/d3c/d77/fb2 zero size 2024-03-04T16:04:14.426 INFO:tasks.workunit.client.0.smithi116.stdout:8/559: symlink d2/de/d16/d78/d29/lb7 0 2024-03-04T16:04:14.426 INFO:tasks.workunit.client.0.smithi116.stdout:2/582: creat d7/d2f/d76/fda x:0 0 0 2024-03-04T16:04:14.435 INFO:tasks.workunit.client.0.smithi116.stdout:4/579: write d1/d3/dd/f4a [2232313,7060] 0 2024-03-04T16:04:14.438 INFO:tasks.workunit.client.0.smithi116.stdout:2/583: mkdir d7/d2f/d6c/ddb 0 2024-03-04T16:04:14.439 INFO:tasks.workunit.client.0.smithi116.stdout:2/584: chown d7/d13/d3e/d98/dce 768434 1 2024-03-04T16:04:14.440 INFO:tasks.workunit.client.0.smithi116.stdout:1/433: write d5/d9/f41 [1439233,117013] 0 2024-03-04T16:04:14.442 INFO:tasks.workunit.client.0.smithi116.stdout:4/580: symlink d1/d3/d15/d20/d1c/d62/d81/daf/lcf 0 2024-03-04T16:04:14.444 INFO:tasks.workunit.client.0.smithi116.stdout:4/581: dread - d1/d3/d40/d52/d84/d88/f8b zero size 2024-03-04T16:04:14.444 INFO:tasks.workunit.client.0.smithi116.stdout:8/560: dread d2/d13/d3c/d77/d97/f44 [0,4194304] 0 2024-03-04T16:04:14.446 INFO:tasks.workunit.client.0.smithi116.stdout:2/585: rename d7/d15/d35/c5d to d7/d2f/d6c/ddb/cdc 0 2024-03-04T16:04:14.450 INFO:tasks.workunit.client.0.smithi116.stdout:2/586: symlink d7/d13/d3e/d98/dce/d47/d64/ldd 0 2024-03-04T16:04:14.451 INFO:tasks.workunit.client.0.smithi116.stdout:1/434: getdents d5/d9/d46 0 2024-03-04T16:04:14.453 INFO:tasks.workunit.client.0.smithi116.stdout:2/587: creat d7/d15/d5e/fde x:0 0 0 2024-03-04T16:04:14.454 INFO:tasks.workunit.client.0.smithi116.stdout:8/561: dread d2/de/d16/d2b/d37/f49 [0,4194304] 0 2024-03-04T16:04:14.454 INFO:tasks.workunit.client.0.smithi116.stdout:1/435: truncate d5/d9/d21/d24/f44 552306 0 2024-03-04T16:04:14.456 INFO:tasks.workunit.client.0.smithi116.stdout:8/562: write d2/d13/d3c/d77/d97/d42/f5d [959151,54922] 0 2024-03-04T16:04:14.458 INFO:tasks.workunit.client.0.smithi116.stdout:8/563: stat d2/d13/d3c/d77/d97/d75 0 2024-03-04T16:04:14.459 INFO:tasks.workunit.client.0.smithi116.stdout:8/564: write d2/de/f10 [3005358,34689] 0 2024-03-04T16:04:14.460 INFO:tasks.workunit.client.0.smithi116.stdout:1/436: dread d5/d9/d21/d27/d32/d49/d4a/f78 [0,4194304] 0 2024-03-04T16:04:14.462 INFO:tasks.workunit.client.0.smithi116.stdout:1/437: fdatasync d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/f62 0 2024-03-04T16:04:14.463 INFO:tasks.workunit.client.0.smithi116.stdout:2/588: rename d7/d2f/d6c/da4/l73 to d7/d81/dc2/ldf 0 2024-03-04T16:04:14.467 INFO:tasks.workunit.client.0.smithi116.stdout:4/582: dread d1/d3/d15/d20/f56 [0,4194304] 0 2024-03-04T16:04:14.470 INFO:tasks.workunit.client.0.smithi116.stdout:1/438: mkdir d5/d9/d21/d27/d32/d35/d55/d75/d91 0 2024-03-04T16:04:14.472 INFO:tasks.workunit.client.0.smithi116.stdout:2/589: creat d7/d2f/d6c/ddb/fe0 x:0 0 0 2024-03-04T16:04:14.479 INFO:tasks.workunit.client.0.smithi116.stdout:1/439: mknod d5/d9/d21/d27/d32/d49/d4c/d67/c92 0 2024-03-04T16:04:14.481 INFO:tasks.workunit.client.0.smithi116.stdout:4/583: truncate d1/d3/de/f13 2089420 0 2024-03-04T16:04:14.483 INFO:tasks.workunit.client.0.smithi116.stdout:1/440: unlink d5/d9/d21/d24/f3a 0 2024-03-04T16:04:14.485 INFO:tasks.workunit.client.0.smithi116.stdout:2/590: truncate d7/d81/f22 6904626 0 2024-03-04T16:04:14.485 INFO:tasks.workunit.client.0.smithi116.stdout:2/591: chown d7/d15/l8c 242484 1 2024-03-04T16:04:14.486 INFO:tasks.workunit.client.0.smithi116.stdout:2/592: dread - d7/d2f/d6c/da4/fc3 zero size 2024-03-04T16:04:14.487 INFO:tasks.workunit.client.0.smithi116.stdout:4/584: creat d1/d3/d15/d20/d1c/d62/d81/d97/fd0 x:0 0 0 2024-03-04T16:04:14.487 INFO:tasks.workunit.client.0.smithi116.stdout:2/593: readlink d7/d2f/d76/d7c/la9 0 2024-03-04T16:04:14.491 INFO:tasks.workunit.client.0.smithi116.stdout:8/565: truncate d2/d13/d3c/f69 3406568 0 2024-03-04T16:04:14.492 INFO:tasks.workunit.client.0.smithi116.stdout:4/585: dwrite d1/d3/dd/d24/d51/f71 [0,4194304] 0 2024-03-04T16:04:14.492 INFO:tasks.workunit.client.0.smithi116.stdout:2/594: mkdir d7/dd3/de1 0 2024-03-04T16:04:14.499 INFO:tasks.workunit.client.0.smithi116.stdout:1/441: link d5/f1d d5/d9/d21/d27/d32/d49/d4a/f93 0 2024-03-04T16:04:14.499 INFO:tasks.workunit.client.0.smithi116.stdout:8/566: dread d2/de/d16/f22 [0,4194304] 0 2024-03-04T16:04:14.508 INFO:tasks.workunit.client.0.smithi116.stdout:1/442: mkdir d5/d9/d21/d27/d94 0 2024-03-04T16:04:14.511 INFO:tasks.workunit.client.0.smithi116.stdout:4/586: creat d1/d3/d15/fd1 x:0 0 0 2024-03-04T16:04:14.512 INFO:tasks.workunit.client.0.smithi116.stdout:4/587: chown d1/f80 7 1 2024-03-04T16:04:14.513 INFO:tasks.workunit.client.0.smithi116.stdout:8/567: creat d2/de/d16/d78/d29/d7a/fb8 x:0 0 0 2024-03-04T16:04:14.517 INFO:tasks.workunit.client.0.smithi116.stdout:4/588: dwrite d1/d3/de/f30 [0,4194304] 0 2024-03-04T16:04:14.520 INFO:tasks.workunit.client.0.smithi116.stdout:0/454: sync 2024-03-04T16:04:14.520 INFO:tasks.workunit.client.0.smithi116.stdout:9/496: sync 2024-03-04T16:04:14.520 INFO:tasks.workunit.client.0.smithi116.stdout:6/463: sync 2024-03-04T16:04:14.521 INFO:tasks.workunit.client.0.smithi116.stdout:0/455: chown da/d16/d19/d27/c44 1 1 2024-03-04T16:04:14.521 INFO:tasks.workunit.client.0.smithi116.stdout:0/456: chown da/d16/d28/d40/d48/d77 32195 1 2024-03-04T16:04:14.527 INFO:tasks.workunit.client.0.smithi116.stdout:2/595: truncate d7/d15/f20 511011 0 2024-03-04T16:04:14.531 INFO:tasks.workunit.client.0.smithi116.stdout:9/497: write d2/d33/d65/d6c/f66 [2062559,20467] 0 2024-03-04T16:04:14.536 INFO:tasks.workunit.client.0.smithi116.stdout:1/443: rmdir d5 39 2024-03-04T16:04:14.538 INFO:tasks.workunit.client.0.smithi116.stdout:8/568: truncate d2/d13/d3c/d46/d4f/f5b 1676140 0 2024-03-04T16:04:14.539 INFO:tasks.workunit.client.0.smithi116.stdout:8/569: write d2/d13/d3c/d77/d97/d42/f5d [207830,21606] 0 2024-03-04T16:04:14.540 INFO:tasks.workunit.client.0.smithi116.stdout:2/596: creat d7/dd3/fe2 x:0 0 0 2024-03-04T16:04:14.548 INFO:tasks.workunit.client.0.smithi116.stdout:9/498: mkdir d2/d19/d27/d3b/d4a/d6a/d94 0 2024-03-04T16:04:14.549 INFO:tasks.workunit.client.0.smithi116.stdout:0/457: dwrite da/d16/d20/d42/f86 [0,4194304] 0 2024-03-04T16:04:14.551 INFO:tasks.workunit.client.0.smithi116.stdout:0/458: read da/f35 [60120,36961] 0 2024-03-04T16:04:14.553 INFO:tasks.workunit.client.0.smithi116.stdout:4/589: rename d1/d3/dd/l17 to d1/d3/d15/d20/ld2 0 2024-03-04T16:04:14.556 INFO:tasks.workunit.client.0.smithi116.stdout:4/590: dread d1/d3/dd/d24/d51/f71 [0,4194304] 0 2024-03-04T16:04:14.558 INFO:tasks.workunit.client.0.smithi116.stdout:6/464: creat d6/de/d25/d36/d4e/d50/d92/f9a x:0 0 0 2024-03-04T16:04:14.561 INFO:tasks.workunit.client.0.smithi116.stdout:6/465: dread d6/de/d25/d36/f27 [0,4194304] 0 2024-03-04T16:04:14.568 INFO:tasks.workunit.client.0.smithi116.stdout:8/570: mkdir d2/d13/db9 0 2024-03-04T16:04:14.570 INFO:tasks.workunit.client.0.smithi116.stdout:2/597: write d7/d15/f3f [527306,125147] 0 2024-03-04T16:04:14.570 INFO:tasks.workunit.client.0.smithi116.stdout:4/591: symlink d1/d3/d15/d20/d1c/d62/d81/d97/ld3 0 2024-03-04T16:04:14.570 INFO:tasks.workunit.client.0.smithi116.stdout:2/598: readlink d7/d13/d3e/d98/dce/d47/d64/lc7 0 2024-03-04T16:04:14.580 INFO:tasks.workunit.client.0.smithi116.stdout:6/466: rename d6/de/d25/d36/d4e/c87 to d6/de/d25/d4a/d63/c9b 0 2024-03-04T16:04:14.581 INFO:tasks.workunit.client.0.smithi116.stdout:6/467: chown d6/de/d25/d4a/d63/d8c/l94 10134 1 2024-03-04T16:04:14.583 INFO:tasks.workunit.client.0.smithi116.stdout:8/571: fsync d2/de/d3e/fa8 0 2024-03-04T16:04:14.584 INFO:tasks.workunit.client.0.smithi116.stdout:8/572: chown d2/d38/d41/dab 240849 1 2024-03-04T16:04:14.584 INFO:tasks.workunit.client.0.smithi116.stdout:4/592: creat d1/d3/d40/d52/fd4 x:0 0 0 2024-03-04T16:04:14.586 INFO:tasks.workunit.client.0.smithi116.stdout:1/444: truncate d5/d9/d46/f7a 196575 0 2024-03-04T16:04:14.587 INFO:tasks.workunit.client.0.smithi116.stdout:9/499: dwrite d2/f1a [0,4194304] 0 2024-03-04T16:04:14.589 INFO:tasks.workunit.client.0.smithi116.stdout:4/593: truncate d1/d3/d15/d20/f82 722576 0 2024-03-04T16:04:14.592 INFO:tasks.workunit.client.0.smithi116.stdout:8/573: dwrite d2/d13/f90 [0,4194304] 0 2024-03-04T16:04:14.596 INFO:tasks.workunit.client.0.smithi116.stdout:9/500: dread d2/d19/d5f/f61 [0,4194304] 0 2024-03-04T16:04:14.596 INFO:tasks.workunit.client.0.smithi116.stdout:9/501: fsync d2/d49/d3c/f76 0 2024-03-04T16:04:14.599 INFO:tasks.workunit.client.0.smithi116.stdout:2/599: truncate d7/d13/d3e/d98/dce/d47/f7a 76382 0 2024-03-04T16:04:14.602 INFO:tasks.workunit.client.0.smithi116.stdout:0/459: getdents da/d16/d28/d40/d48/d57 0 2024-03-04T16:04:14.609 INFO:tasks.workunit.client.0.smithi116.stdout:9/502: truncate d2/f15 3909439 0 2024-03-04T16:04:14.610 INFO:tasks.workunit.client.0.smithi116.stdout:2/600: creat d7/d2f/d6c/da4/fe3 x:0 0 0 2024-03-04T16:04:14.612 INFO:tasks.workunit.client.0.smithi116.stdout:1/445: creat d5/d9/d21/d27/d94/f95 x:0 0 0 2024-03-04T16:04:14.613 INFO:tasks.workunit.client.0.smithi116.stdout:6/468: creat d6/d56/d58/d72/f9c x:0 0 0 2024-03-04T16:04:14.615 INFO:tasks.workunit.client.0.smithi116.stdout:0/460: unlink da/d16/d28/d40/d48/d57/f7f 0 2024-03-04T16:04:14.616 INFO:tasks.workunit.client.0.smithi116.stdout:0/461: readlink da/d16/d28/d40/d48/d77/l89 0 2024-03-04T16:04:14.617 INFO:tasks.workunit.client.0.smithi116.stdout:3/598: sync 2024-03-04T16:04:14.618 INFO:tasks.workunit.client.0.smithi116.stdout:4/594: dread d1/d3/f39 [0,4194304] 0 2024-03-04T16:04:14.618 INFO:tasks.workunit.client.0.smithi116.stdout:6/469: dread d6/de/f4d [0,4194304] 0 2024-03-04T16:04:14.621 INFO:tasks.workunit.client.0.smithi116.stdout:9/503: unlink d2/d19/d27/d3b/d4a/d5b/f2f 0 2024-03-04T16:04:14.631 INFO:tasks.workunit.client.0.smithi116.stdout:6/470: unlink d6/d56/d58/d72/f9c 0 2024-03-04T16:04:14.632 INFO:tasks.workunit.client.0.smithi116.stdout:6/471: stat d6/de/d15 0 2024-03-04T16:04:14.633 INFO:tasks.workunit.client.0.smithi116.stdout:6/472: write d6/de/d1e/f48 [7342721,15889] 0 2024-03-04T16:04:14.636 INFO:tasks.workunit.client.0.smithi116.stdout:4/595: dread d1/f75 [0,4194304] 0 2024-03-04T16:04:14.639 INFO:tasks.workunit.client.0.smithi116.stdout:3/599: link dc/df/f79 dc/d34/dc1/fcb 0 2024-03-04T16:04:14.639 INFO:tasks.workunit.client.0.smithi116.stdout:0/462: write da/d16/d28/d3d/f66 [8688619,26652] 0 2024-03-04T16:04:14.641 INFO:tasks.workunit.client.0.smithi116.stdout:9/504: mkdir d2/d19/d27/d3b/d4a/d6a/d94/d95 0 2024-03-04T16:04:14.641 INFO:tasks.workunit.client.0.smithi116.stdout:4/596: read d1/d3/dd/d45/d50/f96 [650885,38131] 0 2024-03-04T16:04:14.642 INFO:tasks.workunit.client.0.smithi116.stdout:1/446: unlink d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/l56 0 2024-03-04T16:04:14.642 INFO:tasks.workunit.client.0.smithi116.stdout:6/473: dwrite d6/de/d15/f26 [0,4194304] 0 2024-03-04T16:04:14.643 INFO:tasks.workunit.client.0.smithi116.stdout:8/574: dwrite d2/de/fc [0,4194304] 0 2024-03-04T16:04:14.644 INFO:tasks.workunit.client.0.smithi116.stdout:0/463: write da/d16/d20/d42/f86 [1991045,109656] 0 2024-03-04T16:04:14.645 INFO:tasks.workunit.client.0.smithi116.stdout:3/600: creat dc/df/db8/fcc x:0 0 0 2024-03-04T16:04:14.646 INFO:tasks.workunit.client.0.smithi116.stdout:9/505: stat d2/d33/d65/d6c/l7d 0 2024-03-04T16:04:14.647 INFO:tasks.workunit.client.0.smithi116.stdout:4/597: chown d1/d3/de/d47/l70 960768 1 2024-03-04T16:04:14.648 INFO:tasks.workunit.client.0.smithi116.stdout:8/575: mkdir d2/d13/d3c/d77/dba 0 2024-03-04T16:04:14.650 INFO:tasks.workunit.client.0.smithi116.stdout:6/474: rmdir d6/de/d25/d4a/d63 39 2024-03-04T16:04:14.650 INFO:tasks.workunit.client.0.smithi116.stdout:1/447: chown d5/d9/d21/d27/d32/d35/d55/d58/d7e/f54 301543242 1 2024-03-04T16:04:14.650 INFO:tasks.workunit.client.0.smithi116.stdout:3/601: write dc/d34/d44/d6a/fa0 [8289,87525] 0 2024-03-04T16:04:14.651 INFO:tasks.workunit.client.0.smithi116.stdout:0/464: symlink da/d16/d19/d27/d70/l95 0 2024-03-04T16:04:14.651 INFO:tasks.workunit.client.0.smithi116.stdout:6/475: chown d6/f24 344 1 2024-03-04T16:04:14.653 INFO:tasks.workunit.client.0.smithi116.stdout:3/602: chown dc/d34/d44 54141 1 2024-03-04T16:04:14.653 INFO:tasks.workunit.client.0.smithi116.stdout:9/506: symlink d2/d19/d27/d3b/l96 0 2024-03-04T16:04:14.659 INFO:tasks.workunit.client.0.smithi116.stdout:6/476: creat d6/de/d25/d4a/f9d x:0 0 0 2024-03-04T16:04:14.659 INFO:tasks.workunit.client.0.smithi116.stdout:4/598: dwrite d1/d3/d15/d20/d1c/f6a [0,4194304] 0 2024-03-04T16:04:14.661 INFO:tasks.workunit.client.0.smithi116.stdout:4/599: truncate d1/d3/dd/f98 738879 0 2024-03-04T16:04:14.661 INFO:tasks.workunit.client.0.smithi116.stdout:8/576: dread d2/d38/d41/f6c [0,4194304] 0 2024-03-04T16:04:14.662 INFO:tasks.workunit.client.0.smithi116.stdout:3/603: getdents dc/d34/d44/d6a/db5 0 2024-03-04T16:04:14.663 INFO:tasks.workunit.client.0.smithi116.stdout:9/507: creat d2/d19/d27/d3b/d4a/d5b/f97 x:0 0 0 2024-03-04T16:04:14.664 INFO:tasks.workunit.client.0.smithi116.stdout:4/600: truncate d1/d3/d40/d52/d84/fa0 826791 0 2024-03-04T16:04:14.664 INFO:tasks.workunit.client.0.smithi116.stdout:9/508: dread - d2/d49/f89 zero size 2024-03-04T16:04:14.664 INFO:tasks.workunit.client.0.smithi116.stdout:6/477: mkdir d6/d9e 0 2024-03-04T16:04:14.666 INFO:tasks.workunit.client.0.smithi116.stdout:6/478: write d6/d56/d58/f98 [498292,1583] 0 2024-03-04T16:04:14.666 INFO:tasks.workunit.client.0.smithi116.stdout:6/479: readlink d6/d56/d58/d72/l7a 0 2024-03-04T16:04:14.667 INFO:tasks.workunit.client.0.smithi116.stdout:6/480: chown d6/de/d25/d36/d44 83 1 2024-03-04T16:04:14.668 INFO:tasks.workunit.client.0.smithi116.stdout:3/604: rmdir dc 39 2024-03-04T16:04:14.670 INFO:tasks.workunit.client.0.smithi116.stdout:8/577: creat d2/d13/d3c/d46/d6e/fbb x:0 0 0 2024-03-04T16:04:14.673 INFO:tasks.workunit.client.0.smithi116.stdout:9/509: write d2/d19/f37 [2849202,52845] 0 2024-03-04T16:04:14.674 INFO:tasks.workunit.client.0.smithi116.stdout:5/553: sync 2024-03-04T16:04:14.674 INFO:tasks.workunit.client.0.smithi116.stdout:7/532: sync 2024-03-04T16:04:14.675 INFO:tasks.workunit.client.0.smithi116.stdout:5/554: write dd/f38 [1766895,27746] 0 2024-03-04T16:04:14.675 INFO:tasks.workunit.client.0.smithi116.stdout:7/533: dread - d3/d4/d34/d5d/f8b zero size 2024-03-04T16:04:14.675 INFO:tasks.workunit.client.0.smithi116.stdout:4/601: truncate d1/d3/de/f23 4690255 0 2024-03-04T16:04:14.675 INFO:tasks.workunit.client.0.smithi116.stdout:1/448: link d5/c65 d5/d9/d21/d27/d32/d49/d4c/d67/c96 0 2024-03-04T16:04:14.676 INFO:tasks.workunit.client.0.smithi116.stdout:6/481: rmdir d6/d56/d58/d72/d7b 39 2024-03-04T16:04:14.677 INFO:tasks.workunit.client.0.smithi116.stdout:8/578: creat d2/d13/d3c/d46/d6e/fbc x:0 0 0 2024-03-04T16:04:14.678 INFO:tasks.workunit.client.0.smithi116.stdout:9/510: dread d2/d19/d27/d3b/d4a/d5b/f25 [0,4194304] 0 2024-03-04T16:04:14.685 INFO:tasks.workunit.client.0.smithi116.stdout:2/601: dwrite d7/d13/f56 [4194304,4194304] 0 2024-03-04T16:04:14.689 INFO:tasks.workunit.client.0.smithi116.stdout:6/482: creat d6/de/d29/d3a/f9f x:0 0 0 2024-03-04T16:04:14.698 INFO:tasks.workunit.client.0.smithi116.stdout:8/579: creat d2/de/d16/d78/fbd x:0 0 0 2024-03-04T16:04:14.699 INFO:tasks.workunit.client.0.smithi116.stdout:1/449: fsync d5/d9/d21/f2c 0 2024-03-04T16:04:14.699 INFO:tasks.workunit.client.0.smithi116.stdout:4/602: symlink d1/d3/dd/ld5 0 2024-03-04T16:04:14.701 INFO:tasks.workunit.client.0.smithi116.stdout:2/602: mknod d7/d15/dd7/ce4 0 2024-03-04T16:04:14.702 INFO:tasks.workunit.client.0.smithi116.stdout:1/450: dread d5/d9/d21/d27/d32/d35/f74 [0,4194304] 0 2024-03-04T16:04:14.704 INFO:tasks.workunit.client.0.smithi116.stdout:6/483: truncate d6/f14 4601490 0 2024-03-04T16:04:14.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:14 smithi116 ceph-mon[29120]: pgmap v23: 65 pgs: 65 active+clean; 820 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 15 MiB/s rd, 102 MiB/s wr, 308 op/s 2024-03-04T16:04:14.756 INFO:tasks.workunit.client.0.smithi116.stdout:4/603: fdatasync d1/f75 0 2024-03-04T16:04:14.756 INFO:tasks.workunit.client.0.smithi116.stdout:2/603: mknod d7/d81/ce5 0 2024-03-04T16:04:14.756 INFO:tasks.workunit.client.0.smithi116.stdout:1/451: mknod d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/c97 0 2024-03-04T16:04:14.757 INFO:tasks.workunit.client.0.smithi116.stdout:5/555: dread dd/f22 [0,4194304] 0 2024-03-04T16:04:14.757 INFO:tasks.workunit.client.0.smithi116.stdout:6/484: creat d6/de/d29/fa0 x:0 0 0 2024-03-04T16:04:14.757 INFO:tasks.workunit.client.0.smithi116.stdout:7/534: dread d3/d4/d8/d4a/f6b [0,4194304] 0 2024-03-04T16:04:14.758 INFO:tasks.workunit.client.0.smithi116.stdout:0/465: write da/d16/d28/d3d/d41/f73 [268797,58198] 0 2024-03-04T16:04:14.760 INFO:tasks.workunit.client.0.smithi116.stdout:5/556: fsync dd/d1f/d55/d73/d2d/d8d/fa6 0 2024-03-04T16:04:14.762 INFO:tasks.workunit.client.0.smithi116.stdout:0/466: write da/d16/d19/f68 [715584,57945] 0 2024-03-04T16:04:14.763 INFO:tasks.workunit.client.0.smithi116.stdout:5/557: fsync dd/d1f/d55/d73/d2d/d31/f74 0 2024-03-04T16:04:14.763 INFO:tasks.workunit.client.0.smithi116.stdout:2/604: rmdir d7/d13/d3e/d98/dce/d47/d64/d75/d77 39 2024-03-04T16:04:14.764 INFO:tasks.workunit.client.0.smithi116.stdout:7/535: symlink d3/d4/d8/d4a/d88/la9 0 2024-03-04T16:04:14.765 INFO:tasks.workunit.client.0.smithi116.stdout:1/452: mknod d5/d9/d21/d27/d32/d35/d55/d58/d7e/d61/c98 0 2024-03-04T16:04:14.765 INFO:tasks.workunit.client.0.smithi116.stdout:2/605: stat d7/d2f/d76/d7c/la9 0 2024-03-04T16:04:14.766 INFO:tasks.workunit.client.0.smithi116.stdout:4/604: rename d1/d3/dd/d24/l9b to d1/d3/d15/d20/d1c/d62/d81/d97/dc6/ld6 0 2024-03-04T16:04:14.766 INFO:tasks.workunit.client.0.smithi116.stdout:4/605: dread - d1/d3/d15/d20/d1c/d62/d81/fb0 zero size 2024-03-04T16:04:14.770 INFO:tasks.workunit.client.0.smithi116.stdout:0/467: write da/d16/d19/f1d [1321766,85232] 0 2024-03-04T16:04:14.779 INFO:tasks.workunit.client.0.smithi116.stdout:2/606: dwrite d7/d15/dd7/f88 [0,4194304] 0 2024-03-04T16:04:14.780 INFO:tasks.workunit.client.0.smithi116.stdout:2/607: write f6 [2426262,127731] 0 2024-03-04T16:04:14.784 INFO:tasks.workunit.client.0.smithi116.stdout:7/536: unlink d3/d4/d8/d1a/f28 0 2024-03-04T16:04:14.788 INFO:tasks.workunit.client.0.smithi116.stdout:4/606: mkdir d1/d3/dd/d45/d50/dad/dd7 0 2024-03-04T16:04:14.789 INFO:tasks.workunit.client.0.smithi116.stdout:4/607: write d1/d3/d15/d20/d1c/d62/d81/daf/fca [478610,10586] 0 2024-03-04T16:04:14.796 INFO:tasks.workunit.client.0.smithi116.stdout:6/485: link d6/c83 d6/d56/d58/ca1 0 2024-03-04T16:04:14.798 INFO:tasks.workunit.client.0.smithi116.stdout:5/558: creat dd/d1f/fb2 x:0 0 0 2024-03-04T16:04:14.802 INFO:tasks.workunit.client.0.smithi116.stdout:7/537: symlink d3/d4/d8/d4a/d88/laa 0 2024-03-04T16:04:14.806 INFO:tasks.workunit.client.0.smithi116.stdout:7/538: dread d3/d4/d8/d1a/f42 [0,4194304] 0 2024-03-04T16:04:14.807 INFO:tasks.workunit.client.0.smithi116.stdout:2/608: symlink d7/dd3/de1/le6 0 2024-03-04T16:04:14.808 INFO:tasks.workunit.client.0.smithi116.stdout:2/609: chown d7/d81/f79 73 1 2024-03-04T16:04:14.809 INFO:tasks.workunit.client.0.smithi116.stdout:6/486: mknod d6/de/d29/ca2 0 2024-03-04T16:04:14.810 INFO:tasks.workunit.client.0.smithi116.stdout:5/559: mknod dd/d45/cb3 0 2024-03-04T16:04:14.811 INFO:tasks.workunit.client.0.smithi116.stdout:0/468: write da/d16/d19/d27/f34 [1515519,60662] 0 2024-03-04T16:04:14.811 INFO:tasks.workunit.client.0.smithi116.stdout:0/469: chown da/d16/d28 2184464 1 2024-03-04T16:04:14.812 INFO:tasks.workunit.client.0.smithi116.stdout:1/453: getdents d5/d9/d21/d27/d32/d35/d55 0 2024-03-04T16:04:14.813 INFO:tasks.workunit.client.0.smithi116.stdout:1/454: chown d5/d9/d46/f7f 78312522 1 2024-03-04T16:04:14.814 INFO:tasks.workunit.client.0.smithi116.stdout:1/455: stat f2 0 2024-03-04T16:04:14.815 INFO:tasks.workunit.client.0.smithi116.stdout:1/456: chown d5/d9/d21/d27/d32/d49 103855 1 2024-03-04T16:04:14.819 INFO:tasks.workunit.client.0.smithi116.stdout:4/608: symlink d1/d3/d40/d52/d84/d88/d8d/ld8 0 2024-03-04T16:04:14.820 INFO:tasks.workunit.client.0.smithi116.stdout:2/610: mkdir d7/d81/dc2/de7 0 2024-03-04T16:04:14.821 INFO:tasks.workunit.client.0.smithi116.stdout:5/560: symlink dd/d1f/d55/d73/d2d/d7c/lb4 0 2024-03-04T16:04:14.821 INFO:tasks.workunit.client.0.smithi116.stdout:1/457: mknod d5/d9/d21/d27/d94/c99 0 2024-03-04T16:04:14.823 INFO:tasks.workunit.client.0.smithi116.stdout:5/561: chown dd/d1f/d5c/d5e/f67 31 1 2024-03-04T16:04:14.823 INFO:tasks.workunit.client.0.smithi116.stdout:2/611: dread - d7/d13/d3e/d98/dce/d47/d64/d75/fd8 zero size 2024-03-04T16:04:14.823 INFO:tasks.workunit.client.0.smithi116.stdout:7/539: rename d3/d4/d8/d4a/d65/f72 to d3/d4/d8/d1a/d85/fab 0 2024-03-04T16:04:14.826 INFO:tasks.workunit.client.0.smithi116.stdout:2/612: write d7/d15/f3f [1345419,40729] 0 2024-03-04T16:04:14.827 INFO:tasks.workunit.client.0.smithi116.stdout:6/487: rename d6/f24 to d6/de/d25/d4a/d63/d8c/fa3 0 2024-03-04T16:04:14.827 INFO:tasks.workunit.client.0.smithi116.stdout:7/540: dread d3/d4/d8/f25 [0,4194304] 0 2024-03-04T16:04:14.830 INFO:tasks.workunit.client.0.smithi116.stdout:7/541: dread - d3/d4/d34/d5d/f8b zero size 2024-03-04T16:04:14.832 INFO:tasks.workunit.client.0.smithi116.stdout:1/458: dwrite d5/d9/d21/d27/d32/d35/d55/f82 [0,4194304] 0 2024-03-04T16:04:14.832 INFO:tasks.workunit.client.0.smithi116.stdout:6/488: unlink d6/f17 0 2024-03-04T16:04:14.834 INFO:tasks.workunit.client.0.smithi116.stdout:1/459: write d5/d9/fb [3384476,27879] 0 2024-03-04T16:04:14.838 INFO:tasks.workunit.client.0.smithi116.stdout:4/609: rename d1/d3/dd/d45/d50/dad/dd7 to d1/d3/dd/dd9 0 2024-03-04T16:04:14.839 INFO:tasks.workunit.client.0.smithi116.stdout:6/489: stat d6/d56/d58/d72/l7a 0 2024-03-04T16:04:14.841 INFO:tasks.workunit.client.0.smithi116.stdout:6/490: write d6/de/f52 [1289350,127547] 0 2024-03-04T16:04:14.842 INFO:tasks.workunit.client.0.smithi116.stdout:7/542: mknod d3/d4/d9e/cac 0 2024-03-04T16:04:14.843 INFO:tasks.workunit.client.0.smithi116.stdout:7/543: fsync d3/d4/d8/d4a/d86/f8d 0 2024-03-04T16:04:14.845 INFO:tasks.workunit.client.0.smithi116.stdout:2/613: rename d7/d13/d3e/d98/dce/d47/d64/lc7 to d7/d81/dc2/de7/le8 0 2024-03-04T16:04:14.847 INFO:tasks.workunit.client.0.smithi116.stdout:4/610: dread d1/d3/dd/d45/d50/f96 [0,4194304] 0 2024-03-04T16:04:14.848 INFO:tasks.workunit.client.0.smithi116.stdout:4/611: write d1/d3/f91 [2034045,10043] 0 2024-03-04T16:04:14.848 INFO:tasks.workunit.client.0.smithi116.stdout:6/491: chown d6/de/d5b/c90 142 1 2024-03-04T16:04:14.849 INFO:tasks.workunit.client.0.smithi116.stdout:6/492: chown d6/de/d25/d4a/d63 11876064 1 2024-03-04T16:04:14.850 INFO:tasks.workunit.client.0.smithi116.stdout:6/493: chown d6 315789046 1 2024-03-04T16:04:14.851 INFO:tasks.workunit.client.0.smithi116.stdout:5/562: dread dd/d1f/d72/d8e/f98 [0,4194304] 0 2024-03-04T16:04:14.853 INFO:tasks.workunit.client.0.smithi116.stdout:7/544: getdents d3/d4/d8/d52/d5e/d9a 0 2024-03-04T16:04:14.855 INFO:tasks.workunit.client.0.smithi116.stdout:1/460: getdents d5/d9/d21/d27/d32/d49/d4c/d67 0 2024-03-04T16:04:14.856 INFO:tasks.workunit.client.0.smithi116.stdout:2/614: dwrite d7/f2b [0,4194304] 0 2024-03-04T16:04:14.856 INFO:tasks.workunit.client.0.smithi116.stdout:4/612: rename d1/d3/d15/d20/d1c/d62 to d1/d3/d15/d20/d1c/d62/d81/dda 22 2024-03-04T16:04:14.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:14 smithi152 ceph-mon[36876]: pgmap v23: 65 pgs: 65 active+clean; 820 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 15 MiB/s rd, 102 MiB/s wr, 308 op/s 2024-03-04T16:04:14.858 INFO:tasks.workunit.client.0.smithi116.stdout:6/494: mknod d6/de/ca4 0 2024-03-04T16:04:14.858 INFO:tasks.workunit.client.0.smithi116.stdout:4/613: chown d1/d3/d15/d20/d1c/d62/d81/d97/dc6 247574 1 2024-03-04T16:04:14.860 INFO:tasks.workunit.client.0.smithi116.stdout:7/545: creat d3/d4/d8/d52/fad x:0 0 0 2024-03-04T16:04:14.862 INFO:tasks.workunit.client.0.smithi116.stdout:7/546: write d3/d4/d11/f51 [838001,73654] 0 2024-03-04T16:04:14.863 INFO:tasks.workunit.client.0.smithi116.stdout:4/614: mkdir d1/d3/ddb 0 2024-03-04T16:04:14.864 INFO:tasks.workunit.client.0.smithi116.stdout:7/547: chown d3/d4/d8/f40 1014 1 2024-03-04T16:04:14.865 INFO:tasks.workunit.client.0.smithi116.stdout:2/615: dwrite d7/d13/f1a [0,4194304] 0 2024-03-04T16:04:14.867 INFO:tasks.workunit.client.0.smithi116.stdout:4/615: truncate d1/d3/d15/d20/d1c/fce 374554 0 2024-03-04T16:04:14.868 INFO:tasks.workunit.client.0.smithi116.stdout:4/616: write d1/d6e/d74/f7b [228590,60621] 0 2024-03-04T16:04:14.868 INFO:tasks.workunit.client.0.smithi116.stdout:5/563: rename dd/d45/f99 to dd/d1f/fb5 0 2024-03-04T16:04:14.868 INFO:tasks.workunit.client.0.smithi116.stdout:4/617: readlink d1/d6c/l92 0 2024-03-04T16:04:14.870 INFO:tasks.workunit.client.0.smithi116.stdout:2/616: mknod d7/d2f/d6c/ce9 0 2024-03-04T16:04:14.876 INFO:tasks.workunit.client.0.smithi116.stdout:3/605: sync 2024-03-04T16:04:14.876 INFO:tasks.workunit.client.0.smithi116.stdout:5/564: creat dd/d45/fb6 x:0 0 0 2024-03-04T16:04:14.924 INFO:tasks.workunit.client.0.smithi116.stdout:1/461: rename d5/d9/d21/d27/d32/c3d to d5/d9/d21/d27/d32/d35/d55/d75/c9a 0 2024-03-04T16:04:14.926 INFO:tasks.workunit.client.0.smithi116.stdout:5/565: chown dd/d1f/d5c/c8b 9929976 1 2024-03-04T16:04:14.927 INFO:tasks.workunit.client.0.smithi116.stdout:5/566: write dd/d1f/f4d [5813083,94868] 0 2024-03-04T16:04:14.929 INFO:tasks.workunit.client.0.smithi116.stdout:5/567: write dd/d1f/d55/d73/f94 [2043702,35009] 0 2024-03-04T16:04:14.929 INFO:tasks.workunit.client.0.smithi116.stdout:2/617: rename d7/d15/d35/c66 to d7/d2f/d6c/cea 0 2024-03-04T16:04:14.930 INFO:tasks.workunit.client.0.smithi116.stdout:2/618: fsync d7/d2f/d6c/da4/fc3 0 2024-03-04T16:04:14.931 INFO:tasks.workunit.client.0.smithi116.stdout:8/580: sync 2024-03-04T16:04:14.931 INFO:tasks.workunit.client.0.smithi116.stdout:9/511: sync 2024-03-04T16:04:14.931 INFO:tasks.workunit.client.0.smithi116.stdout:0/470: sync 2024-03-04T16:04:14.931 INFO:tasks.workunit.client.0.smithi116.stdout:2/619: fsync d7/d15/f3f 0 2024-03-04T16:04:14.932 INFO:tasks.workunit.client.0.smithi116.stdout:3/606: creat dc/d23/fcd x:0 0 0 2024-03-04T16:04:14.933 INFO:tasks.workunit.client.0.smithi116.stdout:1/462: dread d5/d9/d21/f2c [0,4194304] 0 2024-03-04T16:04:14.934 INFO:tasks.workunit.client.0.smithi116.stdout:0/471: read da/d16/d20/d42/f86 [559616,80983] 0 2024-03-04T16:04:14.936 INFO:tasks.workunit.client.0.smithi116.stdout:5/568: dread dd/f13 [0,4194304] 0 2024-03-04T16:04:14.937 INFO:tasks.workunit.client.0.smithi116.stdout:2/620: dread d7/d15/f59 [0,4194304] 0 2024-03-04T16:04:14.938 INFO:tasks.workunit.client.0.smithi116.stdout:6/495: write d6/f14 [5138227,95208] 0 2024-03-04T16:04:14.940 INFO:tasks.workunit.client.0.smithi116.stdout:9/512: dwrite d2/d33/f88 [0,4194304] 0 2024-03-04T16:04:14.949 INFO:tasks.workunit.client.0.smithi116.stdout:4/618: write d1/d3/de/f30 [5079927,46025] 0 2024-03-04T16:04:14.950 INFO:tasks.workunit.client.0.smithi116.stdout:4/619: chown d1/d3/d15/d20/d1c/fce 4122786 1 2024-03-04T16:04:14.951 INFO:tasks.workunit.client.0.smithi116.stdout:7/548: truncate d3/d4/d8/f1c 3388537 0 2024-03-04T16:04:14.961 INFO:tasks.workunit.client.0.smithi116.stdout:6/496: dread d6/f21 [0,4194304] 0 2024-03-04T16:04:14.962 INFO:tasks.workunit.client.0.smithi116.stdout:9/513: mknod d2/d49/c98 0 2024-03-04T16:04:14.963 INFO:tasks.workunit.client.0.smithi116.stdout:9/514: write d2/d33/d65/f92 [6884,79119] 0 2024-03-04T16:04:14.967 INFO:tasks.workunit.client.0.smithi116.stdout:8/581: mkdir d2/de/d16/d78/d29/d7a/db1/dbe 0 2024-03-04T16:04:14.971 INFO:tasks.workunit.client.0.smithi116.stdout:7/549: rmdir d3/d4/d34 39 2024-03-04T16:04:14.975 INFO:tasks.workunit.client.0.smithi116.stdout:1/463: rename d5/d9/d21/c30 to d5/d9/d21/d27/d32/d49/d4c/d67/c9b 0 2024-03-04T16:04:14.975 INFO:tasks.workunit.client.0.smithi116.stdout:0/472: dwrite da/d16/d28/d40/d48/f52 [0,4194304] 0 2024-03-04T16:04:14.975 INFO:tasks.workunit.client.0.smithi116.stdout:1/464: fdatasync f2 0 2024-03-04T16:04:14.979 INFO:tasks.workunit.client.0.smithi116.stdout:3/607: creat dc/d34/fce x:0 0 0 2024-03-04T16:04:14.984 INFO:tasks.workunit.client.0.smithi116.stdout:4/620: mkdir d1/d3/dd/d8a/ddc 0 2024-03-04T16:04:14.990 INFO:tasks.workunit.client.0.smithi116.stdout:0/473: mknod da/d16/d28/d40/d48/d57/c96 0 2024-03-04T16:04:14.991 INFO:tasks.workunit.client.0.smithi116.stdout:4/621: dwrite d1/d3/d40/d52/fbc [0,4194304] 0 2024-03-04T16:04:14.994 INFO:tasks.workunit.client.0.smithi116.stdout:8/582: mkdir d2/d13/db9/dbf 0 2024-03-04T16:04:14.997 INFO:tasks.workunit.client.0.smithi116.stdout:6/497: dread d6/de/f52 [0,4194304] 0 2024-03-04T16:04:15.000 INFO:tasks.workunit.client.0.smithi116.stdout:5/569: rename dd/d1f/d55/d73/d2d/l3a to dd/d1f/d55/d73/d2d/d59/lb7 0 2024-03-04T16:04:15.004 INFO:tasks.workunit.client.0.smithi116.stdout:2/621: rename d7/d13/c58 to d7/d13/ceb 0 2024-03-04T16:04:15.006 INFO:tasks.workunit.client.0.smithi116.stdout:0/474: symlink da/d16/d28/d40/d48/d57/l97 0 2024-03-04T16:04:15.007 INFO:tasks.workunit.client.0.smithi116.stdout:0/475: fsync da/d16/d20/d42/d4e/f8f 0 2024-03-04T16:04:15.015 INFO:tasks.workunit.client.0.smithi116.stdout:7/550: creat d3/d4/d8/fae x:0 0 0 2024-03-04T16:04:15.016 INFO:tasks.workunit.client.0.smithi116.stdout:8/583: dread d2/d13/d5e/f87 [0,4194304] 0 2024-03-04T16:04:15.016 INFO:tasks.workunit.client.0.smithi116.stdout:7/551: fsync d3/f6 0 2024-03-04T16:04:15.019 INFO:tasks.workunit.client.0.smithi116.stdout:5/570: creat dd/d1f/d55/d73/d2d/d8d/fb8 x:0 0 0 2024-03-04T16:04:15.020 INFO:tasks.workunit.client.0.smithi116.stdout:2/622: rmdir d7/d13/d3e/d98/dce/d47 39 2024-03-04T16:04:15.021 INFO:tasks.workunit.client.0.smithi116.stdout:5/571: read dd/d1f/f4d [6329832,93826] 0 2024-03-04T16:04:15.027 INFO:tasks.workunit.client.0.smithi116.stdout:4/622: truncate d1/d3/d15/d20/f5e 2786633 0 2024-03-04T16:04:15.033 INFO:tasks.workunit.client.0.smithi116.stdout:6/498: mkdir d6/d9e/da5 0 2024-03-04T16:04:15.035 INFO:tasks.workunit.client.0.smithi116.stdout:8/584: creat d2/d13/d5e/fc0 x:0 0 0 2024-03-04T16:04:15.036 INFO:tasks.workunit.client.0.smithi116.stdout:8/585: write d2/d38/d55/f65 [879619,40411] 0 2024-03-04T16:04:15.037 INFO:tasks.workunit.client.0.smithi116.stdout:7/552: write d3/d4/d8/d1a/f5f [998585,27454] 0 2024-03-04T16:04:15.038 INFO:tasks.workunit.client.0.smithi116.stdout:8/586: write d2/d13/f90 [2692639,17171] 0 2024-03-04T16:04:15.041 INFO:tasks.workunit.client.0.smithi116.stdout:5/572: stat dd/d1f/d55/d73/d2d/d59/lb7 0 2024-03-04T16:04:15.041 INFO:tasks.workunit.client.0.smithi116.stdout:1/465: getdents d5/d9/d21/d27/d32/d49 0 2024-03-04T16:04:15.043 INFO:tasks.workunit.client.0.smithi116.stdout:4/623: dread d1/d3/dd/d24/f29 [0,4194304] 0 2024-03-04T16:04:15.047 INFO:tasks.workunit.client.0.smithi116.stdout:9/515: rename d2/d19/d27/d3b/d4a/d5b/l20 to d2/d19/d27/l99 0 2024-03-04T16:04:15.049 INFO:tasks.workunit.client.0.smithi116.stdout:8/587: creat d2/d13/db9/fc1 x:0 0 0 2024-03-04T16:04:15.051 INFO:tasks.workunit.client.0.smithi116.stdout:8/588: write d2/f32 [1393440,123637] 0 2024-03-04T16:04:15.051 INFO:tasks.workunit.client.0.smithi116.stdout:2/623: mkdir d7/d13/dec 0 2024-03-04T16:04:15.052 INFO:tasks.workunit.client.0.smithi116.stdout:8/589: fdatasync d2/d13/d3c/d77/d97/d5a/f9d 0 2024-03-04T16:04:15.053 INFO:tasks.workunit.client.0.smithi116.stdout:8/590: stat d2/de/d3e/f50 0 2024-03-04T16:04:15.055 INFO:tasks.workunit.client.0.smithi116.stdout:1/466: creat d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/f9c x:0 0 0 2024-03-04T16:04:15.056 INFO:tasks.workunit.client.0.smithi116.stdout:5/573: dread dd/d1f/d5c/d5e/f62 [0,4194304] 0 2024-03-04T16:04:15.058 INFO:tasks.workunit.client.0.smithi116.stdout:6/499: creat d6/de/d25/d36/fa6 x:0 0 0 2024-03-04T16:04:15.058 INFO:tasks.workunit.client.0.smithi116.stdout:1/467: dread d5/d9/f18 [0,4194304] 0 2024-03-04T16:04:15.059 INFO:tasks.workunit.client.0.smithi116.stdout:1/468: write d5/d9/d21/d27/d94/f95 [738495,96461] 0 2024-03-04T16:04:15.062 INFO:tasks.workunit.client.0.smithi116.stdout:6/500: dread f2 [4194304,4194304] 0 2024-03-04T16:04:15.063 INFO:tasks.workunit.client.0.smithi116.stdout:2/624: write d7/d13/d3e/d98/dce/d47/d64/d75/fb6 [971248,13960] 0 2024-03-04T16:04:15.067 INFO:tasks.workunit.client.0.smithi116.stdout:0/476: getdents da/d16/d19/d27/d70 0 2024-03-04T16:04:15.069 INFO:tasks.workunit.client.0.smithi116.stdout:8/591: creat d2/d13/d3c/d77/d97/fc2 x:0 0 0 2024-03-04T16:04:15.071 INFO:tasks.workunit.client.0.smithi116.stdout:4/624: mkdir d1/d3/d15/d20/d1c/ddd 0 2024-03-04T16:04:15.071 INFO:tasks.workunit.client.0.smithi116.stdout:4/625: chown d1/d3/d15/d20/d1c/d6f/cb2 3188490 1 2024-03-04T16:04:15.072 INFO:tasks.workunit.client.0.smithi116.stdout:4/626: stat d1/d3/dd/d45/f4c 0 2024-03-04T16:04:15.072 INFO:tasks.workunit.client.0.smithi116.stdout:5/574: fsync dd/d1f/d55/d73/d2d/d31/f7b 0 2024-03-04T16:04:15.077 INFO:tasks.workunit.client.0.smithi116.stdout:1/469: write d5/f5a [215189,10530] 0 2024-03-04T16:04:15.081 INFO:tasks.workunit.client.0.smithi116.stdout:6/501: dread d6/de/d1e/d80/f57 [0,4194304] 0 2024-03-04T16:04:15.082 INFO:tasks.workunit.client.0.smithi116.stdout:6/502: write d6/de/d25/d4a/f9d [753720,118132] 0 2024-03-04T16:04:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:8/592: unlink d2/d38/f3d 0 2024-03-04T16:04:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:5/575: write dd/d6d/f88 [10450,81219] 0 2024-03-04T16:04:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:1/470: creat d5/d9/d21/d27/d32/d35/d55/d58/d7e/f9d x:0 0 0 2024-03-04T16:04:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:6/503: unlink d6/f37 0 2024-03-04T16:04:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:7/553: truncate d3/d4/d8/f40 1046716 0 2024-03-04T16:04:15.101 INFO:tasks.workunit.client.0.smithi116.stdout:0/477: symlink da/d16/d19/d27/l98 0 2024-03-04T16:04:15.106 INFO:tasks.workunit.client.0.smithi116.stdout:0/478: dwrite da/d16/d28/d40/d5d/d7e/f83 [0,4194304] 0 2024-03-04T16:04:15.107 INFO:tasks.workunit.client.0.smithi116.stdout:9/516: getdents d2/d33/d65/d7e 0 2024-03-04T16:04:15.111 INFO:tasks.workunit.client.0.smithi116.stdout:3/608: sync 2024-03-04T16:04:15.114 INFO:tasks.workunit.client.0.smithi116.stdout:2/625: rename d7/d2f to d7/d13/d3e/ded 0 2024-03-04T16:04:15.114 INFO:tasks.workunit.client.0.smithi116.stdout:2/626: stat d7/d15/d35/f54 0 2024-03-04T16:04:15.115 INFO:tasks.workunit.client.0.smithi116.stdout:2/627: write d7/d13/d3e/d98/dce/d47/d64/d75/fb7 [446040,4202] 0 2024-03-04T16:04:15.116 INFO:tasks.workunit.client.0.smithi116.stdout:2/628: chown d7/d13/d3e/d98/lbe 1558 1 2024-03-04T16:04:15.119 INFO:tasks.workunit.client.0.smithi116.stdout:7/554: mkdir d3/d4/d8/d1a/d85/daf 0 2024-03-04T16:04:15.121 INFO:tasks.workunit.client.0.smithi116.stdout:8/593: mkdir d2/de/d16/dc3 0 2024-03-04T16:04:15.125 INFO:tasks.workunit.client.0.smithi116.stdout:5/576: symlink dd/d1f/lb9 0 2024-03-04T16:04:15.129 INFO:tasks.workunit.client.0.smithi116.stdout:9/517: creat d2/d33/d65/d7e/f9a x:0 0 0 2024-03-04T16:04:15.129 INFO:tasks.workunit.client.0.smithi116.stdout:9/518: stat d2/d49/c98 0 2024-03-04T16:04:15.130 INFO:tasks.workunit.client.0.smithi116.stdout:1/471: fdatasync d5/f4d 0 2024-03-04T16:04:15.135 INFO:tasks.workunit.client.0.smithi116.stdout:2/629: chown d7/d25/fbd 210 1 2024-03-04T16:04:15.137 INFO:tasks.workunit.client.0.smithi116.stdout:6/504: creat d6/de/d84/fa7 x:0 0 0 2024-03-04T16:04:15.139 INFO:tasks.workunit.client.0.smithi116.stdout:7/555: symlink d3/d4/d9e/lb0 0 2024-03-04T16:04:15.142 INFO:tasks.workunit.client.0.smithi116.stdout:4/627: truncate d1/d3/f39 3696171 0 2024-03-04T16:04:15.143 INFO:tasks.workunit.client.0.smithi116.stdout:5/577: truncate dd/d1f/f4d 2009637 0 2024-03-04T16:04:15.149 INFO:tasks.workunit.client.0.smithi116.stdout:1/472: truncate d5/d9/d21/d27/d32/f47 2027704 0 2024-03-04T16:04:15.151 INFO:tasks.workunit.client.0.smithi116.stdout:3/609: truncate dc/d23/d84/d7f/d93/fb1 838051 0 2024-03-04T16:04:15.152 INFO:tasks.workunit.client.0.smithi116.stdout:2/630: creat d7/d15/d5e/d9d/dd4/fee x:0 0 0 2024-03-04T16:04:15.153 INFO:tasks.workunit.client.0.smithi116.stdout:2/631: fdatasync d7/d15/d35/f8f 0 2024-03-04T16:04:15.153 INFO:tasks.workunit.client.0.smithi116.stdout:6/505: mknod d6/de/d25/d36/d4e/d50/ca8 0 2024-03-04T16:04:15.156 INFO:tasks.workunit.client.0.smithi116.stdout:3/610: dread dc/d34/f3e [0,4194304] 0 2024-03-04T16:04:15.158 INFO:tasks.workunit.client.0.smithi116.stdout:7/556: symlink d3/d4/d11/lb1 0 2024-03-04T16:04:15.159 INFO:tasks.workunit.client.0.smithi116.stdout:0/479: write da/d16/d28/d3d/d41/f59 [1284038,42679] 0 2024-03-04T16:04:15.162 INFO:tasks.workunit.client.0.smithi116.stdout:4/628: rename d1/d3/dc5 to d1/d3/d40/d52/d84/dde 0 2024-03-04T16:04:15.165 INFO:tasks.workunit.client.0.smithi116.stdout:9/519: creat d2/d19/d27/d3b/d4a/d6a/d94/d95/f9b x:0 0 0 2024-03-04T16:04:15.169 INFO:tasks.workunit.client.0.smithi116.stdout:8/594: write d2/de/d3e/f2a [3237426,16139] 0 2024-03-04T16:04:15.169 INFO:tasks.workunit.client.0.smithi116.stdout:9/520: dwrite d2/d19/d27/d3b/f3d [0,4194304] 0 2024-03-04T16:04:15.182 INFO:tasks.workunit.client.0.smithi116.stdout:4/629: creat d1/d3/dd/d24/d51/fdf x:0 0 0 2024-03-04T16:04:15.182 INFO:tasks.workunit.client.0.smithi116.stdout:2/632: rmdir d7/d15/d5e/d9d/dd4 39 2024-03-04T16:04:15.188 INFO:tasks.workunit.client.0.smithi116.stdout:2/633: dwrite d7/d13/d3e/d78/f7e [0,4194304] 0 2024-03-04T16:04:15.189 INFO:tasks.workunit.client.0.smithi116.stdout:2/634: truncate d7/d25/f62 93746 0 2024-03-04T16:04:15.191 INFO:tasks.workunit.client.0.smithi116.stdout:1/473: creat d5/d9/d21/d27/d32/d35/d55/d75/d91/f9e x:0 0 0 2024-03-04T16:04:15.193 INFO:tasks.workunit.client.0.smithi116.stdout:8/595: creat d2/d13/d5e/fc4 x:0 0 0 2024-03-04T16:04:15.195 INFO:tasks.workunit.client.0.smithi116.stdout:1/474: read d5/f29 [2591483,123320] 0 2024-03-04T16:04:15.198 INFO:tasks.workunit.client.0.smithi116.stdout:1/475: dread d5/d9/d21/d27/d94/f95 [0,4194304] 0 2024-03-04T16:04:15.200 INFO:tasks.workunit.client.0.smithi116.stdout:0/480: unlink da/d16/l3e 0 2024-03-04T16:04:15.201 INFO:tasks.workunit.client.0.smithi116.stdout:7/557: symlink d3/d4/d8/d1a/d85/daf/lb2 0 2024-03-04T16:04:15.203 INFO:tasks.workunit.client.0.smithi116.stdout:4/630: write d1/d3/dd/d24/d51/f71 [1155265,53294] 0 2024-03-04T16:04:15.204 INFO:tasks.workunit.client.0.smithi116.stdout:4/631: fdatasync d1/d3/de/f26 0 2024-03-04T16:04:15.205 INFO:tasks.workunit.client.0.smithi116.stdout:4/632: write d1/d3/de/d47/f60 [1739123,96657] 0 2024-03-04T16:04:15.206 INFO:tasks.workunit.client.0.smithi116.stdout:4/633: chown d1/d6c/f38 891 1 2024-03-04T16:04:15.210 INFO:tasks.workunit.client.0.smithi116.stdout:8/596: symlink d2/d13/d3c/d46/lc5 0 2024-03-04T16:04:15.211 INFO:tasks.workunit.client.0.smithi116.stdout:4/634: dwrite d1/d3/d15/fb6 [0,4194304] 0 2024-03-04T16:04:15.217 INFO:tasks.workunit.client.0.smithi116.stdout:3/611: rmdir dc/d23/d84/d7f/d93/db6/dc2 0 2024-03-04T16:04:15.222 INFO:tasks.workunit.client.0.smithi116.stdout:0/481: dread da/d16/d28/d40/f5c [0,4194304] 0 2024-03-04T16:04:15.228 INFO:tasks.workunit.client.0.smithi116.stdout:5/578: getdents dd/d1f/d55/d73/d2d/d59 0 2024-03-04T16:04:15.230 INFO:tasks.workunit.client.0.smithi116.stdout:5/579: write dd/d1f/d55/d73/d2d/d31/f3c [3474065,122545] 0 2024-03-04T16:04:15.231 INFO:tasks.workunit.client.0.smithi116.stdout:2/635: rename d7/fb to d7/d15/dd7/fef 0 2024-03-04T16:04:15.236 INFO:tasks.workunit.client.0.smithi116.stdout:6/506: getdents d6/d56/d58/d72 0 2024-03-04T16:04:15.237 INFO:tasks.workunit.client.0.smithi116.stdout:6/507: stat d6/de/d25/d36/d44/f4b 0 2024-03-04T16:04:15.238 INFO:tasks.workunit.client.0.smithi116.stdout:1/476: write d5/d9/d46/d50/f71 [1974047,114203] 0 2024-03-04T16:04:15.238 INFO:tasks.workunit.client.0.smithi116.stdout:4/635: mkdir d1/d6e/d74/de0 0 2024-03-04T16:04:15.242 INFO:tasks.workunit.client.0.smithi116.stdout:3/612: rmdir dc/df/db8 39 2024-03-04T16:04:15.245 INFO:tasks.workunit.client.0.smithi116.stdout:8/597: dread d2/d13/d3c/fa3 [0,4194304] 0 2024-03-04T16:04:15.247 INFO:tasks.workunit.client.0.smithi116.stdout:0/482: rmdir da/d16 39 2024-03-04T16:04:15.249 INFO:tasks.workunit.client.0.smithi116.stdout:7/558: rmdir d3/d4/d34 39 2024-03-04T16:04:15.250 INFO:tasks.workunit.client.0.smithi116.stdout:7/559: write d3/d4/d8/d52/fad [832407,20961] 0 2024-03-04T16:04:15.252 INFO:tasks.workunit.client.0.smithi116.stdout:5/580: mkdir dd/d1f/d55/d73/dba 0 2024-03-04T16:04:15.252 INFO:tasks.workunit.client.0.smithi116.stdout:5/581: fsync dd/d6d/f80 0 2024-03-04T16:04:15.255 INFO:tasks.workunit.client.0.smithi116.stdout:2/636: rename d7/d13/d3e/d98/dce/c92 to d7/d13/d3e/ded/d76/cf0 0 2024-03-04T16:04:15.256 INFO:tasks.workunit.client.0.smithi116.stdout:7/560: dwrite d3/d4/f92 [0,4194304] 0 2024-03-04T16:04:15.259 INFO:tasks.workunit.client.0.smithi116.stdout:3/613: mknod dc/df/d14/ccf 0 2024-03-04T16:04:15.265 INFO:tasks.workunit.client.0.smithi116.stdout:5/582: fdatasync dd/d1f/d55/d73/d2d/d31/f7b 0 2024-03-04T16:04:15.266 INFO:tasks.workunit.client.0.smithi116.stdout:5/583: chown dd/d1f/d55/d73/d2d/d59 1973777 1 2024-03-04T16:04:15.267 INFO:tasks.workunit.client.0.smithi116.stdout:5/584: dread dd/d6d/f88 [0,4194304] 0 2024-03-04T16:04:15.269 INFO:tasks.workunit.client.0.smithi116.stdout:6/508: mknod d6/d56/d58/ca9 0 2024-03-04T16:04:15.324 INFO:tasks.workunit.client.0.smithi116.stdout:3/614: unlink dc/d34/d44/cb9 0 2024-03-04T16:04:15.325 INFO:tasks.workunit.client.0.smithi116.stdout:0/483: rename da/d16/d28/d3d/d41/f73 to da/d16/d20/d79/f99 0 2024-03-04T16:04:15.327 INFO:tasks.workunit.client.0.smithi116.stdout:5/585: creat dd/d1f/d5c/fbb x:0 0 0 2024-03-04T16:04:15.327 INFO:tasks.workunit.client.0.smithi116.stdout:2/637: mkdir d7/d15/d5e/d9d/df1 0 2024-03-04T16:04:15.331 INFO:tasks.workunit.client.0.smithi116.stdout:6/509: creat d6/de/d25/faa x:0 0 0 2024-03-04T16:04:15.332 INFO:tasks.workunit.client.0.smithi116.stdout:1/477: dwrite d5/d9/d21/f2c [0,4194304] 0 2024-03-04T16:04:15.334 INFO:tasks.workunit.client.0.smithi116.stdout:8/598: dwrite d2/de/d16/f2c [0,4194304] 0 2024-03-04T16:04:15.334 INFO:tasks.workunit.client.0.smithi116.stdout:2/638: write d7/d13/d3e/d98/dce/d47/d64/f86 [3512669,75041] 0 2024-03-04T16:04:15.337 INFO:tasks.workunit.client.0.smithi116.stdout:2/639: fdatasync d7/d15/dd7/fb4 0 2024-03-04T16:04:15.338 INFO:tasks.workunit.client.0.smithi116.stdout:0/484: dwrite f7 [0,4194304] 0 2024-03-04T16:04:15.339 INFO:tasks.workunit.client.0.smithi116.stdout:0/485: write da/d16/d28/d3d/d41/f94 [906570,112225] 0 2024-03-04T16:04:15.344 INFO:tasks.workunit.client.0.smithi116.stdout:8/599: readlink d2/de/d16/d78/d29/l52 0 2024-03-04T16:04:15.348 INFO:tasks.workunit.client.0.smithi116.stdout:0/486: dread da/d16/d19/d27/f78 [0,4194304] 0 2024-03-04T16:04:15.355 INFO:tasks.workunit.client.0.smithi116.stdout:5/586: creat dd/d1f/d55/d73/d2d/d59/db0/fbc x:0 0 0 2024-03-04T16:04:15.355 INFO:tasks.workunit.client.0.smithi116.stdout:5/587: write dd/d1f/d5c/fbb [322587,121324] 0 2024-03-04T16:04:15.361 INFO:tasks.workunit.client.0.smithi116.stdout:6/510: truncate d6/de/d25/d4a/f71 2421981 0 2024-03-04T16:04:15.362 INFO:tasks.workunit.client.0.smithi116.stdout:2/640: mknod d7/d13/d3e/d98/dce/d47/dd9/cf2 0 2024-03-04T16:04:15.365 INFO:tasks.workunit.client.0.smithi116.stdout:9/521: sync 2024-03-04T16:04:15.365 INFO:tasks.workunit.client.0.smithi116.stdout:2/641: read d7/d81/f79 [798072,9332] 0 2024-03-04T16:04:15.366 INFO:tasks.workunit.client.0.smithi116.stdout:9/522: write d2/d33/f88 [3203964,59366] 0 2024-03-04T16:04:15.368 INFO:tasks.workunit.client.0.smithi116.stdout:0/487: creat da/d16/d19/d27/d70/d92/f9a x:0 0 0 2024-03-04T16:04:15.370 INFO:tasks.workunit.client.0.smithi116.stdout:6/511: mknod d6/de/d25/d4a/d63/d8c/cab 0 2024-03-04T16:04:15.371 INFO:tasks.workunit.client.0.smithi116.stdout:2/642: dwrite f4 [0,4194304] 0 2024-03-04T16:04:15.372 INFO:tasks.workunit.client.0.smithi116.stdout:6/512: write d6/de/d84/fa7 [940913,26947] 0 2024-03-04T16:04:15.378 INFO:tasks.workunit.client.0.smithi116.stdout:0/488: write da/d16/d28/d3d/f64 [2281744,26445] 0 2024-03-04T16:04:15.379 INFO:tasks.workunit.client.0.smithi116.stdout:0/489: write da/d16/f6e [112615,32212] 0 2024-03-04T16:04:15.379 INFO:tasks.workunit.client.0.smithi116.stdout:2/643: mknod d7/d13/d3e/ded/d76/cf3 0 2024-03-04T16:04:15.382 INFO:tasks.workunit.client.0.smithi116.stdout:6/513: rmdir d6/d56/d58/d72 39 2024-03-04T16:04:15.383 INFO:tasks.workunit.client.0.smithi116.stdout:6/514: write d6/de/d1e/d80/f89 [426921,6288] 0 2024-03-04T16:04:15.385 INFO:tasks.workunit.client.0.smithi116.stdout:3/615: truncate dc/df/d14/f72 241234 0 2024-03-04T16:04:15.386 INFO:tasks.workunit.client.0.smithi116.stdout:3/616: chown dc/d23/d84/d7c/f80 12 1 2024-03-04T16:04:15.387 INFO:tasks.workunit.client.0.smithi116.stdout:0/490: unlink da/d16/d20/d79/f99 0 2024-03-04T16:04:15.388 INFO:tasks.workunit.client.0.smithi116.stdout:9/523: creat d2/d19/f9c x:0 0 0 2024-03-04T16:04:15.391 INFO:tasks.workunit.client.0.smithi116.stdout:3/617: rename dc/d23/d84/d7f/d8b/l97 to dc/d34/d44/d6a/da3/ld0 0 2024-03-04T16:04:15.394 INFO:tasks.workunit.client.0.smithi116.stdout:0/491: dwrite da/d16/d20/f3b [0,4194304] 0 2024-03-04T16:04:15.396 INFO:tasks.workunit.client.0.smithi116.stdout:6/515: dwrite d6/de/d25/d4a/d63/d8c/fa3 [0,4194304] 0 2024-03-04T16:04:15.396 INFO:tasks.workunit.client.0.smithi116.stdout:3/618: creat dc/d23/d84/fd1 x:0 0 0 2024-03-04T16:04:15.400 INFO:tasks.workunit.client.0.smithi116.stdout:0/492: symlink da/d16/d28/d39/l9b 0 2024-03-04T16:04:15.402 INFO:tasks.workunit.client.0.smithi116.stdout:2/644: dread d7/d81/f26 [0,4194304] 0 2024-03-04T16:04:15.405 INFO:tasks.workunit.client.0.smithi116.stdout:6/516: rename d6/de/d25/d36/d4e/d50/c68 to d6/d9e/da5/cac 0 2024-03-04T16:04:15.407 INFO:tasks.workunit.client.0.smithi116.stdout:0/493: mknod da/d16/d20/d53/c9c 0 2024-03-04T16:04:15.409 INFO:tasks.workunit.client.0.smithi116.stdout:5/588: dwrite dd/d1f/d55/d73/d2d/d31/f65 [0,4194304] 0 2024-03-04T16:04:15.414 INFO:tasks.workunit.client.0.smithi116.stdout:8/600: dwrite d2/d13/d3c/d46/d4f/f5b [0,4194304] 0 2024-03-04T16:04:15.414 INFO:tasks.workunit.client.0.smithi116.stdout:3/619: dread dc/d23/d84/f8e [0,4194304] 0 2024-03-04T16:04:15.421 INFO:tasks.workunit.client.0.smithi116.stdout:5/589: rename dd/d1f/l3e to dd/d1f/d51/lbd 0 2024-03-04T16:04:15.421 INFO:tasks.workunit.client.0.smithi116.stdout:6/517: dwrite d6/d56/f4c [0,4194304] 0 2024-03-04T16:04:15.423 INFO:tasks.workunit.client.0.smithi116.stdout:2/645: dwrite d7/d13/d3e/d78/fb1 [0,4194304] 0 2024-03-04T16:04:15.423 INFO:tasks.workunit.client.0.smithi116.stdout:8/601: write d2/d13/f90 [2333470,57973] 0 2024-03-04T16:04:15.430 INFO:tasks.workunit.client.0.smithi116.stdout:6/518: mknod d6/de/d84/cad 0 2024-03-04T16:04:15.431 INFO:tasks.workunit.client.0.smithi116.stdout:0/494: getdents da/d16/d28/d40/d5d/d7e 0 2024-03-04T16:04:15.431 INFO:tasks.workunit.client.0.smithi116.stdout:8/602: creat d2/d13/d5e/fc6 x:0 0 0 2024-03-04T16:04:15.437 INFO:tasks.workunit.client.0.smithi116.stdout:7/561: sync 2024-03-04T16:04:15.437 INFO:tasks.workunit.client.0.smithi116.stdout:4/636: sync 2024-03-04T16:04:15.437 INFO:tasks.workunit.client.0.smithi116.stdout:9/524: dwrite d2/d19/d5f/f91 [0,4194304] 0 2024-03-04T16:04:15.442 INFO:tasks.workunit.client.0.smithi116.stdout:3/620: dwrite dc/df/d14/f5d [0,4194304] 0 2024-03-04T16:04:15.445 INFO:tasks.workunit.client.0.smithi116.stdout:3/621: truncate dc/d23/d84/d7f/d8b/fc7 281108 0 2024-03-04T16:04:15.449 INFO:tasks.workunit.client.0.smithi116.stdout:2/646: mknod d7/d13/d3e/cf4 0 2024-03-04T16:04:15.450 INFO:tasks.workunit.client.0.smithi116.stdout:6/519: mknod d6/de/d25/d36/d4e/d50/d92/cae 0 2024-03-04T16:04:15.452 INFO:tasks.workunit.client.0.smithi116.stdout:8/603: rename d2/d13/d3c/d77/dba to d2/d4a/dc7 0 2024-03-04T16:04:15.454 INFO:tasks.workunit.client.0.smithi116.stdout:3/622: read - dc/df/db8/fcc zero size 2024-03-04T16:04:15.456 INFO:tasks.workunit.client.0.smithi116.stdout:5/590: creat dd/d1f/d55/d73/d2d/dae/fbe x:0 0 0 2024-03-04T16:04:15.457 INFO:tasks.workunit.client.0.smithi116.stdout:8/604: read d2/de/d3e/f50 [920520,1292] 0 2024-03-04T16:04:15.457 INFO:tasks.workunit.client.0.smithi116.stdout:2/647: chown d7/d15/d5e/c9e 19974217 1 2024-03-04T16:04:15.458 INFO:tasks.workunit.client.0.smithi116.stdout:5/591: chown dd/d1f/d55/d73/d2d/d31/f3c 307050 1 2024-03-04T16:04:15.459 INFO:tasks.workunit.client.0.smithi116.stdout:7/562: mkdir d3/d4/d8/d4a/db3 0 2024-03-04T16:04:15.461 INFO:tasks.workunit.client.0.smithi116.stdout:8/605: dread d2/d38/d41/f6c [0,4194304] 0 2024-03-04T16:04:15.461 INFO:tasks.workunit.client.0.smithi116.stdout:3/623: dwrite dc/df/d47/f61 [0,4194304] 0 2024-03-04T16:04:15.463 INFO:tasks.workunit.client.0.smithi116.stdout:0/495: creat da/d16/d19/d27/f9d x:0 0 0 2024-03-04T16:04:15.463 INFO:tasks.workunit.client.0.smithi116.stdout:7/563: write d3/d4/d8/d4a/f84 [889555,65581] 0 2024-03-04T16:04:15.465 INFO:tasks.workunit.client.0.smithi116.stdout:8/606: chown d2/de/fc 19 1 2024-03-04T16:04:15.465 INFO:tasks.workunit.client.0.smithi116.stdout:6/520: creat d6/d56/d58/d72/d7b/faf x:0 0 0 2024-03-04T16:04:15.466 INFO:tasks.workunit.client.0.smithi116.stdout:0/496: dread - da/d16/d28/d40/d48/d57/f8d zero size 2024-03-04T16:04:15.468 INFO:tasks.workunit.client.0.smithi116.stdout:5/592: dwrite dd/d1f/d51/f93 [0,4194304] 0 2024-03-04T16:04:15.470 INFO:tasks.workunit.client.0.smithi116.stdout:2/648: mkdir d7/d13/df5 0 2024-03-04T16:04:15.475 INFO:tasks.workunit.client.0.smithi116.stdout:9/525: getdents d2/d26 0 2024-03-04T16:04:15.476 INFO:tasks.workunit.client.0.smithi116.stdout:0/497: dwrite da/d16/d20/d79/f7c [0,4194304] 0 2024-03-04T16:04:15.477 INFO:tasks.workunit.client.0.smithi116.stdout:8/607: creat d2/d4a/dc7/fc8 x:0 0 0 2024-03-04T16:04:15.481 INFO:tasks.workunit.client.0.smithi116.stdout:0/498: write da/d16/d19/f1d [2076713,25771] 0 2024-03-04T16:04:15.484 INFO:tasks.workunit.client.0.smithi116.stdout:2/649: dwrite d7/d13/d3e/d98/dce/f2d [0,4194304] 0 2024-03-04T16:04:15.486 INFO:tasks.workunit.client.0.smithi116.stdout:6/521: mknod d6/de/d25/d4a/cb0 0 2024-03-04T16:04:15.487 INFO:tasks.workunit.client.0.smithi116.stdout:0/499: truncate da/d16/d28/d3d/d41/f59 1528651 0 2024-03-04T16:04:15.489 INFO:tasks.workunit.client.0.smithi116.stdout:5/593: truncate dd/d1f/f69 2694130 0 2024-03-04T16:04:15.491 INFO:tasks.workunit.client.0.smithi116.stdout:0/500: fdatasync da/d16/d28/d3d/d41/f94 0 2024-03-04T16:04:15.494 INFO:tasks.workunit.client.0.smithi116.stdout:2/650: dwrite d7/dd3/fe2 [0,4194304] 0 2024-03-04T16:04:15.494 INFO:tasks.workunit.client.0.smithi116.stdout:0/501: dread - da/d16/d19/d27/d70/d92/f9a zero size 2024-03-04T16:04:15.495 INFO:tasks.workunit.client.0.smithi116.stdout:5/594: dread dd/d1f/f54 [0,4194304] 0 2024-03-04T16:04:15.495 INFO:tasks.workunit.client.0.smithi116.stdout:6/522: dread - d6/de/d29/fa0 zero size 2024-03-04T16:04:15.498 INFO:tasks.workunit.client.0.smithi116.stdout:5/595: write dd/d1f/f29 [1035920,99359] 0 2024-03-04T16:04:15.500 INFO:tasks.workunit.client.0.smithi116.stdout:5/596: dread - dd/d1f/fb2 zero size 2024-03-04T16:04:15.500 INFO:tasks.workunit.client.0.smithi116.stdout:7/564: getdents d3/d4 0 2024-03-04T16:04:15.501 INFO:tasks.workunit.client.0.smithi116.stdout:3/624: write dc/f4b [2447768,107853] 0 2024-03-04T16:04:15.501 INFO:tasks.workunit.client.0.smithi116.stdout:7/565: dread - d3/d4/d11/d61/f81 zero size 2024-03-04T16:04:15.502 INFO:tasks.workunit.client.0.smithi116.stdout:9/526: creat d2/d19/d27/f9d x:0 0 0 2024-03-04T16:04:15.502 INFO:tasks.workunit.client.0.smithi116.stdout:3/625: fdatasync dc/d23/d84/d7f/d93/db6/fbb 0 2024-03-04T16:04:15.504 INFO:tasks.workunit.client.0.smithi116.stdout:2/651: creat d7/d81/dc2/ff6 x:0 0 0 2024-03-04T16:04:15.504 INFO:tasks.workunit.client.0.smithi116.stdout:9/527: write d2/d19/d27/f9d [670167,119090] 0 2024-03-04T16:04:15.506 INFO:tasks.workunit.client.0.smithi116.stdout:2/652: dread d7/d25/f62 [0,4194304] 0 2024-03-04T16:04:15.508 INFO:tasks.workunit.client.0.smithi116.stdout:2/653: truncate d7/d13/d3e/d98/fba 663448 0 2024-03-04T16:04:15.510 INFO:tasks.workunit.client.0.smithi116.stdout:8/608: rename d2/d13/d3c/d46/d4f/l57 to d2/d13/d3c/lc9 0 2024-03-04T16:04:15.515 INFO:tasks.workunit.client.0.smithi116.stdout:3/626: fdatasync dc/df/f7e 0 2024-03-04T16:04:15.520 INFO:tasks.workunit.client.0.smithi116.stdout:5/597: rename f4 to dd/d1f/d51/fbf 0 2024-03-04T16:04:15.524 INFO:tasks.workunit.client.0.smithi116.stdout:9/528: mknod d2/d19/c9e 0 2024-03-04T16:04:15.525 INFO:tasks.workunit.client.0.smithi116.stdout:3/627: dread dc/d34/f9a [0,4194304] 0 2024-03-04T16:04:15.528 INFO:tasks.workunit.client.0.smithi116.stdout:8/609: truncate d2/f60 5038367 0 2024-03-04T16:04:15.531 INFO:tasks.workunit.client.0.smithi116.stdout:3/628: dwrite dc/df/d14/f58 [0,4194304] 0 2024-03-04T16:04:15.532 INFO:tasks.workunit.client.0.smithi116.stdout:0/502: dread da/d16/d19/d27/f55 [0,4194304] 0 2024-03-04T16:04:15.534 INFO:tasks.workunit.client.0.smithi116.stdout:3/629: write dc/d34/d44/d6a/f6b [829306,38007] 0 2024-03-04T16:04:15.534 INFO:tasks.workunit.client.0.smithi116.stdout:9/529: symlink d2/d19/d27/d3b/d4a/d6a/l9f 0 2024-03-04T16:04:15.537 INFO:tasks.workunit.client.0.smithi116.stdout:6/523: dwrite d6/de/d25/d36/d4e/d50/f5f [4194304,4194304] 0 2024-03-04T16:04:15.537 INFO:tasks.workunit.client.0.smithi116.stdout:0/503: truncate da/d16/d28/d3d/d41/f7d 254255 0 2024-03-04T16:04:15.542 INFO:tasks.workunit.client.0.smithi116.stdout:8/610: rmdir d2/de 39 2024-03-04T16:04:15.542 INFO:tasks.workunit.client.0.smithi116.stdout:9/530: dwrite d2/d19/f9c [0,4194304] 0 2024-03-04T16:04:15.547 INFO:tasks.workunit.client.0.smithi116.stdout:7/566: getdents d3/d4/d8/d52 0 2024-03-04T16:04:15.551 INFO:tasks.workunit.client.0.smithi116.stdout:3/630: readlink dc/d23/d84/d7f/d93/db6/dc5/lc6 0 2024-03-04T16:04:15.552 INFO:tasks.workunit.client.0.smithi116.stdout:2/654: truncate d7/d13/f56 2388656 0 2024-03-04T16:04:15.554 INFO:tasks.workunit.client.0.smithi116.stdout:3/631: chown dc/l24 34301638 1 2024-03-04T16:04:15.555 INFO:tasks.workunit.client.0.smithi116.stdout:7/567: dwrite d3/d4/d11/d61/f81 [0,4194304] 0 2024-03-04T16:04:15.556 INFO:tasks.workunit.client.0.smithi116.stdout:6/524: chown d6/de/d25/d4a/d63/c9b 147944890 1 2024-03-04T16:04:15.557 INFO:tasks.workunit.client.0.smithi116.stdout:1/478: sync 2024-03-04T16:04:15.557 INFO:tasks.workunit.client.0.smithi116.stdout:0/504: creat da/d16/d20/d53/f9e x:0 0 0 2024-03-04T16:04:15.558 INFO:tasks.workunit.client.0.smithi116.stdout:2/655: truncate d7/d13/d3e/d98/dce/d47/d64/f86 4876253 0 2024-03-04T16:04:15.562 INFO:tasks.workunit.client.0.smithi116.stdout:6/525: chown d6/de/d25/d36/l32 0 1 2024-03-04T16:04:15.568 INFO:tasks.workunit.client.0.smithi116.stdout:7/568: rmdir d3/d4/d8/d52/d5e 39 2024-03-04T16:04:15.568 INFO:tasks.workunit.client.0.smithi116.stdout:2/656: creat d7/d25/ff7 x:0 0 0 2024-03-04T16:04:15.569 INFO:tasks.workunit.client.0.smithi116.stdout:2/657: chown d7/d15/d5e 675956 1 2024-03-04T16:04:15.569 INFO:tasks.workunit.client.0.smithi116.stdout:5/598: getdents dd/d1f/d72 0 2024-03-04T16:04:15.569 INFO:tasks.workunit.client.0.smithi116.stdout:9/531: dread d2/d19/d27/d3b/d4a/f7b [0,4194304] 0 2024-03-04T16:04:15.571 INFO:tasks.workunit.client.0.smithi116.stdout:6/526: mknod d6/de/d1e/d80/cb1 0 2024-03-04T16:04:15.571 INFO:tasks.workunit.client.0.smithi116.stdout:1/479: dwrite d5/d9/d21/d27/d32/d35/d55/d75/d91/f9e [0,4194304] 0 2024-03-04T16:04:15.572 INFO:tasks.workunit.client.0.smithi116.stdout:6/527: write d6/d56/f4c [4107066,129172] 0 2024-03-04T16:04:15.575 INFO:tasks.workunit.client.0.smithi116.stdout:8/611: creat d2/de/d16/d2b/d37/fca x:0 0 0 2024-03-04T16:04:15.577 INFO:tasks.workunit.client.0.smithi116.stdout:2/658: mknod d7/d13/d3e/d71/cf8 0 2024-03-04T16:04:15.577 INFO:tasks.workunit.client.0.smithi116.stdout:9/532: creat d2/d26/fa0 x:0 0 0 2024-03-04T16:04:15.584 INFO:tasks.workunit.client.0.smithi116.stdout:5/599: dwrite dd/d1f/d55/d73/f3f [4194304,4194304] 0 2024-03-04T16:04:15.587 INFO:tasks.workunit.client.0.smithi116.stdout:7/569: truncate d3/d4/d8/f1c 3247444 0 2024-03-04T16:04:15.587 INFO:tasks.workunit.client.0.smithi116.stdout:7/570: chown d3/d4/f92 0 1 2024-03-04T16:04:15.595 INFO:tasks.workunit.client.0.smithi116.stdout:0/505: dread da/d16/f4c [4194304,4194304] 0 2024-03-04T16:04:15.595 INFO:tasks.workunit.client.0.smithi116.stdout:9/533: rename d2/d19/c9e to d2/d33/d65/d7e/ca1 0 2024-03-04T16:04:15.596 INFO:tasks.workunit.client.0.smithi116.stdout:3/632: link dc/df/c66 dc/d34/d44/d6a/db5/cd2 0 2024-03-04T16:04:15.599 INFO:tasks.workunit.client.0.smithi116.stdout:5/600: mkdir dd/d1f/d55/d73/d2d/dc0 0 2024-03-04T16:04:15.603 INFO:tasks.workunit.client.0.smithi116.stdout:5/601: dwrite dd/d1f/f5f [0,4194304] 0 2024-03-04T16:04:15.604 INFO:tasks.workunit.client.0.smithi116.stdout:5/602: stat dd/d1f/d55/f77 0 2024-03-04T16:04:15.606 INFO:tasks.workunit.client.0.smithi116.stdout:9/534: mknod d2/d19/d5f/ca2 0 2024-03-04T16:04:15.606 INFO:tasks.workunit.client.0.smithi116.stdout:3/633: mkdir dc/dd3 0 2024-03-04T16:04:15.607 INFO:tasks.workunit.client.0.smithi116.stdout:9/535: dread - d2/d49/d3c/f93 zero size 2024-03-04T16:04:15.608 INFO:tasks.workunit.client.0.smithi116.stdout:1/480: dread d5/f5a [0,4194304] 0 2024-03-04T16:04:15.610 INFO:tasks.workunit.client.0.smithi116.stdout:7/571: mkdir d3/d4/d8/d52/db4 0 2024-03-04T16:04:15.616 INFO:tasks.workunit.client.0.smithi116.stdout:3/634: rename dc/d23/d84/f5e to dc/df/d47/d91/fd4 0 2024-03-04T16:04:15.620 INFO:tasks.workunit.client.0.smithi116.stdout:7/572: symlink d3/d4/d7e/lb5 0 2024-03-04T16:04:15.623 INFO:tasks.workunit.client.0.smithi116.stdout:2/659: write d7/d15/d5e/f67 [858871,130287] 0 2024-03-04T16:04:15.624 INFO:tasks.workunit.client.0.smithi116.stdout:6/528: dwrite d6/f21 [0,4194304] 0 2024-03-04T16:04:15.625 INFO:tasks.workunit.client.0.smithi116.stdout:6/529: fsync d6/de/d1e/f88 0 2024-03-04T16:04:15.628 INFO:tasks.workunit.client.0.smithi116.stdout:9/536: rename d2/d26/f39 to d2/d19/d5f/fa3 0 2024-03-04T16:04:15.629 INFO:tasks.workunit.client.0.smithi116.stdout:9/537: read d2/d19/d27/d3b/d4a/d5b/f43 [295006,49664] 0 2024-03-04T16:04:15.637 INFO:tasks.workunit.client.0.smithi116.stdout:8/612: link d2/d13/d3c/d77/d97/cb4 d2/de/d16/ccb 0 2024-03-04T16:04:15.638 INFO:tasks.workunit.client.0.smithi116.stdout:2/660: mknod d7/d13/d3e/ded/cf9 0 2024-03-04T16:04:15.644 INFO:tasks.workunit.client.0.smithi116.stdout:7/573: rename d3/d4/d11/d61 to d3/d4/d8/d4a/db3/db6 0 2024-03-04T16:04:15.644 INFO:tasks.workunit.client.0.smithi116.stdout:2/661: dwrite d7/d25/d41/fcf [0,4194304] 0 2024-03-04T16:04:15.645 INFO:tasks.workunit.client.0.smithi116.stdout:3/635: symlink dc/df/d47/ld5 0 2024-03-04T16:04:15.646 INFO:tasks.workunit.client.0.smithi116.stdout:7/574: read - d3/d4/d34/d5d/f8b zero size 2024-03-04T16:04:15.646 INFO:tasks.workunit.client.0.smithi116.stdout:5/603: dwrite dd/f22 [0,4194304] 0 2024-03-04T16:04:15.648 INFO:tasks.workunit.client.0.smithi116.stdout:7/575: write d3/d95/f6f [33216,24349] 0 2024-03-04T16:04:15.653 INFO:tasks.workunit.client.0.smithi116.stdout:7/576: write d3/d4/d8/d52/f54 [1384312,30088] 0 2024-03-04T16:04:15.654 INFO:tasks.workunit.client.0.smithi116.stdout:7/577: chown d3/d95/f6f 847923388 1 2024-03-04T16:04:15.656 INFO:tasks.workunit.client.0.smithi116.stdout:5/604: dwrite dd/d1f/d55/d73/d2d/dae/fbe [0,4194304] 0 2024-03-04T16:04:15.661 INFO:tasks.workunit.client.0.smithi116.stdout:6/530: creat d6/de/d15/fb2 x:0 0 0 2024-03-04T16:04:15.663 INFO:tasks.workunit.client.0.smithi116.stdout:7/578: dwrite d3/d4/d8/d4a/d65/f80 [0,4194304] 0 2024-03-04T16:04:15.671 INFO:tasks.workunit.client.0.smithi116.stdout:6/531: dwrite d6/f21 [4194304,4194304] 0 2024-03-04T16:04:15.673 INFO:tasks.workunit.client.0.smithi116.stdout:1/481: getdents d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42 0 2024-03-04T16:04:15.673 INFO:tasks.workunit.client.0.smithi116.stdout:3/636: dwrite dc/d23/d84/d7c/f65 [0,4194304] 0 2024-03-04T16:04:15.674 INFO:tasks.workunit.client.0.smithi116.stdout:1/482: stat d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/f62 0 2024-03-04T16:04:15.680 INFO:tasks.workunit.client.0.smithi116.stdout:4/637: sync 2024-03-04T16:04:15.682 INFO:tasks.workunit.client.0.smithi116.stdout:8/613: mknod d2/de/d16/d78/d29/d7a/db1/dbe/ccc 0 2024-03-04T16:04:15.682 INFO:tasks.workunit.client.0.smithi116.stdout:5/605: mknod dd/d1f/d55/d73/d2d/cc1 0 2024-03-04T16:04:15.683 INFO:tasks.workunit.client.0.smithi116.stdout:7/579: creat d3/d4/d11/fb7 x:0 0 0 2024-03-04T16:04:15.685 INFO:tasks.workunit.client.0.smithi116.stdout:7/580: write d3/d4/d34/d5d/fa7 [881856,77685] 0 2024-03-04T16:04:15.689 INFO:tasks.workunit.client.0.smithi116.stdout:3/637: symlink dc/d23/d84/d7f/d93/ld6 0 2024-03-04T16:04:15.690 INFO:tasks.workunit.client.0.smithi116.stdout:6/532: stat d6/de/f4d 0 2024-03-04T16:04:15.690 INFO:tasks.workunit.client.0.smithi116.stdout:8/614: dwrite d2/d13/d5e/fc6 [0,4194304] 0 2024-03-04T16:04:15.691 INFO:tasks.workunit.client.0.smithi116.stdout:4/638: rename d1/d3/d15/d20/d1c/d6f/fcd to d1/d3/dd/d8a/ddc/fe1 0 2024-03-04T16:04:15.692 INFO:tasks.workunit.client.0.smithi116.stdout:3/638: chown dc/df/d47/f61 263066118 1 2024-03-04T16:04:15.693 INFO:tasks.workunit.client.0.smithi116.stdout:6/533: truncate d6/de/d15/f26 4794048 0 2024-03-04T16:04:15.696 INFO:tasks.workunit.client.0.smithi116.stdout:6/534: dread - d6/de/d29/fa0 zero size 2024-03-04T16:04:15.696 INFO:tasks.workunit.client.0.smithi116.stdout:4/639: chown d1/d3/d15/d20/d1c/d62/d81/d97/cab 2026 1 2024-03-04T16:04:15.697 INFO:tasks.workunit.client.0.smithi116.stdout:2/662: creat d7/d13/d3e/d98/dce/ffa x:0 0 0 2024-03-04T16:04:15.697 INFO:tasks.workunit.client.0.smithi116.stdout:1/483: mknod d5/c9f 0 2024-03-04T16:04:15.698 INFO:tasks.workunit.client.0.smithi116.stdout:7/581: write d3/f59 [651624,3242] 0 2024-03-04T16:04:15.699 INFO:tasks.workunit.client.0.smithi116.stdout:8/615: read d2/d13/d3c/d46/d4f/f70 [78875,62587] 0 2024-03-04T16:04:15.701 INFO:tasks.workunit.client.0.smithi116.stdout:3/639: fdatasync dc/d23/d84/d7c/f62 0 2024-03-04T16:04:15.701 INFO:tasks.workunit.client.0.smithi116.stdout:2/663: chown d7/dd3/de1/le6 2 1 2024-03-04T16:04:15.705 INFO:tasks.workunit.client.0.smithi116.stdout:7/582: rmdir d3/d4/d8/d1a 39 2024-03-04T16:04:15.706 INFO:tasks.workunit.client.0.smithi116.stdout:5/606: dwrite dd/d1f/fb5 [4194304,4194304] 0 2024-03-04T16:04:15.707 INFO:tasks.workunit.client.0.smithi116.stdout:7/583: readlink d3/d4/d34/l62 0 2024-03-04T16:04:15.709 INFO:tasks.workunit.client.0.smithi116.stdout:5/607: stat dd/d1f/d55/d73/la8 0 2024-03-04T16:04:15.712 INFO:tasks.workunit.client.0.smithi116.stdout:1/484: dread d5/d9/d21/d27/d32/d35/d55/d58/d7e/f69 [0,4194304] 0 2024-03-04T16:04:15.713 INFO:tasks.workunit.client.0.smithi116.stdout:3/640: rename dc/df/d14/fb7 to dc/df/d47/d91/fd7 0 2024-03-04T16:04:15.715 INFO:tasks.workunit.client.0.smithi116.stdout:1/485: dread - d5/d9/d46/d50/f8e zero size 2024-03-04T16:04:15.715 INFO:tasks.workunit.client.0.smithi116.stdout:3/641: read dc/d23/d84/f76 [609786,63273] 0 2024-03-04T16:04:15.716 INFO:tasks.workunit.client.0.smithi116.stdout:7/584: stat d3/d4/d8/d52/d5e 0 2024-03-04T16:04:15.718 INFO:tasks.workunit.client.0.smithi116.stdout:7/585: fdatasync d3/f90 0 2024-03-04T16:04:15.718 INFO:tasks.workunit.client.0.smithi116.stdout:4/640: creat d1/d3/d40/d52/d84/d88/fe2 x:0 0 0 2024-03-04T16:04:15.718 INFO:tasks.workunit.client.0.smithi116.stdout:5/608: dwrite dd/d45/f95 [0,4194304] 0 2024-03-04T16:04:15.723 INFO:tasks.workunit.client.0.smithi116.stdout:3/642: read dc/d23/d84/d7f/d93/fb1 [451988,121942] 0 2024-03-04T16:04:15.723 INFO:tasks.workunit.client.0.smithi116.stdout:4/641: write d1/d3/d15/d20/d1c/d62/d81/fb0 [202919,26198] 0 2024-03-04T16:04:15.723 INFO:tasks.workunit.client.0.smithi116.stdout:5/609: truncate dd/d1f/d55/d73/d2d/d7c/f9d 653787 0 2024-03-04T16:04:15.723 INFO:tasks.workunit.client.0.smithi116.stdout:8/616: rename d2/d13/d3c/d77/d97/d42/l89 to d2/d13/d5e/lcd 0 2024-03-04T16:04:15.724 INFO:tasks.workunit.client.0.smithi116.stdout:5/610: fsync dd/fe 0 2024-03-04T16:04:15.730 INFO:tasks.workunit.client.0.smithi116.stdout:5/611: dwrite dd/d6d/f80 [0,4194304] 0 2024-03-04T16:04:15.730 INFO:tasks.workunit.client.0.smithi116.stdout:1/486: fdatasync d5/d9/d21/d27/d32/f83 0 2024-03-04T16:04:15.737 INFO:tasks.workunit.client.0.smithi116.stdout:7/586: dread d3/f18 [0,4194304] 0 2024-03-04T16:04:15.737 INFO:tasks.workunit.client.0.smithi116.stdout:7/587: dread - d3/f55 zero size 2024-03-04T16:04:15.740 INFO:tasks.workunit.client.0.smithi116.stdout:4/642: fsync d1/f58 0 2024-03-04T16:04:15.746 INFO:tasks.workunit.client.0.smithi116.stdout:8/617: symlink d2/de/d16/d78/d29/d7a/db1/dbe/lce 0 2024-03-04T16:04:15.747 INFO:tasks.workunit.client.0.smithi116.stdout:8/618: write d2/d13/d3c/d46/d6e/fbb [394975,100389] 0 2024-03-04T16:04:15.748 INFO:tasks.workunit.client.0.smithi116.stdout:8/619: truncate d2/de/fa 4668805 0 2024-03-04T16:04:15.750 INFO:tasks.workunit.client.0.smithi116.stdout:6/535: write d6/de/d1e/f3f [134295,12736] 0 2024-03-04T16:04:15.750 INFO:tasks.workunit.client.0.smithi116.stdout:7/588: dread d3/d4/d8/d52/f54 [0,4194304] 0 2024-03-04T16:04:15.751 INFO:tasks.workunit.client.0.smithi116.stdout:2/664: link d7/ca d7/d13/d3e/d71/dad/cfb 0 2024-03-04T16:04:15.752 INFO:tasks.workunit.client.0.smithi116.stdout:2/665: write d7/d13/d3e/d78/fb1 [1161654,74462] 0 2024-03-04T16:04:15.753 INFO:tasks.workunit.client.0.smithi116.stdout:5/612: unlink dd/d1f/d5c/d5e/f67 0 2024-03-04T16:04:15.760 INFO:tasks.workunit.client.0.smithi116.stdout:1/487: rename d5/d9/d21/d27/d32/d35/d55/f82 to d5/d9/d46/fa0 0 2024-03-04T16:04:15.767 INFO:tasks.workunit.client.0.smithi116.stdout:2/666: dread d7/d15/d35/f70 [0,4194304] 0 2024-03-04T16:04:15.771 INFO:tasks.workunit.client.0.smithi116.stdout:8/620: mkdir d2/d13/d3c/d77/d97/d5a/dcf 0 2024-03-04T16:04:15.772 INFO:tasks.workunit.client.0.smithi116.stdout:8/621: dread - d2/d13/d5e/fc0 zero size 2024-03-04T16:04:15.777 INFO:tasks.workunit.client.0.smithi116.stdout:6/536: mkdir d6/d9e/da5/db3 0 2024-03-04T16:04:15.777 INFO:tasks.workunit.client.0.smithi116.stdout:6/537: readlink d6/de/d29/l3c 0 2024-03-04T16:04:15.783 INFO:tasks.workunit.client.0.smithi116.stdout:5/613: mkdir dd/d1f/d5c/d5e/dc2 0 2024-03-04T16:04:15.787 INFO:tasks.workunit.client.0.smithi116.stdout:1/488: rmdir d5/d9 39 2024-03-04T16:04:15.800 INFO:tasks.workunit.client.0.smithi116.stdout:3/643: dwrite dc/d23/d84/d7c/f69 [0,4194304] 0 2024-03-04T16:04:15.804 INFO:tasks.workunit.client.0.smithi116.stdout:7/589: dwrite d3/f44 [0,4194304] 0 2024-03-04T16:04:15.806 INFO:tasks.workunit.client.0.smithi116.stdout:7/590: chown d3/d4/d8/l7d 1871832921 1 2024-03-04T16:04:15.809 INFO:tasks.workunit.client.0.smithi116.stdout:5/614: mkdir dd/d1f/dc3 0 2024-03-04T16:04:15.809 INFO:tasks.workunit.client.0.smithi116.stdout:4/643: write d1/d3/de/f23 [4186772,5289] 0 2024-03-04T16:04:15.811 INFO:tasks.workunit.client.0.smithi116.stdout:1/489: readlink d5/d9/d21/d27/d32/d49/d4a/l6e 0 2024-03-04T16:04:15.812 INFO:tasks.workunit.client.0.smithi116.stdout:4/644: write d1/d3/d40/d52/d84/d88/fe2 [415657,20121] 0 2024-03-04T16:04:15.816 INFO:tasks.workunit.client.0.smithi116.stdout:2/667: rename d7/d25/d41/fcf to d7/d15/d5e/d9d/ffc 0 2024-03-04T16:04:15.817 INFO:tasks.workunit.client.0.smithi116.stdout:6/538: write d6/de/d25/d36/f99 [1158483,31097] 0 2024-03-04T16:04:15.818 INFO:tasks.workunit.client.0.smithi116.stdout:1/490: dread d5/d9/d21/d27/d32/d35/f43 [0,4194304] 0 2024-03-04T16:04:15.818 INFO:tasks.workunit.client.0.smithi116.stdout:1/491: chown d5/d9/fb 124907 1 2024-03-04T16:04:15.820 INFO:tasks.workunit.client.0.smithi116.stdout:1/492: dread - d5/d9/d21/d27/d32/f83 zero size 2024-03-04T16:04:15.820 INFO:tasks.workunit.client.0.smithi116.stdout:3/644: write dc/df/f74 [1065454,18671] 0 2024-03-04T16:04:15.823 INFO:tasks.workunit.client.0.smithi116.stdout:9/538: sync 2024-03-04T16:04:15.823 INFO:tasks.workunit.client.0.smithi116.stdout:0/506: sync 2024-03-04T16:04:15.823 INFO:tasks.workunit.client.0.smithi116.stdout:4/645: unlink d1/d3/c9e 0 2024-03-04T16:04:15.826 INFO:tasks.workunit.client.0.smithi116.stdout:6/539: dwrite d6/d56/d58/d72/d7b/faf [0,4194304] 0 2024-03-04T16:04:15.831 INFO:tasks.workunit.client.0.smithi116.stdout:8/622: rename d2/d38/d41/c4b to d2/de/d16/d2b/cd0 0 2024-03-04T16:04:15.832 INFO:tasks.workunit.client.0.smithi116.stdout:5/615: mknod dd/d1f/d55/d73/d2d/dc0/cc4 0 2024-03-04T16:04:15.834 INFO:tasks.workunit.client.0.smithi116.stdout:8/623: truncate d2/de/d16/d2b/d37/fca 289847 0 2024-03-04T16:04:15.835 INFO:tasks.workunit.client.0.smithi116.stdout:3/645: write dc/d23/f85 [386201,67289] 0 2024-03-04T16:04:15.838 INFO:tasks.workunit.client.0.smithi116.stdout:4/646: truncate d1/f58 531095 0 2024-03-04T16:04:15.838 INFO:tasks.workunit.client.0.smithi116.stdout:7/591: creat d3/d4/d8/d52/fb8 x:0 0 0 2024-03-04T16:04:15.839 INFO:tasks.workunit.client.0.smithi116.stdout:1/493: mkdir d5/da1 0 2024-03-04T16:04:15.842 INFO:tasks.workunit.client.0.smithi116.stdout:6/540: rename d6/de/d25/d36/d4e/d50 to d6/de/d84/db4 0 2024-03-04T16:04:15.844 INFO:tasks.workunit.client.0.smithi116.stdout:1/494: dwrite d5/d9/d21/d84/f90 [0,4194304] 0 2024-03-04T16:04:15.846 INFO:tasks.workunit.client.0.smithi116.stdout:1/495: dread - d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/f88 zero size 2024-03-04T16:04:15.847 INFO:tasks.workunit.client.0.smithi116.stdout:0/507: mkdir da/d16/d19/d27/d82/d9f 0 2024-03-04T16:04:15.850 INFO:tasks.workunit.client.0.smithi116.stdout:7/592: mkdir d3/d4/d34/d5d/db9 0 2024-03-04T16:04:15.852 INFO:tasks.workunit.client.0.smithi116.stdout:4/647: symlink d1/d3/d15/d20/d1c/d62/d81/da6/le3 0 2024-03-04T16:04:15.859 INFO:tasks.workunit.client.0.smithi116.stdout:8/624: mkdir d2/de/d16/dd1 0 2024-03-04T16:04:15.860 INFO:tasks.workunit.client.0.smithi116.stdout:8/625: dread - d2/d13/d3c/d46/fb0 zero size 2024-03-04T16:04:15.861 INFO:tasks.workunit.client.0.smithi116.stdout:2/668: getdents d7/d13/d3e/d98/dce/d47/d64/da5 0 2024-03-04T16:04:15.861 INFO:tasks.workunit.client.0.smithi116.stdout:2/669: dread - d7/d15/d5e/fde zero size 2024-03-04T16:04:15.867 INFO:tasks.workunit.client.0.smithi116.stdout:1/496: mknod d5/d9/d21/d27/d32/d35/ca2 0 2024-03-04T16:04:15.868 INFO:tasks.workunit.client.0.smithi116.stdout:2/670: mkdir d7/d13/d3e/ded/dfd 0 2024-03-04T16:04:15.870 INFO:tasks.workunit.client.0.smithi116.stdout:7/593: symlink d3/lba 0 2024-03-04T16:04:15.871 INFO:tasks.workunit.client.0.smithi116.stdout:5/616: write dd/d1f/d55/d73/d2d/f42 [450143,93616] 0 2024-03-04T16:04:15.872 INFO:tasks.workunit.client.0.smithi116.stdout:5/617: chown dd/d1f/d55/d73/d2d/d59/f81 127513 1 2024-03-04T16:04:15.874 INFO:tasks.workunit.client.0.smithi116.stdout:9/539: rename d2/d19/d27/d3b/d4a/l8c to d2/d33/la4 0 2024-03-04T16:04:15.879 INFO:tasks.workunit.client.0.smithi116.stdout:7/594: dwrite d3/d4/d34/d5d/f8b [0,4194304] 0 2024-03-04T16:04:15.880 INFO:tasks.workunit.client.0.smithi116.stdout:7/595: chown d3/d4/d8/c4b 1 1 2024-03-04T16:04:15.881 INFO:tasks.workunit.client.0.smithi116.stdout:6/541: getdents d6/de/d25 0 2024-03-04T16:04:15.881 INFO:tasks.workunit.client.0.smithi116.stdout:7/596: chown d3/fd 100687 1 2024-03-04T16:04:15.885 INFO:tasks.workunit.client.0.smithi116.stdout:7/597: truncate d3/d4/d8/f71 323429 0 2024-03-04T16:04:15.887 INFO:tasks.workunit.client.0.smithi116.stdout:1/497: dread d5/d9/d21/d27/d32/d35/f53 [0,4194304] 0 2024-03-04T16:04:15.890 INFO:tasks.workunit.client.0.smithi116.stdout:5/618: dread dd/f13 [0,4194304] 0 2024-03-04T16:04:15.893 INFO:tasks.workunit.client.0.smithi116.stdout:5/619: write dd/d1f/d55/d73/d2d/d31/f3c [527895,101576] 0 2024-03-04T16:04:15.893 INFO:tasks.workunit.client.0.smithi116.stdout:7/598: dwrite d3/d4/d8/f45 [0,4194304] 0 2024-03-04T16:04:15.895 INFO:tasks.workunit.client.0.smithi116.stdout:9/540: truncate d2/f15 1010733 0 2024-03-04T16:04:15.899 INFO:tasks.workunit.client.0.smithi116.stdout:0/508: rename da/d16/l23 to da/d16/d28/la0 0 2024-03-04T16:04:15.901 INFO:tasks.workunit.client.0.smithi116.stdout:6/542: link d6/de/d25/d4a/d63/d8c/fa3 d6/d56/d58/d72/d7b/fb5 0 2024-03-04T16:04:15.902 INFO:tasks.workunit.client.0.smithi116.stdout:2/671: dwrite d7/d15/d35/f3b [0,4194304] 0 2024-03-04T16:04:15.903 INFO:tasks.workunit.client.0.smithi116.stdout:6/543: write d6/d56/d58/d72/d7b/faf [1515534,31730] 0 2024-03-04T16:04:15.904 INFO:tasks.workunit.client.0.smithi116.stdout:5/620: rmdir dd/d1f/d55/d73/d2d/d59/d71 39 2024-03-04T16:04:15.911 INFO:tasks.workunit.client.0.smithi116.stdout:8/626: rename d2/de/f10 to d2/de/d16/d78/d29/d7a/fd2 0 2024-03-04T16:04:15.913 INFO:tasks.workunit.client.0.smithi116.stdout:0/509: mkdir da/d16/d19/d27/d70/da1 0 2024-03-04T16:04:15.925 INFO:tasks.workunit.client.0.smithi116.stdout:9/541: fsync d2/f1a 0 2024-03-04T16:04:15.926 INFO:tasks.workunit.client.0.smithi116.stdout:9/542: chown d2/d33/d65/d6c 34831 1 2024-03-04T16:04:15.927 INFO:tasks.workunit.client.0.smithi116.stdout:6/544: dread d6/d56/f8b [0,4194304] 0 2024-03-04T16:04:15.928 INFO:tasks.workunit.client.0.smithi116.stdout:2/672: dread d7/d13/d3e/ded/d6c/f90 [0,4194304] 0 2024-03-04T16:04:15.929 INFO:tasks.workunit.client.0.smithi116.stdout:5/621: dread dd/d45/f49 [0,4194304] 0 2024-03-04T16:04:15.932 INFO:tasks.workunit.client.0.smithi116.stdout:0/510: unlink da/d16/f84 0 2024-03-04T16:04:15.933 INFO:tasks.workunit.client.0.smithi116.stdout:7/599: creat d3/fbb x:0 0 0 2024-03-04T16:04:15.933 INFO:tasks.workunit.client.0.smithi116.stdout:6/545: dwrite d6/de/d15/f26 [4194304,4194304] 0 2024-03-04T16:04:15.935 INFO:tasks.workunit.client.0.smithi116.stdout:1/498: truncate d5/d9/d21/d27/d32/d49/f6d 2185769 0 2024-03-04T16:04:15.939 INFO:tasks.workunit.client.0.smithi116.stdout:7/600: dwrite d3/d4/d8/d4a/d86/f8d [0,4194304] 0 2024-03-04T16:04:15.940 INFO:tasks.workunit.client.0.smithi116.stdout:7/601: chown d3/fbb 5972625 1 2024-03-04T16:04:15.941 INFO:tasks.workunit.client.0.smithi116.stdout:7/602: stat d3/d4/d8/d4a/d88/c99 0 2024-03-04T16:04:15.941 INFO:tasks.workunit.client.0.smithi116.stdout:8/627: mknod d2/d13/d3c/d77/d97/d5a/dcf/cd3 0 2024-03-04T16:04:15.943 INFO:tasks.workunit.client.0.smithi116.stdout:2/673: creat d7/d13/d3e/d98/dce/d47/d64/da5/ffe x:0 0 0 2024-03-04T16:04:15.943 INFO:tasks.workunit.client.0.smithi116.stdout:8/628: chown d2/d13/d3c/d77/d97/d5a/f91 17241 1 2024-03-04T16:04:15.944 INFO:tasks.workunit.client.0.smithi116.stdout:2/674: chown d7/d13/d3e/d98/dce/d47/d64/ldd 11906 1 2024-03-04T16:04:15.945 INFO:tasks.workunit.client.0.smithi116.stdout:8/629: read d2/d13/d3c/d77/d97/d5a/f9d [921620,45786] 0 2024-03-04T16:04:15.945 INFO:tasks.workunit.client.0.smithi116.stdout:6/546: creat d6/d56/d58/d72/d7b/fb6 x:0 0 0 2024-03-04T16:04:15.947 INFO:tasks.workunit.client.0.smithi116.stdout:8/630: write d2/d13/d3c/f83 [1235016,38747] 0 2024-03-04T16:04:15.948 INFO:tasks.workunit.client.0.smithi116.stdout:9/543: symlink d2/d33/d90/la5 0 2024-03-04T16:04:15.949 INFO:tasks.workunit.client.0.smithi116.stdout:8/631: read - d2/d13/d3c/d77/d97/fc2 zero size 2024-03-04T16:04:15.949 INFO:tasks.workunit.client.0.smithi116.stdout:2/675: read d7/d13/d3e/d98/fba [106488,106058] 0 2024-03-04T16:04:15.952 INFO:tasks.workunit.client.0.smithi116.stdout:0/511: dread da/d16/d19/d27/f78 [0,4194304] 0 2024-03-04T16:04:15.954 INFO:tasks.workunit.client.0.smithi116.stdout:8/632: read d2/d13/d3c/f80 [3930685,14149] 0 2024-03-04T16:04:15.954 INFO:tasks.workunit.client.0.smithi116.stdout:2/676: read d7/d25/d41/f9f [3688706,2813] 0 2024-03-04T16:04:15.954 INFO:tasks.workunit.client.0.smithi116.stdout:7/603: creat d3/d4/d8/d4a/d88/fbc x:0 0 0 2024-03-04T16:04:15.955 INFO:tasks.workunit.client.0.smithi116.stdout:5/622: symlink dd/d1f/d55/d73/d2d/d59/lc5 0 2024-03-04T16:04:15.955 INFO:tasks.workunit.client.0.smithi116.stdout:6/547: dwrite d6/de/d1e/d80/f89 [0,4194304] 0 2024-03-04T16:04:15.957 INFO:tasks.workunit.client.0.smithi116.stdout:5/623: chown dd/d1f/d5c/d5e 0 1 2024-03-04T16:04:15.957 INFO:tasks.workunit.client.0.smithi116.stdout:2/677: chown d7/d13/d3e/d78/f7e 4423197 1 2024-03-04T16:04:15.957 INFO:tasks.workunit.client.0.smithi116.stdout:8/633: write d2/d13/d3c/d77/d97/d42/f5d [4371970,119557] 0 2024-03-04T16:04:15.960 INFO:tasks.workunit.client.0.smithi116.stdout:5/624: write dd/d1f/d55/f58 [726859,29247] 0 2024-03-04T16:04:15.960 INFO:tasks.workunit.client.0.smithi116.stdout:5/625: dread - dd/d45/fb6 zero size 2024-03-04T16:04:15.960 INFO:tasks.workunit.client.0.smithi116.stdout:2/678: chown d7/d13/d3e/d98/dce/d47/f49 6 1 2024-03-04T16:04:15.962 INFO:tasks.workunit.client.0.smithi116.stdout:9/544: mknod d2/d33/d65/d6c/ca6 0 2024-03-04T16:04:15.967 INFO:tasks.workunit.client.0.smithi116.stdout:9/545: dwrite d2/d49/d3c/f8f [0,4194304] 0 2024-03-04T16:04:15.968 INFO:tasks.workunit.client.0.smithi116.stdout:9/546: stat d2/d33/d65/l86 0 2024-03-04T16:04:15.975 INFO:tasks.workunit.client.0.smithi116.stdout:7/604: read d3/d4/d8/d1a/f42 [1055030,26227] 0 2024-03-04T16:04:15.982 INFO:tasks.workunit.client.0.smithi116.stdout:5/626: mkdir dd/d6d/dc6 0 2024-03-04T16:04:15.989 INFO:tasks.workunit.client.0.smithi116.stdout:9/547: unlink d2/d19/d27/d3b/l96 0 2024-03-04T16:04:15.990 INFO:tasks.workunit.client.0.smithi116.stdout:0/512: symlink da/d16/d28/d40/la2 0 2024-03-04T16:04:15.991 INFO:tasks.workunit.client.0.smithi116.stdout:0/513: dread - da/d16/d20/d42/f88 zero size 2024-03-04T16:04:15.991 INFO:tasks.workunit.client.0.smithi116.stdout:1/499: truncate d5/d9/d21/d27/d32/d35/d55/d58/d7e/f54 4055347 0 2024-03-04T16:04:15.992 INFO:tasks.workunit.client.0.smithi116.stdout:7/605: symlink d3/d4/d8/d4a/d86/lbd 0 2024-03-04T16:04:15.993 INFO:tasks.workunit.client.0.smithi116.stdout:6/548: getdents d6/d9e/da5/db3 0 2024-03-04T16:04:15.993 INFO:tasks.workunit.client.0.smithi116.stdout:7/606: read - d3/d4/d8/d4a/db3/db6/f7c zero size 2024-03-04T16:04:15.995 INFO:tasks.workunit.client.0.smithi116.stdout:6/549: truncate d6/de/d25/d36/fa6 415971 0 2024-03-04T16:04:16.006 INFO:tasks.workunit.client.0.smithi116.stdout:7/607: dwrite d3/d4/d8/d1a/f42 [0,4194304] 0 2024-03-04T16:04:16.024 INFO:tasks.workunit.client.0.smithi116.stdout:9/548: write d2/d19/d27/d3b/d4a/d5b/f24 [4725254,128120] 0 2024-03-04T16:04:16.025 INFO:tasks.workunit.client.0.smithi116.stdout:9/549: dread - d2/d19/d27/d3b/f82 zero size 2024-03-04T16:04:16.025 INFO:tasks.workunit.client.0.smithi116.stdout:9/550: write d2/d19/d27/d3b/d4a/d6a/f75 [493715,66278] 0 2024-03-04T16:04:16.026 INFO:tasks.workunit.client.0.smithi116.stdout:0/514: creat da/d16/d28/fa3 x:0 0 0 2024-03-04T16:04:16.029 INFO:tasks.workunit.client.0.smithi116.stdout:1/500: rename d5/d9/d21/d27/d32/d35/f43 to d5/d9/d21/d27/d32/fa3 0 2024-03-04T16:04:16.030 INFO:tasks.workunit.client.0.smithi116.stdout:8/634: link d2/f60 d2/d4a/d8a/fd4 0 2024-03-04T16:04:16.033 INFO:tasks.workunit.client.0.smithi116.stdout:5/627: write dd/d1f/f4d [742334,5740] 0 2024-03-04T16:04:16.035 INFO:tasks.workunit.client.0.smithi116.stdout:6/550: dwrite d6/d56/f6c [0,4194304] 0 2024-03-04T16:04:16.036 INFO:tasks.workunit.client.0.smithi116.stdout:6/551: fsync d6/de/d25/f85 0 2024-03-04T16:04:16.037 INFO:tasks.workunit.client.0.smithi116.stdout:1/501: rmdir d5/d9/d21/d27/d32/d35/d55/d58/d7e/d61 39 2024-03-04T16:04:16.038 INFO:tasks.workunit.client.0.smithi116.stdout:6/552: chown d6/de/d15/l75 4966 1 2024-03-04T16:04:16.044 INFO:tasks.workunit.client.0.smithi116.stdout:9/551: dread d2/f4f [0,4194304] 0 2024-03-04T16:04:16.047 INFO:tasks.workunit.client.0.smithi116.stdout:5/628: unlink dd/d1f/d55/d73/l18 0 2024-03-04T16:04:16.049 INFO:tasks.workunit.client.0.smithi116.stdout:8/635: symlink d2/d13/ld5 0 2024-03-04T16:04:16.050 INFO:tasks.workunit.client.0.smithi116.stdout:8/636: dread - d2/de/d16/d78/d29/d7a/fb8 zero size 2024-03-04T16:04:16.053 INFO:tasks.workunit.client.0.smithi116.stdout:1/502: fdatasync d5/d9/d21/d27/d32/d49/f66 0 2024-03-04T16:04:16.054 INFO:tasks.workunit.client.0.smithi116.stdout:7/608: write d3/d4/d8/f40 [1178314,104536] 0 2024-03-04T16:04:16.055 INFO:tasks.workunit.client.0.smithi116.stdout:6/553: dread d6/de/f52 [0,4194304] 0 2024-03-04T16:04:16.059 INFO:tasks.workunit.client.0.smithi116.stdout:5/629: rename dd/d1f/d55/d73/f94 to dd/d6d/fc7 0 2024-03-04T16:04:16.060 INFO:tasks.workunit.client.0.smithi116.stdout:8/637: fdatasync d2/de/d16/f2c 0 2024-03-04T16:04:16.064 INFO:tasks.workunit.client.0.smithi116.stdout:7/609: write d3/d4/f16 [1555402,129703] 0 2024-03-04T16:04:16.066 INFO:tasks.workunit.client.0.smithi116.stdout:1/503: mknod d5/d9/d21/d27/d94/ca4 0 2024-03-04T16:04:16.068 INFO:tasks.workunit.client.0.smithi116.stdout:6/554: symlink d6/d56/d58/d72/lb7 0 2024-03-04T16:04:16.074 INFO:tasks.workunit.client.0.smithi116.stdout:8/638: mknod d2/de/d3e/cd6 0 2024-03-04T16:04:16.074 INFO:tasks.workunit.client.0.smithi116.stdout:8/639: stat d2/de/d16/d78/l54 0 2024-03-04T16:04:16.079 INFO:tasks.workunit.client.0.smithi116.stdout:7/610: dwrite d3/d4/d8/d4a/d87/f9d [0,4194304] 0 2024-03-04T16:04:16.080 INFO:tasks.workunit.client.0.smithi116.stdout:7/611: truncate d3/f55 486769 0 2024-03-04T16:04:16.087 INFO:tasks.workunit.client.0.smithi116.stdout:8/640: creat d2/de/d16/d78/d29/d7a/fd7 x:0 0 0 2024-03-04T16:04:16.094 INFO:tasks.workunit.client.0.smithi116.stdout:9/552: getdents d2 0 2024-03-04T16:04:16.097 INFO:tasks.workunit.client.0.smithi116.stdout:6/555: write d6/de/d15/f66 [688595,59189] 0 2024-03-04T16:04:16.098 INFO:tasks.workunit.client.0.smithi116.stdout:7/612: dread d3/d4/d8/d52/f54 [0,4194304] 0 2024-03-04T16:04:16.099 INFO:tasks.workunit.client.0.smithi116.stdout:7/613: fsync d3/d95/f6f 0 2024-03-04T16:04:16.099 INFO:tasks.workunit.client.0.smithi116.stdout:7/614: fdatasync d3/fd 0 2024-03-04T16:04:16.100 INFO:tasks.workunit.client.0.smithi116.stdout:9/553: rmdir d2/d26 39 2024-03-04T16:04:16.102 INFO:tasks.workunit.client.0.smithi116.stdout:5/630: truncate dd/d1f/d55/d73/d2d/dae/fbe 871222 0 2024-03-04T16:04:16.103 INFO:tasks.workunit.client.0.smithi116.stdout:5/631: dread - dd/d1f/d72/fa0 zero size 2024-03-04T16:04:16.103 INFO:tasks.workunit.client.0.smithi116.stdout:6/556: creat d6/fb8 x:0 0 0 2024-03-04T16:04:16.104 INFO:tasks.workunit.client.0.smithi116.stdout:6/557: write d6/de/d84/db4/d92/f9a [576610,59170] 0 2024-03-04T16:04:16.106 INFO:tasks.workunit.client.0.smithi116.stdout:1/504: truncate d5/f38 1957885 0 2024-03-04T16:04:16.109 INFO:tasks.workunit.client.0.smithi116.stdout:6/558: dwrite d6/de/d25/d36/f27 [0,4194304] 0 2024-03-04T16:04:16.110 INFO:tasks.workunit.client.0.smithi116.stdout:7/615: mknod d3/d4/d8/d4a/d86/cbe 0 2024-03-04T16:04:16.110 INFO:tasks.workunit.client.0.smithi116.stdout:6/559: chown d6/de/d1e/d80/cb1 0 1 2024-03-04T16:04:16.110 INFO:tasks.workunit.client.0.smithi116.stdout:7/616: stat d3/d4/d8/d4a/d65/l91 0 2024-03-04T16:04:16.115 INFO:tasks.workunit.client.0.smithi116.stdout:8/641: creat d2/de/d16/fd8 x:0 0 0 2024-03-04T16:04:16.122 INFO:tasks.workunit.client.0.smithi116.stdout:6/560: creat d6/de/d15/fb9 x:0 0 0 2024-03-04T16:04:16.126 INFO:tasks.workunit.client.0.smithi116.stdout:7/617: readlink d3/d4/d8/d1a/l30 0 2024-03-04T16:04:16.128 INFO:tasks.workunit.client.0.smithi116.stdout:8/642: creat d2/d13/d3c/d77/fd9 x:0 0 0 2024-03-04T16:04:16.129 INFO:tasks.workunit.client.0.smithi116.stdout:5/632: write fa [1530279,22307] 0 2024-03-04T16:04:16.131 INFO:tasks.workunit.client.0.smithi116.stdout:1/505: unlink d5/d9/d21/d27/d32/d35/d55/d75/c9a 0 2024-03-04T16:04:16.133 INFO:tasks.workunit.client.0.smithi116.stdout:6/561: fsync d6/de/f4f 0 2024-03-04T16:04:16.133 INFO:tasks.workunit.client.0.smithi116.stdout:1/506: read d5/d9/d21/d84/f90 [1364338,28724] 0 2024-03-04T16:04:16.135 INFO:tasks.workunit.client.0.smithi116.stdout:8/643: dwrite d2/de/d16/d78/fbd [0,4194304] 0 2024-03-04T16:04:16.136 INFO:tasks.workunit.client.0.smithi116.stdout:8/644: write d2/de/f8 [7454743,129560] 0 2024-03-04T16:04:16.139 INFO:tasks.workunit.client.0.smithi116.stdout:8/645: chown d2/de/d16/d78/d29/d7a/c98 51 1 2024-03-04T16:04:16.144 INFO:tasks.workunit.client.0.smithi116.stdout:3/646: sync 2024-03-04T16:04:16.144 INFO:tasks.workunit.client.0.smithi116.stdout:4/648: sync 2024-03-04T16:04:16.145 INFO:tasks.workunit.client.0.smithi116.stdout:3/647: chown dc/d23/d84/d7c/f69 304 1 2024-03-04T16:04:16.146 INFO:tasks.workunit.client.0.smithi116.stdout:9/554: unlink d2/d19/c4d 0 2024-03-04T16:04:16.148 INFO:tasks.workunit.client.0.smithi116.stdout:7/618: getdents d3/d4/d8/d52/d5e/d9a 0 2024-03-04T16:04:16.150 INFO:tasks.workunit.client.0.smithi116.stdout:6/562: creat d6/de/d25/d36/d44/fba x:0 0 0 2024-03-04T16:04:16.152 INFO:tasks.workunit.client.0.smithi116.stdout:1/507: rename d5/d9/d21/d27/d94/ca4 to d5/d86/ca5 0 2024-03-04T16:04:16.153 INFO:tasks.workunit.client.0.smithi116.stdout:1/508: dread - d5/d9/d21/d27/d32/d49/f66 zero size 2024-03-04T16:04:16.153 INFO:tasks.workunit.client.0.smithi116.stdout:1/509: chown d5/d9/d21/d27/d32/d49/d4c/d67 4577460 1 2024-03-04T16:04:16.157 INFO:tasks.workunit.client.0.smithi116.stdout:8/646: rmdir d2/de/d16/d78/d29 39 2024-03-04T16:04:16.162 INFO:tasks.workunit.client.0.smithi116.stdout:7/619: mkdir d3/d4/dbf 0 2024-03-04T16:04:16.164 INFO:tasks.workunit.client.0.smithi116.stdout:6/563: mkdir d6/de/d25/d36/d44/dbb 0 2024-03-04T16:04:16.166 INFO:tasks.workunit.client.0.smithi116.stdout:5/633: rmdir dd/d1f/d5c/d91 0 2024-03-04T16:04:16.167 INFO:tasks.workunit.client.0.smithi116.stdout:8/647: mknod d2/d13/d3c/d46/d6e/cda 0 2024-03-04T16:04:16.168 INFO:tasks.workunit.client.0.smithi116.stdout:4/649: mknod d1/d6e/d74/de0/ce4 0 2024-03-04T16:04:16.169 INFO:tasks.workunit.client.0.smithi116.stdout:3/648: symlink dc/d23/ld8 0 2024-03-04T16:04:16.170 INFO:tasks.workunit.client.0.smithi116.stdout:9/555: mkdir d2/d33/d90/da7 0 2024-03-04T16:04:16.172 INFO:tasks.workunit.client.0.smithi116.stdout:5/634: dwrite dd/d1f/d55/d73/d2d/f50 [0,4194304] 0 2024-03-04T16:04:16.180 INFO:tasks.workunit.client.0.smithi116.stdout:8/648: dread d2/de/d16/d2b/fa0 [0,4194304] 0 2024-03-04T16:04:16.185 INFO:tasks.workunit.client.0.smithi116.stdout:8/649: dwrite d2/de/f8 [4194304,4194304] 0 2024-03-04T16:04:16.190 INFO:tasks.workunit.client.0.smithi116.stdout:5/635: creat dd/d1f/d55/d73/d2d/fc8 x:0 0 0 2024-03-04T16:04:16.191 INFO:tasks.workunit.client.0.smithi116.stdout:5/636: truncate dd/d1f/d5c/d5e/faa 37232 0 2024-03-04T16:04:16.193 INFO:tasks.workunit.client.0.smithi116.stdout:1/510: creat d5/d9/d21/d27/d32/d35/fa6 x:0 0 0 2024-03-04T16:04:16.194 INFO:tasks.workunit.client.0.smithi116.stdout:1/511: dread - d5/d9/d46/f70 zero size 2024-03-04T16:04:16.197 INFO:tasks.workunit.client.0.smithi116.stdout:5/637: dwrite dd/d1f/d55/d73/d2d/d8d/fa6 [0,4194304] 0 2024-03-04T16:04:16.198 INFO:tasks.workunit.client.0.smithi116.stdout:4/650: fdatasync d1/d6c/f42 0 2024-03-04T16:04:16.204 INFO:tasks.workunit.client.0.smithi116.stdout:5/638: dwrite dd/d1f/d55/d73/d2d/d31/f3c [0,4194304] 0 2024-03-04T16:04:16.210 INFO:tasks.workunit.client.0.smithi116.stdout:3/649: mknod dc/d23/d84/d7f/d93/db6/cd9 0 2024-03-04T16:04:16.215 INFO:tasks.workunit.client.0.smithi116.stdout:3/650: dread dc/d34/dc1/fcb [0,4194304] 0 2024-03-04T16:04:16.216 INFO:tasks.workunit.client.0.smithi116.stdout:3/651: write dc/d23/d84/fd1 [845225,74460] 0 2024-03-04T16:04:16.220 INFO:tasks.workunit.client.0.smithi116.stdout:7/620: readlink d3/d4/d8/l23 0 2024-03-04T16:04:16.225 INFO:tasks.workunit.client.0.smithi116.stdout:4/651: creat d1/d3/d15/d20/d1c/d6f/fe5 x:0 0 0 2024-03-04T16:04:16.227 INFO:tasks.workunit.client.0.smithi116.stdout:5/639: mknod dd/d6d/cc9 0 2024-03-04T16:04:16.231 INFO:tasks.workunit.client.0.smithi116.stdout:6/564: rename d6/d56/d58/d72/lb7 to d6/de/lbc 0 2024-03-04T16:04:16.232 INFO:tasks.workunit.client.0.smithi116.stdout:5/640: dwrite dd/d1f/d55/d73/d2d/d31/f74 [0,4194304] 0 2024-03-04T16:04:16.233 INFO:tasks.workunit.client.0.smithi116.stdout:5/641: readlink dd/l23 0 2024-03-04T16:04:16.234 INFO:tasks.workunit.client.0.smithi116.stdout:9/556: link d2/f87 d2/d19/d27/d3b/d4a/d5b/fa8 0 2024-03-04T16:04:16.234 INFO:tasks.workunit.client.0.smithi116.stdout:5/642: truncate dd/d1f/d5c/d5e/faa 219118 0 2024-03-04T16:04:16.235 INFO:tasks.workunit.client.0.smithi116.stdout:5/643: chown dd/d1f/fb2 47740008 1 2024-03-04T16:04:16.236 INFO:tasks.workunit.client.0.smithi116.stdout:5/644: stat dd/d1f/d5c/f75 0 2024-03-04T16:04:16.242 INFO:tasks.workunit.client.0.smithi116.stdout:8/650: link d2/f32 d2/d13/d5e/fdb 0 2024-03-04T16:04:16.246 INFO:tasks.workunit.client.0.smithi116.stdout:1/512: rename d5/f4d to d5/d9/d21/d27/d32/d35/d55/d58/d7e/d42/d8b/fa7 0 2024-03-04T16:04:16.247 INFO:tasks.workunit.client.0.smithi116.stdout:2/679: sync 2024-03-04T16:04:16.247 INFO:tasks.workunit.client.0.smithi116.stdout:0/515: sync 2024-03-04T16:04:16.248 INFO:tasks.workunit.client.0.smithi116.stdout:6/565: unlink d6/de/d25/d4a/d63/c9b 0 2024-03-04T16:04:16.248 INFO:tasks.workunit.client.0.smithi116.stdout:0/516: write da/f1a [3276033,101287] 0 2024-03-04T16:04:16.250 INFO:tasks.workunit.client.0.smithi116.stdout:0/517: write da/d16/d20/f5b [795277,32502] 0 2024-03-04T16:04:16.251 INFO:tasks.workunit.client.0.smithi116.stdout:0/518: write da/d16/d28/d3d/d41/f7d [1167375,59229] 0 2024-03-04T16:04:16.252 INFO:tasks.workunit.client.0.smithi116.stdout:5/645: mkdir dd/d1f/d55/d73/d2d/d59/db0/dca 0 2024-03-04T16:04:16.265 INFO:tasks.workunit.client.0.smithi116.stdout:3/652: creat dc/d23/fda x:0 0 0 2024-03-04T16:04:16.266 INFO:tasks.workunit.client.0.smithi116.stdout:6/566: dread d6/de/f4f [0,4194304] 0 2024-03-04T16:04:16.267 INFO:tasks.workunit.client.0.smithi116.stdout:5/646: mknod dd/d1f/d72/d8e/ccb 0 2024-03-04T16:04:16.267 INFO:tasks.workunit.client.0.smithi116.stdout:9/557: dread d2/f1a [0,4194304] 0 2024-03-04T16:04:16.269 INFO:tasks.workunit.client.0.smithi116.stdout:7/621: link d3/f44 d3/d4/d34/fc0 0 2024-03-04T16:04:16.269 INFO:tasks.workunit.client.0.smithi116.stdout:4/652: creat d1/d3/d15/d20/d1c/d62/d81/fe6 x:0 0 0 2024-03-04T16:04:16.271 INFO:tasks.workunit.client.0.smithi116.stdout:1/513: symlink d5/d9/d21/d27/d32/la8 0 2024-03-04T16:04:16.277 INFO:tasks.workunit.client.0.smithi116.stdout:4/653: mknod d1/d3/d15/d20/d1c/d62/ce7 0 2024-03-04T16:04:16.279 INFO:tasks.workunit.client.0.smithi116.stdout:8/651: link d2/d13/d3c/fae d2/de/d16/fdc 0 2024-03-04T16:04:16.280 INFO:tasks.workunit.client.0.smithi116.stdout:8/652: fsync d2/d13/d3c/d77/fb2 0 2024-03-04T16:04:16.283 INFO:tasks.workunit.client.0.smithi116.stdout:3/653: mknod dc/dd3/cdb 0 2024-03-04T16:04:16.285 INFO:tasks.workunit.client.0.smithi116.stdout:0/519: creat da/fa4 x:0 0 0 2024-03-04T16:04:16.289 INFO:tasks.workunit.client.0.smithi116.stdout:6/567: creat d6/de/d1e/d7c/fbd x:0 0 0 2024-03-04T16:04:16.290 INFO:tasks.workunit.client.0.smithi116.stdout:0/520: dread da/d16/d28/d3d/d41/f59 [0,4194304] 0 2024-03-04T16:04:16.290 INFO:tasks.workunit.client.0.smithi116.stdout:6/568: write d6/de/d84/f8f [719956,100302] 0 2024-03-04T16:04:16.291 INFO:tasks.workunit.client.0.smithi116.stdout:0/521: readlink da/d16/d20/d42/d4e/l50 0 2024-03-04T16:04:16.292 INFO:tasks.workunit.client.0.smithi116.stdout:7/622: mknod d3/d4/d7e/cc1 0 2024-03-04T16:04:16.294 INFO:tasks.workunit.client.0.smithi116.stdout:4/654: unlink d1/d3/d15/d20/d1c/d62/la5 0 2024-03-04T16:04:16.295 INFO:tasks.workunit.client.0.smithi116.stdout:4/655: read - d1/d6c/f38 zero size 2024-03-04T16:04:16.296 INFO:tasks.workunit.client.0.smithi116.stdout:2/680: getdents d7/d13/d3e/d71/dad 0 2024-03-04T16:04:16.299 INFO:tasks.workunit.client.0.smithi116.stdout:8/653: creat d2/de/d3e/d7b/fdd x:0 0 0 2024-03-04T16:04:16.299 INFO:tasks.workunit.client.0.smithi116.stdout:3/654: creat dc/d23/d84/d7f/d93/db6/fdc x:0 0 0 2024-03-04T16:04:16.303 INFO:tasks.workunit.client.0.smithi116.stdout:1/514: fdatasync d5/d9/d21/d27/d32/d35/d55/d58/d7e/f4e 0 2024-03-04T16:04:16.306 INFO:tasks.workunit.client.0.smithi116.stdout:6/569: creat d6/de/d29/d3a/fbe x:0 0 0 2024-03-04T16:04:16.308 INFO:tasks.workunit.client.0.smithi116.stdout:0/522: symlink da/d16/d28/d40/d48/d77/la5 0 2024-03-04T16:04:16.311 INFO:tasks.workunit.client.0.smithi116.stdout:4/656: rename d1/d3/d15/d20/d1c/d62/d81/d97/fd0 to d1/d3/d15/d20/d1c/d62/d81/d97/dc6/fe8 0 2024-03-04T16:04:16.311 INFO:tasks.workunit.client.0.smithi116.stdout:4/657: chown d1 3041779 1 2024-03-04T16:04:16.312 INFO:tasks.workunit.client.0.smithi116.stdout:5/647: write dd/d1f/d55/f77 [1260613,35155] 0 2024-03-04T16:04:16.312 INFO:tasks.workunit.client.0.smithi116.stdout:7/623: dread f2 [0,4194304] 0 2024-03-04T16:04:16.315 INFO:tasks.workunit.client.0.smithi116.stdout:3/655: rmdir dc/d34/d44/d6a/da3 39 2024-03-04T16:04:16.320 INFO:tasks.workunit.client.0.smithi116.stdout:9/558: dwrite d2/d19/d27/d3b/d4a/d5b/fa8 [0,4194304] 0 2024-03-04T16:04:16.323 INFO:tasks.workunit.client.0.smithi116.stdout:0/523: mknod da/d16/d28/d40/d5d/ca6 0 2024-03-04T16:04:16.326 INFO:tasks.workunit.client.0.smithi116.stdout:4/658: symlink d1/d3/dd/d24/db4/le9 0 2024-03-04T16:04:16.327 INFO:tasks.workunit.client.0.smithi116.stdout:3/656: dwrite dc/df/d47/d91/f9f [0,4194304] 0 2024-03-04T16:04:16.329 INFO:tasks.workunit.client.0.smithi116.stdout:0/524: write da/d16/d28/d3d/d41/f94 [1241764,12193] 0 2024-03-04T16:04:16.330 INFO:tasks.workunit.client.0.smithi116.stdout:1/515: dwrite d5/d9/d21/d27/d32/d49/d4a/f78 [0,4194304] 0 2024-03-04T16:04:16.332 INFO:tasks.workunit.client.0.smithi116.stdout:8/654: fdatasync d2/d13/d3c/f69 0 2024-03-04T16:04:16.344 INFO:tasks.workunit.client.0.smithi116.stdout:4/659: unlink d1/d3/d40/d52/d84/d88/fe2 0 2024-03-04T16:04:16.345 INFO:tasks.workunit.client.0.smithi116.stdout:2/681: dwrite d7/d25/fbd [0,4194304] 0 2024-03-04T16:04:16.345 INFO:tasks.workunit.client.0.smithi116.stdout:4/660: chown d1/d3/de/l73 2472219 1 2024-03-04T16:04:16.345 INFO:tasks.workunit.client.0.smithi116.stdout:4/661: chown d1/d6c 1270573 1 2024-03-04T16:04:16.347 INFO:tasks.workunit.client.0.smithi116.stdout:4/662: truncate d1/d6e/f86 5055573 0 2024-03-04T16:04:16.350 INFO:tasks.workunit.client.0.smithi116.stdout:0/525: unlink da/d16/d28/fa3 0 2024-03-04T16:04:16.353 INFO:tasks.workunit.client.0.smithi116.stdout:1/516: chown d5/d9/d21/d27/d32/l80 88518483 1 2024-03-04T16:04:16.359 INFO:tasks.workunit.client.0.smithi116.stdout:9/559: readlink d2/d49/l30 0 2024-03-04T16:04:16.361 INFO:tasks.workunit.client.0.smithi116.stdout:9/560: dread d2/d19/d5f/fa3 [0,4194304] 0 2024-03-04T16:04:16.361 INFO:tasks.workunit.client.0.smithi116.stdout:7/624: truncate d3/d4/d8/f71 103393 0 2024-03-04T16:04:16.363 INFO:tasks.workunit.client.0.smithi116.stdout:3/657: mkdir dc/d34/ddd 0 2024-03-04T16:04:16.364 INFO:tasks.workunit.client.0.smithi116.stdout:3/658: chown dc/d34/d44/fb4 0 1 2024-03-04T16:04:16.365 INFO:tasks.workunit.client.0.smithi116.stdout:2/682: mknod d7/d81/dc2/de7/cff 0 2024-03-04T16:04:16.366 INFO:tasks.workunit.client.0.smithi116.stdout:4/663: write d1/d3/d15/d20/d1c/d62/d81/da6/fbd [3010838,59589] 0 2024-03-04T16:04:16.370 INFO:tasks.workunit.client.0.smithi116.stdout:0/526: mkdir da/d16/d28/d40/d48/da7 0 2024-03-04T16:04:16.374 INFO:tasks.workunit.client.0.smithi116.stdout:5/648: link dd/d1f/d72/d8e/laf dd/d6d/dc6/lcc 0 2024-03-04T16:04:16.375 INFO:tasks.workunit.client.0.smithi116.stdout:0/527: dwrite da/f1a [0,4194304] 0 2024-03-04T16:04:16.375 INFO:tasks.workunit.client.0.smithi116.stdout:5/649: chown dd/d1f/f41 3 1 2024-03-04T16:04:16.376 INFO:tasks.workunit.client.0.smithi116.stdout:6/570: link d6/d56/f4c d6/d56/d58/d72/fbf 0 2024-03-04T16:04:16.380 INFO:tasks.workunit.client.0.smithi116.stdout:7/625: mkdir d3/d4/d8/d4a/d88/dc2 0 2024-03-04T16:04:16.383 INFO:tasks.workunit.client.0.smithi116.stdout:4/664: fsync d1/d3/dd/d45/f4c 0 2024-03-04T16:04:16.384 INFO:tasks.workunit.client.0.smithi116.stdout:1/517: symlink d5/d9/d21/d27/d8c/la9 0 2024-03-04T16:04:16.385 INFO:tasks.workunit.client.0.smithi116.stdout:1/518: write d5/d9/d21/d27/d32/d35/fa6 [181372,68583] 0 2024-03-04T16:04:16.394 INFO:tasks.workunit.client.0.smithi116.stdout:2/683: symlink d7/d13/d3e/ded/dfd/l100 0 2024-03-04T16:04:16.394 INFO:tasks.workunit.client.0.smithi116.stdout:7/626: mkdir d3/d4/d8/d4a/db3/dc3 0 2024-03-04T16:04:16.399 INFO:tasks.workunit.client.0.smithi116.stdout:0/528: mkdir da/d16/d28/d40/da8 0 2024-03-04T16:04:16.399 INFO:tasks.workunit.client.0.smithi116.stdout:9/561: write d2/d33/f54 [2203682,43611] 0 2024-03-04T16:04:16.400 INFO:tasks.workunit.client.0.smithi116.stdout:3/659: dread dc/d23/f1b [0,4194304] 0 2024-03-04T16:04:16.400 INFO:tasks.workunit.client.0.smithi116.stdout:1/519: creat d5/d9/d46/faa x:0 0 0 2024-03-04T16:04:16.402 INFO:tasks.workunit.client.0.smithi116.stdout:5/650: mkdir dd/d1f/dcd 0 2024-03-04T16:04:16.403 INFO:tasks.workunit.client.0.smithi116.stdout:5/651: write dd/d1f/f41 [5864739,123282] 0 2024-03-04T16:04:16.404 INFO:tasks.workunit.client.0.smithi116.stdout:6/571: rename d6/d56/f4c to d6/d9e/fc0 0 2024-03-04T16:04:16.405 INFO:tasks.workunit.client.0.smithi116.stdout:6/572: chown d6/de/d1e/f43 26 1 2024-03-04T16:04:16.405 INFO:tasks.workunit.client.0.smithi116.stdout:4/665: write d1/f5 [4038461,47746] 0 2024-03-04T16:04:16.408 INFO:tasks.workunit.client.0.smithi116.stdout:4/666: dread - d1/d3/dd/d8a/ddc/fe1 zero size 2024-03-04T16:04:16.410 INFO:tasks.workunit.client.0.smithi116.stdout:5/652: dwrite dd/f38 [0,4194304] 0 2024-03-04T16:04:16.419 INFO:tasks.workunit.client.0.smithi116.stdout:0/529: symlink da/d16/d28/d7b/la9 0 2024-03-04T16:04:16.420 INFO:tasks.workunit.client.0.smithi116.stdout:8/655: sync 2024-03-04T16:04:16.420 INFO:tasks.workunit.client.0.smithi116.stdout:2/684: dread d7/f4e [0,4194304] 0 2024-03-04T16:04:16.421 INFO:tasks.workunit.client.0.smithi116.stdout:0/530: write da/d16/d19/d27/f74 [556951,19265] 0 2024-03-04T16:04:16.422 INFO:tasks.workunit.client.0.smithi116.stdout:9/562: write d2/d19/d27/d3b/d4a/d5b/f43 [654175,32452] 0 2024-03-04T16:04:16.424 INFO:tasks.workunit.client.0.smithi116.stdout:8/656: mknod d2/d4a/cde 0 2024-03-04T16:04:16.428 INFO:tasks.workunit.client.0.smithi116.stdout:5/653: creat dd/d1f/dc3/fce x:0 0 0 2024-03-04T16:04:16.428 INFO:tasks.workunit.client.0.smithi116.stdout:2/685: dread d7/d15/d35/f3b [0,4194304] 0 2024-03-04T16:04:16.429 INFO:tasks.workunit.client.0.smithi116.stdout:9/563: chown d2/d33/l4e 855530 1 2024-03-04T16:04:16.429 INFO:tasks.workunit.client.0.smithi116.stdout:2/686: chown d7/d81/c89 57673 1 2024-03-04T16:04:16.436 INFO:tasks.workunit.client.0.smithi116.stdout:0/531: dwrite da/d16/d20/d42/d4e/f8f [0,4194304] 0 2024-03-04T16:04:16.436 INFO:tasks.workunit.client.0.smithi116.stdout:1/520: getdents d5 0 2024-03-04T16:04:16.437 INFO:tasks.workunit.client.0.smithi116.stdout:3/660: rename dc/d23/ca4 to dc/d34/cde 0 2024-03-04T16:04:16.437 INFO:tasks.workunit.client.0.smithi116.stdout:9/564: chown d2/d19/f37 16487558 1 2024-03-04T16:04:16.437 INFO:tasks.workunit.client.0.smithi116.stdout:7/627: truncate d3/d4/d8/d4a/d87/f9d 1592827 0 2024-03-04T16:04:16.441 INFO:tasks.workunit.client.0.smithi116.stdout:5/654: truncate dd/d1f/d55/d73/d2d/d59/db0/fbc 671660 0 2024-03-04T16:04:16.441 INFO:tasks.workunit.client.0.smithi116.stdout:7/628: write d3/d4/d8/d52/fad [362411,104310] 0 2024-03-04T16:04:16.442 INFO:tasks.workunit.client.0.smithi116.stdout:3/661: write dc/df/d14/f5d [4899245,68659] 0 2024-03-04T16:04:16.442 INFO:tasks.workunit.client.0.smithi116.stdout:9/565: write d2/d19/d27/d3b/d4a/d5b/f97 [921448,95652] 0 2024-03-04T16:04:16.444 INFO:tasks.workunit.client.0.smithi116.stdout:1/521: truncate d5/d9/d21/d27/d32/d35/d55/d58/d7e/f9d 8422 0 2024-03-04T16:04:16.449 INFO:tasks.workunit.client.0.smithi116.stdout:6/573: dread d6/de/d25/d4a/d63/d8c/fa3 [0,4194304] 0 2024-03-04T16:04:16.462 INFO:tasks.workunit.client.0.smithi116.stdout:8/657: dwrite d2/de/d16/d78/d29/d7a/fd2 [0,4194304] 0 2024-03-04T16:04:16.472 INFO:tasks.workunit.client.0.smithi116.stdout:9/566: creat d2/d19/d27/d3b/d4a/d5b/fa9 x:0 0 0 2024-03-04T16:04:16.472 INFO:tasks.workunit.client.0.smithi116.stdout:9/567: chown d2/d19/d27 359724704 1 2024-03-04T16:04:16.475 INFO:tasks.workunit.client.0.smithi116.stdout:1/522: creat d5/d86/fab x:0 0 0 2024-03-04T16:04:16.478 INFO:tasks.workunit.client.0.smithi116.stdout:0/532: mkdir da/d16/d28/d40/d48/d57/d8a/daa 0 2024-03-04T16:04:16.479 INFO:tasks.workunit.client.0.smithi116.stdout:8/658: symlink d2/d13/d3c/d46/ldf 0 2024-03-04T16:04:16.480 INFO:tasks.workunit.client.0.smithi116.stdout:7/629: creat d3/d4/d11/fc4 x:0 0 0 2024-03-04T16:04:16.484 INFO:tasks.workunit.client.0.smithi116.stdout:8/659: dwrite d2/de/f8 [4194304,4194304] 0 2024-03-04T16:04:16.486 INFO:tasks.workunit.client.0.smithi116.stdout:9/568: mknod d2/d33/d40/caa 0 2024-03-04T16:04:16.488 INFO:tasks.workunit.client.0.smithi116.stdout:9/569: chown d2/d19/d27/d3b/d4a/d5b/f25 3962 1 2024-03-04T16:04:16.491 INFO:tasks.workunit.client.0.smithi116.stdout:1/523: fdatasync d5/d9/d21/d27/d32/d35/d55/d58/d7e/f69 0 2024-03-04T16:04:16.494 INFO:tasks.workunit.client.0.smithi116.stdout:9/570: dwrite d2/d19/d27/d3b/d4a/d5b/fa9 [0,4194304] 0 2024-03-04T16:04:16.499 INFO:tasks.workunit.client.0.smithi116.stdout:7/630: creat d3/d4/d8/d52/d5e/fc5 x:0 0 0 2024-03-04T16:04:16.499 INFO:tasks.workunit.client.0.smithi116.stdout:8/660: symlink d2/d4a/dc7/le0 0 2024-03-04T16:04:16.500 INFO:tasks.workunit.client.0.smithi116.stdout:6/574: creat d6/de/d25/d36/d4e/fc1 x:0 0 0 2024-03-04T16:04:16.503 INFO:tasks.workunit.client.0.smithi116.stdout:1/524: dwrite f2 [0,4194304] 0 2024-03-04T16:04:16.507 INFO:tasks.workunit.client.0.smithi116.stdout:1/525: chown d5/d9/d21/d27/d32/f47 36369727 1 2024-03-04T16:04:16.508 INFO:tasks.workunit.client.0.smithi116.stdout:2/687: dwrite d7/d15/d35/f54 [0,4194304] 0 2024-03-04T16:04:16.512 INFO:tasks.workunit.client.0.smithi116.stdout:5/655: dwrite dd/d1f/d5c/f75 [0,4194304] 0 2024-03-04T16:04:16.512 INFO:tasks.workunit.client.0.smithi116.stdout:8/661: unlink d2/d13/d3c/d46/d4f/cac 0 2024-03-04T16:04:16.516 INFO:tasks.workunit.client.0.smithi116.stdout:3/662: write dc/d23/f1b [2735598,64888] 0 2024-03-04T16:04:16.518 INFO:tasks.workunit.client.0.smithi116.stdout:1/526: mkdir d5/d86/dac 0 2024-03-04T16:04:16.523 INFO:tasks.workunit.client.0.smithi116.stdout:7/631: creat d3/d4/d8/d4a/db3/dc3/fc6 x:0 0 0 2024-03-04T16:04:16.524 INFO:tasks.workunit.client.0.smithi116.stdout:0/533: dread da/d16/d20/d42/f86 [0,4194304] 0 2024-03-04T16:04:16.527 INFO:tasks.workunit.client.0.smithi116.stdout:9/571: rename d2/d49 to d2/d19/d27/d3b/dab 0 2024-03-04T16:04:16.530 INFO:tasks.workunit.client.0.smithi116.stdout:8/662: truncate d2/d13/d3c/d77/fb3 190722 0 2024-03-04T16:04:16.536 INFO:tasks.workunit.client.0.smithi116.stdout:0/534: unlink da/d16/d28/l3a 0 2024-03-04T16:04:16.540 INFO:tasks.workunit.client.0.smithi116.stdout:1/527: rename d5/d9/d21/d24/c57 to d5/d9/d21/d27/d32/d49/d4c/d67/cad 0 2024-03-04T16:04:16.541 INFO:tasks.workunit.client.0.smithi116.stdout:9/572: mknod d2/d19/d27/d3b/d4a/d6a/cac 0 2024-03-04T16:04:16.544 INFO:tasks.workunit.client.0.smithi116.stdout:3/663: mknod dc/d34/d44/cdf 0 2024-03-04T16:04:16.548 INFO:tasks.workunit.client.0.smithi116.stdout:2/688: link d7/d13/d3e/ded/d6c/f90 d7/d13/dec/f101 0 2024-03-04T16:04:16.548 INFO:tasks.workunit.client.0.smithi116.stdout:2/689: dread - d7/d13/fbf zero size 2024-03-04T16:04:16.549 INFO:tasks.workunit.client.0.smithi116.stdout:1/528: read d5/d9/d21/d84/f90 [2321114,130455] 0 2024-03-04T16:04:16.552 INFO:tasks.workunit.client.0.smithi116.stdout:6/575: link d6/de/d25/d36/d44/l64 d6/de/d29/d3a/lc2 0 2024-03-04T16:04:16.555 INFO:tasks.workunit.client.0.smithi116.stdout:9/573: dread d2/d19/d27/d3b/dab/f6f [0,4194304] 0 2024-03-04T16:04:16.555 INFO:tasks.workunit.client.0.smithi116.stdout:3/664: rename dc/d34/d44/d6a to dc/d23/d84/d7f/d93/db6/dc5/de0 0 2024-03-04T16:04:16.556 INFO:tasks.workunit.client.0.smithi116.stdout:3/665: write dc/df/d47/fb3 [61758,39789] 0 2024-03-04T16:04:16.556 INFO:tasks.workunit.client.0.smithi116.stdout:9/574: write d2/d19/d27/d3b/d4a/d5b/f24 [9058222,61312] 0 2024-03-04T16:04:16.558 INFO:tasks.workunit.client.0.smithi116.stdout:3/666: truncate dc/d23/d84/f8e 518270 0 2024-03-04T16:04:16.562 INFO:tasks.workunit.client.0.smithi116.stdout:7/632: write d3/d4/d8/d4a/f6b [2356005,94000] 0 2024-03-04T16:04:16.563 INFO:tasks.workunit.client.0.smithi116.stdout:7/633: readlink d3/d4/d8/l23 0 2024-03-04T16:04:16.564 INFO:tasks.workunit.client.0.smithi116.stdout:3/667: dwrite dc/df/d14/fc0 [0,4194304] 0 2024-03-04T16:04:16.566 INFO:tasks.workunit.client.0.smithi116.stdout:6/576: mknod d6/de/d1e/d7c/cc3 0 2024-03-04T16:04:16.567 INFO:tasks.workunit.client.0.smithi116.stdout:3/668: truncate dc/df/d14/fc0 4246950 0 2024-03-04T16:04:16.571 INFO:tasks.workunit.client.0.smithi116.stdout:8/663: link l0 d2/d38/d55/le1 0 2024-03-04T16:04:16.572 INFO:tasks.workunit.client.0.smithi116.stdout:7/634: dwrite d3/d4/d11/fb7 [0,4194304] 0 2024-03-04T16:04:16.586 INFO:tasks.workunit.client.0.smithi116.stdout:9/575: mkdir d2/d19/d27/d3b/d4a/dad 0 2024-03-04T16:04:16.586 INFO:tasks.workunit.client.0.smithi116.stdout:9/576: read d2/f1a [4115612,12679] 0 2024-03-04T16:04:16.587 INFO:tasks.workunit.client.0.smithi116.stdout:1/529: write d5/d9/d21/d27/d32/d35/d55/d58/d7e/f69 [1082464,73687] 0 2024-03-04T16:04:16.591 INFO:tasks.workunit.client.0.smithi116.stdout:2/690: dwrite d7/d13/d3e/d78/f7e [4194304,4194304] 0 2024-03-04T16:04:16.591 INFO:tasks.workunit.client.0.smithi116.stdout:8/664: unlink d2/de/d16/d2b/d37/fca 0 2024-03-04T16:04:16.596 INFO:tasks.workunit.client.0.smithi116.stdout:6/577: dread d6/f21 [4194304,4194304] 0 2024-03-04T16:04:16.596 INFO:tasks.workunit.client.0.smithi116.stdout:3/669: mknod dc/d23/d84/d7f/d93/db6/dc5/de0/db5/ce1 0 2024-03-04T16:04:16.597 INFO:tasks.workunit.client.0.smithi116.stdout:3/670: read dc/f33 [1783849,101793] 0 2024-03-04T16:04:16.597 INFO:tasks.workunit.client.0.smithi116.stdout:3/671: stat dc/dd3/cdb 0 2024-03-04T16:04:16.598 INFO:tasks.workunit.client.0.smithi116.stdout:7/635: symlink d3/d4/d8/lc7 0 2024-03-04T16:04:16.604 INFO:tasks.workunit.client.0.smithi116.stdout:7/636: dwrite d3/d4/d8/f40 [0,4194304] 0 2024-03-04T16:04:16.605 INFO:tasks.workunit.client.0.smithi116.stdout:7/637: fdatasync d3/d4/d34/d5d/fa7 0 2024-03-04T16:04:16.605 INFO:tasks.workunit.client.0.smithi116.stdout:1/530: rename d5/d9/d21/d27/d32/d35/d55/d58 to d5/d9/d21/d27/d32/d35/d55/dae 0 2024-03-04T16:04:16.606 INFO:tasks.workunit.client.0.smithi116.stdout:8/665: creat d2/de/d16/d78/d29/d7a/db1/fe2 x:0 0 0 2024-03-04T16:04:16.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:16 smithi152 ceph-mon[36876]: pgmap v24: 65 pgs: 65 active+clean; 1.2 GiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 45 MiB/s rd, 178 MiB/s wr, 523 op/s 2024-03-04T16:04:16.608 INFO:tasks.workunit.client.0.smithi116.stdout:8/666: chown d2/de/d16/d2b/d37 510578 1 2024-03-04T16:04:16.611 INFO:tasks.workunit.client.0.smithi116.stdout:2/691: dread d7/d81/f26 [0,4194304] 0 2024-03-04T16:04:16.616 INFO:tasks.workunit.client.0.smithi116.stdout:7/638: unlink d3/d4/d8/d52/f54 0 2024-03-04T16:04:16.617 INFO:tasks.workunit.client.0.smithi116.stdout:6/578: rename d6/de/f38 to d6/d9e/da5/fc4 0 2024-03-04T16:04:16.618 INFO:tasks.workunit.client.0.smithi116.stdout:6/579: readlink d6/de/d15/l59 0 2024-03-04T16:04:16.619 INFO:tasks.workunit.client.0.smithi116.stdout:9/577: creat d2/d19/d27/fae x:0 0 0 2024-03-04T16:04:16.619 INFO:tasks.workunit.client.0.smithi116.stdout:7/639: dread d3/d4/d34/f74 [0,4194304] 0 2024-03-04T16:04:16.622 INFO:tasks.workunit.client.0.smithi116.stdout:2/692: dread d7/d25/f62 [0,4194304] 0 2024-03-04T16:04:16.622 INFO:tasks.workunit.client.0.smithi116.stdout:3/672: dread dc/d23/d84/f3b [0,4194304] 0 2024-03-04T16:04:16.625 INFO:tasks.workunit.client.0.smithi116.stdout:6/580: dread d6/de/d1e/d80/f57 [0,4194304] 0 2024-03-04T16:04:16.625 INFO:tasks.workunit.client.0.smithi116.stdout:9/578: mknod d2/d33/d65/d6c/caf 0 2024-03-04T16:04:16.627 INFO:tasks.workunit.client.0.smithi116.stdout:9/579: chown d2/d33/d65/d7e/c7f 1 1 2024-03-04T16:04:16.628 INFO:tasks.workunit.client.0.smithi116.stdout:1/531: truncate d5/d9/d21/d27/d32/d35/d55/dae/d7e/f54 1786391 0 2024-03-04T16:04:16.629 INFO:tasks.workunit.client.0.smithi116.stdout:2/693: dwrite d7/d13/d3e/ded/d6c/da4/fe3 [0,4194304] 0 2024-03-04T16:04:16.636 INFO:tasks.workunit.client.0.smithi116.stdout:6/581: dwrite d6/de/d25/d36/fa6 [0,4194304] 0 2024-03-04T16:04:16.644 INFO:tasks.workunit.client.0.smithi116.stdout:3/673: dread dc/f5f [0,4194304] 0 2024-03-04T16:04:16.644 INFO:tasks.workunit.client.0.smithi116.stdout:1/532: dwrite d5/d9/d21/d27/d32/d35/d55/dae/d7e/d42/f9c [0,4194304] 0 2024-03-04T16:04:16.646 INFO:tasks.workunit.client.0.smithi116.stdout:8/667: dwrite d2/de/d3e/d7b/f8e [0,4194304] 0 2024-03-04T16:04:16.647 INFO:tasks.workunit.client.0.smithi116.stdout:7/640: write d3/d4/d8/f1c [3838487,68257] 0 2024-03-04T16:04:16.647 INFO:tasks.workunit.client.0.smithi116.stdout:9/580: symlink d2/d19/d27/d3b/dab/lb0 0 2024-03-04T16:04:16.649 INFO:tasks.workunit.client.0.smithi116.stdout:3/674: fsync dc/d23/d84/d7c/f69 0 2024-03-04T16:04:16.649 INFO:tasks.workunit.client.0.smithi116.stdout:6/582: symlink d6/d9e/lc5 0 2024-03-04T16:04:16.652 INFO:tasks.workunit.client.0.smithi116.stdout:3/675: readlink dc/d34/d44/la9 0 2024-03-04T16:04:16.652 INFO:tasks.workunit.client.0.smithi116.stdout:9/581: write d2/d33/d40/f8b [385198,121604] 0 2024-03-04T16:04:16.656 INFO:tasks.workunit.client.0.smithi116.stdout:8/668: fdatasync d2/d13/d3c/d77/d97/f93 0 2024-03-04T16:04:16.658 INFO:tasks.workunit.client.0.smithi116.stdout:5/656: sync 2024-03-04T16:04:16.658 INFO:tasks.workunit.client.0.smithi116.stdout:4/667: sync 2024-03-04T16:04:16.659 INFO:tasks.workunit.client.0.smithi116.stdout:1/533: dwrite d5/d9/d21/d27/d32/d35/d55/dae/d7e/d42/f88 [0,4194304] 0 2024-03-04T16:04:16.660 INFO:tasks.workunit.client.0.smithi116.stdout:5/657: write dd/d1f/f29 [1088963,4753] 0 2024-03-04T16:04:16.661 INFO:tasks.workunit.client.0.smithi116.stdout:7/641: mknod d3/d4/d8/d4a/d87/cc8 0 2024-03-04T16:04:16.662 INFO:tasks.workunit.client.0.smithi116.stdout:5/658: stat dd/d1f/d55/d73/c1c 0 2024-03-04T16:04:16.664 INFO:tasks.workunit.client.0.smithi116.stdout:2/694: creat d7/d13/f102 x:0 0 0 2024-03-04T16:04:16.664 INFO:tasks.workunit.client.0.smithi116.stdout:1/534: dread - d5/d9/d21/d27/d32/d49/d4c/d67/f73 zero size 2024-03-04T16:04:16.668 INFO:tasks.workunit.client.0.smithi116.stdout:7/642: dread d3/fd [0,4194304] 0 2024-03-04T16:04:16.670 INFO:tasks.workunit.client.0.smithi116.stdout:7/643: write d3/d4/d8/d1a/f5f [576585,2189] 0 2024-03-04T16:04:16.671 INFO:tasks.workunit.client.0.smithi116.stdout:1/535: dwrite f2 [0,4194304] 0 2024-03-04T16:04:16.673 INFO:tasks.workunit.client.0.smithi116.stdout:7/644: write d3/f59 [1328829,122837] 0 2024-03-04T16:04:16.676 INFO:tasks.workunit.client.0.smithi116.stdout:4/668: truncate d1/d3/d40/d52/f76 888075 0 2024-03-04T16:04:16.680 INFO:tasks.workunit.client.0.smithi116.stdout:9/582: creat d2/d26/fb1 x:0 0 0 2024-03-04T16:04:16.685 INFO:tasks.workunit.client.0.smithi116.stdout:8/669: mkdir d2/de3 0 2024-03-04T16:04:16.689 INFO:tasks.workunit.client.0.smithi116.stdout:1/536: symlink d5/d9/d21/d27/d32/d49/d4c/laf 0 2024-03-04T16:04:16.692 INFO:tasks.workunit.client.0.smithi116.stdout:2/695: dread d7/d13/f1a [0,4194304] 0 2024-03-04T16:04:16.698 INFO:tasks.workunit.client.0.smithi116.stdout:9/583: write d2/d19/d27/d3b/dab/f42 [1311997,6692] 0 2024-03-04T16:04:16.699 INFO:tasks.workunit.client.0.smithi116.stdout:8/670: fsync d2/d13/d3c/f69 0 2024-03-04T16:04:16.701 INFO:tasks.workunit.client.0.smithi116.stdout:9/584: truncate d2/d19/d27/d3b/d4a/d6a/d94/d95/f9b 873624 0 2024-03-04T16:04:16.704 INFO:tasks.workunit.client.0.smithi116.stdout:6/583: write d6/d56/d58/d72/f79 [732470,63003] 0 2024-03-04T16:04:16.707 INFO:tasks.workunit.client.0.smithi116.stdout:5/659: rename dd/d1f/d55/d73/c37 to dd/d1f/ccf 0 2024-03-04T16:04:16.708 INFO:tasks.workunit.client.0.smithi116.stdout:3/676: getdents dc/dd3 0 2024-03-04T16:04:16.708 INFO:tasks.workunit.client.0.smithi116.stdout:7/645: dwrite d3/d4/d34/fc0 [0,4194304] 0 2024-03-04T16:04:16.709 INFO:tasks.workunit.client.0.smithi116.stdout:5/660: dread - dd/d1f/dc3/fce zero size 2024-03-04T16:04:16.710 INFO:tasks.workunit.client.0.smithi116.stdout:5/661: dread - dd/d1f/fb2 zero size 2024-03-04T16:04:16.714 INFO:tasks.workunit.client.0.smithi116.stdout:4/669: creat d1/d3/d15/fea x:0 0 0 2024-03-04T16:04:16.716 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:16 smithi116 ceph-mon[29120]: pgmap v24: 65 pgs: 65 active+clean; 1.2 GiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 45 MiB/s rd, 178 MiB/s wr, 523 op/s 2024-03-04T16:04:16.716 INFO:tasks.workunit.client.0.smithi116.stdout:0/535: sync 2024-03-04T16:04:16.717 INFO:tasks.workunit.client.0.smithi116.stdout:6/584: unlink d6/d56/d58/ca9 0 2024-03-04T16:04:16.718 INFO:tasks.workunit.client.0.smithi116.stdout:0/536: read da/d16/d28/d3d/f66 [3168609,98860] 0 2024-03-04T16:04:16.720 INFO:tasks.workunit.client.0.smithi116.stdout:2/696: dread d7/d13/d3e/d98/dce/d47/f49 [0,4194304] 0 2024-03-04T16:04:16.721 INFO:tasks.workunit.client.0.smithi116.stdout:8/671: rename d2/de/d16/d78/d29/lb7 to d2/d13/d3c/d77/d97/d75/le4 0 2024-03-04T16:04:16.721 INFO:tasks.workunit.client.0.smithi116.stdout:2/697: truncate d7/d81/dc2/fd0 319253 0 2024-03-04T16:04:16.722 INFO:tasks.workunit.client.0.smithi116.stdout:7/646: fsync d3/d4/d8/d1a/f32 0 2024-03-04T16:04:16.724 INFO:tasks.workunit.client.0.smithi116.stdout:3/677: mkdir dc/d23/d84/d7f/d93/db6/dc5/de0/db5/de2 0 2024-03-04T16:04:16.726 INFO:tasks.workunit.client.0.smithi116.stdout:9/585: mknod d2/d19/d27/d3b/d4a/cb2 0 2024-03-04T16:04:16.729 INFO:tasks.workunit.client.0.smithi116.stdout:1/537: dwrite d5/d9/d21/d27/d32/fa3 [0,4194304] 0 2024-03-04T16:04:16.730 INFO:tasks.workunit.client.0.smithi116.stdout:2/698: dwrite d7/d81/dc2/fd0 [0,4194304] 0 2024-03-04T16:04:16.733 INFO:tasks.workunit.client.0.smithi116.stdout:2/699: fsync d7/d25/ff7 0 2024-03-04T16:04:16.738 INFO:tasks.workunit.client.0.smithi116.stdout:8/672: mkdir d2/de/d16/d78/d29/de5 0 2024-03-04T16:04:16.740 INFO:tasks.workunit.client.0.smithi116.stdout:5/662: dwrite dd/d1f/d55/d73/d2d/d31/f6a [0,4194304] 0 2024-03-04T16:04:16.747 INFO:tasks.workunit.client.0.smithi116.stdout:6/585: dwrite d6/d56/f8b [0,4194304] 0 2024-03-04T16:04:16.751 INFO:tasks.workunit.client.0.smithi116.stdout:1/538: mkdir d5/d9/d21/d84/db0 0 2024-03-04T16:04:16.753 INFO:tasks.workunit.client.0.smithi116.stdout:0/537: mknod da/d16/cab 0 2024-03-04T16:04:16.755 INFO:tasks.workunit.client.0.smithi116.stdout:1/539: stat d5/d9/d21/d27/l2d 0 2024-03-04T16:04:16.758 INFO:tasks.workunit.client.0.smithi116.stdout:9/586: dwrite d2/d19/d27/d3b/dab/d3c/f76 [0,4194304] 0 2024-03-04T16:04:16.758 INFO:tasks.workunit.client.0.smithi116.stdout:7/647: creat d3/d4/d34/d5d/db9/fc9 x:0 0 0 2024-03-04T16:04:16.763 INFO:tasks.workunit.client.0.smithi116.stdout:4/670: creat d1/d3/d15/d20/d1c/d62/d81/feb x:0 0 0 2024-03-04T16:04:16.764 INFO:tasks.workunit.client.0.smithi116.stdout:3/678: creat dc/d34/dc1/fe3 x:0 0 0 2024-03-04T16:04:16.766 INFO:tasks.workunit.client.0.smithi116.stdout:1/540: dwrite d5/d9/fb [4194304,4194304] 0 2024-03-04T16:04:16.767 INFO:tasks.workunit.client.0.smithi116.stdout:8/673: creat d2/de/d16/dc3/fe6 x:0 0 0 2024-03-04T16:04:16.770 INFO:tasks.workunit.client.0.smithi116.stdout:3/679: chown dc/d23/d84/d7c/f96 164363555 1 2024-03-04T16:04:16.770 INFO:tasks.workunit.client.0.smithi116.stdout:4/671: truncate d1/d3/d40/d52/d84/d88/fb1 963114 0 2024-03-04T16:04:16.771 INFO:tasks.workunit.client.0.smithi116.stdout:5/663: rename dd/l4c to dd/d1f/d55/d73/d2d/d59/db0/ld0 0 2024-03-04T16:04:16.771 INFO:tasks.workunit.client.0.smithi116.stdout:4/672: chown d1/d3/ddb 1 1 2024-03-04T16:04:16.773 INFO:tasks.workunit.client.0.smithi116.stdout:9/587: unlink d2/c6d 0 2024-03-04T16:04:16.774 INFO:tasks.workunit.client.0.smithi116.stdout:9/588: readlink d2/d26/l78 0 2024-03-04T16:04:16.774 INFO:tasks.workunit.client.0.smithi116.stdout:1/541: stat d5/d9/d21/d24/f44 0 2024-03-04T16:04:16.774 INFO:tasks.workunit.client.0.smithi116.stdout:4/673: write d1/d6e/fb7 [356046,71698] 0 2024-03-04T16:04:16.778 INFO:tasks.workunit.client.0.smithi116.stdout:0/538: rename da/d16/d20/d42/d4e/f72 to da/d16/d28/d40/d5d/d7e/fac 0 2024-03-04T16:04:16.779 INFO:tasks.workunit.client.0.smithi116.stdout:0/539: readlink da/d16/d28/d39/l3f 0 2024-03-04T16:04:16.779 INFO:tasks.workunit.client.0.smithi116.stdout:2/700: read d7/d13/d3e/d98/dce/d47/f51 [2107771,128992] 0 2024-03-04T16:04:16.781 INFO:tasks.workunit.client.0.smithi116.stdout:0/540: fsync da/d16/d20/d42/d4e/f8f 0 2024-03-04T16:04:16.782 INFO:tasks.workunit.client.0.smithi116.stdout:0/541: chown da/fa4 13 1 2024-03-04T16:04:16.782 INFO:tasks.workunit.client.0.smithi116.stdout:3/680: creat dc/df/db8/fe4 x:0 0 0 2024-03-04T16:04:16.784 INFO:tasks.workunit.client.0.smithi116.stdout:6/586: truncate d6/de/d5b/f8e 4181667 0 2024-03-04T16:04:16.785 INFO:tasks.workunit.client.0.smithi116.stdout:6/587: write d6/de/d1e/d80/f89 [211563,102462] 0 2024-03-04T16:04:16.785 INFO:tasks.workunit.client.0.smithi116.stdout:1/542: read - d5/d9/d46/f63 zero size 2024-03-04T16:04:16.786 INFO:tasks.workunit.client.0.smithi116.stdout:5/664: dread dd/d6d/f88 [0,4194304] 0 2024-03-04T16:04:16.787 INFO:tasks.workunit.client.0.smithi116.stdout:4/674: creat d1/d6e/d74/dc2/fec x:0 0 0 2024-03-04T16:04:16.788 INFO:tasks.workunit.client.0.smithi116.stdout:4/675: write d1/d3/d15/d20/f82 [977273,6539] 0 2024-03-04T16:04:16.789 INFO:tasks.workunit.client.0.smithi116.stdout:8/674: mkdir d2/d38/d41/dab/de7 0 2024-03-04T16:04:16.790 INFO:tasks.workunit.client.0.smithi116.stdout:7/648: rename d3/d4/d34/l62 to d3/d4/d8/d4a/d65/lca 0 2024-03-04T16:04:16.791 INFO:tasks.workunit.client.0.smithi116.stdout:2/701: fdatasync d7/d25/fa2 0 2024-03-04T16:04:16.791 INFO:tasks.workunit.client.0.smithi116.stdout:7/649: stat d3/d4/d34/ca0 0 2024-03-04T16:04:16.794 INFO:tasks.workunit.client.0.smithi116.stdout:3/681: creat dc/df/d14/fe5 x:0 0 0 2024-03-04T16:04:16.796 INFO:tasks.workunit.client.0.smithi116.stdout:4/676: dwrite d1/d3/de/f10 [0,4194304] 0 2024-03-04T16:04:16.796 INFO:tasks.workunit.client.0.smithi116.stdout:3/682: chown dc/df/d47/ld5 11 1 2024-03-04T16:04:16.801 INFO:tasks.workunit.client.0.smithi116.stdout:5/665: creat dd/d1f/d51/fd1 x:0 0 0 2024-03-04T16:04:16.807 INFO:tasks.workunit.client.0.smithi116.stdout:5/666: dread dd/d1f/d55/f77 [0,4194304] 0 2024-03-04T16:04:16.811 INFO:tasks.workunit.client.0.smithi116.stdout:0/542: rename da/d16/d20/f5b to da/d16/d28/d40/d5d/fad 0 2024-03-04T16:04:16.814 INFO:tasks.workunit.client.0.smithi116.stdout:7/650: fsync d3/d4/d7e/f9b 0 2024-03-04T16:04:16.816 INFO:tasks.workunit.client.0.smithi116.stdout:7/651: write d3/d4/d8/d4a/d88/fbc [130501,61202] 0 2024-03-04T16:04:16.821 INFO:tasks.workunit.client.0.smithi116.stdout:6/588: dread f2 [0,4194304] 0 2024-03-04T16:04:16.821 INFO:tasks.workunit.client.0.smithi116.stdout:4/677: mknod d1/d3/dd/d45/d50/dad/ced 0 2024-03-04T16:04:16.824 INFO:tasks.workunit.client.0.smithi116.stdout:4/678: truncate d1/d3/d15/d20/d1c/d62/d81/d97/fb9 550342 0 2024-03-04T16:04:16.824 INFO:tasks.workunit.client.0.smithi116.stdout:3/683: dread dc/d23/f21 [0,4194304] 0 2024-03-04T16:04:16.827 INFO:tasks.workunit.client.0.smithi116.stdout:8/675: creat d2/de3/fe8 x:0 0 0 2024-03-04T16:04:16.827 INFO:tasks.workunit.client.0.smithi116.stdout:3/684: fsync dc/df/db8/fe4 0 2024-03-04T16:04:16.830 INFO:tasks.workunit.client.0.smithi116.stdout:8/676: fsync d2/d13/d3c/d77/d97/d5a/f6f 0 2024-03-04T16:04:16.830 INFO:tasks.workunit.client.0.smithi116.stdout:3/685: dread - dc/d23/d84/d7f/d93/db6/fdc zero size 2024-03-04T16:04:16.832 INFO:tasks.workunit.client.0.smithi116.stdout:9/589: getdents d2/d33/d65/d7e 0 2024-03-04T16:04:16.835 INFO:tasks.workunit.client.0.smithi116.stdout:9/590: chown d2/d33/d65/d7e/f9a 19738 1 2024-03-04T16:04:16.835 INFO:tasks.workunit.client.0.smithi116.stdout:4/679: dwrite d1/d3/d40/d52/d84/d88/f8b [0,4194304] 0 2024-03-04T16:04:16.838 INFO:tasks.workunit.client.0.smithi116.stdout:1/543: link d5/d9/d21/d27/d32/d35/d55/dae/d7e/d42/c97 d5/d86/dac/cb1 0 2024-03-04T16:04:16.840 INFO:tasks.workunit.client.0.smithi116.stdout:2/702: dwrite d7/d13/d3e/d98/dce/d47/f49 [0,4194304] 0 2024-03-04T16:04:16.843 INFO:tasks.workunit.client.0.smithi116.stdout:9/591: dwrite d2/d33/f88 [0,4194304] 0 2024-03-04T16:04:16.845 INFO:tasks.workunit.client.0.smithi116.stdout:9/592: truncate d2/d19/d27/d3b/f82 160894 0 2024-03-04T16:04:16.850 INFO:tasks.workunit.client.0.smithi116.stdout:6/589: mkdir d6/d56/d58/d72/d7b/dc6 0 2024-03-04T16:04:16.856 INFO:tasks.workunit.client.0.smithi116.stdout:4/680: unlink d1/d3/d40/d52/cbb 0 2024-03-04T16:04:16.857 INFO:tasks.workunit.client.0.smithi116.stdout:3/686: creat dc/df/dbc/fe6 x:0 0 0 2024-03-04T16:04:16.859 INFO:tasks.workunit.client.0.smithi116.stdout:2/703: creat d7/d25/d41/f103 x:0 0 0 2024-03-04T16:04:16.861 INFO:tasks.workunit.client.0.smithi116.stdout:1/544: rename d5/d9/d21/d27/d32/d35/d55/dae/d7e/d42/d8b/fa7 to d5/d9/d21/d27/d32/d35/d55/d75/d91/fb2 0 2024-03-04T16:04:16.862 INFO:tasks.workunit.client.0.smithi116.stdout:0/543: dwrite da/d16/d28/d40/f5c [0,4194304] 0 2024-03-04T16:04:16.866 INFO:tasks.workunit.client.0.smithi116.stdout:1/545: read d5/d9/d21/d27/d32/d35/f87 [15349,95384] 0 2024-03-04T16:04:16.866 INFO:tasks.workunit.client.0.smithi116.stdout:6/590: mknod d6/de/d15/cc7 0 2024-03-04T16:04:16.867 INFO:tasks.workunit.client.0.smithi116.stdout:4/681: symlink d1/d6c/lee 0 2024-03-04T16:04:16.873 INFO:tasks.workunit.client.0.smithi116.stdout:2/704: mkdir d7/d15/dd7/d104 0 2024-03-04T16:04:16.875 INFO:tasks.workunit.client.0.smithi116.stdout:7/652: write d3/d4/d8/d1a/d85/fab [4321918,120673] 0 2024-03-04T16:04:16.879 INFO:tasks.workunit.client.0.smithi116.stdout:0/544: mknod da/d16/d28/d39/cae 0 2024-03-04T16:04:16.879 INFO:tasks.workunit.client.0.smithi116.stdout:0/545: chown da/d16/d19/d27/l6c 517288 1 2024-03-04T16:04:16.881 INFO:tasks.workunit.client.0.smithi116.stdout:1/546: creat d5/d9/d21/d27/d8c/fb3 x:0 0 0 2024-03-04T16:04:16.882 INFO:tasks.workunit.client.0.smithi116.stdout:1/547: readlink d5/d9/d21/d27/d32/d49/d4c/laf 0 2024-03-04T16:04:16.883 INFO:tasks.workunit.client.0.smithi116.stdout:4/682: creat d1/d3/dd/d24/fef x:0 0 0 2024-03-04T16:04:16.883 INFO:tasks.workunit.client.0.smithi116.stdout:6/591: dread d6/de/d15/f6f [0,4194304] 0 2024-03-04T16:04:16.884 INFO:tasks.workunit.client.0.smithi116.stdout:3/687: mkdir dc/d34/de7 0 2024-03-04T16:04:16.886 INFO:tasks.workunit.client.0.smithi116.stdout:3/688: read dc/d34/dc1/fcb [336825,64398] 0 2024-03-04T16:04:16.886 INFO:tasks.workunit.client.0.smithi116.stdout:2/705: creat d7/dd3/de1/f105 x:0 0 0 2024-03-04T16:04:16.888 INFO:tasks.workunit.client.0.smithi116.stdout:8/677: getdents d2/d13/d3c/d77/d97/d5a 0 2024-03-04T16:04:16.889 INFO:tasks.workunit.client.0.smithi116.stdout:8/678: stat d2/d13/d3c/d46/d4f/f5b 0 2024-03-04T16:04:16.889 INFO:tasks.workunit.client.0.smithi116.stdout:4/683: dwrite d1/d3/d15/fea [0,4194304] 0 2024-03-04T16:04:16.890 INFO:tasks.workunit.client.0.smithi116.stdout:8/679: chown d2/de3/fe8 1988197 1 2024-03-04T16:04:16.895 INFO:tasks.workunit.client.0.smithi116.stdout:1/548: rename d5/d9/d46 to d5/d9/d21/d27/d32/d35/d55/db4 0 2024-03-04T16:04:16.897 INFO:tasks.workunit.client.0.smithi116.stdout:1/549: write d5/d9/d21/d27/d32/d35/d55/d75/d91/f9e [3049414,77205] 0 2024-03-04T16:04:16.900 INFO:tasks.workunit.client.0.smithi116.stdout:1/550: dread d5/d9/d21/d27/d32/d35/d55/dae/d7e/f9d [0,4194304] 0 2024-03-04T16:04:16.901 INFO:tasks.workunit.client.0.smithi116.stdout:9/593: truncate d2/d33/d65/f92 85166 0 2024-03-04T16:04:16.902 INFO:tasks.workunit.client.0.smithi116.stdout:3/689: fsync dc/d23/d84/d7f/d93/fb1 0 2024-03-04T16:04:16.903 INFO:tasks.workunit.client.0.smithi116.stdout:3/690: truncate dc/f64 1344710 0 2024-03-04T16:04:16.903 INFO:tasks.workunit.client.0.smithi116.stdout:3/691: stat dc/d23/d84/d7f/d93/db6/cd9 0 2024-03-04T16:04:16.906 INFO:tasks.workunit.client.0.smithi116.stdout:7/653: mkdir d3/d4/d8/d1a/dcb 0 2024-03-04T16:04:16.907 INFO:tasks.workunit.client.0.smithi116.stdout:7/654: chown d3/d4/d34/f74 17138679 1 2024-03-04T16:04:16.908 INFO:tasks.workunit.client.0.smithi116.stdout:7/655: write d3/d4/d8/d52/d5e/f6e [1433694,88242] 0 2024-03-04T16:04:16.909 INFO:tasks.workunit.client.0.smithi116.stdout:8/680: symlink d2/de/d3e/le9 0 2024-03-04T16:04:16.920 INFO:tasks.workunit.client.0.smithi116.stdout:9/594: mknod d2/d19/d5f/cb3 0 2024-03-04T16:04:16.921 INFO:tasks.workunit.client.0.smithi116.stdout:9/595: read d2/d19/d5f/fa3 [31596,32184] 0 2024-03-04T16:04:16.922 INFO:tasks.workunit.client.0.smithi116.stdout:8/681: dread d2/d13/d3c/d77/d97/d5a/f9d [0,4194304] 0 2024-03-04T16:04:16.922 INFO:tasks.workunit.client.0.smithi116.stdout:9/596: chown d2/d19/d27/d3b/l41 129390062 1 2024-03-04T16:04:16.924 INFO:tasks.workunit.client.0.smithi116.stdout:3/692: mkdir dc/d23/d84/d7f/de8 0 2024-03-04T16:04:16.930 INFO:tasks.workunit.client.0.smithi116.stdout:7/656: mknod d3/d95/ccc 0 2024-03-04T16:04:16.930 INFO:tasks.workunit.client.0.smithi116.stdout:0/546: link da/l1c da/d16/d28/d7b/laf 0 2024-03-04T16:04:16.930 INFO:tasks.workunit.client.0.smithi116.stdout:7/657: chown d3/d4/d11/l48 922409 1 2024-03-04T16:04:16.931 INFO:tasks.workunit.client.0.smithi116.stdout:7/658: readlink d3/d4/d8/l7d 0 2024-03-04T16:04:16.934 INFO:tasks.workunit.client.0.smithi116.stdout:6/592: creat d6/de/d5b/fc8 x:0 0 0 2024-03-04T16:04:16.936 INFO:tasks.workunit.client.0.smithi116.stdout:0/547: dwrite da/ff [0,4194304] 0 2024-03-04T16:04:16.937 INFO:tasks.workunit.client.0.smithi116.stdout:7/659: dwrite d3/d4/d7e/f9b [4194304,4194304] 0 2024-03-04T16:04:16.938 INFO:tasks.workunit.client.0.smithi116.stdout:8/682: creat d2/de/d3e/d7b/fea x:0 0 0 2024-03-04T16:04:16.939 INFO:tasks.workunit.client.0.smithi116.stdout:9/597: fdatasync d2/d19/d27/d3b/d4a/f7b 0 2024-03-04T16:04:16.941 INFO:tasks.workunit.client.0.smithi116.stdout:3/693: creat dc/d34/d44/dab/fe9 x:0 0 0 2024-03-04T16:04:16.942 INFO:tasks.workunit.client.0.smithi116.stdout:0/548: write da/d16/d28/d40/d48/d57/f8c [515335,38996] 0 2024-03-04T16:04:16.943 INFO:tasks.workunit.client.0.smithi116.stdout:4/684: write d1/d3/f39 [440039,126359] 0 2024-03-04T16:04:16.945 INFO:tasks.workunit.client.0.smithi116.stdout:3/694: stat dc/df/d14/f5d 0 2024-03-04T16:04:16.945 INFO:tasks.workunit.client.0.smithi116.stdout:2/706: link d7/d13/fbf d7/d13/d3e/d98/dce/f106 0 2024-03-04T16:04:16.947 INFO:tasks.workunit.client.0.smithi116.stdout:0/549: chown da/d16/d28/d40/d48/l4d 448 1 2024-03-04T16:04:16.950 INFO:tasks.workunit.client.0.smithi116.stdout:1/551: write d5/d9/d21/d27/d32/d35/d55/dae/d7e/f54 [948049,3415] 0 2024-03-04T16:04:16.951 INFO:tasks.workunit.client.0.smithi116.stdout:8/683: dwrite d2/d13/d3c/d77/d97/d42/f5d [0,4194304] 0 2024-03-04T16:04:16.955 INFO:tasks.workunit.client.0.smithi116.stdout:7/660: creat d3/d4/d7e/fcd x:0 0 0 2024-03-04T16:04:16.955 INFO:tasks.workunit.client.0.smithi116.stdout:9/598: mknod d2/d26/cb4 0 2024-03-04T16:04:16.957 INFO:tasks.workunit.client.0.smithi116.stdout:1/552: read d5/d9/d21/d27/d32/d35/f76 [2056290,29873] 0 2024-03-04T16:04:16.958 INFO:tasks.workunit.client.0.smithi116.stdout:4/685: mknod d1/d3/dd/dd9/cf0 0 2024-03-04T16:04:16.960 INFO:tasks.workunit.client.0.smithi116.stdout:2/707: mkdir d7/d13/d3e/d78/d107 0 2024-03-04T16:04:16.961 INFO:tasks.workunit.client.0.smithi116.stdout:8/684: readlink d2/de/l1a 0 2024-03-04T16:04:16.963 INFO:tasks.workunit.client.0.smithi116.stdout:8/685: dread - d2/d13/d3c/d46/d6e/fbc zero size 2024-03-04T16:04:16.964 INFO:tasks.workunit.client.0.smithi116.stdout:8/686: dread - d2/de/d16/d78/d29/d7a/fb8 zero size 2024-03-04T16:04:16.970 INFO:tasks.workunit.client.0.smithi116.stdout:9/599: rmdir d2/d19/d27/d3b/d4a 39 2024-03-04T16:04:16.973 INFO:tasks.workunit.client.0.smithi116.stdout:4/686: unlink d1/d6e/d74/f7b 0 2024-03-04T16:04:16.974 INFO:tasks.workunit.client.0.smithi116.stdout:7/661: dread d3/d4/d8/d1a/d85/fab [0,4194304] 0 2024-03-04T16:04:16.976 INFO:tasks.workunit.client.0.smithi116.stdout:1/553: creat d5/d9/d21/d84/fb5 x:0 0 0 2024-03-04T16:04:16.978 INFO:tasks.workunit.client.0.smithi116.stdout:2/708: mknod d7/d81/dc2/c108 0 2024-03-04T16:04:16.980 INFO:tasks.workunit.client.0.smithi116.stdout:2/709: dread - d7/d13/f102 zero size 2024-03-04T16:04:16.980 INFO:tasks.workunit.client.0.smithi116.stdout:6/593: dwrite d6/de/d1e/d80/f57 [0,4194304] 0 2024-03-04T16:04:16.981 INFO:tasks.workunit.client.0.smithi116.stdout:4/687: dwrite d1/f5 [0,4194304] 0 2024-03-04T16:04:16.983 INFO:tasks.workunit.client.0.smithi116.stdout:6/594: write d6/de/d25/f96 [95051,73507] 0 2024-03-04T16:04:16.986 INFO:tasks.workunit.client.0.smithi116.stdout:9/600: write d2/d19/d27/d3b/d4a/f7b [394618,83778] 0 2024-03-04T16:04:16.988 INFO:tasks.workunit.client.0.smithi116.stdout:3/695: dread dc/d23/d84/d7c/f62 [0,4194304] 0 2024-03-04T16:04:16.988 INFO:tasks.workunit.client.0.smithi116.stdout:7/662: symlink d3/d4/d8/d1a/d85/daf/lce 0 2024-03-04T16:04:16.992 INFO:tasks.workunit.client.0.smithi116.stdout:0/550: creat da/d16/d28/fb0 x:0 0 0 2024-03-04T16:04:16.992 INFO:tasks.workunit.client.0.smithi116.stdout:3/696: dread dc/d23/d84/f3b [0,4194304] 0 2024-03-04T16:04:16.993 INFO:tasks.workunit.client.0.smithi116.stdout:3/697: chown dc/d34/d44/fac 31451 1 2024-03-04T16:04:16.995 INFO:tasks.workunit.client.0.smithi116.stdout:8/687: truncate d2/d13/d3c/d77/fb3 629833 0 2024-03-04T16:04:16.996 INFO:tasks.workunit.client.0.smithi116.stdout:8/688: read d2/d38/d41/f6c [3804,95342] 0 2024-03-04T16:04:16.998 INFO:tasks.workunit.client.0.smithi116.stdout:2/710: rename d7/d15/f38 to d7/d13/dec/f109 0 2024-03-04T16:04:17.005 INFO:tasks.workunit.client.0.smithi116.stdout:9/601: creat d2/d19/d27/d3b/d4a/d6a/d94/d95/fb5 x:0 0 0 2024-03-04T16:04:17.006 INFO:tasks.workunit.client.0.smithi116.stdout:9/602: write d2/d19/d27/f9d [1505127,107716] 0 2024-03-04T16:04:17.007 INFO:tasks.workunit.client.0.smithi116.stdout:7/663: mknod d3/d4/d8/d4a/d65/ccf 0 2024-03-04T16:04:17.015 INFO:tasks.workunit.client.0.smithi116.stdout:8/689: mkdir d2/d13/d3c/d77/d97/deb 0 2024-03-04T16:04:17.016 INFO:tasks.workunit.client.0.smithi116.stdout:1/554: dread d5/d9/d21/f2c [0,4194304] 0 2024-03-04T16:04:17.018 INFO:tasks.workunit.client.0.smithi116.stdout:2/711: rmdir d7/d13/d3e/d98/dce/d47/d64/da5 39 2024-03-04T16:04:17.019 INFO:tasks.workunit.client.0.smithi116.stdout:5/667: sync 2024-03-04T16:04:17.024 INFO:tasks.workunit.client.0.smithi116.stdout:6/595: write d6/f45 [1084090,75109] 0 2024-03-04T16:04:17.028 INFO:tasks.workunit.client.0.smithi116.stdout:7/664: mkdir d3/d4/d8/d1a/d85/dd0 0 2024-03-04T16:04:17.029 INFO:tasks.workunit.client.0.smithi116.stdout:0/551: write da/d16/d19/d27/f78 [1300169,104666] 0 2024-03-04T16:04:17.030 INFO:tasks.workunit.client.0.smithi116.stdout:3/698: getdents dc/dd3 0 2024-03-04T16:04:17.034 INFO:tasks.workunit.client.0.smithi116.stdout:1/555: rename d5/d9/d21/d27/d32/d35/d55/dae/d7e to d5/d9/d21/d27/d94/db6 0 2024-03-04T16:04:17.035 INFO:tasks.workunit.client.0.smithi116.stdout:1/556: dread d5/f6f [0,4194304] 0 2024-03-04T16:04:17.038 INFO:tasks.workunit.client.0.smithi116.stdout:6/596: creat d6/d9e/da5/fc9 x:0 0 0 2024-03-04T16:04:17.039 INFO:tasks.workunit.client.0.smithi116.stdout:9/603: symlink d2/d19/d27/d3b/d4a/lb6 0 2024-03-04T16:04:17.040 INFO:tasks.workunit.client.0.smithi116.stdout:7/665: mkdir d3/d4/d8/d1a/d85/dd1 0 2024-03-04T16:04:17.042 INFO:tasks.workunit.client.0.smithi116.stdout:3/699: creat dc/d23/d84/d7f/d93/fea x:0 0 0 2024-03-04T16:04:17.043 INFO:tasks.workunit.client.0.smithi116.stdout:3/700: chown dc/d34/d44/fb4 5316 1 2024-03-04T16:04:17.044 INFO:tasks.workunit.client.0.smithi116.stdout:9/604: dwrite d2/d19/d27/d3b/d4a/f73 [4194304,4194304] 0 2024-03-04T16:04:17.045 INFO:tasks.workunit.client.0.smithi116.stdout:9/605: write d2/d19/f84 [600585,4862] 0 2024-03-04T16:04:17.046 INFO:tasks.workunit.client.0.smithi116.stdout:2/712: creat d7/d15/d5e/d9d/dd4/f10a x:0 0 0 2024-03-04T16:04:17.048 INFO:tasks.workunit.client.0.smithi116.stdout:1/557: write d5/d9/d21/d27/d32/d35/d55/d75/d91/fb2 [1276702,97252] 0 2024-03-04T16:04:17.049 INFO:tasks.workunit.client.0.smithi116.stdout:4/688: getdents d1/d3/dd/d8a/ddc 0 2024-03-04T16:04:17.049 INFO:tasks.workunit.client.0.smithi116.stdout:4/689: fsync d1/d3/dd/d45/d50/f96 0 2024-03-04T16:04:17.051 INFO:tasks.workunit.client.0.smithi116.stdout:6/597: mkdir d6/d9e/da5/dca 0 2024-03-04T16:04:17.054 INFO:tasks.workunit.client.0.smithi116.stdout:1/558: dread d5/d9/d21/d27/d32/f47 [0,4194304] 0 2024-03-04T16:04:17.058 INFO:tasks.workunit.client.0.smithi116.stdout:3/701: rename dc/d23/d84/d7f/d93/db6/dc5/de0/db5 to dc/d34/de7/deb 0 2024-03-04T16:04:17.058 INFO:tasks.workunit.client.0.smithi116.stdout:6/598: dwrite d6/de/d84/f8f [0,4194304] 0 2024-03-04T16:04:17.059 INFO:tasks.workunit.client.0.smithi116.stdout:3/702: dread - dc/f88 zero size 2024-03-04T16:04:17.060 INFO:tasks.workunit.client.0.smithi116.stdout:5/668: creat dd/fd2 x:0 0 0 2024-03-04T16:04:17.061 INFO:tasks.workunit.client.0.smithi116.stdout:9/606: mkdir d2/d19/d27/d3b/d4a/d6a/d94/d95/db7 0 2024-03-04T16:04:17.063 INFO:tasks.workunit.client.0.smithi116.stdout:4/690: creat d1/d3/dd/d8a/ff1 x:0 0 0 2024-03-04T16:04:17.064 INFO:tasks.workunit.client.0.smithi116.stdout:4/691: chown d1/d6c/dbf 59 1 2024-03-04T16:04:17.068 INFO:tasks.workunit.client.0.smithi116.stdout:0/552: creat da/d16/fb1 x:0 0 0 2024-03-04T16:04:17.070 INFO:tasks.workunit.client.0.smithi116.stdout:8/690: getdents d2/de/d3e 0 2024-03-04T16:04:17.072 INFO:tasks.workunit.client.0.smithi116.stdout:6/599: creat d6/de/d84/fcb x:0 0 0 2024-03-04T16:04:17.073 INFO:tasks.workunit.client.0.smithi116.stdout:3/703: creat dc/d23/d84/d7f/d93/db6/fec x:0 0 0 2024-03-04T16:04:17.074 INFO:tasks.workunit.client.0.smithi116.stdout:6/600: chown d6/de/d84/db4/d92/f9a 208 1 2024-03-04T16:04:17.074 INFO:tasks.workunit.client.0.smithi116.stdout:0/553: dwrite da/d16/d19/d27/f78 [0,4194304] 0 2024-03-04T16:04:17.075 INFO:tasks.workunit.client.0.smithi116.stdout:6/601: chown d6/de/d29/l81 400089321 1 2024-03-04T16:04:17.075 INFO:tasks.workunit.client.0.smithi116.stdout:0/554: write da/fa4 [806111,127218] 0 2024-03-04T16:04:17.076 INFO:tasks.workunit.client.0.smithi116.stdout:9/607: rename d2/d19/d27/d3b/d4a/dad to d2/d33/d65/d6c/db8 0 2024-03-04T16:04:17.083 INFO:tasks.workunit.client.0.smithi116.stdout:8/691: unlink d2/d13/d3c/d77/l7d 0 2024-03-04T16:04:17.084 INFO:tasks.workunit.client.0.smithi116.stdout:8/692: truncate d2/d13/d5e/fc0 617317 0 2024-03-04T16:04:17.086 INFO:tasks.workunit.client.0.smithi116.stdout:5/669: symlink dd/d1f/ld3 0 2024-03-04T16:04:17.091 INFO:tasks.workunit.client.0.smithi116.stdout:0/555: mkdir da/d16/d19/d27/d70/d92/db2 0 2024-03-04T16:04:17.092 INFO:tasks.workunit.client.0.smithi116.stdout:9/608: rmdir d2/d19/d27/d3b/dab/d3c 39 2024-03-04T16:04:17.096 INFO:tasks.workunit.client.0.smithi116.stdout:7/666: getdents d3/d4/d8 0 2024-03-04T16:04:17.097 INFO:tasks.workunit.client.0.smithi116.stdout:3/704: mknod dc/df/ced 0 2024-03-04T16:04:17.101 INFO:tasks.workunit.client.0.smithi116.stdout:5/670: rename dd/d1f/l44 to dd/d1f/d55/d73/d2d/ld4 0 2024-03-04T16:04:17.101 INFO:tasks.workunit.client.0.smithi116.stdout:7/667: dwrite d3/d4/d8/d1a/f5f [0,4194304] 0 2024-03-04T16:04:17.102 INFO:tasks.workunit.client.0.smithi116.stdout:6/602: truncate d6/d9e/da5/fc4 146679 0 2024-03-04T16:04:17.103 INFO:tasks.workunit.client.0.smithi116.stdout:7/668: write d3/d4/d34/d5d/db9/fc9 [1022846,128419] 0 2024-03-04T16:04:17.104 INFO:tasks.workunit.client.0.smithi116.stdout:9/609: symlink d2/d19/d27/d3b/d4a/d6a/d94/lb9 0 2024-03-04T16:04:17.110 INFO:tasks.workunit.client.0.smithi116.stdout:3/705: mkdir dc/d23/d84/d7f/d93/db6/dee 0 2024-03-04T16:04:17.122 INFO:tasks.workunit.client.0.smithi116.stdout:9/610: mknod d2/d33/d65/d7e/cba 0 2024-03-04T16:04:17.122 INFO:tasks.workunit.client.0.smithi116.stdout:9/611: write d2/d26/fa0 [382593,94849] 0 2024-03-04T16:04:17.125 INFO:tasks.workunit.client.0.smithi116.stdout:4/692: getdents d1/d3/dd 0 2024-03-04T16:04:17.130 INFO:tasks.workunit.client.0.smithi116.stdout:5/671: creat dd/d1f/d55/d73/dba/fd5 x:0 0 0 2024-03-04T16:04:17.132 INFO:tasks.workunit.client.0.smithi116.stdout:6/603: mknod d6/d56/d58/d72/ccc 0 2024-03-04T16:04:17.133 INFO:tasks.workunit.client.0.smithi116.stdout:9/612: rmdir d2/d26/d28 39 2024-03-04T16:04:17.137 INFO:tasks.workunit.client.0.smithi116.stdout:3/706: symlink dc/d23/d84/d7f/d93/db6/dee/lef 0 2024-03-04T16:04:17.139 INFO:tasks.workunit.client.0.smithi116.stdout:5/672: dwrite dd/d1f/d55/d73/d2d/fc8 [0,4194304] 0 2024-03-04T16:04:17.139 INFO:tasks.workunit.client.0.smithi116.stdout:7/669: dread d3/d4/d8/d1a/f46 [0,4194304] 0 2024-03-04T16:04:17.140 INFO:tasks.workunit.client.0.smithi116.stdout:5/673: readlink dd/d1f/d55/l8c 0 2024-03-04T16:04:17.141 INFO:tasks.workunit.client.0.smithi116.stdout:9/613: symlink d2/d33/d65/d6c/db8/lbb 0 2024-03-04T16:04:17.141 INFO:tasks.workunit.client.0.smithi116.stdout:6/604: dwrite d6/fb8 [0,4194304] 0 2024-03-04T16:04:17.141 INFO:tasks.workunit.client.0.smithi116.stdout:3/707: chown dc/d23/d84/d7f/d93/db6/dc5/cca 1102202155 1 2024-03-04T16:04:17.142 INFO:tasks.workunit.client.0.smithi116.stdout:5/674: fsync dd/d1f/fb2 0 2024-03-04T16:04:17.150 INFO:tasks.workunit.client.0.smithi116.stdout:9/614: dwrite d2/d33/f3a [0,4194304] 0 2024-03-04T16:04:17.150 INFO:tasks.workunit.client.0.smithi116.stdout:3/708: dread dc/d34/dc1/fcb [0,4194304] 0 2024-03-04T16:04:17.151 INFO:tasks.workunit.client.0.smithi116.stdout:3/709: dread - dc/d23/fcd zero size 2024-03-04T16:04:17.154 INFO:tasks.workunit.client.0.smithi116.stdout:3/710: dread dc/d23/d84/d7f/d8b/fc7 [0,4194304] 0 2024-03-04T16:04:17.228 INFO:tasks.workunit.client.0.smithi116.stdout:0/556: rename da/d16/d19/d27/d70/d92 to da/d16/d28/d40/d48/d57/db3 0 2024-03-04T16:04:17.232 INFO:tasks.workunit.client.0.smithi116.stdout:3/711: unlink dc/d23/d84/d7f/d93/db6/dc5/de0/la2 0 2024-03-04T16:04:17.239 INFO:tasks.workunit.client.0.smithi116.stdout:2/713: sync 2024-03-04T16:04:17.242 INFO:tasks.workunit.client.0.smithi116.stdout:0/557: rmdir da/d16/d28/d7b 39 2024-03-04T16:04:17.247 INFO:tasks.workunit.client.0.smithi116.stdout:5/675: link dd/d45/l79 dd/d6d/ld6 0 2024-03-04T16:04:17.249 INFO:tasks.workunit.client.0.smithi116.stdout:2/714: creat d7/d13/d3e/ded/dfd/f10b x:0 0 0 2024-03-04T16:04:17.251 INFO:tasks.workunit.client.0.smithi116.stdout:8/693: dwrite d2/de/f15 [0,4194304] 0 2024-03-04T16:04:17.265 INFO:tasks.workunit.client.0.smithi116.stdout:8/694: symlink d2/d13/d5e/da7/lec 0 2024-03-04T16:04:17.267 INFO:tasks.workunit.client.0.smithi116.stdout:2/715: link d7/d13/d3e/d98/caf d7/d13/d3e/ded/dfd/c10c 0 2024-03-04T16:04:17.267 INFO:tasks.workunit.client.0.smithi116.stdout:4/693: dwrite d1/d3/dd/d45/f4c [4194304,4194304] 0 2024-03-04T16:04:17.268 INFO:tasks.workunit.client.0.smithi116.stdout:2/716: write d7/d15/d35/f54 [4423438,51396] 0 2024-03-04T16:04:17.270 INFO:tasks.workunit.client.0.smithi116.stdout:8/695: symlink d2/d13/d3c/d77/d97/d5a/led 0 2024-03-04T16:04:17.270 INFO:tasks.workunit.client.0.smithi116.stdout:2/717: write d7/d13/d3e/d78/f84 [3172541,5003] 0 2024-03-04T16:04:17.271 INFO:tasks.workunit.client.0.smithi116.stdout:8/696: fsync d2/d38/d41/f6c 0 2024-03-04T16:04:17.285 INFO:tasks.workunit.client.0.smithi116.stdout:8/697: truncate d2/d13/d3c/fae 3207376 0 2024-03-04T16:04:17.292 INFO:tasks.workunit.client.0.smithi116.stdout:8/698: symlink d2/de/lee 0 2024-03-04T16:04:17.298 INFO:tasks.workunit.client.0.smithi116.stdout:8/699: mknod d2/de/d16/d78/d29/d7a/cef 0 2024-03-04T16:04:17.309 INFO:tasks.workunit.client.0.smithi116.stdout:8/700: dread d2/de/f8 [0,4194304] 0 2024-03-04T16:04:17.310 INFO:tasks.workunit.client.0.smithi116.stdout:8/701: read d2/d38/d41/f6c [200093,29961] 0 2024-03-04T16:04:17.313 INFO:tasks.workunit.client.0.smithi116.stdout:8/702: write d2/d13/d3c/d46/d4f/f70 [525479,9216] 0 2024-03-04T16:04:17.313 INFO:tasks.workunit.client.0.smithi116.stdout:7/670: rmdir d3/d4/d8/d4a/d86 39 2024-03-04T16:04:17.322 INFO:tasks.workunit.client.0.smithi116.stdout:1/559: sync 2024-03-04T16:04:17.322 INFO:tasks.workunit.client.0.smithi116.stdout:1/560: chown d5/d9/d21/d27/d32/d35/ca2 158 1 2024-03-04T16:04:17.324 INFO:tasks.workunit.client.0.smithi116.stdout:6/605: symlink d6/de/d25/d4a/lcd 0 2024-03-04T16:04:17.328 INFO:tasks.workunit.client.0.smithi116.stdout:1/561: mknod d5/d9/d21/d27/d32/d35/d55/d75/cb7 0 2024-03-04T16:04:17.329 INFO:tasks.workunit.client.0.smithi116.stdout:6/606: dwrite d6/de/d25/d36/fa6 [0,4194304] 0 2024-03-04T16:04:17.329 INFO:tasks.workunit.client.0.smithi116.stdout:6/607: chown d6/de/d1e/d80 2490567 1 2024-03-04T16:04:17.330 INFO:tasks.workunit.client.0.smithi116.stdout:5/676: mknod dd/d1f/d5c/cd7 0 2024-03-04T16:04:17.333 INFO:tasks.workunit.client.0.smithi116.stdout:1/562: creat d5/d9/d21/d27/d32/d35/d55/fb8 x:0 0 0 2024-03-04T16:04:17.334 INFO:tasks.workunit.client.0.smithi116.stdout:1/563: readlink d5/d9/d21/d27/d8c/la9 0 2024-03-04T16:04:17.340 INFO:tasks.workunit.client.0.smithi116.stdout:5/677: unlink dd/d1f/d55/d73/d2d/d7c/lb4 0 2024-03-04T16:04:17.341 INFO:tasks.workunit.client.0.smithi116.stdout:5/678: chown dd/d45/l46 1411607 1 2024-03-04T16:04:17.342 INFO:tasks.workunit.client.0.smithi116.stdout:4/694: symlink d1/d3/d40/d52/d84/lf2 0 2024-03-04T16:04:17.344 INFO:tasks.workunit.client.0.smithi116.stdout:6/608: dread d6/de/f1f [0,4194304] 0 2024-03-04T16:04:17.344 INFO:tasks.workunit.client.0.smithi116.stdout:6/609: stat d6/de/d29/fa0 0 2024-03-04T16:04:17.350 INFO:tasks.workunit.client.0.smithi116.stdout:6/610: creat d6/de/d1e/d80/fce x:0 0 0 2024-03-04T16:04:17.353 INFO:tasks.workunit.client.0.smithi116.stdout:5/679: symlink dd/d1f/d5c/d5e/dc2/ld8 0 2024-03-04T16:04:17.364 INFO:tasks.workunit.client.0.smithi116.stdout:7/671: rmdir d3/d4/d8/d52/db4 0 2024-03-04T16:04:17.369 INFO:tasks.workunit.client.0.smithi116.stdout:4/695: getdents d1/d3/d15/d20/d1c/d62/d81/daf 0 2024-03-04T16:04:17.370 INFO:tasks.workunit.client.0.smithi116.stdout:5/680: dread dd/d6d/f80 [0,4194304] 0 2024-03-04T16:04:17.374 INFO:tasks.workunit.client.0.smithi116.stdout:7/672: symlink d3/d4/d9e/ld2 0 2024-03-04T16:04:17.376 INFO:tasks.workunit.client.0.smithi116.stdout:7/673: write d3/d4/d8/d1a/f5c [286420,42265] 0 2024-03-04T16:04:17.377 INFO:tasks.workunit.client.0.smithi116.stdout:5/681: creat dd/d1f/d55/d73/d2d/d7c/fd9 x:0 0 0 2024-03-04T16:04:17.377 INFO:tasks.workunit.client.0.smithi116.stdout:6/611: dread d6/d9e/fc0 [0,4194304] 0 2024-03-04T16:04:17.379 INFO:tasks.workunit.client.0.smithi116.stdout:5/682: truncate dd/d1f/d72/fa0 952839 0 2024-03-04T16:04:17.379 INFO:tasks.workunit.client.0.smithi116.stdout:6/612: truncate d6/de/d84/fcb 499635 0 2024-03-04T16:04:17.379 INFO:tasks.workunit.client.0.smithi116.stdout:6/613: chown d6/l6d 5682 1 2024-03-04T16:04:17.379 INFO:tasks.workunit.client.0.smithi116.stdout:5/683: write fa [180074,97102] 0 2024-03-04T16:04:17.381 INFO:tasks.workunit.client.0.smithi116.stdout:0/558: rmdir da/d16/d28 39 2024-03-04T16:04:17.381 INFO:tasks.workunit.client.0.smithi116.stdout:5/684: dread dd/d1f/d72/fa0 [0,4194304] 0 2024-03-04T16:04:17.384 INFO:tasks.workunit.client.0.smithi116.stdout:4/696: unlink d1/d3/dd/d24/d51/caa 0 2024-03-04T16:04:17.393 INFO:tasks.workunit.client.0.smithi116.stdout:7/674: mkdir d3/d4/d8/d1a/d85/dd0/dd3 0 2024-03-04T16:04:17.393 INFO:tasks.workunit.client.0.smithi116.stdout:9/615: write d2/f48 [108111,123761] 0 2024-03-04T16:04:17.394 INFO:tasks.workunit.client.0.smithi116.stdout:4/697: creat d1/d3/d15/d20/d65/ff3 x:0 0 0 2024-03-04T16:04:17.396 INFO:tasks.workunit.client.0.smithi116.stdout:6/614: truncate d6/de/d29/f8d 2167554 0 2024-03-04T16:04:17.398 INFO:tasks.workunit.client.0.smithi116.stdout:0/559: readlink da/d16/d28/l8b 0 2024-03-04T16:04:17.398 INFO:tasks.workunit.client.0.smithi116.stdout:0/560: dread - da/d16/f93 zero size 2024-03-04T16:04:17.400 INFO:tasks.workunit.client.0.smithi116.stdout:5/685: symlink dd/d6d/dc6/lda 0 2024-03-04T16:04:17.400 INFO:tasks.workunit.client.0.smithi116.stdout:0/561: fsync da/d16/d28/d40/f5c 0 2024-03-04T16:04:17.402 INFO:tasks.workunit.client.0.smithi116.stdout:7/675: creat d3/d4/d34/d5d/db9/fd4 x:0 0 0 2024-03-04T16:04:17.402 INFO:tasks.workunit.client.0.smithi116.stdout:3/712: dwrite dc/d23/d84/d7c/f5c [0,4194304] 0 2024-03-04T16:04:17.412 INFO:tasks.workunit.client.0.smithi116.stdout:0/562: readlink da/l11 0 2024-03-04T16:04:17.412 INFO:tasks.workunit.client.0.smithi116.stdout:5/686: symlink dd/d1f/d55/d73/d2d/ldb 0 2024-03-04T16:04:17.413 INFO:tasks.workunit.client.0.smithi116.stdout:7/676: mkdir d3/d4/d8/d1a/d85/daf/dd5 0 2024-03-04T16:04:17.416 INFO:tasks.workunit.client.0.smithi116.stdout:9/616: link d2/d26/f71 d2/d33/d40/fbc 0 2024-03-04T16:04:17.417 INFO:tasks.workunit.client.0.smithi116.stdout:4/698: creat d1/d6c/dbf/ff4 x:0 0 0 2024-03-04T16:04:17.422 INFO:tasks.workunit.client.0.smithi116.stdout:5/687: chown dd/ca7 249075789 1 2024-03-04T16:04:17.427 INFO:tasks.workunit.client.0.smithi116.stdout:3/713: symlink dc/d23/d84/d7f/d93/db6/dc5/de0/lf0 0 2024-03-04T16:04:17.427 INFO:tasks.workunit.client.0.smithi116.stdout:6/615: link d6/de/d84/cad d6/d9e/da5/ccf 0 2024-03-04T16:04:17.428 INFO:tasks.workunit.client.0.smithi116.stdout:7/677: dwrite d3/f1f [0,4194304] 0 2024-03-04T16:04:17.428 INFO:tasks.workunit.client.0.smithi116.stdout:0/563: symlink da/d16/d28/d40/lb4 0 2024-03-04T16:04:17.429 INFO:tasks.workunit.client.0.smithi116.stdout:3/714: write dc/d23/fda [352748,4005] 0 2024-03-04T16:04:17.430 INFO:tasks.workunit.client.0.smithi116.stdout:7/678: readlink d3/d4/d8/d1a/l30 0 2024-03-04T16:04:17.433 INFO:tasks.workunit.client.0.smithi116.stdout:7/679: write d3/d4/d11/fb7 [2335873,128987] 0 2024-03-04T16:04:17.433 INFO:tasks.workunit.client.0.smithi116.stdout:6/616: creat d6/de/d1e/d7c/fd0 x:0 0 0 2024-03-04T16:04:17.434 INFO:tasks.workunit.client.0.smithi116.stdout:0/564: dread da/d16/f6e [0,4194304] 0 2024-03-04T16:04:17.440 INFO:tasks.workunit.client.0.smithi116.stdout:3/715: mkdir dc/d23/d84/d7f/d93/df1 0 2024-03-04T16:04:17.455 INFO:tasks.workunit.client.0.smithi116.stdout:3/716: dread dc/df/d14/f5d [0,4194304] 0 2024-03-04T16:04:17.458 INFO:tasks.workunit.client.0.smithi116.stdout:3/717: dread dc/f73 [0,4194304] 0 2024-03-04T16:04:17.472 INFO:tasks.workunit.client.0.smithi116.stdout:0/565: mkdir da/d16/d20/d53/db5 0 2024-03-04T16:04:17.474 INFO:tasks.workunit.client.0.smithi116.stdout:6/617: symlink d6/de/ld1 0 2024-03-04T16:04:17.477 INFO:tasks.workunit.client.0.smithi116.stdout:8/703: dwrite d2/f60 [0,4194304] 0 2024-03-04T16:04:17.479 INFO:tasks.workunit.client.0.smithi116.stdout:8/704: write d2/de/d3e/d7b/fea [130946,109957] 0 2024-03-04T16:04:17.486 INFO:tasks.workunit.client.0.smithi116.stdout:8/705: dwrite d2/d13/d3c/d77/fd9 [0,4194304] 0 2024-03-04T16:04:17.487 INFO:tasks.workunit.client.0.smithi116.stdout:3/718: rename dc/d34/la8 to dc/d23/d84/d7f/d93/lf2 0 2024-03-04T16:04:17.491 INFO:tasks.workunit.client.0.smithi116.stdout:8/706: chown d2/lf 248934440 1 2024-03-04T16:04:17.494 INFO:tasks.workunit.client.0.smithi116.stdout:0/566: rename da/d16/d20/d42/f86 to da/d16/d28/d40/d48/d57/db3/db2/fb6 0 2024-03-04T16:04:17.494 INFO:tasks.workunit.client.0.smithi116.stdout:8/707: read d2/d13/d3c/d77/fd9 [3895497,67464] 0 2024-03-04T16:04:17.495 INFO:tasks.workunit.client.0.smithi116.stdout:3/719: dwrite dc/d23/fcd [0,4194304] 0 2024-03-04T16:04:17.500 INFO:tasks.workunit.client.0.smithi116.stdout:6/618: rename d6/de/d1e/f43 to d6/de/d25/d36/d44/dbb/fd2 0 2024-03-04T16:04:17.501 INFO:tasks.workunit.client.0.smithi116.stdout:6/619: write d6/de/d25/d36/fa6 [1818914,101597] 0 2024-03-04T16:04:17.502 INFO:tasks.workunit.client.0.smithi116.stdout:6/620: dread - d6/de/d29/fa0 zero size 2024-03-04T16:04:17.503 INFO:tasks.workunit.client.0.smithi116.stdout:6/621: write d6/de/d1e/f3f [1066714,78730] 0 2024-03-04T16:04:17.519 INFO:tasks.workunit.client.0.smithi116.stdout:1/564: creat d5/d9/d21/d27/fb9 x:0 0 0 2024-03-04T16:04:17.522 INFO:tasks.workunit.client.0.smithi116.stdout:8/708: creat d2/de/d16/d78/d29/ff0 x:0 0 0 2024-03-04T16:04:17.524 INFO:tasks.workunit.client.0.smithi116.stdout:1/565: dwrite d5/d9/f18 [0,4194304] 0 2024-03-04T16:04:17.531 INFO:tasks.workunit.client.0.smithi116.stdout:1/566: dwrite d5/d9/d21/d24/f37 [4194304,4194304] 0 2024-03-04T16:04:17.531 INFO:tasks.workunit.client.0.smithi116.stdout:8/709: rmdir d2/d13/db9/dbf 0 2024-03-04T16:04:17.536 INFO:tasks.workunit.client.0.smithi116.stdout:8/710: fsync d2/d13/d3c/d77/fb3 0 2024-03-04T16:04:17.539 INFO:tasks.workunit.client.0.smithi116.stdout:1/567: dwrite d5/d9/d21/d27/d32/d35/d55/fb8 [0,4194304] 0 2024-03-04T16:04:17.547 INFO:tasks.workunit.client.0.smithi116.stdout:1/568: rename d5/d9/d21/d27/d32/d35/d55/db4/d50/f71 to d5/d9/d21/d27/d32/d35/d55/d75/d91/fba 0 2024-03-04T16:04:17.548 INFO:tasks.workunit.client.0.smithi116.stdout:1/569: chown d5/d9/d21/d27/d94/c99 46068399 1 2024-03-04T16:04:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:1/570: rename d5/d9/d21/d27/d94/db6/c12 to d5/cbb 0 2024-03-04T16:04:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:7/680: getdents d3/d4/d8/d1a/d85/daf 0 2024-03-04T16:04:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:7/681: mknod d3/d4/d34/d5d/cd6 0 2024-03-04T16:04:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:7/682: rename d3/d4/d8/d1a/d85 to d3/d4/d9e/dd7 0 2024-03-04T16:04:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:7/683: write d3/d95/f8e [648764,39186] 0 2024-03-04T16:04:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:9/617: write d2/d19/f2a [168340,70506] 0 2024-03-04T16:04:17.571 INFO:tasks.workunit.client.0.smithi116.stdout:9/618: unlink d2/d19/l5d 0 2024-03-04T16:04:17.574 INFO:tasks.workunit.client.0.smithi116.stdout:9/619: unlink d2/d33/d65/d6c/ca6 0 2024-03-04T16:04:17.581 INFO:tasks.workunit.client.0.smithi116.stdout:2/718: rmdir d7/d13/d3e/d98 39 2024-03-04T16:04:17.584 INFO:tasks.workunit.client.0.smithi116.stdout:2/719: creat d7/d81/f10d x:0 0 0 2024-03-04T16:04:17.590 INFO:tasks.workunit.client.0.smithi116.stdout:9/620: dread d2/f1a [0,4194304] 0 2024-03-04T16:04:17.592 INFO:tasks.workunit.client.0.smithi116.stdout:9/621: mkdir d2/d19/d27/dbd 0 2024-03-04T16:04:17.595 INFO:tasks.workunit.client.0.smithi116.stdout:9/622: mkdir d2/d19/d27/d3b/d4a/d6a/d94/dbe 0 2024-03-04T16:04:17.596 INFO:tasks.workunit.client.0.smithi116.stdout:2/720: dread d7/d13/f1e [0,4194304] 0 2024-03-04T16:04:17.599 INFO:tasks.workunit.client.0.smithi116.stdout:2/721: mknod d7/d13/d3e/c10e 0 2024-03-04T16:04:17.604 INFO:tasks.workunit.client.0.smithi116.stdout:2/722: getdents d7/d13/d3e/ded/d76 0 2024-03-04T16:04:17.607 INFO:tasks.workunit.client.0.smithi116.stdout:2/723: chown d7/d13/d3e/d98/dce/f4b 5717 1 2024-03-04T16:04:17.607 INFO:tasks.workunit.client.0.smithi116.stdout:2/724: chown d7/d25/fa2 111453304 1 2024-03-04T16:04:17.615 INFO:tasks.workunit.client.0.smithi116.stdout:3/720: dwrite dc/df/d14/f49 [0,4194304] 0 2024-03-04T16:04:17.617 INFO:tasks.workunit.client.0.smithi116.stdout:3/721: truncate dc/df/d47/f7d 226705 0 2024-03-04T16:04:17.618 INFO:tasks.workunit.client.0.smithi116.stdout:3/722: truncate dc/df/db8/fcc 474705 0 2024-03-04T16:04:17.618 INFO:tasks.workunit.client.0.smithi116.stdout:3/723: dread - dc/d23/d84/d7f/d93/fea zero size 2024-03-04T16:04:17.625 INFO:tasks.workunit.client.0.smithi116.stdout:3/724: read dc/d23/d84/d7f/d93/db6/dc5/de0/da3/fb2 [149852,93279] 0 2024-03-04T16:04:17.626 INFO:tasks.workunit.client.0.smithi116.stdout:6/622: write d6/de/d29/f8d [659016,78787] 0 2024-03-04T16:04:17.628 INFO:tasks.workunit.client.0.smithi116.stdout:6/623: chown d6/de/d25/d36/d44/fba 535753154 1 2024-03-04T16:04:17.630 INFO:tasks.workunit.client.0.smithi116.stdout:8/711: write d2/fd [2390603,117049] 0 2024-03-04T16:04:17.631 INFO:tasks.workunit.client.0.smithi116.stdout:8/712: chown d2/de/d16/d78/d29/d7a 61955 1 2024-03-04T16:04:17.631 INFO:tasks.workunit.client.0.smithi116.stdout:8/713: dread - d2/de/d16/fd8 zero size 2024-03-04T16:04:17.635 INFO:tasks.workunit.client.0.smithi116.stdout:6/624: mknod d6/de/d25/d4a/cd3 0 2024-03-04T16:04:17.638 INFO:tasks.workunit.client.0.smithi116.stdout:8/714: fdatasync d2/d13/d5e/f87 0 2024-03-04T16:04:17.640 INFO:tasks.workunit.client.0.smithi116.stdout:1/571: symlink d5/d9/lbc 0 2024-03-04T16:04:17.641 INFO:tasks.workunit.client.0.smithi116.stdout:7/684: truncate d3/d4/d8/f40 3531721 0 2024-03-04T16:04:17.642 INFO:tasks.workunit.client.0.smithi116.stdout:6/625: dwrite d6/de/d1e/f48 [4194304,4194304] 0 2024-03-04T16:04:17.645 INFO:tasks.workunit.client.0.smithi116.stdout:1/572: readlink d5/d9/d21/d27/d32/d35/l77 0 2024-03-04T16:04:17.649 INFO:tasks.workunit.client.0.smithi116.stdout:8/715: dwrite d2/d13/d3c/d77/d97/fc2 [0,4194304] 0 2024-03-04T16:04:17.650 INFO:tasks.workunit.client.0.smithi116.stdout:1/573: read d5/d9/d21/d27/d32/d35/f76 [655531,28219] 0 2024-03-04T16:04:17.651 INFO:tasks.workunit.client.0.smithi116.stdout:8/716: read d2/de/d3e/d7b/fea [227355,90343] 0 2024-03-04T16:04:17.654 INFO:tasks.workunit.client.0.smithi116.stdout:6/626: dwrite d6/de/d1e/f48 [4194304,4194304] 0 2024-03-04T16:04:17.657 INFO:tasks.workunit.client.0.smithi116.stdout:6/627: truncate d6/de/d1e/d7c/fd0 821033 0 2024-03-04T16:04:17.658 INFO:tasks.workunit.client.0.smithi116.stdout:6/628: fdatasync d6/de/d84/f8f 0 2024-03-04T16:04:17.658 INFO:tasks.workunit.client.0.smithi116.stdout:1/574: rmdir d5/d9/d21/d27 39 2024-03-04T16:04:17.663 INFO:tasks.workunit.client.0.smithi116.stdout:7/685: link d3/d4/c3d d3/d4/d8/d4a/d88/dc2/cd8 0 2024-03-04T16:04:17.666 INFO:tasks.workunit.client.0.smithi116.stdout:8/717: mknod d2/d13/d3c/d46/d6e/cf1 0 2024-03-04T16:04:17.667 INFO:tasks.workunit.client.0.smithi116.stdout:8/718: chown d2/d13/d3c/d77/laa 1007 1 2024-03-04T16:04:17.671 INFO:tasks.workunit.client.0.smithi116.stdout:6/629: truncate d6/de/d29/d3a/f46 310982 0 2024-03-04T16:04:17.672 INFO:tasks.workunit.client.0.smithi116.stdout:7/686: mknod d3/d4/d8/d4a/d88/cd9 0 2024-03-04T16:04:17.673 INFO:tasks.workunit.client.0.smithi116.stdout:7/687: dread - d3/d4/d8/d4a/d88/f93 zero size 2024-03-04T16:04:17.677 INFO:tasks.workunit.client.0.smithi116.stdout:8/719: rename d2/de/d16/d2b to d2/de/d16/d78/d29/de5/df2 0 2024-03-04T16:04:17.678 INFO:tasks.workunit.client.0.smithi116.stdout:8/720: fsync d2/de/d3e/d7b/fdd 0 2024-03-04T16:04:17.682 INFO:tasks.workunit.client.0.smithi116.stdout:1/575: rename d5/d9/d21/d27/d32/la8 to d5/d9/d21/d27/lbd 0 2024-03-04T16:04:17.683 INFO:tasks.workunit.client.0.smithi116.stdout:3/725: dwrite dc/df/d14/f8c [0,4194304] 0 2024-03-04T16:04:17.683 INFO:tasks.workunit.client.0.smithi116.stdout:1/576: fdatasync d5/d9/d21/d27/d94/db6/f69 0 2024-03-04T16:04:17.685 INFO:tasks.workunit.client.0.smithi116.stdout:8/721: write d2/de/d16/d78/d29/de5/df2/fa0 [1790889,29036] 0 2024-03-04T16:04:17.692 INFO:tasks.workunit.client.0.smithi116.stdout:6/630: rmdir d6/d9e/da5/dca 0 2024-03-04T16:04:17.694 INFO:tasks.workunit.client.0.smithi116.stdout:4/699: sync 2024-03-04T16:04:17.694 INFO:tasks.workunit.client.0.smithi116.stdout:6/631: truncate d6/de/d15/fb2 878777 0 2024-03-04T16:04:17.694 INFO:tasks.workunit.client.0.smithi116.stdout:4/700: fsync d1/d3/d15/fb6 0 2024-03-04T16:04:17.698 INFO:tasks.workunit.client.0.smithi116.stdout:4/701: dread d1/d3/dd/f98 [0,4194304] 0 2024-03-04T16:04:17.701 INFO:tasks.workunit.client.0.smithi116.stdout:1/577: dread d5/d9/d21/d27/d32/d49/d4a/f93 [0,4194304] 0 2024-03-04T16:04:17.703 INFO:tasks.workunit.client.0.smithi116.stdout:8/722: mknod d2/cf3 0 2024-03-04T16:04:17.707 INFO:tasks.workunit.client.0.smithi116.stdout:4/702: mknod d1/d3/dd/d8a/cf5 0 2024-03-04T16:04:17.708 INFO:tasks.workunit.client.0.smithi116.stdout:4/703: chown d1/d3/dd/d45/l59 1436090 1 2024-03-04T16:04:17.709 INFO:tasks.workunit.client.0.smithi116.stdout:1/578: write d5/d9/d21/d27/d32/d35/d55/d75/d91/fb2 [380050,31793] 0 2024-03-04T16:04:17.714 INFO:tasks.workunit.client.0.smithi116.stdout:6/632: link d6/d56/d58/d72/ccc d6/d56/d58/d72/d7b/cd4 0 2024-03-04T16:04:17.714 INFO:tasks.workunit.client.0.smithi116.stdout:3/726: dread dc/df/d47/d91/fd4 [0,4194304] 0 2024-03-04T16:04:17.715 INFO:tasks.workunit.client.0.smithi116.stdout:4/704: dwrite d1/d6c/f38 [0,4194304] 0 2024-03-04T16:04:17.717 INFO:tasks.workunit.client.0.smithi116.stdout:1/579: creat d5/d9/fbe x:0 0 0 2024-03-04T16:04:17.718 INFO:tasks.workunit.client.0.smithi116.stdout:4/705: write d1/d3/de/d47/f60 [208525,50144] 0 2024-03-04T16:04:17.718 INFO:tasks.workunit.client.0.smithi116.stdout:1/580: readlink d5/d9/d21/d27/l2d 0 2024-03-04T16:04:17.720 INFO:tasks.workunit.client.0.smithi116.stdout:7/688: truncate d3/d4/d8/d4a/db3/db6/f81 1667930 0 2024-03-04T16:04:17.722 INFO:tasks.workunit.client.0.smithi116.stdout:6/633: chown d6/d9e/da5/ccf 325 1 2024-03-04T16:04:17.730 INFO:tasks.workunit.client.0.smithi116.stdout:4/706: creat d1/d3/dd/d8a/ddc/ff6 x:0 0 0 2024-03-04T16:04:17.733 INFO:tasks.workunit.client.0.smithi116.stdout:7/689: creat d3/d4/d8/d4a/db3/db6/fda x:0 0 0 2024-03-04T16:04:17.734 INFO:tasks.workunit.client.0.smithi116.stdout:6/634: truncate d6/f9 2124284 0 2024-03-04T16:04:17.740 INFO:tasks.workunit.client.0.smithi116.stdout:0/567: sync 2024-03-04T16:04:17.740 INFO:tasks.workunit.client.0.smithi116.stdout:5/688: sync 2024-03-04T16:04:17.741 INFO:tasks.workunit.client.0.smithi116.stdout:8/723: dwrite d2/d13/d3c/d77/fb3 [0,4194304] 0 2024-03-04T16:04:17.742 INFO:tasks.workunit.client.0.smithi116.stdout:5/689: chown dd/d1f/d55/f77 32296716 1 2024-03-04T16:04:17.742 INFO:tasks.workunit.client.0.smithi116.stdout:0/568: fdatasync da/d16/d19/f1d 0 2024-03-04T16:04:17.744 INFO:tasks.workunit.client.0.smithi116.stdout:0/569: fdatasync da/d16/f93 0 2024-03-04T16:04:17.747 INFO:tasks.workunit.client.0.smithi116.stdout:3/727: write dc/df/d14/f5d [5922499,44462] 0 2024-03-04T16:04:17.752 INFO:tasks.workunit.client.0.smithi116.stdout:7/690: rename d3/d4/d9e/ld2 to d3/d4/d8/d4a/d86/ldb 0 2024-03-04T16:04:17.754 INFO:tasks.workunit.client.0.smithi116.stdout:5/690: creat dd/d1f/d55/d73/d2d/fdc x:0 0 0 2024-03-04T16:04:17.755 INFO:tasks.workunit.client.0.smithi116.stdout:5/691: chown dd/f22 0 1 2024-03-04T16:04:17.756 INFO:tasks.workunit.client.0.smithi116.stdout:0/570: unlink da/d16/d28/d40/d5d/d7e/f83 0 2024-03-04T16:04:17.762 INFO:tasks.workunit.client.0.smithi116.stdout:1/581: truncate d5/d9/d21/d27/d32/fa3 2443977 0 2024-03-04T16:04:17.764 INFO:tasks.workunit.client.0.smithi116.stdout:3/728: mknod dc/d34/cf3 0 2024-03-04T16:04:17.765 INFO:tasks.workunit.client.0.smithi116.stdout:4/707: dread d1/fae [0,4194304] 0 2024-03-04T16:04:17.765 INFO:tasks.workunit.client.0.smithi116.stdout:0/571: unlink da/d16/d19/d27/d70/l7a 0 2024-03-04T16:04:17.765 INFO:tasks.workunit.client.0.smithi116.stdout:4/708: chown d1/d3/dd/d45/d50/c83 0 1 2024-03-04T16:04:17.776 INFO:tasks.workunit.client.0.smithi116.stdout:7/691: write d3/d4/d11/f38 [4393381,17460] 0 2024-03-04T16:04:17.777 INFO:tasks.workunit.client.0.smithi116.stdout:1/582: write d5/d9/d21/d27/d32/f3e [3363999,58096] 0 2024-03-04T16:04:17.777 INFO:tasks.workunit.client.0.smithi116.stdout:7/692: stat d3/d4/d11/f36 0 2024-03-04T16:04:17.779 INFO:tasks.workunit.client.0.smithi116.stdout:7/693: write d3/d4/d8/d1a/f32 [3792519,14969] 0 2024-03-04T16:04:17.786 INFO:tasks.workunit.client.0.smithi116.stdout:5/692: getdents dd/d1f/d55/d73/d2d 0 2024-03-04T16:04:17.787 INFO:tasks.workunit.client.0.smithi116.stdout:3/729: write dc/df/f92 [4918745,128345] 0 2024-03-04T16:04:17.791 INFO:tasks.workunit.client.0.smithi116.stdout:7/694: symlink d3/d4/d9e/dd7/dd0/ldc 0 2024-03-04T16:04:17.795 INFO:tasks.workunit.client.0.smithi116.stdout:0/572: truncate da/d16/d20/f3b 1105222 0 2024-03-04T16:04:17.795 INFO:tasks.workunit.client.0.smithi116.stdout:0/573: chown da/fe 123004903 1 2024-03-04T16:04:17.796 INFO:tasks.workunit.client.0.smithi116.stdout:3/730: mknod dc/dd3/cf4 0 2024-03-04T16:04:17.796 INFO:tasks.workunit.client.0.smithi116.stdout:7/695: dread d3/d4/d8/f3e [0,4194304] 0 2024-03-04T16:04:17.797 INFO:tasks.workunit.client.0.smithi116.stdout:3/731: readlink dc/d23/d84/d7f/d93/db6/dc5/lc6 0 2024-03-04T16:04:17.797 INFO:tasks.workunit.client.0.smithi116.stdout:0/574: chown da/d16/d28/d40/d48/d77/l89 0 1 2024-03-04T16:04:17.798 INFO:tasks.workunit.client.0.smithi116.stdout:1/583: write d5/d9/d21/d84/f90 [897094,116345] 0 2024-03-04T16:04:17.800 INFO:tasks.workunit.client.0.smithi116.stdout:4/709: getdents d1/d3/d40/d52 0 2024-03-04T16:04:17.802 INFO:tasks.workunit.client.0.smithi116.stdout:5/693: dread dd/d1f/d51/f93 [0,4194304] 0 2024-03-04T16:04:17.803 INFO:tasks.workunit.client.0.smithi116.stdout:7/696: chown d3/d4/d11/c47 21 1 2024-03-04T16:04:17.803 INFO:tasks.workunit.client.0.smithi116.stdout:3/732: dread dc/d34/f3e [0,4194304] 0 2024-03-04T16:04:17.805 INFO:tasks.workunit.client.0.smithi116.stdout:0/575: mkdir da/d16/d28/d40/d48/da7/db7 0 2024-03-04T16:04:17.805 INFO:tasks.workunit.client.0.smithi116.stdout:1/584: mkdir d5/d9/d21/d27/d32/d35/dbf 0 2024-03-04T16:04:17.806 INFO:tasks.workunit.client.0.smithi116.stdout:0/576: readlink da/d16/d28/d40/d48/l4d 0 2024-03-04T16:04:17.808 INFO:tasks.workunit.client.0.smithi116.stdout:4/710: creat d1/d3/d15/d20/d1c/d62/d81/da6/ff7 x:0 0 0 2024-03-04T16:04:17.809 INFO:tasks.workunit.client.0.smithi116.stdout:5/694: write dd/d1f/d51/fbf [9054653,11834] 0 2024-03-04T16:04:17.813 INFO:tasks.workunit.client.0.smithi116.stdout:1/585: creat d5/d9/d21/d27/d32/d35/d55/fc0 x:0 0 0 2024-03-04T16:04:17.813 INFO:tasks.workunit.client.0.smithi116.stdout:3/733: dwrite dc/f70 [0,4194304] 0 2024-03-04T16:04:17.815 INFO:tasks.workunit.client.0.smithi116.stdout:5/695: mknod dd/d1f/d51/cdd 0 2024-03-04T16:04:17.817 INFO:tasks.workunit.client.0.smithi116.stdout:0/577: unlink da/d16/d28/d40/c71 0 2024-03-04T16:04:17.820 INFO:tasks.workunit.client.0.smithi116.stdout:3/734: write dc/d23/d84/f8e [1475622,55149] 0 2024-03-04T16:04:17.823 INFO:tasks.workunit.client.0.smithi116.stdout:7/697: creat d3/d4/d8/d4a/fdd x:0 0 0 2024-03-04T16:04:17.823 INFO:tasks.workunit.client.0.smithi116.stdout:1/586: symlink d5/lc1 0 2024-03-04T16:04:17.825 INFO:tasks.workunit.client.0.smithi116.stdout:0/578: dwrite da/d16/fb1 [0,4194304] 0 2024-03-04T16:04:17.826 INFO:tasks.workunit.client.0.smithi116.stdout:0/579: stat da/d16/d28/d40/f5c 0 2024-03-04T16:04:17.827 INFO:tasks.workunit.client.0.smithi116.stdout:3/735: creat dc/d34/de7/deb/ff5 x:0 0 0 2024-03-04T16:04:17.827 INFO:tasks.workunit.client.0.smithi116.stdout:4/711: creat d1/d3/d15/ff8 x:0 0 0 2024-03-04T16:04:17.829 INFO:tasks.workunit.client.0.smithi116.stdout:4/712: stat d1/d3/d15/d20/d65/ff3 0 2024-03-04T16:04:17.830 INFO:tasks.workunit.client.0.smithi116.stdout:4/713: chown d1/d3/dd/d24/d51/c54 33712446 1 2024-03-04T16:04:17.832 INFO:tasks.workunit.client.0.smithi116.stdout:7/698: rename d3/d4/d34/d5d/cd6 to d3/d4/d8/cde 0 2024-03-04T16:04:17.833 INFO:tasks.workunit.client.0.smithi116.stdout:0/580: symlink da/d16/d20/d42/d4e/lb8 0 2024-03-04T16:04:17.834 INFO:tasks.workunit.client.0.smithi116.stdout:0/581: chown da/l1c 139534 1 2024-03-04T16:04:17.836 INFO:tasks.workunit.client.0.smithi116.stdout:1/587: creat d5/d9/d21/d27/d32/d35/d55/d75/fc2 x:0 0 0 2024-03-04T16:04:17.839 INFO:tasks.workunit.client.0.smithi116.stdout:4/714: mkdir d1/d3/de/d47/df9 0 2024-03-04T16:04:17.841 INFO:tasks.workunit.client.0.smithi116.stdout:4/715: write d1/d3/de/f30 [822020,112761] 0 2024-03-04T16:04:17.842 INFO:tasks.workunit.client.0.smithi116.stdout:4/716: dread - d1/d3/d15/d20/d65/f99 zero size 2024-03-04T16:04:17.842 INFO:tasks.workunit.client.0.smithi116.stdout:0/582: fsync da/f35 0 2024-03-04T16:04:17.853 INFO:tasks.workunit.client.0.smithi116.stdout:0/583: creat da/d16/d28/d40/d5d/d7e/fb9 x:0 0 0 2024-03-04T16:04:17.858 INFO:tasks.workunit.client.0.smithi116.stdout:1/588: creat d5/da1/fc3 x:0 0 0 2024-03-04T16:04:17.859 INFO:tasks.workunit.client.0.smithi116.stdout:0/584: mkdir da/d16/d28/d40/d48/d57/d8a/daa/dba 0 2024-03-04T16:04:17.862 INFO:tasks.workunit.client.0.smithi116.stdout:1/589: creat d5/d9/d21/d27/d32/d35/d55/db4/fc4 x:0 0 0 2024-03-04T16:04:17.863 INFO:tasks.workunit.client.0.smithi116.stdout:0/585: rename da/d16/d28/c62 to da/d16/d28/d40/da8/cbb 0 2024-03-04T16:04:17.866 INFO:tasks.workunit.client.0.smithi116.stdout:0/586: rename da/d16/d28/d39/l3f to da/d16/d28/d40/d48/d77/lbc 0 2024-03-04T16:04:17.867 INFO:tasks.workunit.client.0.smithi116.stdout:0/587: chown da/d16/d28/d40/d48/l4d 551 1 2024-03-04T16:04:17.868 INFO:tasks.workunit.client.0.smithi116.stdout:5/696: truncate dd/d1f/d55/d73/d2d/d31/f6a 2751449 0 2024-03-04T16:04:17.868 INFO:tasks.workunit.client.0.smithi116.stdout:5/697: readlink dd/d1f/d5c/l96 0 2024-03-04T16:04:17.873 INFO:tasks.workunit.client.0.smithi116.stdout:1/590: getdents d5/d9 0 2024-03-04T16:04:17.875 INFO:tasks.workunit.client.0.smithi116.stdout:5/698: creat dd/d1f/d55/d73/fde x:0 0 0 2024-03-04T16:04:17.879 INFO:tasks.workunit.client.0.smithi116.stdout:3/736: truncate dc/d23/d84/d7c/f5c 4131108 0 2024-03-04T16:04:17.879 INFO:tasks.workunit.client.0.smithi116.stdout:9/623: sync 2024-03-04T16:04:17.879 INFO:tasks.workunit.client.0.smithi116.stdout:2/725: sync 2024-03-04T16:04:17.880 INFO:tasks.workunit.client.0.smithi116.stdout:2/726: write d7/d15/dd7/fb4 [1082746,73039] 0 2024-03-04T16:04:17.881 INFO:tasks.workunit.client.0.smithi116.stdout:1/591: truncate d5/d9/d21/d27/d32/d49/f66 758927 0 2024-03-04T16:04:17.881 INFO:tasks.workunit.client.0.smithi116.stdout:7/699: fdatasync d3/d4/d8/d52/d5e/fa4 0 2024-03-04T16:04:17.881 INFO:tasks.workunit.client.0.smithi116.stdout:2/727: chown d7/f2b 864472 1 2024-03-04T16:04:17.883 INFO:tasks.workunit.client.0.smithi116.stdout:2/728: fdatasync d7/d13/d3e/ded/d6c/da4/fc3 0 2024-03-04T16:04:17.884 INFO:tasks.workunit.client.0.smithi116.stdout:5/699: rename dd/d1f/d55/d73/d2d/d59/lb7 to dd/d1f/d55/ldf 0 2024-03-04T16:04:17.887 INFO:tasks.workunit.client.0.smithi116.stdout:3/737: chown dc/d23/l32 175372 1 2024-03-04T16:04:17.889 INFO:tasks.workunit.client.0.smithi116.stdout:1/592: creat d5/d9/d21/d27/d32/d35/d55/db4/fc5 x:0 0 0 2024-03-04T16:04:17.890 INFO:tasks.workunit.client.0.smithi116.stdout:4/717: dwrite d1/d3/d40/d52/f76 [0,4194304] 0 2024-03-04T16:04:17.892 INFO:tasks.workunit.client.0.smithi116.stdout:2/729: fdatasync d7/d25/fa2 0 2024-03-04T16:04:17.894 INFO:tasks.workunit.client.0.smithi116.stdout:0/588: link da/d16/d28/d40/d5d/l5e da/d16/d28/d7b/lbd 0 2024-03-04T16:04:17.899 INFO:tasks.workunit.client.0.smithi116.stdout:3/738: write dc/d34/f75 [361934,81404] 0 2024-03-04T16:04:17.901 INFO:tasks.workunit.client.0.smithi116.stdout:9/624: mkdir d2/d33/d65/dbf 0 2024-03-04T16:04:17.902 INFO:tasks.workunit.client.0.smithi116.stdout:7/700: mkdir d3/d4/d9e/dd7/dd0/dd3/ddf 0 2024-03-04T16:04:17.903 INFO:tasks.workunit.client.0.smithi116.stdout:1/593: rmdir d5/d9/d21/d27/d94 39 2024-03-04T16:04:17.905 INFO:tasks.workunit.client.0.smithi116.stdout:4/718: creat d1/d3/dd/d45/ffa x:0 0 0 2024-03-04T16:04:17.907 INFO:tasks.workunit.client.0.smithi116.stdout:7/701: read d3/f6 [791536,49470] 0 2024-03-04T16:04:17.912 INFO:tasks.workunit.client.0.smithi116.stdout:1/594: symlink d5/d9/d21/d27/d8c/lc6 0 2024-03-04T16:04:17.914 INFO:tasks.workunit.client.0.smithi116.stdout:4/719: mknod d1/d3/d40/d52/d84/cfb 0 2024-03-04T16:04:17.916 INFO:tasks.workunit.client.0.smithi116.stdout:4/720: chown d1/d3/d40/d52/fd4 997 1 2024-03-04T16:04:17.916 INFO:tasks.workunit.client.0.smithi116.stdout:4/721: write d1/d3/d15/d20/d1c/d62/d81/da6/fbd [2944207,55749] 0 2024-03-04T16:04:17.917 INFO:tasks.workunit.client.0.smithi116.stdout:4/722: chown d1/d3/dd/dd9/cf0 771 1 2024-03-04T16:04:17.918 INFO:tasks.workunit.client.0.smithi116.stdout:4/723: chown d1/d3/dd/d8a 879 1 2024-03-04T16:04:17.922 INFO:tasks.workunit.client.0.smithi116.stdout:1/595: fsync d5/d9/d21/d27/d32/d35/f5c 0 2024-03-04T16:04:17.923 INFO:tasks.workunit.client.0.smithi116.stdout:7/702: mkdir d3/d4/d9e/dd7/dd0/dd3/ddf/de0 0 2024-03-04T16:04:17.925 INFO:tasks.workunit.client.0.smithi116.stdout:9/625: dread d2/d19/d27/d3b/dab/d3c/f76 [0,4194304] 0 2024-03-04T16:04:17.928 INFO:tasks.workunit.client.0.smithi116.stdout:2/730: write d7/d13/fbf [550005,57854] 0 2024-03-04T16:04:17.934 INFO:tasks.workunit.client.0.smithi116.stdout:9/626: symlink d2/d33/d65/d7e/lc0 0 2024-03-04T16:04:17.936 INFO:tasks.workunit.client.0.smithi116.stdout:3/739: getdents dc/df/d47/d91 0 2024-03-04T16:04:17.937 INFO:tasks.workunit.client.0.smithi116.stdout:3/740: chown dc/d23/d84/d7f/d93/db6/dc5/de0/lf0 2888497 1 2024-03-04T16:04:17.938 INFO:tasks.workunit.client.0.smithi116.stdout:4/724: link d1/d3/de/f26 d1/d6c/dbf/ffc 0 2024-03-04T16:04:17.943 INFO:tasks.workunit.client.0.smithi116.stdout:9/627: rmdir d2/d26 39 2024-03-04T16:04:17.944 INFO:tasks.workunit.client.0.smithi116.stdout:2/731: dread d7/d25/fbd [0,4194304] 0 2024-03-04T16:04:17.944 INFO:tasks.workunit.client.0.smithi116.stdout:3/741: fdatasync dc/d23/d84/f2d 0 2024-03-04T16:04:17.945 INFO:tasks.workunit.client.0.smithi116.stdout:4/725: dwrite d1/d3/d15/d20/d1c/d62/d81/da6/ff7 [0,4194304] 0 2024-03-04T16:04:17.954 INFO:tasks.workunit.client.0.smithi116.stdout:9/628: creat d2/d19/d27/d3b/d4a/d6a/fc1 x:0 0 0 2024-03-04T16:04:17.955 INFO:tasks.workunit.client.0.smithi116.stdout:9/629: write d2/d33/f54 [784662,118686] 0 2024-03-04T16:04:17.955 INFO:tasks.workunit.client.0.smithi116.stdout:2/732: creat d7/d81/dc2/f10f x:0 0 0 2024-03-04T16:04:17.960 INFO:tasks.workunit.client.0.smithi116.stdout:3/742: creat dc/d23/d84/d7f/d93/db6/dee/ff6 x:0 0 0 2024-03-04T16:04:17.960 INFO:tasks.workunit.client.0.smithi116.stdout:1/596: write d5/f3b [827829,29186] 0 2024-03-04T16:04:17.961 INFO:tasks.workunit.client.0.smithi116.stdout:3/743: dread - dc/d23/d84/d7f/d93/db6/fdc zero size 2024-03-04T16:04:17.966 INFO:tasks.workunit.client.0.smithi116.stdout:7/703: dwrite d3/d4/d8/d4a/d87/f9d [0,4194304] 0 2024-03-04T16:04:17.968 INFO:tasks.workunit.client.0.smithi116.stdout:1/597: write d5/d9/d21/d27/d94/db6/d42/f9c [2324829,85996] 0 2024-03-04T16:04:17.972 INFO:tasks.workunit.client.0.smithi116.stdout:9/630: creat d2/d33/d90/da7/fc2 x:0 0 0 2024-03-04T16:04:17.976 INFO:tasks.workunit.client.0.smithi116.stdout:1/598: dwrite d5/d9/d21/d27/d94/db6/f69 [0,4194304] 0 2024-03-04T16:04:17.979 INFO:tasks.workunit.client.0.smithi116.stdout:4/726: dwrite d1/d3/d15/d20/fa3 [0,4194304] 0 2024-03-04T16:04:17.981 INFO:tasks.workunit.client.0.smithi116.stdout:7/704: fsync d3/f68 0 2024-03-04T16:04:17.987 INFO:tasks.workunit.client.0.smithi116.stdout:3/744: dwrite dc/d23/f21 [0,4194304] 0 2024-03-04T16:04:17.994 INFO:tasks.workunit.client.0.smithi116.stdout:1/599: write d5/d9/d21/f2c [3352076,17690] 0 2024-03-04T16:04:17.999 INFO:tasks.workunit.client.0.smithi116.stdout:7/705: symlink d3/d4/d8/d4a/db3/dc3/le1 0 2024-03-04T16:04:18.000 INFO:tasks.workunit.client.0.smithi116.stdout:7/706: write d3/d4/d7e/fcd [656247,53413] 0 2024-03-04T16:04:18.002 INFO:tasks.workunit.client.0.smithi116.stdout:8/724: sync 2024-03-04T16:04:18.002 INFO:tasks.workunit.client.0.smithi116.stdout:6/635: sync 2024-03-04T16:04:18.004 INFO:tasks.workunit.client.0.smithi116.stdout:3/745: fsync dc/df/d47/d91/fd4 0 2024-03-04T16:04:18.005 INFO:tasks.workunit.client.0.smithi116.stdout:2/733: getdents d7/d13/d3e/ded/d6c 0 2024-03-04T16:04:18.008 INFO:tasks.workunit.client.0.smithi116.stdout:7/707: mknod d3/d4/d8/d4a/d87/ce2 0 2024-03-04T16:04:18.009 INFO:tasks.workunit.client.0.smithi116.stdout:6/636: creat d6/fd5 x:0 0 0 2024-03-04T16:04:18.011 INFO:tasks.workunit.client.0.smithi116.stdout:9/631: link d2/d19/f2a d2/fc3 0 2024-03-04T16:04:18.012 INFO:tasks.workunit.client.0.smithi116.stdout:9/632: write d2/d19/f9c [3474563,38610] 0 2024-03-04T16:04:18.016 INFO:tasks.workunit.client.0.smithi116.stdout:8/725: truncate d2/de/d16/d78/d29/de5/df2/d37/f49 847712 0 2024-03-04T16:04:18.017 INFO:tasks.workunit.client.0.smithi116.stdout:9/633: dwrite d2/d19/d27/d3b/d4a/f73 [4194304,4194304] 0 2024-03-04T16:04:18.019 INFO:tasks.workunit.client.0.smithi116.stdout:2/734: unlink d7/d13/d3e/ded/d76/f9a 0 2024-03-04T16:04:18.020 INFO:tasks.workunit.client.0.smithi116.stdout:4/727: link d1/d3/dd/l1d d1/d6e/d74/de0/lfd 0 2024-03-04T16:04:18.022 INFO:tasks.workunit.client.0.smithi116.stdout:7/708: creat d3/d4/d9e/fe3 x:0 0 0 2024-03-04T16:04:18.023 INFO:tasks.workunit.client.0.smithi116.stdout:9/634: dread d2/d33/f3a [0,4194304] 0 2024-03-04T16:04:18.028 INFO:tasks.workunit.client.0.smithi116.stdout:6/637: dwrite d6/de/d25/d4a/d63/d8c/fa3 [0,4194304] 0 2024-03-04T16:04:18.028 INFO:tasks.workunit.client.0.smithi116.stdout:8/726: dread d2/d13/d3c/fa3 [0,4194304] 0 2024-03-04T16:04:18.144 INFO:tasks.workunit.client.0.smithi116.stdout:7/709: rmdir d3/d95 39 2024-03-04T16:04:18.144 INFO:tasks.workunit.client.0.smithi116.stdout:3/746: write dc/f9e [1938815,78624] 0 2024-03-04T16:04:18.145 INFO:tasks.workunit.client.0.smithi116.stdout:9/635: chown d2/d26/l60 93612 1 2024-03-04T16:04:18.145 INFO:tasks.workunit.client.0.smithi116.stdout:6/638: symlink d6/de/d25/ld6 0 2024-03-04T16:04:18.146 INFO:tasks.workunit.client.0.smithi116.stdout:2/735: symlink d7/d13/d3e/d98/dce/d47/d64/l110 0 2024-03-04T16:04:18.147 INFO:tasks.workunit.client.0.smithi116.stdout:9/636: write d2/d19/d27/d3b/f3d [1551498,85507] 0 2024-03-04T16:04:18.147 INFO:tasks.workunit.client.0.smithi116.stdout:3/747: chown dc/df/c4f 491580630 1 2024-03-04T16:04:18.149 INFO:tasks.workunit.client.0.smithi116.stdout:3/748: write dc/df/f74 [2019187,54674] 0 2024-03-04T16:04:18.149 INFO:tasks.workunit.client.0.smithi116.stdout:8/727: creat d2/de/d16/d78/d29/de5/ff4 x:0 0 0 2024-03-04T16:04:18.154 INFO:tasks.workunit.client.0.smithi116.stdout:2/736: fdatasync d7/d81/f1c 0 2024-03-04T16:04:18.155 INFO:tasks.workunit.client.0.smithi116.stdout:7/710: dwrite d3/d4/d8/d1a/f5f [0,4194304] 0 2024-03-04T16:04:18.155 INFO:tasks.workunit.client.0.smithi116.stdout:9/637: symlink d2/d19/d27/d3b/d4a/d5b/lc4 0 2024-03-04T16:04:18.164 INFO:tasks.workunit.client.0.smithi116.stdout:2/737: fsync d7/d15/f59 0 2024-03-04T16:04:18.168 INFO:tasks.workunit.client.0.smithi116.stdout:7/711: creat d3/d4/d9e/dd7/dd0/dd3/ddf/fe4 x:0 0 0 2024-03-04T16:04:18.170 INFO:tasks.workunit.client.0.smithi116.stdout:8/728: truncate d2/d13/d3c/fae 1510145 0 2024-03-04T16:04:18.172 INFO:tasks.workunit.client.0.smithi116.stdout:9/638: fdatasync d2/d19/d27/d3b/f68 0 2024-03-04T16:04:18.173 INFO:tasks.workunit.client.0.smithi116.stdout:9/639: truncate d2/d19/d5f/fa3 272339 0 2024-03-04T16:04:18.176 INFO:tasks.workunit.client.0.smithi116.stdout:2/738: mknod d7/d13/c111 0 2024-03-04T16:04:18.178 INFO:tasks.workunit.client.0.smithi116.stdout:8/729: rename d2/de/d16/d78/d29/de5/df2/d37/c3f to d2/d13/d5e/cf5 0 2024-03-04T16:04:18.181 INFO:tasks.workunit.client.0.smithi116.stdout:7/712: creat d3/d95/fe5 x:0 0 0 2024-03-04T16:04:18.182 INFO:tasks.workunit.client.0.smithi116.stdout:2/739: mkdir d7/d13/d3e/d98/dce/d47/dd9/d112 0 2024-03-04T16:04:18.184 INFO:tasks.workunit.client.0.smithi116.stdout:8/730: creat d2/de/d16/d78/d29/d7a/db1/ff6 x:0 0 0 2024-03-04T16:04:18.185 INFO:tasks.workunit.client.0.smithi116.stdout:9/640: mkdir d2/d33/dc5 0 2024-03-04T16:04:18.186 INFO:tasks.workunit.client.0.smithi116.stdout:9/641: rename d2/d19/d27/d3b/d4a/d6a to d2/d19/d27/d3b/d4a/d6a/d94/d95/db7/dc6 22 2024-03-04T16:04:18.186 INFO:tasks.workunit.client.0.smithi116.stdout:7/713: unlink d3/d4/d11/fc4 0 2024-03-04T16:04:18.188 INFO:tasks.workunit.client.0.smithi116.stdout:8/731: fdatasync d2/d13/d3c/d46/fb0 0 2024-03-04T16:04:18.190 INFO:tasks.workunit.client.0.smithi116.stdout:9/642: symlink d2/d33/d65/d7e/lc7 0 2024-03-04T16:04:18.191 INFO:tasks.workunit.client.0.smithi116.stdout:2/740: mkdir d7/d15/d5e/d9d/df1/d113 0 2024-03-04T16:04:18.193 INFO:tasks.workunit.client.0.smithi116.stdout:8/732: mknod d2/d13/d5e/da7/cf7 0 2024-03-04T16:04:18.194 INFO:tasks.workunit.client.0.smithi116.stdout:9/643: chown d2/d19/d27/d3b/dab/f89 3 1 2024-03-04T16:04:18.195 INFO:tasks.workunit.client.0.smithi116.stdout:9/644: write d2/d26/f44 [750894,11201] 0 2024-03-04T16:04:18.200 INFO:tasks.workunit.client.0.smithi116.stdout:8/733: rmdir d2/d13/d3c/d77/d97/d5a 39 2024-03-04T16:04:18.204 INFO:tasks.workunit.client.0.smithi116.stdout:8/734: mkdir d2/de/d16/d78/d29/de5/df2/d37/df8 0 2024-03-04T16:04:18.205 INFO:tasks.workunit.client.0.smithi116.stdout:9/645: mkdir d2/d26/dc8 0 2024-03-04T16:04:18.206 INFO:tasks.workunit.client.0.smithi116.stdout:9/646: readlink d2/d33/d90/la5 0 2024-03-04T16:04:18.206 INFO:tasks.workunit.client.0.smithi116.stdout:9/647: chown d2/d26 181775 1 2024-03-04T16:04:18.207 INFO:tasks.workunit.client.0.smithi116.stdout:8/735: unlink d2/d13/d3c/d77/d97/c7e 0 2024-03-04T16:04:18.208 INFO:tasks.workunit.client.0.smithi116.stdout:8/736: chown d2/d13/d5e/da7 34181 1 2024-03-04T16:04:18.212 INFO:tasks.workunit.client.0.smithi116.stdout:9/648: link d2/d33/d65/d7e/lc0 d2/d19/d27/dbd/lc9 0 2024-03-04T16:04:18.212 INFO:tasks.workunit.client.0.smithi116.stdout:8/737: dwrite d2/d38/f3b [0,4194304] 0 2024-03-04T16:04:18.213 INFO:tasks.workunit.client.0.smithi116.stdout:9/649: dread - d2/d26/fb1 zero size 2024-03-04T16:04:18.224 INFO:tasks.workunit.client.0.smithi116.stdout:8/738: link d2/d13/d3c/d77/d97/d75/f79 d2/d38/d41/ff9 0 2024-03-04T16:04:18.225 INFO:tasks.workunit.client.0.smithi116.stdout:9/650: link d2/d19/d27/dbd/lc9 d2/d19/d27/d3b/dab/lca 0 2024-03-04T16:04:18.229 INFO:tasks.workunit.client.0.smithi116.stdout:9/651: creat d2/d19/d27/d3b/d4a/d6a/d94/fcb x:0 0 0 2024-03-04T16:04:18.229 INFO:tasks.workunit.client.0.smithi116.stdout:8/739: dwrite d2/d38/f64 [0,4194304] 0 2024-03-04T16:04:18.232 INFO:tasks.workunit.client.0.smithi116.stdout:8/740: readlink d2/d4a/dc7/le0 0 2024-03-04T16:04:18.236 INFO:tasks.workunit.client.0.smithi116.stdout:9/652: dwrite d2/d19/d27/d3b/d4a/d6a/d94/fcb [0,4194304] 0 2024-03-04T16:04:18.239 INFO:tasks.workunit.client.0.smithi116.stdout:8/741: link d2/de/c59 d2/d4a/cfa 0 2024-03-04T16:04:18.241 INFO:tasks.workunit.client.0.smithi116.stdout:9/653: mkdir d2/d19/d27/d3b/d4a/d5b/dcc 0 2024-03-04T16:04:18.242 INFO:tasks.workunit.client.0.smithi116.stdout:1/600: sync 2024-03-04T16:04:18.242 INFO:tasks.workunit.client.0.smithi116.stdout:5/700: sync 2024-03-04T16:04:18.242 INFO:tasks.workunit.client.0.smithi116.stdout:4/728: sync 2024-03-04T16:04:18.242 INFO:tasks.workunit.client.0.smithi116.stdout:0/589: sync 2024-03-04T16:04:18.243 INFO:tasks.workunit.client.0.smithi116.stdout:8/742: symlink d2/d13/d3c/d46/d96/lfb 0 2024-03-04T16:04:18.245 INFO:tasks.workunit.client.0.smithi116.stdout:1/601: write d5/d9/d21/d27/d94/db6/f45 [4961525,121971] 0 2024-03-04T16:04:18.247 INFO:tasks.workunit.client.0.smithi116.stdout:9/654: mknod d2/d26/ccd 0 2024-03-04T16:04:18.250 INFO:tasks.workunit.client.0.smithi116.stdout:4/729: dwrite d1/d3/dd/f4a [0,4194304] 0 2024-03-04T16:04:18.251 INFO:tasks.workunit.client.0.smithi116.stdout:8/743: dread d2/d13/d3c/d77/d97/f43 [4194304,4194304] 0 2024-03-04T16:04:18.258 INFO:tasks.workunit.client.0.smithi116.stdout:9/655: rename d2/d19/d27/d3b/f3d to d2/d33/dc5/fce 0 2024-03-04T16:04:18.259 INFO:tasks.workunit.client.0.smithi116.stdout:8/744: stat d2/d13/d3c/lc9 0 2024-03-04T16:04:18.259 INFO:tasks.workunit.client.0.smithi116.stdout:9/656: dread - d2/d19/d27/d3b/dab/d3c/f93 zero size 2024-03-04T16:04:18.263 INFO:tasks.workunit.client.0.smithi116.stdout:0/590: fdatasync da/d16/d28/d40/d48/d57/db3/db2/fb6 0 2024-03-04T16:04:18.264 INFO:tasks.workunit.client.0.smithi116.stdout:0/591: write da/fe [534519,125376] 0 2024-03-04T16:04:18.264 INFO:tasks.workunit.client.0.smithi116.stdout:1/602: mkdir d5/d9/d21/dc7 0 2024-03-04T16:04:18.269 INFO:tasks.workunit.client.0.smithi116.stdout:8/745: symlink d2/de/d3e/d7b/lfc 0 2024-03-04T16:04:18.271 INFO:tasks.workunit.client.0.smithi116.stdout:9/657: rename d2/d19/d5f/f91 to d2/d19/d27/dbd/fcf 0 2024-03-04T16:04:18.271 INFO:tasks.workunit.client.0.smithi116.stdout:0/592: rename da/d16/d28/d40/d48/d57 to da/d16/d28/d40/d48/d57/db3/dbe 22 2024-03-04T16:04:18.271 INFO:tasks.workunit.client.0.smithi116.stdout:4/730: dread d1/d6e/d74/dc2/fc9 [0,4194304] 0 2024-03-04T16:04:18.273 INFO:tasks.workunit.client.0.smithi116.stdout:4/731: fsync d1/d3/d15/d20/fa3 0 2024-03-04T16:04:18.274 INFO:tasks.workunit.client.0.smithi116.stdout:4/732: truncate d1/d3/d15/d20/d65/fc7 425189 0 2024-03-04T16:04:18.275 INFO:tasks.workunit.client.0.smithi116.stdout:5/701: dread dd/d1f/f41 [4194304,4194304] 0 2024-03-04T16:04:18.275 INFO:tasks.workunit.client.0.smithi116.stdout:5/702: fsync dd/d1f/f5f 0 2024-03-04T16:04:18.276 INFO:tasks.workunit.client.0.smithi116.stdout:5/703: write dd/f38 [881982,81742] 0 2024-03-04T16:04:18.277 INFO:tasks.workunit.client.0.smithi116.stdout:5/704: dread dd/d1f/d5c/d5e/faa [0,4194304] 0 2024-03-04T16:04:18.353 INFO:tasks.workunit.client.0.smithi116.stdout:8/746: unlink d2/d13/d5e/f73 0 2024-03-04T16:04:18.353 INFO:tasks.workunit.client.0.smithi116.stdout:4/733: rmdir d1/d3/d40/d52/d84 39 2024-03-04T16:04:18.354 INFO:tasks.workunit.client.0.smithi116.stdout:1/603: rename d5/d9/d21/d24/c28 to d5/d86/dac/cc8 0 2024-03-04T16:04:18.354 INFO:tasks.workunit.client.0.smithi116.stdout:5/705: readlink dd/d1f/d55/d73/d2d/l33 0 2024-03-04T16:04:18.355 INFO:tasks.workunit.client.0.smithi116.stdout:7/714: truncate d3/d4/d8/d1a/f42 1643078 0 2024-03-04T16:04:18.360 INFO:tasks.workunit.client.0.smithi116.stdout:6/639: fdatasync d6/de/d25/d36/d44/fba 0 2024-03-04T16:04:18.360 INFO:tasks.workunit.client.0.smithi116.stdout:2/741: dwrite d7/d81/f79 [0,4194304] 0 2024-03-04T16:04:18.361 INFO:tasks.workunit.client.0.smithi116.stdout:2/742: readlink d7/d13/d3e/ded/d6c/lc6 0 2024-03-04T16:04:18.363 INFO:tasks.workunit.client.0.smithi116.stdout:1/604: fdatasync d5/d9/d21/d27/d94/db6/f6a 0 2024-03-04T16:04:18.363 INFO:tasks.workunit.client.0.smithi116.stdout:6/640: truncate d6/de/d25/d36/d44/fba 723131 0 2024-03-04T16:04:18.366 INFO:tasks.workunit.client.0.smithi116.stdout:7/715: rename d3/d4/d9e/dd7/dd0 to d3/d4/dbf/de6 0 2024-03-04T16:04:18.367 INFO:tasks.workunit.client.0.smithi116.stdout:8/747: dwrite d2/d13/d3c/d77/d97/f44 [4194304,4194304] 0 2024-03-04T16:04:18.381 INFO:tasks.workunit.client.0.smithi116.stdout:8/748: dread d2/de/d3e/f4e [0,4194304] 0 2024-03-04T16:04:18.383 INFO:tasks.workunit.client.0.smithi116.stdout:2/743: write d7/d13/d3e/d98/dce/d47/d64/f87 [3514387,51923] 0 2024-03-04T16:04:18.384 INFO:tasks.workunit.client.0.smithi116.stdout:6/641: creat d6/d56/fd7 x:0 0 0 2024-03-04T16:04:18.388 INFO:tasks.workunit.client.0.smithi116.stdout:1/605: truncate d5/d9/d21/d27/d32/d35/f74 1344314 0 2024-03-04T16:04:18.389 INFO:tasks.workunit.client.0.smithi116.stdout:1/606: chown d5/d9/d21/d27/d8c 3861 1 2024-03-04T16:04:18.390 INFO:tasks.workunit.client.0.smithi116.stdout:7/716: mknod d3/d4/d34/d5d/db9/ce7 0 2024-03-04T16:04:18.390 INFO:tasks.workunit.client.0.smithi116.stdout:5/706: dwrite dd/d1f/f54 [0,4194304] 0 2024-03-04T16:04:18.392 INFO:tasks.workunit.client.0.smithi116.stdout:9/658: getdents d2/d19/d5f 0 2024-03-04T16:04:18.393 INFO:tasks.workunit.client.0.smithi116.stdout:3/749: sync 2024-03-04T16:04:18.396 INFO:tasks.workunit.client.0.smithi116.stdout:3/750: stat dc/df/d47/d91/f98 0 2024-03-04T16:04:18.397 INFO:tasks.workunit.client.0.smithi116.stdout:2/744: mknod d7/d13/d3e/d78/c114 0 2024-03-04T16:04:18.398 INFO:tasks.workunit.client.0.smithi116.stdout:7/717: dwrite d3/d4/d34/d5d/fa7 [0,4194304] 0 2024-03-04T16:04:18.399 INFO:tasks.workunit.client.0.smithi116.stdout:7/718: readlink d3/d4/d8/d4a/l4f 0 2024-03-04T16:04:18.401 INFO:tasks.workunit.client.0.smithi116.stdout:1/607: rmdir d5/d9/d21/d27/d32/d35/d55/db4/d50 39 2024-03-04T16:04:18.402 INFO:tasks.workunit.client.0.smithi116.stdout:4/734: getdents d1/d3/dd/d45/d50/dad 0 2024-03-04T16:04:18.406 INFO:tasks.workunit.client.0.smithi116.stdout:5/707: rename dd/d1f/d55/d73/d2d/d8d/fb8 to dd/d1f/d55/d73/d2d/d8d/fe0 0 2024-03-04T16:04:18.408 INFO:tasks.workunit.client.0.smithi116.stdout:6/642: mknod d6/de/d25/cd8 0 2024-03-04T16:04:18.411 INFO:tasks.workunit.client.0.smithi116.stdout:7/719: symlink d3/d4/dbf/de6/le8 0 2024-03-04T16:04:18.411 INFO:tasks.workunit.client.0.smithi116.stdout:7/720: write d3/d4/d34/d5d/fa7 [2709629,127243] 0 2024-03-04T16:04:18.413 INFO:tasks.workunit.client.0.smithi116.stdout:1/608: mkdir d5/d9/d21/d27/d94/db6/d42/d8b/dc9 0 2024-03-04T16:04:18.414 INFO:tasks.workunit.client.0.smithi116.stdout:9/659: symlink d2/d19/d27/d3b/ld0 0 2024-03-04T16:04:18.415 INFO:tasks.workunit.client.0.smithi116.stdout:9/660: write d2/d19/d27/d3b/f82 [566142,124899] 0 2024-03-04T16:04:18.417 INFO:tasks.workunit.client.0.smithi116.stdout:4/735: mkdir d1/d3/d15/d20/d65/dfe 0 2024-03-04T16:04:18.419 INFO:tasks.workunit.client.0.smithi116.stdout:8/749: rename d2/de/d16/d78/d29/de5/df2/c2d to d2/de/cfd 0 2024-03-04T16:04:18.421 INFO:tasks.workunit.client.0.smithi116.stdout:5/708: symlink dd/d1f/d55/d73/d2d/dc0/le1 0 2024-03-04T16:04:18.421 INFO:tasks.workunit.client.0.smithi116.stdout:5/709: readlink dd/d1f/d55/d73/d2d/ldb 0 2024-03-04T16:04:18.424 INFO:tasks.workunit.client.0.smithi116.stdout:3/751: mknod dc/d34/ddd/cf7 0 2024-03-04T16:04:18.424 INFO:tasks.workunit.client.0.smithi116.stdout:0/593: sync 2024-03-04T16:04:18.432 INFO:tasks.workunit.client.0.smithi116.stdout:2/745: rename d7/dd3/de1 to d7/d81/d115 0 2024-03-04T16:04:18.435 INFO:tasks.workunit.client.0.smithi116.stdout:2/746: truncate d7/d13/d3e/ded/d6c/ddb/fe0 328957 0 2024-03-04T16:04:18.437 INFO:tasks.workunit.client.0.smithi116.stdout:3/752: symlink dc/d23/d84/d7f/d93/lf8 0 2024-03-04T16:04:18.439 INFO:tasks.workunit.client.0.smithi116.stdout:3/753: read dc/df/d47/d91/fd7 [1035783,28500] 0 2024-03-04T16:04:18.439 INFO:tasks.workunit.client.0.smithi116.stdout:3/754: write dc/d23/f1b [2759674,23931] 0 2024-03-04T16:04:18.444 INFO:tasks.workunit.client.0.smithi116.stdout:7/721: mkdir d3/de9 0 2024-03-04T16:04:18.445 INFO:tasks.workunit.client.0.smithi116.stdout:4/736: creat d1/d3/d15/d20/d1c/ddd/fff x:0 0 0 2024-03-04T16:04:18.447 INFO:tasks.workunit.client.0.smithi116.stdout:1/609: rename d5/d9/d21/d27/d32/d49/d4c/laf to d5/d9/d21/d27/d94/db6/d42/d8b/dc9/lca 0 2024-03-04T16:04:18.450 INFO:tasks.workunit.client.0.smithi116.stdout:8/750: truncate d2/d13/d5e/f87 2179926 0 2024-03-04T16:04:18.453 INFO:tasks.workunit.client.0.smithi116.stdout:2/747: creat d7/d15/dd7/f116 x:0 0 0 2024-03-04T16:04:18.454 INFO:tasks.workunit.client.0.smithi116.stdout:2/748: truncate d7/d13/d3e/ded/d6c/ddb/fe0 916925 0 2024-03-04T16:04:18.466 INFO:tasks.workunit.client.0.smithi116.stdout:5/710: truncate dd/d1f/d55/d73/d2d/d7c/f86 965196 0 2024-03-04T16:04:18.466 INFO:tasks.workunit.client.0.smithi116.stdout:5/711: stat dd/d1f/d5c/d5e/faa 0 2024-03-04T16:04:18.471 INFO:tasks.workunit.client.0.smithi116.stdout:0/594: dwrite da/d16/d28/d3d/f4a [0,4194304] 0 2024-03-04T16:04:18.483 INFO:tasks.workunit.client.0.smithi116.stdout:9/661: link d2/d19/d27/d3b/dab/f6f d2/d33/d65/d6c/db8/fd1 0 2024-03-04T16:04:18.484 INFO:tasks.workunit.client.0.smithi116.stdout:7/722: fsync d3/d4/d8/f3e 0 2024-03-04T16:04:18.485 INFO:tasks.workunit.client.0.smithi116.stdout:3/755: symlink dc/d34/de7/deb/lf9 0 2024-03-04T16:04:18.487 INFO:tasks.workunit.client.0.smithi116.stdout:4/737: unlink d1/d3/f39 0 2024-03-04T16:04:18.488 INFO:tasks.workunit.client.0.smithi116.stdout:7/723: read d3/d4/d8/d52/f53 [1287027,89617] 0 2024-03-04T16:04:18.488 INFO:tasks.workunit.client.0.smithi116.stdout:7/724: chown d3/d4/dbf/de6/le8 197 1 2024-03-04T16:04:18.489 INFO:tasks.workunit.client.0.smithi116.stdout:8/751: mknod d2/d13/db9/cfe 0 2024-03-04T16:04:18.491 INFO:tasks.workunit.client.0.smithi116.stdout:6/643: getdents d6/d56/d58/d72 0 2024-03-04T16:04:18.492 INFO:tasks.workunit.client.0.smithi116.stdout:5/712: symlink dd/d1f/d55/le2 0 2024-03-04T16:04:18.499 INFO:tasks.workunit.client.0.smithi116.stdout:0/595: mknod da/d16/d20/cbf 0 2024-03-04T16:04:18.503 INFO:tasks.workunit.client.0.smithi116.stdout:4/738: fsync d1/d3/d40/d52/d84/d88/f8b 0 2024-03-04T16:04:18.503 INFO:tasks.workunit.client.0.smithi116.stdout:0/596: stat da/d16/d19/d27/c85 0 2024-03-04T16:04:18.503 INFO:tasks.workunit.client.0.smithi116.stdout:2/749: symlink d7/l117 0 2024-03-04T16:04:18.504 INFO:tasks.workunit.client.0.smithi116.stdout:7/725: dwrite d3/d4/d11/f38 [0,4194304] 0 2024-03-04T16:04:18.505 INFO:tasks.workunit.client.0.smithi116.stdout:1/610: dread d5/f38 [0,4194304] 0 2024-03-04T16:04:18.509 INFO:tasks.workunit.client.0.smithi116.stdout:6/644: dwrite d6/f14 [4194304,4194304] 0 2024-03-04T16:04:18.509 INFO:tasks.workunit.client.0.smithi116.stdout:0/597: read da/f1a [1288554,19855] 0 2024-03-04T16:04:18.510 INFO:tasks.workunit.client.0.smithi116.stdout:4/739: write d1/d6c/dbf/ff4 [443296,2607] 0 2024-03-04T16:04:18.511 INFO:tasks.workunit.client.0.smithi116.stdout:5/713: chown dd/d1f/d55/d73/d2d/c30 3473 1 2024-03-04T16:04:18.513 INFO:tasks.workunit.client.0.smithi116.stdout:8/752: mknod d2/de/d16/d78/cff 0 2024-03-04T16:04:18.513 INFO:tasks.workunit.client.0.smithi116.stdout:6/645: chown d6/de/d29/l3c 168 1 2024-03-04T16:04:18.513 INFO:tasks.workunit.client.0.smithi116.stdout:4/740: chown d1/d6c/l89 44027184 1 2024-03-04T16:04:18.523 INFO:tasks.workunit.client.0.smithi116.stdout:5/714: mkdir dd/d1f/d5c/d5e/dc2/de3 0 2024-03-04T16:04:18.526 INFO:tasks.workunit.client.0.smithi116.stdout:0/598: creat da/d16/d28/d40/d48/d57/d8a/daa/fc0 x:0 0 0 2024-03-04T16:04:18.528 INFO:tasks.workunit.client.0.smithi116.stdout:8/753: mknod d2/d13/d3c/d46/c100 0 2024-03-04T16:04:18.533 INFO:tasks.workunit.client.0.smithi116.stdout:2/750: creat d7/d13/d3e/d98/dce/d47/dd9/d112/f118 x:0 0 0 2024-03-04T16:04:18.534 INFO:tasks.workunit.client.0.smithi116.stdout:1/611: mknod d5/d9/d21/dc7/ccb 0 2024-03-04T16:04:18.534 INFO:tasks.workunit.client.0.smithi116.stdout:9/662: link d2/d33/la4 d2/d19/d27/d3b/d4a/d6a/d94/d95/db7/ld2 0 2024-03-04T16:04:18.541 INFO:tasks.workunit.client.0.smithi116.stdout:2/751: chown d7/d13/d3e/d98/dce/d47/d64/d75/d77/cc8 13162895 1 2024-03-04T16:04:18.542 INFO:tasks.workunit.client.0.smithi116.stdout:0/599: dwrite da/d16/f4c [0,4194304] 0 2024-03-04T16:04:18.548 INFO:tasks.workunit.client.0.smithi116.stdout:6/646: dread d6/de/f1b [0,4194304] 0 2024-03-04T16:04:18.549 INFO:tasks.workunit.client.0.smithi116.stdout:9/663: dwrite d2/d19/d27/dbd/fcf [0,4194304] 0 2024-03-04T16:04:18.550 INFO:tasks.workunit.client.0.smithi116.stdout:9/664: write d2/d26/fb1 [749544,74816] 0 2024-03-04T16:04:18.551 INFO:tasks.workunit.client.0.smithi116.stdout:2/752: truncate d7/d13/d3e/d98/dce/f3a 142919 0 2024-03-04T16:04:18.558 INFO:tasks.workunit.client.0.smithi116.stdout:4/741: rename d1/d3/dd/d24/d51/ca4 to d1/d3/de/c100 0 2024-03-04T16:04:18.560 INFO:tasks.workunit.client.0.smithi116.stdout:7/726: write d3/d4/d11/f36 [2825609,54021] 0 2024-03-04T16:04:18.562 INFO:tasks.workunit.client.0.smithi116.stdout:9/665: mknod d2/d33/d90/cd3 0 2024-03-04T16:04:18.567 INFO:tasks.workunit.client.0.smithi116.stdout:4/742: rename d1/d3/de/d47/c67 to d1/d3/d40/c101 0 2024-03-04T16:04:18.572 INFO:tasks.workunit.client.0.smithi116.stdout:0/600: getdents da/d16/d28/d7b 0 2024-03-04T16:04:18.573 INFO:tasks.workunit.client.0.smithi116.stdout:0/601: chown da/d16/d28/d40/d5d/d7e/fac 114151277 1 2024-03-04T16:04:18.574 INFO:tasks.workunit.client.0.smithi116.stdout:5/715: truncate dd/f13 2938552 0 2024-03-04T16:04:18.575 INFO:tasks.workunit.client.0.smithi116.stdout:5/716: write dd/d1f/d55/d73/d2d/f42 [1432979,50961] 0 2024-03-04T16:04:18.576 INFO:tasks.workunit.client.0.smithi116.stdout:5/717: chown dd/d1f/d5c/d5e/dc2 4703553 1 2024-03-04T16:04:18.579 INFO:tasks.workunit.client.0.smithi116.stdout:5/718: fdatasync dd/d1f/d55/d73/d2d/d31/f3c 0 2024-03-04T16:04:18.579 INFO:tasks.workunit.client.0.smithi116.stdout:9/666: rename d2/d19/d27/d3b to d2/d26/dd4 0 2024-03-04T16:04:18.580 INFO:tasks.workunit.client.0.smithi116.stdout:2/753: dread d7/d13/d3e/ded/f50 [0,4194304] 0 2024-03-04T16:04:18.580 INFO:tasks.workunit.client.0.smithi116.stdout:1/612: write d5/d9/d21/d27/d32/d35/f74 [599699,107717] 0 2024-03-04T16:04:18.581 INFO:tasks.workunit.client.0.smithi116.stdout:0/602: creat da/d16/d28/d40/d48/d57/d8a/fc1 x:0 0 0 2024-03-04T16:04:18.586 INFO:tasks.workunit.client.0.smithi116.stdout:6/647: write d6/de/d15/f6f [648845,75681] 0 2024-03-04T16:04:18.590 INFO:tasks.workunit.client.0.smithi116.stdout:4/743: dread d1/d3/d15/d20/f1e [0,4194304] 0 2024-03-04T16:04:18.591 INFO:tasks.workunit.client.0.smithi116.stdout:2/754: mkdir d7/d13/d3e/d98/dce/d47/d64/d75/d77/d119 0 2024-03-04T16:04:18.593 INFO:tasks.workunit.client.0.smithi116.stdout:0/603: mkdir da/d16/d28/d40/d48/d57/db3/db2/dc2 0 2024-03-04T16:04:18.593 INFO:tasks.workunit.client.0.smithi116.stdout:4/744: readlink d1/d3/d15/d20/d1c/d62/d81/daf/lcf 0 2024-03-04T16:04:18.595 INFO:tasks.workunit.client.0.smithi116.stdout:1/613: dread d5/d9/d21/d27/d94/db6/f45 [0,4194304] 0 2024-03-04T16:04:18.596 INFO:tasks.workunit.client.0.smithi116.stdout:0/604: write f7 [1179311,107418] 0 2024-03-04T16:04:18.597 INFO:tasks.workunit.client.0.smithi116.stdout:6/648: dwrite d6/de/d84/fcb [0,4194304] 0 2024-03-04T16:04:18.599 INFO:tasks.workunit.client.0.smithi116.stdout:1/614: fdatasync d5/d9/d21/d27/d32/d35/d55/d75/d91/f9e 0 2024-03-04T16:04:18.602 INFO:tasks.workunit.client.0.smithi116.stdout:2/755: fdatasync d7/d81/f21 0 2024-03-04T16:04:18.604 INFO:tasks.workunit.client.0.smithi116.stdout:0/605: dwrite da/d16/d19/d27/f9d [0,4194304] 0 2024-03-04T16:04:18.605 INFO:tasks.workunit.client.0.smithi116.stdout:2/756: chown d7/d13/d3e/d98/dce/d47/d64/f6f 52334 1 2024-03-04T16:04:18.606 INFO:tasks.workunit.client.0.smithi116.stdout:0/606: fsync da/d16/d28/d40/f5c 0 2024-03-04T16:04:18.620 INFO:tasks.workunit.client.0.smithi116.stdout:2/757: mkdir d7/d13/d3e/ded/d6c/da4/d11a 0 2024-03-04T16:04:18.620 INFO:tasks.workunit.client.0.smithi116.stdout:4/745: write d1/f80 [1342789,62404] 0 2024-03-04T16:04:18.622 INFO:tasks.workunit.client.0.smithi116.stdout:0/607: rename da/d16/d28/d39/cae to da/d16/d19/d27/d82/cc3 0 2024-03-04T16:04:18.623 INFO:tasks.workunit.client.0.smithi116.stdout:5/719: link dd/d1f/d55/d73/d2d/cc1 dd/d1f/ce4 0 2024-03-04T16:04:18.623 INFO:tasks.workunit.client.0.smithi116.stdout:5/720: dread - dd/d1f/d51/fd1 zero size 2024-03-04T16:04:18.624 INFO:tasks.workunit.client.0.smithi116.stdout:5/721: write dd/d1f/d51/fbf [2976477,129923] 0 2024-03-04T16:04:18.625 INFO:tasks.workunit.client.0.smithi116.stdout:6/649: symlink d6/d56/d58/ld9 0 2024-03-04T16:04:18.626 INFO:tasks.workunit.client.0.smithi116.stdout:1/615: symlink d5/d9/d21/d27/d32/d35/dbf/lcc 0 2024-03-04T16:04:18.629 INFO:tasks.workunit.client.0.smithi116.stdout:2/758: fdatasync d7/d25/d41/f9f 0 2024-03-04T16:04:18.630 INFO:tasks.workunit.client.0.smithi116.stdout:9/667: truncate d2/d26/d28/f81 858715 0 2024-03-04T16:04:18.633 INFO:tasks.workunit.client.0.smithi116.stdout:1/616: dwrite d5/d9/d21/d84/fb5 [0,4194304] 0 2024-03-04T16:04:18.635 INFO:tasks.workunit.client.0.smithi116.stdout:1/617: fdatasync d5/d9/d21/d27/d94/db6/d42/f88 0 2024-03-04T16:04:18.636 INFO:tasks.workunit.client.0.smithi116.stdout:0/608: dread da/d16/d20/d79/f7c [0,4194304] 0 2024-03-04T16:04:18.637 INFO:tasks.workunit.client.0.smithi116.stdout:0/609: chown da/d16/d19/d27/f55 12800 1 2024-03-04T16:04:18.637 INFO:tasks.workunit.client.0.smithi116.stdout:6/650: chown d6/de/d25/d36/d4e 1834181 1 2024-03-04T16:04:18.644 INFO:tasks.workunit.client.0.smithi116.stdout:2/759: rmdir d7/d15/d35 39 2024-03-04T16:04:18.648 INFO:tasks.workunit.client.0.smithi116.stdout:2/760: dwrite d7/d81/dc2/f10f [0,4194304] 0 2024-03-04T16:04:18.650 INFO:tasks.workunit.client.0.smithi116.stdout:5/722: mkdir dd/d1f/de5 0 2024-03-04T16:04:18.655 INFO:tasks.workunit.client.0.smithi116.stdout:2/761: dread d7/d25/fbd [0,4194304] 0 2024-03-04T16:04:18.655 INFO:tasks.workunit.client.0.smithi116.stdout:4/746: dwrite d1/d3/de/f13 [0,4194304] 0 2024-03-04T16:04:18.658 INFO:tasks.workunit.client.0.smithi116.stdout:4/747: truncate d1/d3/dd/d8a/ddc/fe1 782531 0 2024-03-04T16:04:18.660 INFO:tasks.workunit.client.0.smithi116.stdout:1/618: unlink d5/d9/d21/d27/d32/d49/d4c/d67/cad 0 2024-03-04T16:04:18.661 INFO:tasks.workunit.client.0.smithi116.stdout:4/748: write d1/d3/d15/d20/d1c/d62/d81/d97/fb9 [914029,18664] 0 2024-03-04T16:04:18.663 INFO:tasks.workunit.client.0.smithi116.stdout:6/651: creat d6/d9e/da5/db3/fda x:0 0 0 2024-03-04T16:04:18.664 INFO:tasks.workunit.client.0.smithi116.stdout:9/668: creat d2/d26/dd4/fd5 x:0 0 0 2024-03-04T16:04:18.668 INFO:tasks.workunit.client.0.smithi116.stdout:9/669: dread d2/d19/d27/dbd/fcf [0,4194304] 0 2024-03-04T16:04:18.668 INFO:tasks.workunit.client.0.smithi116.stdout:1/619: truncate d5/d9/d21/f2f 452419 0 2024-03-04T16:04:18.669 INFO:tasks.workunit.client.0.smithi116.stdout:4/749: mknod d1/d3/d40/d52/d84/dde/c102 0 2024-03-04T16:04:18.672 INFO:tasks.workunit.client.0.smithi116.stdout:6/652: mknod d6/de/d25/d36/d44/cdb 0 2024-03-04T16:04:18.675 INFO:tasks.workunit.client.0.smithi116.stdout:9/670: dread d2/d19/d27/f9d [0,4194304] 0 2024-03-04T16:04:18.675 INFO:tasks.workunit.client.0.smithi116.stdout:2/762: rename d7/f24 to d7/d15/d5e/d9d/df1/d113/f11b 0 2024-03-04T16:04:18.676 INFO:tasks.workunit.client.0.smithi116.stdout:0/610: getdents da/d16/d28/d40/d48 0 2024-03-04T16:04:18.677 INFO:tasks.workunit.client.0.smithi116.stdout:2/763: truncate d7/d13/d3e/ded/d6c/da4/fc3 298507 0 2024-03-04T16:04:18.679 INFO:tasks.workunit.client.0.smithi116.stdout:1/620: dread - d5/d9/d21/d27/d32/d35/d55/db4/d50/f8e zero size 2024-03-04T16:04:18.680 INFO:tasks.workunit.client.0.smithi116.stdout:4/750: rmdir d1/d3/d15/d20/d1c/d62/d81 39 2024-03-04T16:04:18.681 INFO:tasks.workunit.client.0.smithi116.stdout:6/653: readlink d6/de/d29/d3a/lc2 0 2024-03-04T16:04:18.685 INFO:tasks.workunit.client.0.smithi116.stdout:1/621: creat d5/d9/d21/d27/d94/fcd x:0 0 0 2024-03-04T16:04:18.689 INFO:tasks.workunit.client.0.smithi116.stdout:0/611: mknod da/d16/d20/d42/cc4 0 2024-03-04T16:04:18.690 INFO:tasks.workunit.client.0.smithi116.stdout:0/612: chown da/d16/d20/d42/f88 48 1 2024-03-04T16:04:18.692 INFO:tasks.workunit.client.0.smithi116.stdout:0/613: write da/d16/d28/d3d/d41/f94 [249757,126317] 0 2024-03-04T16:04:18.696 INFO:tasks.workunit.client.0.smithi116.stdout:2/764: creat d7/d13/d3e/d98/dce/d47/d64/f11c x:0 0 0 2024-03-04T16:04:18.698 INFO:tasks.workunit.client.0.smithi116.stdout:0/614: dread da/d16/d19/f68 [0,4194304] 0 2024-03-04T16:04:18.702 INFO:tasks.workunit.client.0.smithi116.stdout:1/622: dread f2 [0,4194304] 0 2024-03-04T16:04:18.703 INFO:tasks.workunit.client.0.smithi116.stdout:0/615: chown da/d16/d28/d40/d5d/l5e 485 1 2024-03-04T16:04:18.707 INFO:tasks.workunit.client.0.smithi116.stdout:2/765: symlink d7/d13/d3e/l11d 0 2024-03-04T16:04:18.707 INFO:tasks.workunit.client.0.smithi116.stdout:6/654: dread d6/de/f3d [0,4194304] 0 2024-03-04T16:04:18.708 INFO:tasks.workunit.client.0.smithi116.stdout:1/623: stat d5/f5a 0 2024-03-04T16:04:18.708 INFO:tasks.workunit.client.0.smithi116.stdout:9/671: truncate d2/d33/f88 1502774 0 2024-03-04T16:04:18.709 INFO:tasks.workunit.client.0.smithi116.stdout:9/672: write d2/d26/dd4/d4a/d5b/f24 [7418457,16148] 0 2024-03-04T16:04:18.714 INFO:tasks.workunit.client.0.smithi116.stdout:6/655: readlink d6/de/d25/d4a/d63/l86 0 2024-03-04T16:04:18.714 INFO:tasks.workunit.client.0.smithi116.stdout:4/751: write d1/d3/d40/d52/d84/d88/fc4 [1616310,115632] 0 2024-03-04T16:04:18.716 INFO:tasks.workunit.client.0.smithi116.stdout:9/673: dwrite d2/d26/dd4/dab/f42 [4194304,4194304] 0 2024-03-04T16:04:18.717 INFO:tasks.workunit.client.0.smithi116.stdout:3/756: sync 2024-03-04T16:04:18.718 INFO:tasks.workunit.client.0.smithi116.stdout:3/757: stat dc/d23/d84/d7f/d93/db6/dc5/lc6 0 2024-03-04T16:04:18.722 INFO:tasks.workunit.client.0.smithi116.stdout:6/656: read d6/f9 [688231,5331] 0 2024-03-04T16:04:18.723 INFO:tasks.workunit.client.0.smithi116.stdout:4/752: creat d1/d3/de/d47/f103 x:0 0 0 2024-03-04T16:04:18.723 INFO:tasks.workunit.client.0.smithi116.stdout:4/753: fsync d1/d6e/d74/dc2/fec 0 2024-03-04T16:04:18.724 INFO:tasks.workunit.client.0.smithi116.stdout:4/754: truncate d1/d3/de/f12 4862720 0 2024-03-04T16:04:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:18 smithi116 ceph-mon[29120]: pgmap v25: 65 pgs: 65 active+clean; 1.2 GiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 44 MiB/s rd, 152 MiB/s wr, 387 op/s 2024-03-04T16:04:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:18 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:18.789 INFO:tasks.workunit.client.0.smithi116.stdout:9/674: rmdir d2/d26/dd4 39 2024-03-04T16:04:18.789 INFO:tasks.workunit.client.0.smithi116.stdout:6/657: creat d6/de/d29/fdc x:0 0 0 2024-03-04T16:04:18.790 INFO:tasks.workunit.client.0.smithi116.stdout:1/624: write d5/d9/f23 [835900,88844] 0 2024-03-04T16:04:18.791 INFO:tasks.workunit.client.0.smithi116.stdout:4/755: dwrite d1/d3/de/f23 [4194304,4194304] 0 2024-03-04T16:04:18.794 INFO:tasks.workunit.client.0.smithi116.stdout:1/625: dread - d5/d86/fab zero size 2024-03-04T16:04:18.794 INFO:tasks.workunit.client.0.smithi116.stdout:9/675: creat d2/d33/dc5/fd6 x:0 0 0 2024-03-04T16:04:18.798 INFO:tasks.workunit.client.0.smithi116.stdout:0/616: dwrite da/d16/d20/d53/f63 [0,4194304] 0 2024-03-04T16:04:18.801 INFO:tasks.workunit.client.0.smithi116.stdout:4/756: truncate d1/d3/d40/d52/d84/fa0 1783801 0 2024-03-04T16:04:18.803 INFO:tasks.workunit.client.0.smithi116.stdout:2/766: dwrite d7/d81/f26 [0,4194304] 0 2024-03-04T16:04:18.807 INFO:tasks.workunit.client.0.smithi116.stdout:6/658: dwrite d6/de/d1e/d80/fce [0,4194304] 0 2024-03-04T16:04:18.811 INFO:tasks.workunit.client.0.smithi116.stdout:1/626: unlink d5/d9/d21/d27/d94/db6/f45 0 2024-03-04T16:04:18.819 INFO:tasks.workunit.client.0.smithi116.stdout:4/757: dread d1/d3/d40/d52/d84/d88/fc4 [0,4194304] 0 2024-03-04T16:04:18.820 INFO:tasks.workunit.client.0.smithi116.stdout:6/659: rename d6/d56/d58/d72 to d6/de/d25/d36/d44/dbb/ddd 0 2024-03-04T16:04:18.822 INFO:tasks.workunit.client.0.smithi116.stdout:1/627: chown d5/f6f 10 1 2024-03-04T16:04:18.829 INFO:tasks.workunit.client.0.smithi116.stdout:4/758: mknod d1/d3/de/d47/df9/c104 0 2024-03-04T16:04:18.832 INFO:tasks.workunit.client.0.smithi116.stdout:4/759: dread d1/d3/d15/d20/d1c/fce [0,4194304] 0 2024-03-04T16:04:18.832 INFO:tasks.workunit.client.0.smithi116.stdout:0/617: dread f7 [0,4194304] 0 2024-03-04T16:04:18.835 INFO:tasks.workunit.client.0.smithi116.stdout:6/660: link d6/de/d15/l49 d6/de/d84/lde 0 2024-03-04T16:04:18.836 INFO:tasks.workunit.client.0.smithi116.stdout:1/628: getdents d5/d9/d21/d27/d94/db6/d42 0 2024-03-04T16:04:18.838 INFO:tasks.workunit.client.0.smithi116.stdout:4/760: mknod d1/d6c/dbf/c105 0 2024-03-04T16:04:18.839 INFO:tasks.workunit.client.0.smithi116.stdout:4/761: chown d1/d3/de/c77 617791566 1 2024-03-04T16:04:18.843 INFO:tasks.workunit.client.0.smithi116.stdout:6/661: mknod d6/de/d1e/d80/cdf 0 2024-03-04T16:04:18.847 INFO:tasks.workunit.client.0.smithi116.stdout:6/662: unlink d6/de/d29/fa0 0 2024-03-04T16:04:18.848 INFO:tasks.workunit.client.0.smithi116.stdout:8/754: sync 2024-03-04T16:04:18.848 INFO:tasks.workunit.client.0.smithi116.stdout:7/727: sync 2024-03-04T16:04:18.851 INFO:tasks.workunit.client.0.smithi116.stdout:9/676: dwrite d2/d19/d27/dbd/fcf [0,4194304] 0 2024-03-04T16:04:18.855 INFO:tasks.workunit.client.0.smithi116.stdout:4/762: creat d1/d3/d15/d20/d1c/d62/d81/f106 x:0 0 0 2024-03-04T16:04:18.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:18 smithi152 ceph-mon[36876]: pgmap v25: 65 pgs: 65 active+clean; 1.2 GiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 44 MiB/s rd, 152 MiB/s wr, 387 op/s 2024-03-04T16:04:18.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:18 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:18.859 INFO:tasks.workunit.client.0.smithi116.stdout:4/763: rename d1/d3/d15/d20/d1c/d6f to d1/d3/d15/d20/d1c/d6f/d107 22 2024-03-04T16:04:18.859 INFO:tasks.workunit.client.0.smithi116.stdout:2/767: dwrite d7/d13/d3e/d98/dce/d47/f51 [0,4194304] 0 2024-03-04T16:04:18.860 INFO:tasks.workunit.client.0.smithi116.stdout:7/728: creat d3/d4/dbf/de6/dd3/ddf/de0/fea x:0 0 0 2024-03-04T16:04:18.865 INFO:tasks.workunit.client.0.smithi116.stdout:6/663: link d6/de/d1e/d80/f57 d6/de/d25/d4a/d63/fe0 0 2024-03-04T16:04:18.865 INFO:tasks.workunit.client.0.smithi116.stdout:8/755: creat d2/de/f101 x:0 0 0 2024-03-04T16:04:18.866 INFO:tasks.workunit.client.0.smithi116.stdout:8/756: truncate d2/de/d16/d78/d29/de5/ff4 437401 0 2024-03-04T16:04:18.867 INFO:tasks.workunit.client.0.smithi116.stdout:4/764: mkdir d1/d3/dd/d45/d108 0 2024-03-04T16:04:18.870 INFO:tasks.workunit.client.0.smithi116.stdout:4/765: fdatasync d1/d3/d40/d52/fd4 0 2024-03-04T16:04:18.870 INFO:tasks.workunit.client.0.smithi116.stdout:4/766: chown d1/d3/d40/d52/d84 2674 1 2024-03-04T16:04:18.876 INFO:tasks.workunit.client.0.smithi116.stdout:6/664: creat d6/de/d25/d36/d44/dbb/ddd/d7b/dc6/fe1 x:0 0 0 2024-03-04T16:04:18.877 INFO:tasks.workunit.client.0.smithi116.stdout:6/665: readlink d6/d56/l78 0 2024-03-04T16:04:18.879 INFO:tasks.workunit.client.0.smithi116.stdout:1/629: truncate f2 702724 0 2024-03-04T16:04:18.881 INFO:tasks.workunit.client.0.smithi116.stdout:7/729: creat d3/d4/d8/d1a/feb x:0 0 0 2024-03-04T16:04:18.882 INFO:tasks.workunit.client.0.smithi116.stdout:4/767: truncate d1/d3/de/f26 2852684 0 2024-03-04T16:04:18.884 INFO:tasks.workunit.client.0.smithi116.stdout:7/730: read d3/d4/d8/d1a/f32 [2832039,126574] 0 2024-03-04T16:04:18.885 INFO:tasks.workunit.client.0.smithi116.stdout:6/666: dwrite d6/de/d1e/d80/fce [4194304,4194304] 0 2024-03-04T16:04:18.885 INFO:tasks.workunit.client.0.smithi116.stdout:4/768: chown d1/d3/dd/d45/d50/c83 93 1 2024-03-04T16:04:18.887 INFO:tasks.workunit.client.0.smithi116.stdout:1/630: creat d5/d9/fce x:0 0 0 2024-03-04T16:04:18.892 INFO:tasks.workunit.client.0.smithi116.stdout:6/667: creat d6/de/d84/db4/fe2 x:0 0 0 2024-03-04T16:04:18.894 INFO:tasks.workunit.client.0.smithi116.stdout:9/677: dread d2/d26/f44 [0,4194304] 0 2024-03-04T16:04:18.894 INFO:tasks.workunit.client.0.smithi116.stdout:1/631: mkdir d5/d9/d21/d27/d32/d49/d4c/d67/dcf 0 2024-03-04T16:04:18.895 INFO:tasks.workunit.client.0.smithi116.stdout:7/731: symlink d3/d4/d8/d4a/lec 0 2024-03-04T16:04:18.895 INFO:tasks.workunit.client.0.smithi116.stdout:2/768: dread d7/d13/dec/f101 [0,4194304] 0 2024-03-04T16:04:18.898 INFO:tasks.workunit.client.0.smithi116.stdout:8/757: write d2/d13/d5e/fdb [837129,94846] 0 2024-03-04T16:04:18.900 INFO:tasks.workunit.client.0.smithi116.stdout:4/769: dwrite d1/d3/d15/d20/d1c/d62/d81/feb [0,4194304] 0 2024-03-04T16:04:18.901 INFO:tasks.workunit.client.0.smithi116.stdout:9/678: chown d2/d26/dd4/dab/f42 22 1 2024-03-04T16:04:18.903 INFO:tasks.workunit.client.0.smithi116.stdout:7/732: rmdir d3/d4/d34/d5d/db9 39 2024-03-04T16:04:18.904 INFO:tasks.workunit.client.0.smithi116.stdout:2/769: readlink d7/l72 0 2024-03-04T16:04:18.906 INFO:tasks.workunit.client.0.smithi116.stdout:1/632: creat d5/d9/d21/d27/d32/d35/d55/dae/fd0 x:0 0 0 2024-03-04T16:04:18.907 INFO:tasks.workunit.client.0.smithi116.stdout:7/733: write d3/d4/d8/d4a/d88/f93 [952663,65805] 0 2024-03-04T16:04:18.907 INFO:tasks.workunit.client.0.smithi116.stdout:5/723: sync 2024-03-04T16:04:18.911 INFO:tasks.workunit.client.0.smithi116.stdout:5/724: fdatasync dd/d1f/d5c/fbb 0 2024-03-04T16:04:18.912 INFO:tasks.workunit.client.0.smithi116.stdout:8/758: dwrite d2/de/d16/d78/d29/de5/df2/fa0 [0,4194304] 0 2024-03-04T16:04:18.919 INFO:tasks.workunit.client.0.smithi116.stdout:4/770: fsync d1/d3/d15/d20/f5e 0 2024-03-04T16:04:18.921 INFO:tasks.workunit.client.0.smithi116.stdout:1/633: dwrite d5/d9/d21/d27/d32/d49/d4a/f78 [4194304,4194304] 0 2024-03-04T16:04:18.924 INFO:tasks.workunit.client.0.smithi116.stdout:2/770: dwrite d7/d15/d5e/d9d/ffc [0,4194304] 0 2024-03-04T16:04:18.931 INFO:tasks.workunit.client.0.smithi116.stdout:8/759: fdatasync d2/d13/d3c/f69 0 2024-03-04T16:04:18.936 INFO:tasks.workunit.client.0.smithi116.stdout:7/734: mkdir d3/d4/d34/d5d/ded 0 2024-03-04T16:04:18.938 INFO:tasks.workunit.client.0.smithi116.stdout:5/725: dread fa [0,4194304] 0 2024-03-04T16:04:18.940 INFO:tasks.workunit.client.0.smithi116.stdout:4/771: truncate d1/d3/d15/d20/f4e 1344321 0 2024-03-04T16:04:18.942 INFO:tasks.workunit.client.0.smithi116.stdout:1/634: mkdir d5/d9/d21/d27/d94/db6/d42/d8b/dc9/dd1 0 2024-03-04T16:04:18.943 INFO:tasks.workunit.client.0.smithi116.stdout:2/771: symlink d7/d25/l11e 0 2024-03-04T16:04:18.944 INFO:tasks.workunit.client.0.smithi116.stdout:7/735: dwrite d3/d4/d34/d5d/fa7 [4194304,4194304] 0 2024-03-04T16:04:18.947 INFO:tasks.workunit.client.0.smithi116.stdout:8/760: fdatasync d2/de/d16/f1b 0 2024-03-04T16:04:18.950 INFO:tasks.workunit.client.0.smithi116.stdout:9/679: dwrite d2/d26/dd4/dab/f6f [0,4194304] 0 2024-03-04T16:04:18.953 INFO:tasks.workunit.client.0.smithi116.stdout:5/726: mknod dd/d1f/d55/d73/d2d/dc0/ce6 0 2024-03-04T16:04:18.954 INFO:tasks.workunit.client.0.smithi116.stdout:6/668: truncate d6/de/d1e/f35 434683 0 2024-03-04T16:04:18.954 INFO:tasks.workunit.client.0.smithi116.stdout:9/680: dread d2/d19/d27/dbd/fcf [0,4194304] 0 2024-03-04T16:04:18.959 INFO:tasks.workunit.client.0.smithi116.stdout:3/758: sync 2024-03-04T16:04:18.959 INFO:tasks.workunit.client.0.smithi116.stdout:0/618: sync 2024-03-04T16:04:18.960 INFO:tasks.workunit.client.0.smithi116.stdout:2/772: fsync d7/d13/d3e/ded/d6c/f90 0 2024-03-04T16:04:18.960 INFO:tasks.workunit.client.0.smithi116.stdout:9/681: dwrite d2/d33/f54 [0,4194304] 0 2024-03-04T16:04:18.962 INFO:tasks.workunit.client.0.smithi116.stdout:9/682: write d2/d26/dd4/d4a/d6a/d94/d95/f9b [937089,110611] 0 2024-03-04T16:04:18.972 INFO:tasks.workunit.client.0.smithi116.stdout:5/727: creat dd/d6d/fe7 x:0 0 0 2024-03-04T16:04:18.978 INFO:tasks.workunit.client.0.smithi116.stdout:0/619: dread da/d16/d19/f68 [0,4194304] 0 2024-03-04T16:04:18.979 INFO:tasks.workunit.client.0.smithi116.stdout:3/759: creat dc/df/dbc/ffa x:0 0 0 2024-03-04T16:04:18.985 INFO:tasks.workunit.client.0.smithi116.stdout:7/736: unlink d3/d4/d8/d4a/d86/ldb 0 2024-03-04T16:04:18.994 INFO:tasks.workunit.client.0.smithi116.stdout:8/761: dwrite d2/de/d16/f2c [0,4194304] 0 2024-03-04T16:04:18.996 INFO:tasks.workunit.client.0.smithi116.stdout:1/635: write d5/d9/d21/d27/d32/d49/f66 [414813,84025] 0 2024-03-04T16:04:18.997 INFO:tasks.workunit.client.0.smithi116.stdout:6/669: mknod d6/de/d25/d36/d4e/ce3 0 2024-03-04T16:04:19.003 INFO:tasks.workunit.client.0.smithi116.stdout:0/620: creat da/d16/d28/d40/d5d/d7e/fc5 x:0 0 0 2024-03-04T16:04:19.006 INFO:tasks.workunit.client.0.smithi116.stdout:4/772: truncate d1/d3/d15/d20/d1c/f6a 1960809 0 2024-03-04T16:04:19.009 INFO:tasks.workunit.client.0.smithi116.stdout:3/760: unlink dc/d23/d84/d7f/d93/lf8 0 2024-03-04T16:04:19.011 INFO:tasks.workunit.client.0.smithi116.stdout:9/683: creat d2/d33/d65/dbf/fd7 x:0 0 0 2024-03-04T16:04:19.012 INFO:tasks.workunit.client.0.smithi116.stdout:7/737: rmdir d3/d4/d8/d4a/d88/dc2 39 2024-03-04T16:04:19.013 INFO:tasks.workunit.client.0.smithi116.stdout:8/762: creat d2/d38/d41/f102 x:0 0 0 2024-03-04T16:04:19.017 INFO:tasks.workunit.client.0.smithi116.stdout:6/670: unlink d6/de/d25/ld6 0 2024-03-04T16:04:19.020 INFO:tasks.workunit.client.0.smithi116.stdout:0/621: unlink c8 0 2024-03-04T16:04:19.023 INFO:tasks.workunit.client.0.smithi116.stdout:2/773: rename d7/d25/d41 to d7/d15/d5e/d9d/d11f 0 2024-03-04T16:04:19.026 INFO:tasks.workunit.client.0.smithi116.stdout:3/761: symlink dc/d23/d84/d7f/d93/db6/lfb 0 2024-03-04T16:04:19.030 INFO:tasks.workunit.client.0.smithi116.stdout:3/762: dwrite dc/d34/d44/fbd [0,4194304] 0 2024-03-04T16:04:19.031 INFO:tasks.workunit.client.0.smithi116.stdout:3/763: readlink dc/d34/l60 0 2024-03-04T16:04:19.032 INFO:tasks.workunit.client.0.smithi116.stdout:3/764: fsync dc/d23/d84/d7f/d93/db6/fec 0 2024-03-04T16:04:19.054 INFO:tasks.workunit.client.0.smithi116.stdout:5/728: write dd/f15 [1664180,16922] 0 2024-03-04T16:04:19.054 INFO:tasks.workunit.client.0.smithi116.stdout:9/684: rename d2/d26/dd4/ld0 to d2/d26/dd4/dab/d3c/ld8 0 2024-03-04T16:04:19.054 INFO:tasks.workunit.client.0.smithi116.stdout:1/636: write d5/d9/f3f [36527,31330] 0 2024-03-04T16:04:19.055 INFO:tasks.workunit.client.0.smithi116.stdout:4/773: symlink d1/d3/d15/d20/d1c/l109 0 2024-03-04T16:04:19.061 INFO:tasks.workunit.client.0.smithi116.stdout:5/729: write dd/d1f/d55/d73/d2d/d7c/fd9 [345207,79498] 0 2024-03-04T16:04:19.074 INFO:tasks.workunit.client.0.smithi116.stdout:9/685: unlink d2/d26/dd4/d4a/d6a/fc1 0 2024-03-04T16:04:19.075 INFO:tasks.workunit.client.0.smithi116.stdout:1/637: mknod d5/d9/d21/d27/d32/d49/d4c/d67/cd2 0 2024-03-04T16:04:19.076 INFO:tasks.workunit.client.0.smithi116.stdout:3/765: dwrite dc/df/f92 [4194304,4194304] 0 2024-03-04T16:04:19.076 INFO:tasks.workunit.client.0.smithi116.stdout:8/763: dwrite d2/de/d16/f30 [4194304,4194304] 0 2024-03-04T16:04:19.076 INFO:tasks.workunit.client.0.smithi116.stdout:0/622: getdents da/d16/d20/d42 0 2024-03-04T16:04:19.076 INFO:tasks.workunit.client.0.smithi116.stdout:6/671: dwrite d6/f13 [4194304,4194304] 0 2024-03-04T16:04:19.093 INFO:tasks.workunit.client.0.smithi116.stdout:5/730: chown dd/d1f/d55/d73/d2d/d59/db0/ld0 58 1 2024-03-04T16:04:19.094 INFO:tasks.workunit.client.0.smithi116.stdout:6/672: write d6/de/d29/d3a/fbe [888633,107666] 0 2024-03-04T16:04:19.094 INFO:tasks.workunit.client.0.smithi116.stdout:7/738: dwrite d3/d4/d8/d1a/f42 [0,4194304] 0 2024-03-04T16:04:19.095 INFO:tasks.workunit.client.0.smithi116.stdout:9/686: truncate d2/d19/d27/fae 47236 0 2024-03-04T16:04:19.095 INFO:tasks.workunit.client.0.smithi116.stdout:3/766: creat dc/d34/dc1/ffc x:0 0 0 2024-03-04T16:04:19.095 INFO:tasks.workunit.client.0.smithi116.stdout:8/764: rmdir d2/d13/d3c/d77/d97/d75 39 2024-03-04T16:04:19.095 INFO:tasks.workunit.client.0.smithi116.stdout:6/673: write d6/d56/f8b [2851814,82464] 0 2024-03-04T16:04:19.095 INFO:tasks.workunit.client.0.smithi116.stdout:8/765: write d2/de/d3e/d7b/fdd [435816,16197] 0 2024-03-04T16:04:19.095 INFO:tasks.workunit.client.0.smithi116.stdout:0/623: creat da/d16/d20/d42/fc6 x:0 0 0 2024-03-04T16:04:19.095 INFO:tasks.workunit.client.0.smithi116.stdout:6/674: write d6/de/d15/f66 [319095,123319] 0 2024-03-04T16:04:19.095 INFO:tasks.workunit.client.0.smithi116.stdout:4/774: creat d1/d3/d15/d20/d1c/d62/d81/f10a x:0 0 0 2024-03-04T16:04:19.095 INFO:tasks.workunit.client.0.smithi116.stdout:6/675: readlink d6/de/d25/d4a/d63/l70 0 2024-03-04T16:04:19.097 INFO:tasks.workunit.client.0.smithi116.stdout:7/739: mkdir d3/d4/dbf/de6/dd3/ddf/dee 0 2024-03-04T16:04:19.099 INFO:tasks.workunit.client.0.smithi116.stdout:3/767: chown dc/d23/d84/d7f/d93/db6/dc5/de0/da3/ld0 32033 1 2024-03-04T16:04:19.109 INFO:tasks.workunit.client.0.smithi116.stdout:6/676: dread d6/de/d1e/d80/f89 [0,4194304] 0 2024-03-04T16:04:19.113 INFO:tasks.workunit.client.0.smithi116.stdout:4/775: creat d1/d3/d15/d20/d1c/d62/d81/daf/f10b x:0 0 0 2024-03-04T16:04:19.116 INFO:tasks.workunit.client.0.smithi116.stdout:7/740: chown d3/d4/d34/d5d/db9/ce7 86549781 1 2024-03-04T16:04:19.117 INFO:tasks.workunit.client.0.smithi116.stdout:3/768: mkdir dc/d23/d84/d7f/dfd 0 2024-03-04T16:04:19.123 INFO:tasks.workunit.client.0.smithi116.stdout:8/766: dread d2/d13/d3c/d46/d6e/faf [0,4194304] 0 2024-03-04T16:04:19.124 INFO:tasks.workunit.client.0.smithi116.stdout:6/677: creat d6/de/d25/d4a/d63/fe4 x:0 0 0 2024-03-04T16:04:19.125 INFO:tasks.workunit.client.0.smithi116.stdout:8/767: dread d2/de/d16/d78/d29/de5/ff4 [0,4194304] 0 2024-03-04T16:04:19.126 INFO:tasks.workunit.client.0.smithi116.stdout:8/768: chown d2/de/l1a 6100 1 2024-03-04T16:04:19.127 INFO:tasks.workunit.client.0.smithi116.stdout:9/687: rename d2/d26/dd4/c62 to d2/d33/d65/cd9 0 2024-03-04T16:04:19.128 INFO:tasks.workunit.client.0.smithi116.stdout:4/776: dread d1/d3/d15/d20/f4e [0,4194304] 0 2024-03-04T16:04:19.129 INFO:tasks.workunit.client.0.smithi116.stdout:3/769: symlink dc/dd3/lfe 0 2024-03-04T16:04:19.130 INFO:tasks.workunit.client.0.smithi116.stdout:3/770: chown dc/d23/d84/d7f/d93/db6/dc5/de0/fa0 24731345 1 2024-03-04T16:04:19.131 INFO:tasks.workunit.client.0.smithi116.stdout:5/731: link dd/d1f/d55/d73/d2d/c2e dd/d1f/d55/d73/d2d/d31/ce8 0 2024-03-04T16:04:19.131 INFO:tasks.workunit.client.0.smithi116.stdout:3/771: chown dc/df/l12 22675 1 2024-03-04T16:04:19.132 INFO:tasks.workunit.client.0.smithi116.stdout:6/678: creat d6/de/d29/fe5 x:0 0 0 2024-03-04T16:04:19.140 INFO:tasks.workunit.client.0.smithi116.stdout:7/741: creat d3/d4/d9e/dd7/daf/dd5/fef x:0 0 0 2024-03-04T16:04:19.142 INFO:tasks.workunit.client.0.smithi116.stdout:8/769: dread d2/de/d3e/f2a [0,4194304] 0 2024-03-04T16:04:19.143 INFO:tasks.workunit.client.0.smithi116.stdout:8/770: write d2/d4a/dc7/fc8 [871464,48363] 0 2024-03-04T16:04:19.144 INFO:tasks.workunit.client.0.smithi116.stdout:5/732: creat dd/d1f/d72/d8e/fe9 x:0 0 0 2024-03-04T16:04:19.147 INFO:tasks.workunit.client.0.smithi116.stdout:6/679: symlink d6/de/d84/db4/d92/le6 0 2024-03-04T16:04:19.147 INFO:tasks.workunit.client.0.smithi116.stdout:1/638: truncate d5/d9/d21/d27/d32/d35/d55/fb8 1455543 0 2024-03-04T16:04:19.157 INFO:tasks.workunit.client.0.smithi116.stdout:8/771: unlink d2/d13/d3c/d77/laa 0 2024-03-04T16:04:19.158 INFO:tasks.workunit.client.0.smithi116.stdout:2/774: sync 2024-03-04T16:04:19.159 INFO:tasks.workunit.client.0.smithi116.stdout:5/733: rmdir dd/d1f/d55/d73/d2d/d31 39 2024-03-04T16:04:19.160 INFO:tasks.workunit.client.0.smithi116.stdout:5/734: write dd/d1f/d55/d73/fde [377180,78132] 0 2024-03-04T16:04:19.164 INFO:tasks.workunit.client.0.smithi116.stdout:0/624: dread da/d16/d20/d42/d4e/f8f [0,4194304] 0 2024-03-04T16:04:19.165 INFO:tasks.workunit.client.0.smithi116.stdout:0/625: chown da/d16/d19/c29 8331 1 2024-03-04T16:04:19.170 INFO:tasks.workunit.client.0.smithi116.stdout:7/742: dread d3/d4/d8/f1c [0,4194304] 0 2024-03-04T16:04:19.171 INFO:tasks.workunit.client.0.smithi116.stdout:2/775: dread d7/d81/f7b [0,4194304] 0 2024-03-04T16:04:19.172 INFO:tasks.workunit.client.0.smithi116.stdout:3/772: link dc/d34/l38 dc/d34/de7/deb/de2/lff 0 2024-03-04T16:04:19.174 INFO:tasks.workunit.client.0.smithi116.stdout:9/688: write d2/d26/f71 [1108714,23720] 0 2024-03-04T16:04:19.174 INFO:tasks.workunit.client.0.smithi116.stdout:6/680: mknod d6/d56/d58/ce7 0 2024-03-04T16:04:19.176 INFO:tasks.workunit.client.0.smithi116.stdout:9/689: fsync d2/d26/dd4/d4a/f7b 0 2024-03-04T16:04:19.177 INFO:tasks.workunit.client.0.smithi116.stdout:7/743: dwrite d3/d4/d8/d52/d5e/fc5 [0,4194304] 0 2024-03-04T16:04:19.177 INFO:tasks.workunit.client.0.smithi116.stdout:0/626: symlink da/d16/d20/lc7 0 2024-03-04T16:04:19.180 INFO:tasks.workunit.client.0.smithi116.stdout:4/777: getdents d1/d3/d15/d20/d1c/ddd 0 2024-03-04T16:04:19.185 INFO:tasks.workunit.client.0.smithi116.stdout:5/735: mkdir dd/d1f/d55/d73/d2d/d59/dea 0 2024-03-04T16:04:19.186 INFO:tasks.workunit.client.0.smithi116.stdout:5/736: truncate dd/fd2 72637 0 2024-03-04T16:04:19.187 INFO:tasks.workunit.client.0.smithi116.stdout:4/778: dwrite d1/d3/d40/d52/d84/fa0 [0,4194304] 0 2024-03-04T16:04:19.188 INFO:tasks.workunit.client.0.smithi116.stdout:2/776: creat d7/d81/d115/f120 x:0 0 0 2024-03-04T16:04:19.195 INFO:tasks.workunit.client.0.smithi116.stdout:6/681: creat d6/de/d15/fe8 x:0 0 0 2024-03-04T16:04:19.197 INFO:tasks.workunit.client.0.smithi116.stdout:5/737: dwrite dd/d1f/f29 [4194304,4194304] 0 2024-03-04T16:04:19.198 INFO:tasks.workunit.client.0.smithi116.stdout:9/690: read d2/f4f [224223,10773] 0 2024-03-04T16:04:19.198 INFO:tasks.workunit.client.0.smithi116.stdout:7/744: dread - d3/d4/f49 zero size 2024-03-04T16:04:19.200 INFO:tasks.workunit.client.0.smithi116.stdout:5/738: write dd/d1f/d51/fbf [9372362,64363] 0 2024-03-04T16:04:19.202 INFO:tasks.workunit.client.0.smithi116.stdout:5/739: fdatasync dd/fd2 0 2024-03-04T16:04:19.204 INFO:tasks.workunit.client.0.smithi116.stdout:1/639: dwrite d5/f6f [0,4194304] 0 2024-03-04T16:04:19.206 INFO:tasks.workunit.client.0.smithi116.stdout:5/740: chown dd/d1f/d55/d73/d2d/l28 170792024 1 2024-03-04T16:04:19.207 INFO:tasks.workunit.client.0.smithi116.stdout:3/773: creat dc/d23/d84/d7f/de8/f100 x:0 0 0 2024-03-04T16:04:19.210 INFO:tasks.workunit.client.0.smithi116.stdout:2/777: creat d7/d13/d3e/d98/dce/d47/dd9/f121 x:0 0 0 2024-03-04T16:04:19.212 INFO:tasks.workunit.client.0.smithi116.stdout:0/627: dwrite da/d16/d20/d42/f88 [0,4194304] 0 2024-03-04T16:04:19.215 INFO:tasks.workunit.client.0.smithi116.stdout:9/691: dwrite d2/d26/dd4/d4a/d6a/d94/d95/fb5 [0,4194304] 0 2024-03-04T16:04:19.217 INFO:tasks.workunit.client.0.smithi116.stdout:0/628: write da/d16/d28/fb0 [163057,25843] 0 2024-03-04T16:04:19.219 INFO:tasks.workunit.client.0.smithi116.stdout:8/772: link d2/de/cfd d2/d13/d3c/d46/c103 0 2024-03-04T16:04:19.221 INFO:tasks.workunit.client.0.smithi116.stdout:0/629: chown da/d16/d28/d40/f5c 122 1 2024-03-04T16:04:19.225 INFO:tasks.workunit.client.0.smithi116.stdout:1/640: mknod d5/d9/d21/d27/d94/cd3 0 2024-03-04T16:04:19.226 INFO:tasks.workunit.client.0.smithi116.stdout:0/630: write da/d16/fb1 [4318733,14833] 0 2024-03-04T16:04:19.227 INFO:tasks.workunit.client.0.smithi116.stdout:6/682: dwrite d6/de/d1e/f51 [0,4194304] 0 2024-03-04T16:04:19.229 INFO:tasks.workunit.client.0.smithi116.stdout:1/641: chown d5/d9/d21/d27/fb9 4008 1 2024-03-04T16:04:19.230 INFO:tasks.workunit.client.0.smithi116.stdout:5/741: rename dd/d6d/fe7 to dd/d1f/dc3/feb 0 2024-03-04T16:04:19.230 INFO:tasks.workunit.client.0.smithi116.stdout:9/692: dwrite d2/f87 [4194304,4194304] 0 2024-03-04T16:04:19.231 INFO:tasks.workunit.client.0.smithi116.stdout:3/774: dread dc/d23/d84/f8e [0,4194304] 0 2024-03-04T16:04:19.233 INFO:tasks.workunit.client.0.smithi116.stdout:2/778: creat d7/d13/d3e/d71/f122 x:0 0 0 2024-03-04T16:04:19.234 INFO:tasks.workunit.client.0.smithi116.stdout:1/642: write d5/d9/d21/d27/d32/f83 [290420,39884] 0 2024-03-04T16:04:19.237 INFO:tasks.workunit.client.0.smithi116.stdout:0/631: rmdir da/d16/d28/d40/d48/d57/db3/db2 39 2024-03-04T16:04:19.239 INFO:tasks.workunit.client.0.smithi116.stdout:6/683: creat d6/de/d25/d36/d44/dbb/ddd/fe9 x:0 0 0 2024-03-04T16:04:19.240 INFO:tasks.workunit.client.0.smithi116.stdout:5/742: creat dd/d1f/d72/d8e/fec x:0 0 0 2024-03-04T16:04:19.242 INFO:tasks.workunit.client.0.smithi116.stdout:3/775: rmdir dc/df/db8 39 2024-03-04T16:04:19.245 INFO:tasks.workunit.client.0.smithi116.stdout:7/745: creat d3/d4/d34/d5d/ff0 x:0 0 0 2024-03-04T16:04:19.246 INFO:tasks.workunit.client.0.smithi116.stdout:1/643: dwrite d5/d9/f3f [0,4194304] 0 2024-03-04T16:04:19.246 INFO:tasks.workunit.client.0.smithi116.stdout:7/746: readlink d3/d4/d8/d4a/d65/l70 0 2024-03-04T16:04:19.247 INFO:tasks.workunit.client.0.smithi116.stdout:2/779: unlink d7/d13/d3e/d98/dce/f2d 0 2024-03-04T16:04:19.248 INFO:tasks.workunit.client.0.smithi116.stdout:2/780: write d7/d81/d115/f120 [513136,68459] 0 2024-03-04T16:04:19.254 INFO:tasks.workunit.client.0.smithi116.stdout:5/743: creat dd/d1f/d72/fed x:0 0 0 2024-03-04T16:04:19.256 INFO:tasks.workunit.client.0.smithi116.stdout:3/776: truncate dc/f33 2064240 0 2024-03-04T16:04:19.259 INFO:tasks.workunit.client.0.smithi116.stdout:9/693: creat d2/d26/dd4/dab/fda x:0 0 0 2024-03-04T16:04:19.260 INFO:tasks.workunit.client.0.smithi116.stdout:9/694: readlink d2/d26/dd4/d4a/d5b/l34 0 2024-03-04T16:04:19.261 INFO:tasks.workunit.client.0.smithi116.stdout:9/695: chown d2/d26/dd4/dab/f89 416377 1 2024-03-04T16:04:19.264 INFO:tasks.workunit.client.0.smithi116.stdout:2/781: creat d7/d25/f123 x:0 0 0 2024-03-04T16:04:19.266 INFO:tasks.workunit.client.0.smithi116.stdout:5/744: creat dd/d1f/d55/fee x:0 0 0 2024-03-04T16:04:19.323 INFO:tasks.workunit.client.0.smithi116.stdout:3/777: mkdir dc/df/dbc/d101 0 2024-03-04T16:04:19.324 INFO:tasks.workunit.client.0.smithi116.stdout:7/747: symlink d3/d4/d9e/dd7/dd1/lf1 0 2024-03-04T16:04:19.324 INFO:tasks.workunit.client.0.smithi116.stdout:9/696: rename d2/d26/dd4/f82 to d2/d19/d27/dbd/fdb 0 2024-03-04T16:04:19.325 INFO:tasks.workunit.client.0.smithi116.stdout:2/782: dwrite d7/d81/f79 [0,4194304] 0 2024-03-04T16:04:19.327 INFO:tasks.workunit.client.0.smithi116.stdout:0/632: truncate da/d16/d28/d40/d5d/fad 2217731 0 2024-03-04T16:04:19.328 INFO:tasks.workunit.client.0.smithi116.stdout:1/644: link d5/d9/d21/d27/d32/d35/f87 d5/d9/d21/d27/d94/db6/d42/d8b/dc9/fd4 0 2024-03-04T16:04:19.329 INFO:tasks.workunit.client.0.smithi116.stdout:5/745: symlink dd/lef 0 2024-03-04T16:04:19.330 INFO:tasks.workunit.client.0.smithi116.stdout:9/697: mknod d2/d19/d5f/cdc 0 2024-03-04T16:04:19.341 INFO:tasks.workunit.client.0.smithi116.stdout:3/778: creat dc/df/d47/f102 x:0 0 0 2024-03-04T16:04:19.346 INFO:tasks.workunit.client.0.smithi116.stdout:3/779: unlink dc/df/c3f 0 2024-03-04T16:04:19.348 INFO:tasks.workunit.client.0.smithi116.stdout:1/645: creat d5/d9/d21/d27/d32/fd5 x:0 0 0 2024-03-04T16:04:19.350 INFO:tasks.workunit.client.0.smithi116.stdout:9/698: rename d2/d26/l60 to d2/d26/ldd 0 2024-03-04T16:04:19.352 INFO:tasks.workunit.client.0.smithi116.stdout:3/780: rename dc/d23/d84/l54 to dc/d34/de7/l103 0 2024-03-04T16:04:19.352 INFO:tasks.workunit.client.0.smithi116.stdout:9/699: write d2/d19/f9c [4715162,29596] 0 2024-03-04T16:04:19.356 INFO:tasks.workunit.client.0.smithi116.stdout:1/646: dwrite d5/d9/d21/d27/d32/d35/d55/db4/faa [0,4194304] 0 2024-03-04T16:04:19.360 INFO:tasks.workunit.client.0.smithi116.stdout:0/633: dread da/f1e [0,4194304] 0 2024-03-04T16:04:19.362 INFO:tasks.workunit.client.0.smithi116.stdout:3/781: dwrite dc/d23/d84/d7c/f65 [4194304,4194304] 0 2024-03-04T16:04:19.367 INFO:tasks.workunit.client.0.smithi116.stdout:0/634: symlink da/d16/d28/d40/d48/d57/d8a/daa/dba/lc8 0 2024-03-04T16:04:19.374 INFO:tasks.workunit.client.0.smithi116.stdout:2/783: write d7/d13/d3e/ded/d6c/f97 [848518,76634] 0 2024-03-04T16:04:19.375 INFO:tasks.workunit.client.0.smithi116.stdout:2/784: read - d7/d81/f10d zero size 2024-03-04T16:04:19.377 INFO:tasks.workunit.client.0.smithi116.stdout:2/785: write d7/d13/d3e/d98/dce/f106 [319879,130964] 0 2024-03-04T16:04:19.382 INFO:tasks.workunit.client.0.smithi116.stdout:7/748: dwrite d3/d4/d8/f45 [4194304,4194304] 0 2024-03-04T16:04:19.382 INFO:tasks.workunit.client.0.smithi116.stdout:2/786: rename d7/d15/f59 to d7/d13/d3e/ded/dfd/f124 0 2024-03-04T16:04:19.385 INFO:tasks.workunit.client.0.smithi116.stdout:3/782: dread dc/f33 [0,4194304] 0 2024-03-04T16:04:19.387 INFO:tasks.workunit.client.0.smithi116.stdout:2/787: mkdir d7/d15/d5e/d9d/df1/d125 0 2024-03-04T16:04:19.390 INFO:tasks.workunit.client.0.smithi116.stdout:3/783: creat dc/d23/d84/d7f/d93/f104 x:0 0 0 2024-03-04T16:04:19.392 INFO:tasks.workunit.client.0.smithi116.stdout:9/700: write d2/f1a [4748987,63633] 0 2024-03-04T16:04:19.393 INFO:tasks.workunit.client.0.smithi116.stdout:3/784: creat dc/d23/d84/d7f/d8b/f105 x:0 0 0 2024-03-04T16:04:19.394 INFO:tasks.workunit.client.0.smithi116.stdout:3/785: stat dc/d34/de7/deb/cd2 0 2024-03-04T16:04:19.396 INFO:tasks.workunit.client.0.smithi116.stdout:9/701: symlink d2/d33/d90/da7/lde 0 2024-03-04T16:04:19.399 INFO:tasks.workunit.client.0.smithi116.stdout:9/702: rmdir d2/d33/d40 39 2024-03-04T16:04:19.400 INFO:tasks.workunit.client.0.smithi116.stdout:2/788: dread d7/d13/f1e [0,4194304] 0 2024-03-04T16:04:19.404 INFO:tasks.workunit.client.0.smithi116.stdout:2/789: readlink d7/d13/d3e/d98/dce/d47/d64/da5/lcc 0 2024-03-04T16:04:19.404 INFO:tasks.workunit.client.0.smithi116.stdout:2/790: chown d7/d81/d115 20 1 2024-03-04T16:04:19.408 INFO:tasks.workunit.client.0.smithi116.stdout:2/791: creat d7/d15/dd7/f126 x:0 0 0 2024-03-04T16:04:19.411 INFO:tasks.workunit.client.0.smithi116.stdout:2/792: fdatasync d7/d13/d3e/d71/dad/fc4 0 2024-03-04T16:04:19.413 INFO:tasks.workunit.client.0.smithi116.stdout:9/703: link d2/d33/f46 d2/d26/fdf 0 2024-03-04T16:04:19.413 INFO:tasks.workunit.client.0.smithi116.stdout:2/793: truncate d7/d81/f10d 79857 0 2024-03-04T16:04:19.415 INFO:tasks.workunit.client.0.smithi116.stdout:2/794: readlink d7/l1f 0 2024-03-04T16:04:19.416 INFO:tasks.workunit.client.0.smithi116.stdout:2/795: readlink d7/d13/d3e/d71/lb2 0 2024-03-04T16:04:19.418 INFO:tasks.workunit.client.0.smithi116.stdout:3/786: write dc/d34/dc1/fcb [1051871,104376] 0 2024-03-04T16:04:19.418 INFO:tasks.workunit.client.0.smithi116.stdout:9/704: mknod d2/d26/ce0 0 2024-03-04T16:04:19.420 INFO:tasks.workunit.client.0.smithi116.stdout:9/705: chown d2/d26/dd4/d4a/d6a/d94/d95/fb5 386927006 1 2024-03-04T16:04:19.420 INFO:tasks.workunit.client.0.smithi116.stdout:2/796: chown d7/d15/d35/f70 11498071 1 2024-03-04T16:04:19.424 INFO:tasks.workunit.client.0.smithi116.stdout:9/706: symlink d2/d26/dd4/d4a/le1 0 2024-03-04T16:04:19.427 INFO:tasks.workunit.client.0.smithi116.stdout:9/707: mknod d2/d33/d65/ce2 0 2024-03-04T16:04:19.428 INFO:tasks.workunit.client.0.smithi116.stdout:3/787: write dc/d34/f7a [30766,60691] 0 2024-03-04T16:04:19.430 INFO:tasks.workunit.client.0.smithi116.stdout:9/708: rename d2/d33/d90/da7 to d2/d33/d65/d6c/db8/de3 0 2024-03-04T16:04:19.433 INFO:tasks.workunit.client.0.smithi116.stdout:9/709: fsync d2/d19/f2a 0 2024-03-04T16:04:19.442 INFO:tasks.workunit.client.0.smithi116.stdout:3/788: dwrite dc/d23/d84/d7c/f5c [0,4194304] 0 2024-03-04T16:04:19.450 INFO:tasks.workunit.client.0.smithi116.stdout:9/710: dwrite d2/d26/f47 [0,4194304] 0 2024-03-04T16:04:19.455 INFO:tasks.workunit.client.0.smithi116.stdout:3/789: dread dc/d23/d84/d7f/d93/fb1 [0,4194304] 0 2024-03-04T16:04:19.467 INFO:tasks.workunit.client.0.smithi116.stdout:3/790: chown dc/d23/c43 105 1 2024-03-04T16:04:19.467 INFO:tasks.workunit.client.0.smithi116.stdout:3/791: fdatasync dc/d23/d84/d7f/d93/db6/fbb 0 2024-03-04T16:04:19.471 INFO:tasks.workunit.client.0.smithi116.stdout:3/792: unlink dc/d23/f85 0 2024-03-04T16:04:19.472 INFO:tasks.workunit.client.0.smithi116.stdout:3/793: dread - dc/d23/d84/d7f/d93/f104 zero size 2024-03-04T16:04:19.472 INFO:tasks.workunit.client.0.smithi116.stdout:3/794: chown dc/f5f 17 1 2024-03-04T16:04:19.473 INFO:tasks.workunit.client.0.smithi116.stdout:3/795: write dc/d23/d84/d7f/d93/db6/fbb [880667,17311] 0 2024-03-04T16:04:19.477 INFO:tasks.workunit.client.0.smithi116.stdout:3/796: creat dc/df/db8/f106 x:0 0 0 2024-03-04T16:04:19.479 INFO:tasks.workunit.client.0.smithi116.stdout:3/797: mkdir dc/d23/d84/d7f/d107 0 2024-03-04T16:04:19.480 INFO:tasks.workunit.client.0.smithi116.stdout:3/798: read - dc/d23/d84/d7f/d93/db6/fec zero size 2024-03-04T16:04:19.486 INFO:tasks.workunit.client.0.smithi116.stdout:3/799: dwrite dc/d34/dc1/fe3 [0,4194304] 0 2024-03-04T16:04:19.487 INFO:tasks.workunit.client.0.smithi116.stdout:3/800: write dc/f70 [7404197,86063] 0 2024-03-04T16:04:19.488 INFO:tasks.workunit.client.0.smithi116.stdout:3/801: stat dc/d23/d84/d7c/f65 0 2024-03-04T16:04:19.491 INFO:tasks.workunit.client.0.smithi116.stdout:3/802: creat dc/df/dbc/f108 x:0 0 0 2024-03-04T16:04:19.494 INFO:tasks.workunit.client.0.smithi116.stdout:3/803: symlink dc/d23/d84/d7f/d93/l109 0 2024-03-04T16:04:19.499 INFO:tasks.workunit.client.0.smithi116.stdout:3/804: rmdir dc/d23/d84/d7f/d107 0 2024-03-04T16:04:19.502 INFO:tasks.workunit.client.0.smithi116.stdout:3/805: symlink dc/d23/d84/d7f/d93/db6/dc5/l10a 0 2024-03-04T16:04:19.505 INFO:tasks.workunit.client.0.smithi116.stdout:3/806: symlink dc/d34/dc1/l10b 0 2024-03-04T16:04:19.508 INFO:tasks.workunit.client.0.smithi116.stdout:3/807: rename dc/d23/d84/c6f to dc/d23/d84/d7f/de8/c10c 0 2024-03-04T16:04:19.508 INFO:tasks.workunit.client.0.smithi116.stdout:3/808: read dc/d23/d84/f8e [399415,109564] 0 2024-03-04T16:04:19.511 INFO:tasks.workunit.client.0.smithi116.stdout:3/809: mknod dc/d23/c10d 0 2024-03-04T16:04:19.558 INFO:tasks.workunit.client.0.smithi116.stdout:3/810: dwrite dc/f5f [0,4194304] 0 2024-03-04T16:04:19.565 INFO:tasks.workunit.client.0.smithi116.stdout:3/811: getdents dc/d34/d44/dab 0 2024-03-04T16:04:19.568 INFO:tasks.workunit.client.0.smithi116.stdout:3/812: mknod dc/d34/de7/c10e 0 2024-03-04T16:04:19.569 INFO:tasks.workunit.client.0.smithi116.stdout:3/813: readlink dc/df/d47/d91/la7 0 2024-03-04T16:04:19.569 INFO:tasks.workunit.client.0.smithi116.stdout:3/814: fdatasync dc/df/d14/f49 0 2024-03-04T16:04:19.572 INFO:tasks.workunit.client.0.smithi116.stdout:3/815: symlink dc/d34/ddd/l10f 0 2024-03-04T16:04:19.597 INFO:tasks.workunit.client.0.smithi116.stdout:4/779: sync 2024-03-04T16:04:19.597 INFO:tasks.workunit.client.0.smithi116.stdout:8/773: sync 2024-03-04T16:04:19.599 INFO:tasks.workunit.client.0.smithi116.stdout:8/774: mkdir d2/d13/d3c/d46/d104 0 2024-03-04T16:04:19.601 INFO:tasks.workunit.client.0.smithi116.stdout:4/780: rename d1/d6e/d74/dc2 to d1/d3/dd/d24/db4/d10c 0 2024-03-04T16:04:19.603 INFO:tasks.workunit.client.0.smithi116.stdout:8/775: dread d2/d38/d55/f65 [0,4194304] 0 2024-03-04T16:04:19.609 INFO:tasks.workunit.client.0.smithi116.stdout:8/776: rmdir d2/d13/d3c/d77/d97/d5a 39 2024-03-04T16:04:19.610 INFO:tasks.workunit.client.0.smithi116.stdout:8/777: write d2/d13/d5e/fdb [118590,59194] 0 2024-03-04T16:04:19.611 INFO:tasks.workunit.client.0.smithi116.stdout:8/778: readlink d2/de/d16/l33 0 2024-03-04T16:04:19.612 INFO:tasks.workunit.client.0.smithi116.stdout:8/779: write d2/d38/f64 [2908843,35569] 0 2024-03-04T16:04:19.613 INFO:tasks.workunit.client.0.smithi116.stdout:4/781: link d1/d3/dd/l1d d1/d3/dd/l10d 0 2024-03-04T16:04:19.619 INFO:tasks.workunit.client.0.smithi116.stdout:4/782: dwrite d1/d3/d40/d52/d84/d88/f8b [0,4194304] 0 2024-03-04T16:04:19.623 INFO:tasks.workunit.client.0.smithi116.stdout:4/783: creat d1/d3/dd/d24/db4/d10c/f10e x:0 0 0 2024-03-04T16:04:19.627 INFO:tasks.workunit.client.0.smithi116.stdout:4/784: getdents d1/d3/dd/d45/d50 0 2024-03-04T16:04:19.631 INFO:tasks.workunit.client.0.smithi116.stdout:4/785: mknod d1/d3/dd/d45/d50/c10f 0 2024-03-04T16:04:19.632 INFO:tasks.workunit.client.0.smithi116.stdout:8/780: dread d2/d13/d3c/d77/d97/f44 [4194304,4194304] 0 2024-03-04T16:04:19.635 INFO:tasks.workunit.client.0.smithi116.stdout:4/786: creat d1/d3/d40/d52/d84/f110 x:0 0 0 2024-03-04T16:04:19.638 INFO:tasks.workunit.client.0.smithi116.stdout:8/781: dwrite d2/de/d16/d78/d29/d7a/fd2 [0,4194304] 0 2024-03-04T16:04:19.644 INFO:tasks.workunit.client.0.smithi116.stdout:8/782: dwrite d2/d38/f64 [0,4194304] 0 2024-03-04T16:04:19.669 INFO:tasks.workunit.client.0.smithi116.stdout:8/783: truncate d2/d13/d5e/fc0 502871 0 2024-03-04T16:04:19.670 INFO:tasks.workunit.client.0.smithi116.stdout:8/784: creat d2/de/d3e/f105 x:0 0 0 2024-03-04T16:04:19.673 INFO:tasks.workunit.client.0.smithi116.stdout:8/785: symlink d2/de/d16/dd1/l106 0 2024-03-04T16:04:19.675 INFO:tasks.workunit.client.0.smithi116.stdout:8/786: creat d2/de/d16/dc3/f107 x:0 0 0 2024-03-04T16:04:19.678 INFO:tasks.workunit.client.0.smithi116.stdout:8/787: fdatasync d2/d13/d3c/d77/d97/d5a/f91 0 2024-03-04T16:04:19.679 INFO:tasks.workunit.client.0.smithi116.stdout:8/788: chown d2/de/d16/f1b 593581530 1 2024-03-04T16:04:19.696 INFO:tasks.workunit.client.0.smithi116.stdout:8/789: dwrite d2/d38/d41/ff9 [0,4194304] 0 2024-03-04T16:04:19.698 INFO:tasks.workunit.client.0.smithi116.stdout:8/790: fsync d2/d38/d41/f102 0 2024-03-04T16:04:19.703 INFO:tasks.workunit.client.0.smithi116.stdout:8/791: dwrite d2/d38/f64 [0,4194304] 0 2024-03-04T16:04:19.705 INFO:tasks.workunit.client.0.smithi116.stdout:8/792: creat d2/de3/f108 x:0 0 0 2024-03-04T16:04:19.709 INFO:tasks.workunit.client.0.smithi116.stdout:8/793: rename d2/d13/d3c/f69 to d2/d13/d3c/d77/d97/deb/f109 0 2024-03-04T16:04:19.710 INFO:tasks.workunit.client.0.smithi116.stdout:8/794: chown d2/d13/d5e/da7 105565 1 2024-03-04T16:04:19.712 INFO:tasks.workunit.client.0.smithi116.stdout:8/795: rmdir d2/d13/d3c/d46 39 2024-03-04T16:04:19.715 INFO:tasks.workunit.client.0.smithi116.stdout:8/796: mknod d2/d13/d3c/d46/d6e/c10a 0 2024-03-04T16:04:19.717 INFO:tasks.workunit.client.0.smithi116.stdout:8/797: rmdir d2/de/d16/d78/d29/de5 39 2024-03-04T16:04:19.723 INFO:tasks.workunit.client.0.smithi116.stdout:8/798: dwrite d2/de/d3e/d7b/fdd [0,4194304] 0 2024-03-04T16:04:19.733 INFO:tasks.workunit.client.0.smithi116.stdout:8/799: creat d2/de/d16/d78/d29/de5/df2/da4/f10b x:0 0 0 2024-03-04T16:04:19.735 INFO:tasks.workunit.client.0.smithi116.stdout:8/800: mknod d2/d13/d3c/d77/d97/d42/c10c 0 2024-03-04T16:04:19.736 INFO:tasks.workunit.client.0.smithi116.stdout:6/684: sync 2024-03-04T16:04:19.736 INFO:tasks.workunit.client.0.smithi116.stdout:8/801: read d2/de/d16/d78/d29/de5/df2/fa0 [3925239,88131] 0 2024-03-04T16:04:19.737 INFO:tasks.workunit.client.0.smithi116.stdout:8/802: truncate d2/f32 1995850 0 2024-03-04T16:04:19.738 INFO:tasks.workunit.client.0.smithi116.stdout:8/803: stat d2/d13/db9 0 2024-03-04T16:04:19.739 INFO:tasks.workunit.client.0.smithi116.stdout:8/804: fsync d2/f32 0 2024-03-04T16:04:19.741 INFO:tasks.workunit.client.0.smithi116.stdout:8/805: write d2/d13/d3c/d46/d6e/fbc [772038,92396] 0 2024-03-04T16:04:19.742 INFO:tasks.workunit.client.0.smithi116.stdout:6/685: dwrite d6/de/f4f [4194304,4194304] 0 2024-03-04T16:04:19.748 INFO:tasks.workunit.client.0.smithi116.stdout:8/806: link d2/d13/d3c/d77/d97/d5a/f6f d2/d13/d3c/d77/d97/d5a/f10d 0 2024-03-04T16:04:19.757 INFO:tasks.workunit.client.0.smithi116.stdout:8/807: dwrite d2/d38/d55/fb5 [0,4194304] 0 2024-03-04T16:04:19.769 INFO:tasks.workunit.client.0.smithi116.stdout:6/686: fdatasync d6/de/f4f 0 2024-03-04T16:04:19.777 INFO:tasks.workunit.client.0.smithi116.stdout:6/687: write d6/de/f3b [1769160,31764] 0 2024-03-04T16:04:19.778 INFO:tasks.workunit.client.0.smithi116.stdout:6/688: mknod d6/d9e/cea 0 2024-03-04T16:04:19.779 INFO:tasks.workunit.client.0.smithi116.stdout:6/689: stat d6/de/d25/cd8 0 2024-03-04T16:04:19.786 INFO:tasks.workunit.client.0.smithi116.stdout:6/690: dwrite d6/de/d25/f85 [0,4194304] 0 2024-03-04T16:04:19.792 INFO:tasks.workunit.client.0.smithi116.stdout:6/691: dread d6/de/f52 [0,4194304] 0 2024-03-04T16:04:19.794 INFO:tasks.workunit.client.0.smithi116.stdout:6/692: rmdir d6/d9e/da5/db3 39 2024-03-04T16:04:19.795 INFO:tasks.workunit.client.0.smithi116.stdout:6/693: chown d6/d9e/fc0 7949 1 2024-03-04T16:04:19.804 INFO:tasks.workunit.client.0.smithi116.stdout:6/694: write d6/de/d25/d36/f1a [428485,80999] 0 2024-03-04T16:04:19.808 INFO:tasks.workunit.client.0.smithi116.stdout:6/695: rename d6/de/d84/fcb to d6/de/d25/d4a/d63/d95/feb 0 2024-03-04T16:04:19.861 INFO:tasks.workunit.client.0.smithi116.stdout:5/746: sync 2024-03-04T16:04:19.863 INFO:tasks.workunit.client.0.smithi116.stdout:5/747: truncate dd/d6d/f88 789751 0 2024-03-04T16:04:19.865 INFO:tasks.workunit.client.0.smithi116.stdout:5/748: mkdir dd/d6d/df0 0 2024-03-04T16:04:19.869 INFO:tasks.workunit.client.0.smithi116.stdout:5/749: rename dd/d1f/d55/d73/d2d/d8d/fe0 to dd/d1f/d55/d73/d2d/ff1 0 2024-03-04T16:04:19.870 INFO:tasks.workunit.client.0.smithi116.stdout:5/750: fdatasync dd/d1f/d55/f58 0 2024-03-04T16:04:19.881 INFO:tasks.workunit.client.0.smithi116.stdout:5/751: unlink dd/d1f/d55/d73/d2d/fdc 0 2024-03-04T16:04:19.893 INFO:tasks.workunit.client.0.smithi116.stdout:5/752: dread dd/d45/f95 [0,4194304] 0 2024-03-04T16:04:19.896 INFO:tasks.workunit.client.0.smithi116.stdout:5/753: mknod dd/d1f/d55/d73/d2d/cf2 0 2024-03-04T16:04:19.896 INFO:tasks.workunit.client.0.smithi116.stdout:5/754: write dd/f38 [2957377,121086] 0 2024-03-04T16:04:19.897 INFO:tasks.workunit.client.0.smithi116.stdout:5/755: chown dd/d1f/d55/lab 426 1 2024-03-04T16:04:19.906 INFO:tasks.workunit.client.0.smithi116.stdout:5/756: getdents dd/d1f/d55/d73/d2d/d8d 0 2024-03-04T16:04:19.908 INFO:tasks.workunit.client.0.smithi116.stdout:5/757: dread fa [0,4194304] 0 2024-03-04T16:04:19.910 INFO:tasks.workunit.client.0.smithi116.stdout:5/758: fdatasync dd/d1f/d72/fa0 0 2024-03-04T16:04:19.917 INFO:tasks.workunit.client.0.smithi116.stdout:5/759: link dd/d1f/d55/d73/d2d/d31/la3 dd/d6d/dc6/lf3 0 2024-03-04T16:04:19.921 INFO:tasks.workunit.client.0.smithi116.stdout:5/760: rename dd/d1f/dc3/fce to dd/d1f/d5c/ff4 0 2024-03-04T16:04:19.924 INFO:tasks.workunit.client.0.smithi116.stdout:5/761: write fa [601036,62401] 0 2024-03-04T16:04:19.938 INFO:tasks.workunit.client.0.smithi116.stdout:5/762: dread dd/d1f/f29 [0,4194304] 0 2024-03-04T16:04:19.948 INFO:tasks.workunit.client.0.smithi116.stdout:1/647: sync 2024-03-04T16:04:19.948 INFO:tasks.workunit.client.0.smithi116.stdout:0/635: sync 2024-03-04T16:04:19.952 INFO:tasks.workunit.client.0.smithi116.stdout:1/648: truncate d5/f48 373318 0 2024-03-04T16:04:19.955 INFO:tasks.workunit.client.0.smithi116.stdout:0/636: dwrite da/d16/d20/d42/d4e/f8f [0,4194304] 0 2024-03-04T16:04:19.956 INFO:tasks.workunit.client.0.smithi116.stdout:5/763: dread dd/d1f/d55/d73/d2d/d8d/fa6 [0,4194304] 0 2024-03-04T16:04:19.969 INFO:tasks.workunit.client.0.smithi116.stdout:5/764: mknod dd/d1f/de5/cf5 0 2024-03-04T16:04:19.969 INFO:tasks.workunit.client.0.smithi116.stdout:1/649: write d5/d9/d21/d27/d32/d35/d55/db4/f7f [313727,120190] 0 2024-03-04T16:04:19.970 INFO:tasks.workunit.client.0.smithi116.stdout:0/637: rmdir da/d16/d20/d53/db5 0 2024-03-04T16:04:19.974 INFO:tasks.workunit.client.0.smithi116.stdout:0/638: symlink da/d16/lc9 0 2024-03-04T16:04:19.977 INFO:tasks.workunit.client.0.smithi116.stdout:1/650: rename d5/d9/ce to d5/d9/d21/d27/d32/d35/d55/d75/d91/cd6 0 2024-03-04T16:04:19.979 INFO:tasks.workunit.client.0.smithi116.stdout:1/651: write d5/d9/d21/d27/d32/d35/f52 [420218,130015] 0 2024-03-04T16:04:19.979 INFO:tasks.workunit.client.0.smithi116.stdout:0/639: symlink da/d16/d28/d40/d48/d57/d8a/daa/lca 0 2024-03-04T16:04:19.983 INFO:tasks.workunit.client.0.smithi116.stdout:0/640: mknod da/d16/d19/d27/d45/ccb 0 2024-03-04T16:04:19.984 INFO:tasks.workunit.client.0.smithi116.stdout:5/765: truncate dd/d1f/fb5 2827254 0 2024-03-04T16:04:19.984 INFO:tasks.workunit.client.0.smithi116.stdout:0/641: read - da/d16/d28/d40/d5d/d7e/fc5 zero size 2024-03-04T16:04:19.985 INFO:tasks.workunit.client.0.smithi116.stdout:1/652: creat d5/d9/d21/d27/d94/fd7 x:0 0 0 2024-03-04T16:04:19.986 INFO:tasks.workunit.client.0.smithi116.stdout:5/766: mkdir dd/d1f/d55/d73/d2d/d8d/df6 0 2024-03-04T16:04:19.988 INFO:tasks.workunit.client.0.smithi116.stdout:0/642: unlink da/d16/d28/d3d/d41/f7d 0 2024-03-04T16:04:19.989 INFO:tasks.workunit.client.0.smithi116.stdout:1/653: mknod d5/d9/d21/d27/d32/d35/dbf/cd8 0 2024-03-04T16:04:19.993 INFO:tasks.workunit.client.0.smithi116.stdout:0/643: dread - da/d16/d19/d27/d70/f8e zero size 2024-03-04T16:04:19.996 INFO:tasks.workunit.client.0.smithi116.stdout:5/767: fdatasync dd/d6d/f88 0 2024-03-04T16:04:20.007 INFO:tasks.workunit.client.0.smithi116.stdout:2/797: sync 2024-03-04T16:04:20.007 INFO:tasks.workunit.client.0.smithi116.stdout:3/816: sync 2024-03-04T16:04:20.007 INFO:tasks.workunit.client.0.smithi116.stdout:7/749: sync 2024-03-04T16:04:20.008 INFO:tasks.workunit.client.0.smithi116.stdout:7/750: write d3/f6 [913635,15538] 0 2024-03-04T16:04:20.012 INFO:tasks.workunit.client.0.smithi116.stdout:7/751: chown d3/d4/d8/d4a/d87/f9d 26 1 2024-03-04T16:04:20.014 INFO:tasks.workunit.client.0.smithi116.stdout:2/798: unlink d7/d13/d3e/ded/c83 0 2024-03-04T16:04:20.016 INFO:tasks.workunit.client.0.smithi116.stdout:3/817: dwrite dc/d23/d84/fd1 [0,4194304] 0 2024-03-04T16:04:20.020 INFO:tasks.workunit.client.0.smithi116.stdout:7/752: dwrite d3/d4/d11/f38 [4194304,4194304] 0 2024-03-04T16:04:20.024 INFO:tasks.workunit.client.0.smithi116.stdout:1/654: truncate d5/d9/f3f 1222877 0 2024-03-04T16:04:20.025 INFO:tasks.workunit.client.0.smithi116.stdout:1/655: fdatasync d5/d9/d21/d27/d32/f3e 0 2024-03-04T16:04:20.026 INFO:tasks.workunit.client.0.smithi116.stdout:7/753: dwrite d3/d95/f6f [0,4194304] 0 2024-03-04T16:04:20.028 INFO:tasks.workunit.client.0.smithi116.stdout:3/818: mkdir dc/d34/de7/d110 0 2024-03-04T16:04:20.029 INFO:tasks.workunit.client.0.smithi116.stdout:3/819: readlink dc/d23/d84/d7f/d93/db6/dc5/l10a 0 2024-03-04T16:04:20.031 INFO:tasks.workunit.client.0.smithi116.stdout:2/799: creat d7/d13/d3e/d78/d107/f127 x:0 0 0 2024-03-04T16:04:20.032 INFO:tasks.workunit.client.0.smithi116.stdout:2/800: dread - d7/d13/d3e/d78/d107/f127 zero size 2024-03-04T16:04:20.035 INFO:tasks.workunit.client.0.smithi116.stdout:5/768: write dd/d1f/d72/d8e/f98 [2053620,100332] 0 2024-03-04T16:04:20.036 INFO:tasks.workunit.client.0.smithi116.stdout:5/769: write dd/d1f/d72/fed [516890,31912] 0 2024-03-04T16:04:20.041 INFO:tasks.workunit.client.0.smithi116.stdout:7/754: rename d3/d4/d7e to d3/d4/d8/d4a/d65/df2 0 2024-03-04T16:04:20.043 INFO:tasks.workunit.client.0.smithi116.stdout:3/820: write dc/df/d14/f1a [1819301,90960] 0 2024-03-04T16:04:20.044 INFO:tasks.workunit.client.0.smithi116.stdout:5/770: mkdir dd/d1f/d55/d73/d2d/dc0/df7 0 2024-03-04T16:04:20.044 INFO:tasks.workunit.client.0.smithi116.stdout:0/644: dwrite da/d16/d28/d40/d5d/fad [0,4194304] 0 2024-03-04T16:04:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:5/771: rename dd/lef to dd/d1f/d55/d73/lf8 0 2024-03-04T16:04:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:0/645: mknod da/d16/d19/ccc 0 2024-03-04T16:04:20.053 INFO:tasks.workunit.client.0.smithi116.stdout:5/772: symlink dd/d1f/d5c/lf9 0 2024-03-04T16:04:20.054 INFO:tasks.workunit.client.0.smithi116.stdout:1/656: dread d5/d9/d21/d27/d32/d35/d55/fb8 [0,4194304] 0 2024-03-04T16:04:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:2/801: dread d7/f4d [0,4194304] 0 2024-03-04T16:04:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:5/773: read dd/d1f/d55/d73/d2d/d31/f3c [1487663,15408] 0 2024-03-04T16:04:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:5/774: chown dd/ca4 3179540 1 2024-03-04T16:04:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:5/775: write dd/f15 [298274,124519] 0 2024-03-04T16:04:20.064 INFO:tasks.workunit.client.0.smithi116.stdout:2/802: creat d7/d13/d3e/ded/d6c/da4/da1/f128 x:0 0 0 2024-03-04T16:04:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:6/696: sync 2024-03-04T16:04:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:9/711: sync 2024-03-04T16:04:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:8/808: sync 2024-03-04T16:04:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:4/787: sync 2024-03-04T16:04:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:1/657: mkdir d5/d9/d21/d27/d94/db6/d42/d8b/dc9/dd1/dd9 0 2024-03-04T16:04:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:9/712: stat d2/d19/f9c 0 2024-03-04T16:04:20.071 INFO:tasks.workunit.client.0.smithi116.stdout:2/803: creat d7/d13/d3e/d98/f129 x:0 0 0 2024-03-04T16:04:20.071 INFO:tasks.workunit.client.0.smithi116.stdout:2/804: chown d7/d15/f93 0 1 2024-03-04T16:04:20.076 INFO:tasks.workunit.client.0.smithi116.stdout:6/697: mkdir d6/d9e/dec 0 2024-03-04T16:04:20.076 INFO:tasks.workunit.client.0.smithi116.stdout:0/646: dread da/d16/d28/d3d/f64 [0,4194304] 0 2024-03-04T16:04:20.078 INFO:tasks.workunit.client.0.smithi116.stdout:9/713: fsync d2/d33/d40/fbc 0 2024-03-04T16:04:20.081 INFO:tasks.workunit.client.0.smithi116.stdout:1/658: mkdir d5/d9/d21/d27/d32/d49/dda 0 2024-03-04T16:04:20.082 INFO:tasks.workunit.client.0.smithi116.stdout:9/714: chown d2/d26/dd4/d4a/d6a/d94/d95/fb5 187 1 2024-03-04T16:04:20.083 INFO:tasks.workunit.client.0.smithi116.stdout:5/776: creat dd/d1f/d55/d73/d2d/dc0/df7/ffa x:0 0 0 2024-03-04T16:04:20.084 INFO:tasks.workunit.client.0.smithi116.stdout:2/805: mkdir d7/d81/dc2/d12a 0 2024-03-04T16:04:20.085 INFO:tasks.workunit.client.0.smithi116.stdout:6/698: dwrite d6/de/d29/fe5 [0,4194304] 0 2024-03-04T16:04:20.085 INFO:tasks.workunit.client.0.smithi116.stdout:4/788: dread d1/f5 [0,4194304] 0 2024-03-04T16:04:20.086 INFO:tasks.workunit.client.0.smithi116.stdout:0/647: dwrite da/d16/d28/d40/f5c [0,4194304] 0 2024-03-04T16:04:20.086 INFO:tasks.workunit.client.0.smithi116.stdout:1/659: chown d5/d9/d21/d27/d32/fd5 6741507 1 2024-03-04T16:04:20.086 INFO:tasks.workunit.client.0.smithi116.stdout:3/821: write dc/d23/d84/d7f/d8b/fc9 [248193,3318] 0 2024-03-04T16:04:20.102 INFO:tasks.workunit.client.0.smithi116.stdout:5/777: creat dd/d1f/d55/d73/d2d/dc0/df7/ffb x:0 0 0 2024-03-04T16:04:20.104 INFO:tasks.workunit.client.0.smithi116.stdout:6/699: creat d6/de/d84/db4/d92/fed x:0 0 0 2024-03-04T16:04:20.105 INFO:tasks.workunit.client.0.smithi116.stdout:9/715: dread d2/d26/dd4/d4a/d6a/d94/d95/f9b [0,4194304] 0 2024-03-04T16:04:20.106 INFO:tasks.workunit.client.0.smithi116.stdout:7/755: fdatasync d3/d4/d9e/dd7/f89 0 2024-03-04T16:04:20.107 INFO:tasks.workunit.client.0.smithi116.stdout:9/716: write d2/d26/f71 [1847217,108192] 0 2024-03-04T16:04:20.108 INFO:tasks.workunit.client.0.smithi116.stdout:3/822: mknod dc/df/dbc/c111 0 2024-03-04T16:04:20.108 INFO:tasks.workunit.client.0.smithi116.stdout:9/717: stat d2/d33/d40/caa 0 2024-03-04T16:04:20.112 INFO:tasks.workunit.client.0.smithi116.stdout:0/648: unlink da/d16/d28/d40/d48/l4d 0 2024-03-04T16:04:20.115 INFO:tasks.workunit.client.0.smithi116.stdout:8/809: rename d2/d13/d3c/d77/d97/l56 to d2/d38/l10e 0 2024-03-04T16:04:20.126 INFO:tasks.workunit.client.0.smithi116.stdout:7/756: mkdir d3/d4/dbf/de6/dd3/ddf/df3 0 2024-03-04T16:04:20.130 INFO:tasks.workunit.client.0.smithi116.stdout:8/810: dread d2/de/d16/f1b [0,4194304] 0 2024-03-04T16:04:20.132 INFO:tasks.workunit.client.0.smithi116.stdout:0/649: mknod da/d16/d28/d40/d48/d57/d8a/daa/dba/ccd 0 2024-03-04T16:04:20.132 INFO:tasks.workunit.client.0.smithi116.stdout:0/650: chown f7 45356 1 2024-03-04T16:04:20.133 INFO:tasks.workunit.client.0.smithi116.stdout:1/660: mknod d5/d9/d21/d27/d32/d35/d55/db4/cdb 0 2024-03-04T16:04:20.134 INFO:tasks.workunit.client.0.smithi116.stdout:6/700: rmdir d6/de/d84 39 2024-03-04T16:04:20.136 INFO:tasks.workunit.client.0.smithi116.stdout:2/806: rename d7/d13/d3e/d98/lbe to d7/d13/d3e/d98/dce/d47/d64/d75/d77/d119/l12b 0 2024-03-04T16:04:20.137 INFO:tasks.workunit.client.0.smithi116.stdout:0/651: dwrite da/d16/d28/d40/d48/d57/d8a/daa/fc0 [0,4194304] 0 2024-03-04T16:04:20.137 INFO:tasks.workunit.client.0.smithi116.stdout:5/778: getdents dd/d1f/d55/d73/d2d/d59/db0/dca 0 2024-03-04T16:04:20.176 INFO:tasks.workunit.client.0.smithi116.stdout:4/789: write d1/d3/d15/d20/f1e [1807143,57111] 0 2024-03-04T16:04:20.177 INFO:tasks.workunit.client.0.smithi116.stdout:8/811: mknod d2/d13/d3c/d77/d97/d5a/c10f 0 2024-03-04T16:04:20.177 INFO:tasks.workunit.client.0.smithi116.stdout:6/701: rename d6/l6d to d6/de/d25/d4a/d63/d95/lee 0 2024-03-04T16:04:20.177 INFO:tasks.workunit.client.0.smithi116.stdout:2/807: creat d7/d13/d3e/d98/dce/d47/dd9/d112/f12c x:0 0 0 2024-03-04T16:04:20.179 INFO:tasks.workunit.client.0.smithi116.stdout:0/652: creat da/d16/d20/d79/fce x:0 0 0 2024-03-04T16:04:20.180 INFO:tasks.workunit.client.0.smithi116.stdout:6/702: chown d6/d56/d58/f98 3546 1 2024-03-04T16:04:20.181 INFO:tasks.workunit.client.0.smithi116.stdout:3/823: creat dc/d34/f112 x:0 0 0 2024-03-04T16:04:20.181 INFO:tasks.workunit.client.0.smithi116.stdout:5/779: chown dd/d1f/d55/d73/d2d/d31/ce8 41337632 1 2024-03-04T16:04:20.185 INFO:tasks.workunit.client.0.smithi116.stdout:9/718: dwrite d2/d33/f77 [0,4194304] 0 2024-03-04T16:04:20.185 INFO:tasks.workunit.client.0.smithi116.stdout:2/808: creat d7/d25/f12d x:0 0 0 2024-03-04T16:04:20.189 INFO:tasks.workunit.client.0.smithi116.stdout:8/812: creat d2/de/d16/d78/d29/de5/df2/d37/f110 x:0 0 0 2024-03-04T16:04:20.190 INFO:tasks.workunit.client.0.smithi116.stdout:3/824: dwrite dc/d23/d84/d7c/f69 [4194304,4194304] 0 2024-03-04T16:04:20.190 INFO:tasks.workunit.client.0.smithi116.stdout:0/653: mkdir da/d16/d20/dcf 0 2024-03-04T16:04:20.197 INFO:tasks.workunit.client.0.smithi116.stdout:7/757: mknod d3/d4/d8/d1a/cf4 0 2024-03-04T16:04:20.201 INFO:tasks.workunit.client.0.smithi116.stdout:9/719: creat d2/d19/d5f/fe4 x:0 0 0 2024-03-04T16:04:20.201 INFO:tasks.workunit.client.0.smithi116.stdout:9/720: write d2/d26/dd4/dab/d3c/f93 [121868,94038] 0 2024-03-04T16:04:20.202 INFO:tasks.workunit.client.0.smithi116.stdout:2/809: mkdir d7/d13/d3e/d98/dce/d47/d64/d75/d77/d12e 0 2024-03-04T16:04:20.206 INFO:tasks.workunit.client.0.smithi116.stdout:2/810: dwrite d7/d15/d35/f8f [0,4194304] 0 2024-03-04T16:04:20.207 INFO:tasks.workunit.client.0.smithi116.stdout:3/825: creat dc/d23/d84/d7f/d8b/f113 x:0 0 0 2024-03-04T16:04:20.209 INFO:tasks.workunit.client.0.smithi116.stdout:0/654: unlink da/d16/d28/d40/d48/f52 0 2024-03-04T16:04:20.212 INFO:tasks.workunit.client.0.smithi116.stdout:6/703: symlink d6/d9e/dec/lef 0 2024-03-04T16:04:20.216 INFO:tasks.workunit.client.0.smithi116.stdout:9/721: fsync d2/fc3 0 2024-03-04T16:04:20.218 INFO:tasks.workunit.client.0.smithi116.stdout:8/813: link d2/de/d16/d78/d29/d7a/fb8 d2/de/d3e/f111 0 2024-03-04T16:04:20.227 INFO:tasks.workunit.client.0.smithi116.stdout:6/704: stat d6/de/d25/d36/d44/dbb/ddd/f79 0 2024-03-04T16:04:20.228 INFO:tasks.workunit.client.0.smithi116.stdout:6/705: fdatasync d6/de/d15/f26 0 2024-03-04T16:04:20.235 INFO:tasks.workunit.client.0.smithi116.stdout:4/790: write d1/d3/d15/d20/f4e [1065215,1525] 0 2024-03-04T16:04:20.236 INFO:tasks.workunit.client.0.smithi116.stdout:2/811: fsync d7/d15/dd7/fef 0 2024-03-04T16:04:20.241 INFO:tasks.workunit.client.0.smithi116.stdout:4/791: dwrite d1/d3/d15/d20/d1c/d62/d81/da6/fbd [0,4194304] 0 2024-03-04T16:04:20.241 INFO:tasks.workunit.client.0.smithi116.stdout:6/706: chown d6/de/d25/d36/d44/dbb/ddd/d7b/dc6 8 1 2024-03-04T16:04:20.242 INFO:tasks.workunit.client.0.smithi116.stdout:4/792: write d1/f75 [2364926,99937] 0 2024-03-04T16:04:20.243 INFO:tasks.workunit.client.0.smithi116.stdout:4/793: write d1/d3/d40/d52/fbc [1084498,62873] 0 2024-03-04T16:04:20.250 INFO:tasks.workunit.client.0.smithi116.stdout:0/655: creat da/d16/d28/d40/d48/d57/db3/db2/dc2/fd0 x:0 0 0 2024-03-04T16:04:20.259 INFO:tasks.workunit.client.0.smithi116.stdout:9/722: rename d2/d26/dd4/d4a/d5b to d2/d26/dd4/de5 0 2024-03-04T16:04:20.262 INFO:tasks.workunit.client.0.smithi116.stdout:0/656: stat da/d16/d28/d40/d5d 0 2024-03-04T16:04:20.265 INFO:tasks.workunit.client.0.smithi116.stdout:6/707: mknod d6/de/d84/db4/d92/cf0 0 2024-03-04T16:04:20.266 INFO:tasks.workunit.client.0.smithi116.stdout:6/708: fsync d6/de/d25/d36/d44/dbb/ddd/d7b/faf 0 2024-03-04T16:04:20.266 INFO:tasks.workunit.client.0.smithi116.stdout:6/709: chown d6/fd5 753081 1 2024-03-04T16:04:20.267 INFO:tasks.workunit.client.0.smithi116.stdout:6/710: write d6/d56/d58/f98 [453333,84960] 0 2024-03-04T16:04:20.267 INFO:tasks.workunit.client.0.smithi116.stdout:8/814: rename d2/d4a/cde to d2/d13/d3c/d46/d104/c112 0 2024-03-04T16:04:20.269 INFO:tasks.workunit.client.0.smithi116.stdout:4/794: mkdir d1/d3/d15/d20/d65/dfe/d111 0 2024-03-04T16:04:20.271 INFO:tasks.workunit.client.0.smithi116.stdout:0/657: rmdir da/d16/d28/d40/d48/d77 39 2024-03-04T16:04:20.271 INFO:tasks.workunit.client.0.smithi116.stdout:3/826: write dc/f33 [110156,47924] 0 2024-03-04T16:04:20.271 INFO:tasks.workunit.client.0.smithi116.stdout:0/658: stat da/d16/d19/d27/l98 0 2024-03-04T16:04:20.273 INFO:tasks.workunit.client.0.smithi116.stdout:0/659: stat da/d16/d28/d40/d48/d57/db3/db2/fb6 0 2024-03-04T16:04:20.279 INFO:tasks.workunit.client.0.smithi116.stdout:6/711: unlink d6/de/d1e/d7c/fd0 0 2024-03-04T16:04:20.280 INFO:tasks.workunit.client.0.smithi116.stdout:7/758: write d3/d4/d34/f74 [542720,121550] 0 2024-03-04T16:04:20.281 INFO:tasks.workunit.client.0.smithi116.stdout:8/815: mknod d2/d38/d41/dab/c113 0 2024-03-04T16:04:20.282 INFO:tasks.workunit.client.0.smithi116.stdout:8/816: chown d2/d13/d3c/lc9 45936 1 2024-03-04T16:04:20.282 INFO:tasks.workunit.client.0.smithi116.stdout:8/817: chown d2/d38/d41/dab/c113 81043 1 2024-03-04T16:04:20.284 INFO:tasks.workunit.client.0.smithi116.stdout:2/812: getdents d7/d13/d3e/d98/dce/d47 0 2024-03-04T16:04:20.290 INFO:tasks.workunit.client.0.smithi116.stdout:6/712: mknod d6/de/d25/d36/cf1 0 2024-03-04T16:04:20.293 INFO:tasks.workunit.client.0.smithi116.stdout:4/795: mkdir d1/d3/d15/d20/d1c/d62/d112 0 2024-03-04T16:04:20.296 INFO:tasks.workunit.client.0.smithi116.stdout:4/796: dread d1/d3/d15/d20/d1c/d62/d81/fb0 [0,4194304] 0 2024-03-04T16:04:20.296 INFO:tasks.workunit.client.0.smithi116.stdout:2/813: read d7/d13/d3e/d98/fba [108580,3851] 0 2024-03-04T16:04:20.297 INFO:tasks.workunit.client.0.smithi116.stdout:2/814: chown d7/d13/d3e 5660 1 2024-03-04T16:04:20.305 INFO:tasks.workunit.client.0.smithi116.stdout:4/797: creat d1/f113 x:0 0 0 2024-03-04T16:04:20.307 INFO:tasks.workunit.client.0.smithi116.stdout:3/827: creat dc/d23/d84/d7f/d93/db6/dc5/de0/f114 x:0 0 0 2024-03-04T16:04:20.307 INFO:tasks.workunit.client.0.smithi116.stdout:3/828: stat dc/df/c59 0 2024-03-04T16:04:20.311 INFO:tasks.workunit.client.0.smithi116.stdout:0/660: rename da/d16/d19/d27/l98 to da/d16/d28/d40/d5d/ld1 0 2024-03-04T16:04:20.312 INFO:tasks.workunit.client.0.smithi116.stdout:0/661: stat da/d16/d28/d40/d48/d57/d8a/fc1 0 2024-03-04T16:04:20.312 INFO:tasks.workunit.client.0.smithi116.stdout:0/662: chown da/d16/d28/d39/l9b 1673478 1 2024-03-04T16:04:20.314 INFO:tasks.workunit.client.0.smithi116.stdout:7/759: link d3/d4/d8/d4a/db3/dc3/le1 d3/d4/d9e/dd7/lf5 0 2024-03-04T16:04:20.318 INFO:tasks.workunit.client.0.smithi116.stdout:3/829: symlink dc/d34/dc1/l115 0 2024-03-04T16:04:20.320 INFO:tasks.workunit.client.0.smithi116.stdout:8/818: write d2/d13/d3c/d77/d97/d5a/f91 [125193,53632] 0 2024-03-04T16:04:20.324 INFO:tasks.workunit.client.0.smithi116.stdout:4/798: mknod d1/d3/de/c114 0 2024-03-04T16:04:20.326 INFO:tasks.workunit.client.0.smithi116.stdout:3/830: creat dc/d34/de7/deb/f116 x:0 0 0 2024-03-04T16:04:20.328 INFO:tasks.workunit.client.0.smithi116.stdout:6/713: getdents d6/d56/d58 0 2024-03-04T16:04:20.329 INFO:tasks.workunit.client.0.smithi116.stdout:3/831: chown dc/d23/d84/d7f/d93 560 1 2024-03-04T16:04:20.329 INFO:tasks.workunit.client.0.smithi116.stdout:8/819: mkdir d2/d4a/d114 0 2024-03-04T16:04:20.329 INFO:tasks.workunit.client.0.smithi116.stdout:2/815: write d7/d13/dec/f101 [5246901,27923] 0 2024-03-04T16:04:20.333 INFO:tasks.workunit.client.0.smithi116.stdout:7/760: link d3/d95/fe5 d3/d95/ff6 0 2024-03-04T16:04:20.333 INFO:tasks.workunit.client.0.smithi116.stdout:4/799: dwrite d1/d3/d15/d20/d65/f99 [0,4194304] 0 2024-03-04T16:04:20.333 INFO:tasks.workunit.client.0.smithi116.stdout:2/816: read - d7/d13/d3e/d98/dce/ffa zero size 2024-03-04T16:04:20.334 INFO:tasks.workunit.client.0.smithi116.stdout:7/761: readlink d3/lba 0 2024-03-04T16:04:20.334 INFO:tasks.workunit.client.0.smithi116.stdout:6/714: symlink d6/de/d25/d36/d44/lf2 0 2024-03-04T16:04:20.335 INFO:tasks.workunit.client.0.smithi116.stdout:0/663: link da/d16/d28/c37 da/d16/d20/dcf/cd2 0 2024-03-04T16:04:20.337 INFO:tasks.workunit.client.0.smithi116.stdout:8/820: mknod d2/d38/d55/c115 0 2024-03-04T16:04:20.342 INFO:tasks.workunit.client.0.smithi116.stdout:2/817: dwrite d7/d15/d35/f8f [0,4194304] 0 2024-03-04T16:04:20.342 INFO:tasks.workunit.client.0.smithi116.stdout:3/832: link dc/f4b dc/dd3/f117 0 2024-03-04T16:04:20.343 INFO:tasks.workunit.client.0.smithi116.stdout:3/833: stat dc/d23/d84/d7f/d93/db6/dc5/de0/f6b 0 2024-03-04T16:04:20.353 INFO:tasks.workunit.client.0.smithi116.stdout:7/762: unlink d3/d4/d8/d1a/l26 0 2024-03-04T16:04:20.355 INFO:tasks.workunit.client.0.smithi116.stdout:2/818: mkdir d7/d25/d12f 0 2024-03-04T16:04:20.358 INFO:tasks.workunit.client.0.smithi116.stdout:4/800: creat d1/d3/d15/d20/d65/dfe/d111/f115 x:0 0 0 2024-03-04T16:04:20.360 INFO:tasks.workunit.client.0.smithi116.stdout:4/801: read - d1/d3/dd/d24/db4/d10c/fec zero size 2024-03-04T16:04:20.360 INFO:tasks.workunit.client.0.smithi116.stdout:7/763: dwrite d3/d4/d8/d4a/d65/df2/fcd [0,4194304] 0 2024-03-04T16:04:20.360 INFO:tasks.workunit.client.0.smithi116.stdout:0/664: creat da/d16/d20/dcf/fd3 x:0 0 0 2024-03-04T16:04:20.366 INFO:tasks.workunit.client.0.smithi116.stdout:6/715: write d6/de/d1e/d80/f57 [284048,124198] 0 2024-03-04T16:04:20.369 INFO:tasks.workunit.client.0.smithi116.stdout:1/661: sync 2024-03-04T16:04:20.369 INFO:tasks.workunit.client.0.smithi116.stdout:9/723: sync 2024-03-04T16:04:20.369 INFO:tasks.workunit.client.0.smithi116.stdout:5/780: sync 2024-03-04T16:04:20.370 INFO:tasks.workunit.client.0.smithi116.stdout:4/802: mknod d1/d6e/d74/de0/c116 0 2024-03-04T16:04:20.372 INFO:tasks.workunit.client.0.smithi116.stdout:5/781: write dd/d1f/f4d [451851,20170] 0 2024-03-04T16:04:20.375 INFO:tasks.workunit.client.0.smithi116.stdout:8/821: dread d2/de/d16/d78/d29/d7a/fd2 [4194304,4194304] 0 2024-03-04T16:04:20.377 INFO:tasks.workunit.client.0.smithi116.stdout:6/716: dwrite d6/de/d1e/d7c/fbd [0,4194304] 0 2024-03-04T16:04:20.380 INFO:tasks.workunit.client.0.smithi116.stdout:4/803: creat d1/d6e/d74/f117 x:0 0 0 2024-03-04T16:04:20.381 INFO:tasks.workunit.client.0.smithi116.stdout:9/724: read d2/d26/dd4/dab/d3c/f79 [1421510,40166] 0 2024-03-04T16:04:20.382 INFO:tasks.workunit.client.0.smithi116.stdout:3/834: truncate dc/df/f7e 1598613 0 2024-03-04T16:04:20.384 INFO:tasks.workunit.client.0.smithi116.stdout:7/764: creat d3/d4/d8/d1a/ff7 x:0 0 0 2024-03-04T16:04:20.389 INFO:tasks.workunit.client.0.smithi116.stdout:8/822: mkdir d2/d4a/dc7/d116 0 2024-03-04T16:04:20.389 INFO:tasks.workunit.client.0.smithi116.stdout:2/819: dwrite d7/d15/d35/f48 [0,4194304] 0 2024-03-04T16:04:20.397 INFO:tasks.workunit.client.0.smithi116.stdout:0/665: dwrite da/d16/d19/d27/d70/f8e [0,4194304] 0 2024-03-04T16:04:20.400 INFO:tasks.workunit.client.0.smithi116.stdout:2/820: dread d7/d13/d3e/ded/d6c/ddb/fe0 [0,4194304] 0 2024-03-04T16:04:20.401 INFO:tasks.workunit.client.0.smithi116.stdout:4/804: rename d1/d3/dd/d45/l59 to d1/d3/dd/d8a/l118 0 2024-03-04T16:04:20.401 INFO:tasks.workunit.client.0.smithi116.stdout:3/835: dwrite dc/d23/d84/d7f/d93/db6/dc5/de0/f6b [0,4194304] 0 2024-03-04T16:04:20.403 INFO:tasks.workunit.client.0.smithi116.stdout:7/765: dread - d3/d4/d8/fae zero size 2024-03-04T16:04:20.404 INFO:tasks.workunit.client.0.smithi116.stdout:5/782: fsync dd/d1f/d55/d73/d2d/d31/f6a 0 2024-03-04T16:04:20.407 INFO:tasks.workunit.client.0.smithi116.stdout:8/823: dread d2/d38/d41/ff9 [0,4194304] 0 2024-03-04T16:04:20.409 INFO:tasks.workunit.client.0.smithi116.stdout:6/717: symlink d6/de/d25/d36/d44/dbb/ddd/lf3 0 2024-03-04T16:04:20.409 INFO:tasks.workunit.client.0.smithi116.stdout:1/662: dwrite d5/d9/d21/d27/d32/d35/f5c [0,4194304] 0 2024-03-04T16:04:20.416 INFO:tasks.workunit.client.0.smithi116.stdout:9/725: symlink d2/d33/le6 0 2024-03-04T16:04:20.418 INFO:tasks.workunit.client.0.smithi116.stdout:9/726: readlink d2/d26/dd4/d4a/le1 0 2024-03-04T16:04:20.419 INFO:tasks.workunit.client.0.smithi116.stdout:8/824: dwrite d2/de/d16/d78/d29/ff0 [0,4194304] 0 2024-03-04T16:04:20.423 INFO:tasks.workunit.client.0.smithi116.stdout:1/663: dread d5/d9/d21/d27/d32/d35/d55/db4/f70 [0,4194304] 0 2024-03-04T16:04:20.425 INFO:tasks.workunit.client.0.smithi116.stdout:2/821: symlink d7/d81/d115/l130 0 2024-03-04T16:04:20.427 INFO:tasks.workunit.client.0.smithi116.stdout:7/766: creat d3/d4/d9e/ff8 x:0 0 0 2024-03-04T16:04:20.430 INFO:tasks.workunit.client.0.smithi116.stdout:5/783: truncate dd/d1f/d5c/d5e/faa 539009 0 2024-03-04T16:04:20.432 INFO:tasks.workunit.client.0.smithi116.stdout:1/664: dwrite d5/d9/d21/d27/d32/d35/d55/d75/d91/fb2 [0,4194304] 0 2024-03-04T16:04:20.445 INFO:tasks.workunit.client.0.smithi116.stdout:8/825: chown d2/d13/d3c/d77/d97/cb4 380 1 2024-03-04T16:04:20.447 INFO:tasks.workunit.client.0.smithi116.stdout:4/805: dwrite d1/f85 [0,4194304] 0 2024-03-04T16:04:20.456 INFO:tasks.workunit.client.0.smithi116.stdout:5/784: fdatasync dd/d1f/d5c/d5e/f62 0 2024-03-04T16:04:20.456 INFO:tasks.workunit.client.0.smithi116.stdout:6/718: symlink d6/d9e/da5/db3/lf4 0 2024-03-04T16:04:20.457 INFO:tasks.workunit.client.0.smithi116.stdout:9/727: truncate d2/d33/f88 2518298 0 2024-03-04T16:04:20.459 INFO:tasks.workunit.client.0.smithi116.stdout:8/826: mkdir d2/d4a/dc7/d117 0 2024-03-04T16:04:20.461 INFO:tasks.workunit.client.0.smithi116.stdout:4/806: mkdir d1/d3/dd/d45/d119 0 2024-03-04T16:04:20.463 INFO:tasks.workunit.client.0.smithi116.stdout:5/785: dwrite dd/d1f/d55/d73/d2d/f50 [0,4194304] 0 2024-03-04T16:04:20.464 INFO:tasks.workunit.client.0.smithi116.stdout:5/786: rename dd/d1f/d55/d73/d2d to dd/d1f/d55/d73/d2d/dc0/dfc 22 2024-03-04T16:04:20.470 INFO:tasks.workunit.client.0.smithi116.stdout:1/665: write d5/d9/d21/d27/d32/d35/d55/db4/f63 [351518,44621] 0 2024-03-04T16:04:20.496 INFO:tasks.workunit.client.0.smithi116.stdout:7/767: creat d3/de9/ff9 x:0 0 0 2024-03-04T16:04:20.496 INFO:tasks.workunit.client.0.smithi116.stdout:2/822: symlink d7/d15/d5e/d9d/l131 0 2024-03-04T16:04:20.497 INFO:tasks.workunit.client.0.smithi116.stdout:0/666: getdents da/d16/d28/d40/d48/da7 0 2024-03-04T16:04:20.498 INFO:tasks.workunit.client.0.smithi116.stdout:6/719: dread - d6/de/d29/d3a/f9f zero size 2024-03-04T16:04:20.498 INFO:tasks.workunit.client.0.smithi116.stdout:0/667: readlink da/d16/d28/d40/d48/d57/d8a/daa/dba/lc8 0 2024-03-04T16:04:20.500 INFO:tasks.workunit.client.0.smithi116.stdout:3/836: getdents dc 0 2024-03-04T16:04:20.501 INFO:tasks.workunit.client.0.smithi116.stdout:3/837: write dc/d34/d44/fbd [2967696,130329] 0 2024-03-04T16:04:20.502 INFO:tasks.workunit.client.0.smithi116.stdout:9/728: creat d2/d33/d65/d6c/db8/de3/fe7 x:0 0 0 2024-03-04T16:04:20.506 INFO:tasks.workunit.client.0.smithi116.stdout:4/807: rmdir d1/d3/d15/d20/d1c/d6f 39 2024-03-04T16:04:20.507 INFO:tasks.workunit.client.0.smithi116.stdout:5/787: symlink dd/d1f/d55/d73/lfd 0 2024-03-04T16:04:20.510 INFO:tasks.workunit.client.0.smithi116.stdout:1/666: symlink d5/d9/d21/d27/d32/d35/d55/dae/ldc 0 2024-03-04T16:04:20.515 INFO:tasks.workunit.client.0.smithi116.stdout:0/668: symlink da/d16/d28/d40/d48/d57/ld4 0 2024-03-04T16:04:20.518 INFO:tasks.workunit.client.0.smithi116.stdout:9/729: creat d2/d26/d28/fe8 x:0 0 0 2024-03-04T16:04:20.523 INFO:tasks.workunit.client.0.smithi116.stdout:1/667: symlink d5/da1/ldd 0 2024-03-04T16:04:20.524 INFO:tasks.workunit.client.0.smithi116.stdout:9/730: dread d2/d19/d5f/f61 [0,4194304] 0 2024-03-04T16:04:20.524 INFO:tasks.workunit.client.0.smithi116.stdout:2/823: fsync d7/fb5 0 2024-03-04T16:04:20.524 INFO:tasks.workunit.client.0.smithi116.stdout:9/731: chown d2/d26/f44 1 1 2024-03-04T16:04:20.526 INFO:tasks.workunit.client.0.smithi116.stdout:6/720: symlink d6/de/d25/d4a/lf5 0 2024-03-04T16:04:20.530 INFO:tasks.workunit.client.0.smithi116.stdout:3/838: creat dc/d23/d84/d7f/d93/df1/f118 x:0 0 0 2024-03-04T16:04:20.530 INFO:tasks.workunit.client.0.smithi116.stdout:3/839: stat dc/d23/f1b 0 2024-03-04T16:04:20.531 INFO:tasks.workunit.client.0.smithi116.stdout:5/788: mknod dd/d1f/d55/d73/d2d/d59/dea/cfe 0 2024-03-04T16:04:20.532 INFO:tasks.workunit.client.0.smithi116.stdout:0/669: dread da/d16/d28/d3d/f4a [0,4194304] 0 2024-03-04T16:04:20.535 INFO:tasks.workunit.client.0.smithi116.stdout:2/824: symlink d7/d13/dec/l132 0 2024-03-04T16:04:20.536 INFO:tasks.workunit.client.0.smithi116.stdout:2/825: stat d7/d13/d3e/d98/dce/d47/d64/da5 0 2024-03-04T16:04:20.537 INFO:tasks.workunit.client.0.smithi116.stdout:7/768: rename d3/d4/d8/d4a/d65/f80 to d3/ffa 0 2024-03-04T16:04:20.539 INFO:tasks.workunit.client.0.smithi116.stdout:6/721: rmdir d6/de/d1e/d7c 39 2024-03-04T16:04:20.546 INFO:tasks.workunit.client.0.smithi116.stdout:2/826: fsync d7/d15/dd7/f88 0 2024-03-04T16:04:20.547 INFO:tasks.workunit.client.0.smithi116.stdout:2/827: write d7/d81/f10d [105694,127887] 0 2024-03-04T16:04:20.548 INFO:tasks.workunit.client.0.smithi116.stdout:5/789: dread dd/d6d/f80 [0,4194304] 0 2024-03-04T16:04:20.549 INFO:tasks.workunit.client.0.smithi116.stdout:4/808: rename d1/d3/d15/d20/d65 to d1/d3/dd/dd9/d11a 0 2024-03-04T16:04:20.553 INFO:tasks.workunit.client.0.smithi116.stdout:9/732: creat d2/d19/fe9 x:0 0 0 2024-03-04T16:04:20.557 INFO:tasks.workunit.client.0.smithi116.stdout:4/809: creat d1/d3/dd/d24/d51/f11b x:0 0 0 2024-03-04T16:04:20.558 INFO:tasks.workunit.client.0.smithi116.stdout:7/769: mknod d3/d4/d8/d4a/d88/cfb 0 2024-03-04T16:04:20.559 INFO:tasks.workunit.client.0.smithi116.stdout:4/810: chown d1/d3/d40/d52/d84/dc1 326170171 1 2024-03-04T16:04:20.560 INFO:tasks.workunit.client.0.smithi116.stdout:2/828: getdents d7/d25/d12f 0 2024-03-04T16:04:20.562 INFO:tasks.workunit.client.0.smithi116.stdout:5/790: mkdir dd/d1f/d55/d73/d2d/d59/d71/dff 0 2024-03-04T16:04:20.563 INFO:tasks.workunit.client.0.smithi116.stdout:5/791: dread - dd/d1f/fb2 zero size 2024-03-04T16:04:20.564 INFO:tasks.workunit.client.0.smithi116.stdout:7/770: chown d3/d4/d8/c4b 383640 1 2024-03-04T16:04:20.564 INFO:tasks.workunit.client.0.smithi116.stdout:9/733: dread d2/d26/d28/f81 [0,4194304] 0 2024-03-04T16:04:20.564 INFO:tasks.workunit.client.0.smithi116.stdout:4/811: mknod d1/c11c 0 2024-03-04T16:04:20.566 INFO:tasks.workunit.client.0.smithi116.stdout:2/829: write d7/d15/d35/f3b [1607229,29840] 0 2024-03-04T16:04:20.568 INFO:tasks.workunit.client.0.smithi116.stdout:5/792: read dd/d1f/d55/f58 [103696,129305] 0 2024-03-04T16:04:20.569 INFO:tasks.workunit.client.0.smithi116.stdout:7/771: fsync d3/d4/d8/d4a/d65/df2/f9b 0 2024-03-04T16:04:20.569 INFO:tasks.workunit.client.0.smithi116.stdout:9/734: write d2/d19/d27/fae [770517,43987] 0 2024-03-04T16:04:20.570 INFO:tasks.workunit.client.0.smithi116.stdout:0/670: dread da/d16/d20/f65 [0,4194304] 0 2024-03-04T16:04:20.573 INFO:tasks.workunit.client.0.smithi116.stdout:8/827: dwrite d2/de/d16/d78/d29/d7a/fb8 [0,4194304] 0 2024-03-04T16:04:20.581 INFO:tasks.workunit.client.0.smithi116.stdout:1/668: truncate d5/f5a 2440587 0 2024-03-04T16:04:20.582 INFO:tasks.workunit.client.0.smithi116.stdout:7/772: dwrite d3/d4/d34/fc0 [0,4194304] 0 2024-03-04T16:04:20.584 INFO:tasks.workunit.client.0.smithi116.stdout:1/669: stat d5/d9/d21/d27/d32/d35/d55/dae/ldc 0 2024-03-04T16:04:20.587 INFO:tasks.workunit.client.0.smithi116.stdout:2/830: fdatasync d7/d15/d5e/d9d/df1/d113/f11b 0 2024-03-04T16:04:20.588 INFO:tasks.workunit.client.0.smithi116.stdout:0/671: mkdir da/d16/d28/dd5 0 2024-03-04T16:04:20.590 INFO:tasks.workunit.client.0.smithi116.stdout:7/773: read - d3/d4/f49 zero size 2024-03-04T16:04:20.592 INFO:tasks.workunit.client.0.smithi116.stdout:4/812: rename d1/d3/f44 to d1/d3/d15/d20/d1c/d62/f11d 0 2024-03-04T16:04:20.593 INFO:tasks.workunit.client.0.smithi116.stdout:2/831: chown d7/d13/d3e/d98/dce/d47/d64/ldd 1889062 1 2024-03-04T16:04:20.594 INFO:tasks.workunit.client.0.smithi116.stdout:7/774: fdatasync d3/d4/d8/d1a/ff7 0 2024-03-04T16:04:20.596 INFO:tasks.workunit.client.0.smithi116.stdout:9/735: creat d2/d33/d65/fea x:0 0 0 2024-03-04T16:04:20.599 INFO:tasks.workunit.client.0.smithi116.stdout:5/793: getdents dd/d1f/d55/d73 0 2024-03-04T16:04:20.599 INFO:tasks.workunit.client.0.smithi116.stdout:6/722: dwrite d6/de/d29/d3a/f9f [0,4194304] 0 2024-03-04T16:04:20.600 INFO:tasks.workunit.client.0.smithi116.stdout:8/828: dread d2/d13/d3c/d77/fd9 [0,4194304] 0 2024-03-04T16:04:20.602 INFO:tasks.workunit.client.0.smithi116.stdout:8/829: fdatasync d2/de/d3e/fa8 0 2024-03-04T16:04:20.606 INFO:tasks.workunit.client.0.smithi116.stdout:7/775: mkdir d3/d4/d9e/dd7/daf/dfc 0 2024-03-04T16:04:20.608 INFO:tasks.workunit.client.0.smithi116.stdout:7/776: chown d3/d4/d8/d4a/db3/db6/fda 2874344 1 2024-03-04T16:04:20.612 INFO:tasks.workunit.client.0.smithi116.stdout:5/794: stat dd/d1f/ccf 0 2024-03-04T16:04:20.615 INFO:tasks.workunit.client.0.smithi116.stdout:4/813: getdents d1/d3/dd/d45/d50/dad 0 2024-03-04T16:04:20.618 INFO:tasks.workunit.client.0.smithi116.stdout:6/723: rmdir d6/de/d25/d36/d4e 39 2024-03-04T16:04:20.621 INFO:tasks.workunit.client.0.smithi116.stdout:7/777: read d3/f56 [572253,14653] 0 2024-03-04T16:04:20.623 INFO:tasks.workunit.client.0.smithi116.stdout:1/670: rename d5/d9/d21/d27/l59 to d5/d9/d21/lde 0 2024-03-04T16:04:20.623 INFO:tasks.workunit.client.0.smithi116.stdout:0/672: truncate da/d16/d20/d42/f88 1505280 0 2024-03-04T16:04:20.624 INFO:tasks.workunit.client.0.smithi116.stdout:9/736: creat d2/d26/dd4/d4a/d6a/d94/dbe/feb x:0 0 0 2024-03-04T16:04:20.624 INFO:tasks.workunit.client.0.smithi116.stdout:2/832: dwrite d7/d13/d3e/ded/d76/d7c/f7d [0,4194304] 0 2024-03-04T16:04:20.625 INFO:tasks.workunit.client.0.smithi116.stdout:5/795: symlink dd/d1f/d55/d73/d2d/d8d/l100 0 2024-03-04T16:04:20.629 INFO:tasks.workunit.client.0.smithi116.stdout:2/833: read d7/d15/d5e/d9d/ffc [3030016,23021] 0 2024-03-04T16:04:20.629 INFO:tasks.workunit.client.0.smithi116.stdout:6/724: symlink d6/d56/lf6 0 2024-03-04T16:04:20.631 INFO:tasks.workunit.client.0.smithi116.stdout:6/725: chown d6/de/d84/db4/l76 13123 1 2024-03-04T16:04:20.631 INFO:tasks.workunit.client.0.smithi116.stdout:4/814: rename d1/d3/dd/d8a/cf5 to d1/d6c/c11e 0 2024-03-04T16:04:20.632 INFO:tasks.workunit.client.0.smithi116.stdout:1/671: mknod d5/d9/cdf 0 2024-03-04T16:04:20.635 INFO:tasks.workunit.client.0.smithi116.stdout:7/778: dread d3/d4/d8/f1c [0,4194304] 0 2024-03-04T16:04:20.639 INFO:tasks.workunit.client.0.smithi116.stdout:6/726: creat d6/de/d25/d36/d44/dbb/ff7 x:0 0 0 2024-03-04T16:04:20.641 INFO:tasks.workunit.client.0.smithi116.stdout:4/815: dwrite d1/d3/d15/d20/d1c/d62/d81/fe6 [0,4194304] 0 2024-03-04T16:04:20.645 INFO:tasks.workunit.client.0.smithi116.stdout:8/830: getdents d2/de/d16/d78/d29/de5 0 2024-03-04T16:04:20.646 INFO:tasks.workunit.client.0.smithi116.stdout:8/831: readlink d2/d13/d5e/l8d 0 2024-03-04T16:04:20.647 INFO:tasks.workunit.client.0.smithi116.stdout:8/832: chown d2/de/d16/d78/d29/de5/df2/d37/df8 0 1 2024-03-04T16:04:20.649 INFO:tasks.workunit.client.0.smithi116.stdout:2/834: link d7/d81/d115/f120 d7/d13/d3e/ded/f133 0 2024-03-04T16:04:20.651 INFO:tasks.workunit.client.0.smithi116.stdout:5/796: dread dd/d1f/d72/d8e/f98 [0,4194304] 0 2024-03-04T16:04:20.653 INFO:tasks.workunit.client.0.smithi116.stdout:7/779: rename d3/d4/d8/d4a/d86 to d3/d4/d9e/dd7/dfd 0 2024-03-04T16:04:20.657 INFO:tasks.workunit.client.0.smithi116.stdout:0/673: write da/d16/d20/f65 [1562661,100741] 0 2024-03-04T16:04:20.657 INFO:tasks.workunit.client.0.smithi116.stdout:4/816: symlink d1/d3/dd/d45/d50/l11f 0 2024-03-04T16:04:20.660 INFO:tasks.workunit.client.0.smithi116.stdout:4/817: write d1/d3/d15/d20/d1c/d62/d81/d97/fb9 [1346182,74725] 0 2024-03-04T16:04:20.660 INFO:tasks.workunit.client.0.smithi116.stdout:8/833: mkdir d2/de/d16/d78/d29/de5/d118 0 2024-03-04T16:04:20.662 INFO:tasks.workunit.client.0.smithi116.stdout:6/727: dread d6/de/d25/d4a/f9d [0,4194304] 0 2024-03-04T16:04:20.665 INFO:tasks.workunit.client.0.smithi116.stdout:9/737: truncate d2/d33/dc5/fce 4008096 0 2024-03-04T16:04:20.666 INFO:tasks.workunit.client.0.smithi116.stdout:0/674: mkdir da/d16/d28/d40/d48/d57/dd6 0 2024-03-04T16:04:20.667 INFO:tasks.workunit.client.0.smithi116.stdout:9/738: read d2/d33/f77 [326154,110258] 0 2024-03-04T16:04:20.667 INFO:tasks.workunit.client.0.smithi116.stdout:8/834: dwrite d2/de3/f108 [0,4194304] 0 2024-03-04T16:04:20.668 INFO:tasks.workunit.client.0.smithi116.stdout:9/739: fdatasync d2/d19/d5f/fe4 0 2024-03-04T16:04:20.670 INFO:tasks.workunit.client.0.smithi116.stdout:0/675: truncate da/d16/d19/d27/d70/f8e 4548158 0 2024-03-04T16:04:20.671 INFO:tasks.workunit.client.0.smithi116.stdout:2/835: creat d7/d13/d3e/d98/dce/d47/d64/d75/d77/d12e/f134 x:0 0 0 2024-03-04T16:04:20.672 INFO:tasks.workunit.client.0.smithi116.stdout:4/818: chown d1/d3/de/c100 26920728 1 2024-03-04T16:04:20.675 INFO:tasks.workunit.client.0.smithi116.stdout:5/797: getdents dd/d1f/d55/d73/d2d/d8d/df6 0 2024-03-04T16:04:20.678 INFO:tasks.workunit.client.0.smithi116.stdout:7/780: mknod d3/cfe 0 2024-03-04T16:04:20.680 INFO:tasks.workunit.client.0.smithi116.stdout:1/672: getdents d5/d9/d21/dc7 0 2024-03-04T16:04:20.681 INFO:tasks.workunit.client.0.smithi116.stdout:1/673: write d5/d9/d21/d27/d94/db6/f54 [2605604,87519] 0 2024-03-04T16:04:20.684 INFO:tasks.workunit.client.0.smithi116.stdout:2/836: rmdir d7/d13/d3e/ded/d6c/da4 39 2024-03-04T16:04:20.687 INFO:tasks.workunit.client.0.smithi116.stdout:2/837: dread - d7/d25/fa2 zero size 2024-03-04T16:04:20.687 INFO:tasks.workunit.client.0.smithi116.stdout:7/781: rmdir d3/d4/dbf/de6/dd3/ddf/de0 39 2024-03-04T16:04:20.687 INFO:tasks.workunit.client.0.smithi116.stdout:4/819: read d1/d3/dd/d45/f4b [1371044,41969] 0 2024-03-04T16:04:20.691 INFO:tasks.workunit.client.0.smithi116.stdout:1/674: dwrite d5/d9/d21/d27/d32/d35/f52 [4194304,4194304] 0 2024-03-04T16:04:20.691 INFO:tasks.workunit.client.0.smithi116.stdout:3/840: sync 2024-03-04T16:04:20.696 INFO:tasks.workunit.client.0.smithi116.stdout:6/728: rename d6/d9e to d6/de/d25/d4a/df8 0 2024-03-04T16:04:20.700 INFO:tasks.workunit.client.0.smithi116.stdout:6/729: read - d6/de/d25/d36/d44/dbb/ddd/fe9 zero size 2024-03-04T16:04:20.700 INFO:tasks.workunit.client.0.smithi116.stdout:1/675: dwrite d5/da1/fc3 [0,4194304] 0 2024-03-04T16:04:20.705 INFO:tasks.workunit.client.0.smithi116.stdout:9/740: creat d2/d33/fec x:0 0 0 2024-03-04T16:04:20.707 INFO:tasks.workunit.client.0.smithi116.stdout:0/676: creat da/d16/d28/d40/d48/fd7 x:0 0 0 2024-03-04T16:04:20.714 INFO:tasks.workunit.client.0.smithi116.stdout:7/782: fsync d3/d4/d11/f51 0 2024-03-04T16:04:20.715 INFO:tasks.workunit.client.0.smithi116.stdout:1/676: fsync d5/d9/d21/d24/f44 0 2024-03-04T16:04:20.716 INFO:tasks.workunit.client.0.smithi116.stdout:7/783: write d3/d4/d8/d4a/d88/f93 [1466043,127696] 0 2024-03-04T16:04:20.717 INFO:tasks.workunit.client.0.smithi116.stdout:1/677: fsync d5/d9/d21/d27/d32/fd5 0 2024-03-04T16:04:20.718 INFO:tasks.workunit.client.0.smithi116.stdout:8/835: getdents d2/de/d16/d78 0 2024-03-04T16:04:20.718 INFO:tasks.workunit.client.0.smithi116.stdout:1/678: read - d5/d9/d21/d27/d8c/fb3 zero size 2024-03-04T16:04:20.719 INFO:tasks.workunit.client.0.smithi116.stdout:5/798: truncate dd/f15 1698629 0 2024-03-04T16:04:20.720 INFO:tasks.workunit.client.0.smithi116.stdout:2/838: mknod d7/d13/d3e/ded/d6c/da4/c135 0 2024-03-04T16:04:20.720 INFO:tasks.workunit.client.0.smithi116.stdout:8/836: write d2/de/d16/d78/d29/d7a/fd7 [908686,112815] 0 2024-03-04T16:04:20.723 INFO:tasks.workunit.client.0.smithi116.stdout:1/679: read d5/d9/d21/d27/d32/d49/d4a/f78 [4589323,24430] 0 2024-03-04T16:04:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:20 smithi116 ceph-mon[29120]: pgmap v26: 65 pgs: 65 active+clean; 1.4 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 55 MiB/s rd, 177 MiB/s wr, 448 op/s 2024-03-04T16:04:20.727 INFO:tasks.workunit.client.0.smithi116.stdout:8/837: dread d2/d13/d5e/fdb [0,4194304] 0 2024-03-04T16:04:20.728 INFO:tasks.workunit.client.0.smithi116.stdout:8/838: write d2/d13/d3c/d77/fb3 [3459962,42326] 0 2024-03-04T16:04:20.730 INFO:tasks.workunit.client.0.smithi116.stdout:7/784: creat d3/d4/d8/d4a/d65/fff x:0 0 0 2024-03-04T16:04:20.730 INFO:tasks.workunit.client.0.smithi116.stdout:3/841: write dc/df/faf [866194,118876] 0 2024-03-04T16:04:20.731 INFO:tasks.workunit.client.0.smithi116.stdout:3/842: chown dc/d34/d44/cdf 27109041 1 2024-03-04T16:04:20.736 INFO:tasks.workunit.client.0.smithi116.stdout:7/785: dread d3/d4/d8/d4a/d88/fbc [0,4194304] 0 2024-03-04T16:04:20.738 INFO:tasks.workunit.client.0.smithi116.stdout:4/820: creat d1/d3/d15/d20/d1c/f120 x:0 0 0 2024-03-04T16:04:20.738 INFO:tasks.workunit.client.0.smithi116.stdout:5/799: dwrite dd/d1f/dc3/feb [0,4194304] 0 2024-03-04T16:04:20.739 INFO:tasks.workunit.client.0.smithi116.stdout:6/730: write d6/de/d1e/d80/f89 [2098515,30059] 0 2024-03-04T16:04:20.746 INFO:tasks.workunit.client.0.smithi116.stdout:7/786: dwrite d3/d4/d8/f3e [0,4194304] 0 2024-03-04T16:04:20.750 INFO:tasks.workunit.client.0.smithi116.stdout:9/741: rename d2/d33/dc5/fd6 to d2/d19/fed 0 2024-03-04T16:04:20.752 INFO:tasks.workunit.client.0.smithi116.stdout:9/742: dread - d2/d33/d65/dbf/fd7 zero size 2024-03-04T16:04:20.754 INFO:tasks.workunit.client.0.smithi116.stdout:1/680: getdents d5/d9/d21/d84/db0 0 2024-03-04T16:04:20.758 INFO:tasks.workunit.client.0.smithi116.stdout:9/743: dwrite d2/f1a [0,4194304] 0 2024-03-04T16:04:20.763 INFO:tasks.workunit.client.0.smithi116.stdout:6/731: write d6/de/f52 [1282750,26390] 0 2024-03-04T16:04:20.765 INFO:tasks.workunit.client.0.smithi116.stdout:6/732: fsync d6/de/d25/d36/fa6 0 2024-03-04T16:04:20.766 INFO:tasks.workunit.client.0.smithi116.stdout:6/733: write d6/de/d25/d36/d44/fba [1069761,28394] 0 2024-03-04T16:04:20.772 INFO:tasks.workunit.client.0.smithi116.stdout:0/677: rename da/d16/d28/d40/d48/d57/db3/f9a to da/d16/d20/fd8 0 2024-03-04T16:04:20.773 INFO:tasks.workunit.client.0.smithi116.stdout:0/678: readlink da/d16/d19/l36 0 2024-03-04T16:04:20.774 INFO:tasks.workunit.client.0.smithi116.stdout:5/800: mkdir dd/d1f/d55/d73/d2d/d101 0 2024-03-04T16:04:20.776 INFO:tasks.workunit.client.0.smithi116.stdout:8/839: creat d2/d13/d3c/d77/d97/f119 x:0 0 0 2024-03-04T16:04:20.778 INFO:tasks.workunit.client.0.smithi116.stdout:9/744: creat d2/d33/d65/d6c/db8/fee x:0 0 0 2024-03-04T16:04:20.780 INFO:tasks.workunit.client.0.smithi116.stdout:8/840: dread d2/de3/f108 [0,4194304] 0 2024-03-04T16:04:20.780 INFO:tasks.workunit.client.0.smithi116.stdout:8/841: chown d2/de/d3e/d7b/c84 3785672 1 2024-03-04T16:04:20.781 INFO:tasks.workunit.client.0.smithi116.stdout:8/842: chown d2/d4a/dc7/d116 46273452 1 2024-03-04T16:04:20.785 INFO:tasks.workunit.client.0.smithi116.stdout:4/821: truncate d1/d3/dd/dd9/d11a/f99 184933 0 2024-03-04T16:04:20.786 INFO:tasks.workunit.client.0.smithi116.stdout:2/839: getdents d7/d13/d3e/ded/dfd 0 2024-03-04T16:04:20.790 INFO:tasks.workunit.client.0.smithi116.stdout:1/681: rename d5/d9/d21/d27/d32/d49 to d5/d9/d21/d24/de0 0 2024-03-04T16:04:20.791 INFO:tasks.workunit.client.0.smithi116.stdout:1/682: readlink d5/d9/d21/d27/d32/d35/d55/dae/ldc 0 2024-03-04T16:04:20.791 INFO:tasks.workunit.client.0.smithi116.stdout:0/679: fsync da/f1e 0 2024-03-04T16:04:20.797 INFO:tasks.workunit.client.0.smithi116.stdout:3/843: getdents dc/d34/de7/deb/de2 0 2024-03-04T16:04:20.799 INFO:tasks.workunit.client.0.smithi116.stdout:7/787: dwrite d3/d4/d8/f71 [0,4194304] 0 2024-03-04T16:04:20.800 INFO:tasks.workunit.client.0.smithi116.stdout:5/801: dread dd/d1f/d55/f77 [0,4194304] 0 2024-03-04T16:04:20.801 INFO:tasks.workunit.client.0.smithi116.stdout:9/745: unlink d2/d26/dd4/d4a/cb2 0 2024-03-04T16:04:20.805 INFO:tasks.workunit.client.0.smithi116.stdout:9/746: fsync d2/d33/d65/d6c/db8/fee 0 2024-03-04T16:04:20.806 INFO:tasks.workunit.client.0.smithi116.stdout:3/844: dwrite dc/d23/d84/d7f/d93/db6/dc5/de0/f114 [0,4194304] 0 2024-03-04T16:04:20.810 INFO:tasks.workunit.client.0.smithi116.stdout:3/845: write dc/d23/d84/d7f/d93/db6/dee/ff6 [488368,64802] 0 2024-03-04T16:04:20.812 INFO:tasks.workunit.client.0.smithi116.stdout:4/822: creat d1/d3/dd/d8a/f121 x:0 0 0 2024-03-04T16:04:20.812 INFO:tasks.workunit.client.0.smithi116.stdout:3/846: chown dc/d34/de7/deb 59393457 1 2024-03-04T16:04:20.814 INFO:tasks.workunit.client.0.smithi116.stdout:2/840: creat d7/d13/d3e/d71/f136 x:0 0 0 2024-03-04T16:04:20.815 INFO:tasks.workunit.client.0.smithi116.stdout:2/841: chown d7/d13/dec/f101 18924 1 2024-03-04T16:04:20.817 INFO:tasks.workunit.client.0.smithi116.stdout:6/734: write d6/de/d25/d36/f99 [461177,37159] 0 2024-03-04T16:04:20.819 INFO:tasks.workunit.client.0.smithi116.stdout:7/788: mkdir d3/d4/d9e/dd7/dfd/d100 0 2024-03-04T16:04:20.821 INFO:tasks.workunit.client.0.smithi116.stdout:8/843: dwrite d2/de/d16/fdc [0,4194304] 0 2024-03-04T16:04:20.823 INFO:tasks.workunit.client.0.smithi116.stdout:5/802: rename dd/d1f/l5b to dd/d1f/d55/d73/d2d/d7c/l102 0 2024-03-04T16:04:20.834 INFO:tasks.workunit.client.0.smithi116.stdout:3/847: mkdir dc/d23/d84/d7f/d93/d119 0 2024-03-04T16:04:20.842 INFO:tasks.workunit.client.0.smithi116.stdout:7/789: rmdir d3/d4/dbf/de6/dd3/ddf 39 2024-03-04T16:04:20.845 INFO:tasks.workunit.client.0.smithi116.stdout:9/747: mknod d2/d26/dd4/dab/cef 0 2024-03-04T16:04:20.849 INFO:tasks.workunit.client.0.smithi116.stdout:2/842: symlink d7/d13/d3e/d98/dce/d47/d64/d75/l137 0 2024-03-04T16:04:20.853 INFO:tasks.workunit.client.0.smithi116.stdout:8/844: dread d2/de/d16/f2c [0,4194304] 0 2024-03-04T16:04:20.855 INFO:tasks.workunit.client.0.smithi116.stdout:4/823: creat d1/d3/d15/d20/d1c/d62/d81/f122 x:0 0 0 2024-03-04T16:04:20.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:20 smithi152 ceph-mon[36876]: pgmap v26: 65 pgs: 65 active+clean; 1.4 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 55 MiB/s rd, 177 MiB/s wr, 448 op/s 2024-03-04T16:04:20.859 INFO:tasks.workunit.client.0.smithi116.stdout:1/683: getdents d5/d9/d21/d27/d32/d35/d55 0 2024-03-04T16:04:20.860 INFO:tasks.workunit.client.0.smithi116.stdout:6/735: dread d6/de/d25/f40 [0,4194304] 0 2024-03-04T16:04:20.863 INFO:tasks.workunit.client.0.smithi116.stdout:5/803: creat dd/f103 x:0 0 0 2024-03-04T16:04:20.863 INFO:tasks.workunit.client.0.smithi116.stdout:3/848: truncate dc/df/d14/fc0 3187253 0 2024-03-04T16:04:20.866 INFO:tasks.workunit.client.0.smithi116.stdout:8/845: unlink d2/de/d16/d78/d29/d7a/fd2 0 2024-03-04T16:04:20.870 INFO:tasks.workunit.client.0.smithi116.stdout:1/684: readlink d5/d9/d21/lde 0 2024-03-04T16:04:20.870 INFO:tasks.workunit.client.0.smithi116.stdout:9/748: truncate d2/f4f 3796483 0 2024-03-04T16:04:20.873 INFO:tasks.workunit.client.0.smithi116.stdout:3/849: fdatasync dc/d23/d84/d7f/d93/db6/dc5/de0/fa0 0 2024-03-04T16:04:20.873 INFO:tasks.workunit.client.0.smithi116.stdout:8/846: dwrite d2/de/d16/d78/d29/de5/df2/da4/f10b [0,4194304] 0 2024-03-04T16:04:20.875 INFO:tasks.workunit.client.0.smithi116.stdout:7/790: getdents d3/d4/d8/d1a 0 2024-03-04T16:04:20.875 INFO:tasks.workunit.client.0.smithi116.stdout:7/791: stat d3/d4/d8/d52/ca6 0 2024-03-04T16:04:20.876 INFO:tasks.workunit.client.0.smithi116.stdout:5/804: mknod dd/d1f/d55/d73/d2d/d101/c104 0 2024-03-04T16:04:20.877 INFO:tasks.workunit.client.0.smithi116.stdout:5/805: chown dd/d6d/dc6 56397 1 2024-03-04T16:04:20.880 INFO:tasks.workunit.client.0.smithi116.stdout:3/850: chown dc/c25 10 1 2024-03-04T16:04:20.880 INFO:tasks.workunit.client.0.smithi116.stdout:8/847: mkdir d2/de/d16/d78/d29/de5/df2/d11a 0 2024-03-04T16:04:20.882 INFO:tasks.workunit.client.0.smithi116.stdout:7/792: mkdir d3/d4/dbf/d101 0 2024-03-04T16:04:20.883 INFO:tasks.workunit.client.0.smithi116.stdout:9/749: dwrite d2/d26/dd4/dab/d3c/f76 [0,4194304] 0 2024-03-04T16:04:20.884 INFO:tasks.workunit.client.0.smithi116.stdout:5/806: mknod dd/d1f/d5c/d5e/c105 0 2024-03-04T16:04:20.887 INFO:tasks.workunit.client.0.smithi116.stdout:7/793: creat d3/d4/d8/d4a/db3/f102 x:0 0 0 2024-03-04T16:04:20.889 INFO:tasks.workunit.client.0.smithi116.stdout:1/685: rename d5/f5a to d5/d9/d21/d27/d94/db6/d61/fe1 0 2024-03-04T16:04:20.890 INFO:tasks.workunit.client.0.smithi116.stdout:3/851: fdatasync dc/df/f7e 0 2024-03-04T16:04:20.891 INFO:tasks.workunit.client.0.smithi116.stdout:3/852: stat dc/d34/d44/fb4 0 2024-03-04T16:04:20.894 INFO:tasks.workunit.client.0.smithi116.stdout:3/853: dread dc/d23/d84/d7f/d93/db6/dc5/de0/f77 [4194304,4194304] 0 2024-03-04T16:04:20.895 INFO:tasks.workunit.client.0.smithi116.stdout:1/686: mknod d5/d9/d21/d27/d32/d35/d55/db4/d50/ce2 0 2024-03-04T16:04:20.897 INFO:tasks.workunit.client.0.smithi116.stdout:5/807: creat dd/f106 x:0 0 0 2024-03-04T16:04:20.898 INFO:tasks.workunit.client.0.smithi116.stdout:7/794: truncate d3/d4/d8/f40 1684895 0 2024-03-04T16:04:20.899 INFO:tasks.workunit.client.0.smithi116.stdout:3/854: creat dc/d23/d84/d7f/de8/f11a x:0 0 0 2024-03-04T16:04:20.899 INFO:tasks.workunit.client.0.smithi116.stdout:5/808: chown dd/d1f/d55/d73/d2d/dc0/le1 16 1 2024-03-04T16:04:20.899 INFO:tasks.workunit.client.0.smithi116.stdout:3/855: chown dc/d34/d44/fc3 0 1 2024-03-04T16:04:20.902 INFO:tasks.workunit.client.0.smithi116.stdout:5/809: dread dd/d1f/d55/d73/fde [0,4194304] 0 2024-03-04T16:04:20.903 INFO:tasks.workunit.client.0.smithi116.stdout:7/795: dread d3/d4/f16 [4194304,4194304] 0 2024-03-04T16:04:20.905 INFO:tasks.workunit.client.0.smithi116.stdout:5/810: rename dd/f106 to dd/d1f/d5c/d5e/dc2/f107 0 2024-03-04T16:04:20.907 INFO:tasks.workunit.client.0.smithi116.stdout:7/796: mknod d3/d4/d34/c103 0 2024-03-04T16:04:20.909 INFO:tasks.workunit.client.0.smithi116.stdout:5/811: mkdir dd/d1f/d55/d73/d2d/d108 0 2024-03-04T16:04:20.909 INFO:tasks.workunit.client.1.smithi152.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-04T16:04:20.911 INFO:tasks.workunit.client.0.smithi116.stdout:7/797: rename d3/d95/f6f to d3/d4/d9e/dd7/daf/dfc/f104 0 2024-03-04T16:04:20.913 INFO:tasks.workunit.client.0.smithi116.stdout:5/812: dread dd/d1f/dc3/feb [0,4194304] 0 2024-03-04T16:04:20.914 INFO:tasks.workunit.client.0.smithi116.stdout:5/813: read - dd/d1f/d5c/f9e zero size 2024-03-04T16:04:20.915 INFO:tasks.workunit.client.1.smithi152.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-04T16:04:20.915 INFO:tasks.workunit.client.1.smithi152.stderr:+ make 2024-03-04T16:04:20.916 INFO:tasks.workunit.client.0.smithi116.stdout:5/814: mknod dd/d1f/d55/d73/d2d/d59/d71/dff/c109 0 2024-03-04T16:04:20.922 INFO:tasks.workunit.client.0.smithi116.stdout:5/815: dwrite dd/d1f/d51/fd1 [0,4194304] 0 2024-03-04T16:04:20.923 INFO:tasks.workunit.client.0.smithi116.stdout:5/816: write dd/d1f/d51/fbf [4729847,46790] 0 2024-03-04T16:04:20.927 INFO:tasks.workunit.client.0.smithi116.stdout:5/817: rmdir dd/d45 39 2024-03-04T16:04:20.929 INFO:tasks.workunit.client.0.smithi116.stdout:5/818: read dd/d45/f95 [1172457,14134] 0 2024-03-04T16:04:20.929 INFO:tasks.workunit.client.0.smithi116.stdout:5/819: chown dd/d45/c53 219754 1 2024-03-04T16:04:20.954 INFO:tasks.workunit.client.0.smithi116.stdout:4/824: dwrite d1/d3/f61 [0,4194304] 0 2024-03-04T16:04:20.957 INFO:tasks.workunit.client.0.smithi116.stdout:8/848: truncate d2/d13/d5e/fc6 3324214 0 2024-03-04T16:04:20.960 INFO:tasks.workunit.client.0.smithi116.stdout:4/825: creat d1/d3/dd/d24/db4/f123 x:0 0 0 2024-03-04T16:04:20.960 INFO:tasks.workunit.client.0.smithi116.stdout:4/826: stat d1/d3/d40/d52/d84/dde 0 2024-03-04T16:04:20.962 INFO:tasks.workunit.client.0.smithi116.stdout:4/827: write d1/d3/dd/d24/db4/d10c/fec [230536,71403] 0 2024-03-04T16:04:20.962 INFO:tasks.workunit.client.0.smithi116.stdout:9/750: truncate d2/d26/dd4/d4a/d6a/d94/fcb 3202471 0 2024-03-04T16:04:20.967 INFO:tasks.workunit.client.0.smithi116.stdout:8/849: creat d2/f11b x:0 0 0 2024-03-04T16:04:20.969 INFO:tasks.workunit.client.0.smithi116.stdout:4/828: mkdir d1/d3/de/d124 0 2024-03-04T16:04:20.970 INFO:tasks.workunit.client.0.smithi116.stdout:3/856: write dc/d23/d84/d7f/d93/fb1 [132736,51466] 0 2024-03-04T16:04:20.974 INFO:tasks.workunit.client.0.smithi116.stdout:4/829: getdents d1/d3/dd/d24 0 2024-03-04T16:04:20.977 INFO:tasks.workunit.client.0.smithi116.stdout:4/830: mkdir d1/d3/dd/d45/d119/d125 0 2024-03-04T16:04:20.978 INFO:tasks.workunit.client.0.smithi116.stdout:3/857: read dc/d23/d84/d7f/d93/fb1 [655740,64549] 0 2024-03-04T16:04:20.979 INFO:tasks.workunit.client.0.smithi116.stdout:7/798: write d3/d4/d8/d4a/d88/fbc [957437,10563] 0 2024-03-04T16:04:20.982 INFO:tasks.workunit.client.0.smithi116.stdout:4/831: rmdir d1/d3/d15/d20/d1c/d62/d81/daf 39 2024-03-04T16:04:20.983 INFO:tasks.workunit.client.0.smithi116.stdout:4/832: write d1/d3/dd/d24/d51/f11b [456691,118647] 0 2024-03-04T16:04:20.983 INFO:tasks.workunit.client.0.smithi116.stdout:3/858: mknod dc/df/dbc/c11b 0 2024-03-04T16:04:20.986 INFO:tasks.workunit.client.0.smithi116.stdout:4/833: read d1/d3/dd/d8a/ddc/fe1 [12412,15168] 0 2024-03-04T16:04:20.987 INFO:tasks.workunit.client.0.smithi116.stdout:3/859: fsync dc/df/d47/d91/fd7 0 2024-03-04T16:04:20.993 INFO:tasks.workunit.client.0.smithi116.stdout:0/680: sync 2024-03-04T16:04:20.993 INFO:tasks.workunit.client.0.smithi116.stdout:9/751: dread d2/d33/d65/d6c/db8/fd1 [0,4194304] 0 2024-03-04T16:04:20.994 INFO:tasks.workunit.client.0.smithi116.stdout:1/687: write d5/d9/d21/d27/d94/db6/d61/fe1 [1101629,129993] 0 2024-03-04T16:04:20.996 INFO:tasks.workunit.client.0.smithi116.stdout:4/834: mknod d1/d3/d15/d20/d1c/c126 0 2024-03-04T16:04:20.997 INFO:tasks.workunit.client.1.smithi152.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-04T16:04:21.000 INFO:tasks.workunit.client.0.smithi116.stdout:9/752: mkdir d2/d33/d65/d6c/db8/de3/df0 0 2024-03-04T16:04:21.001 INFO:tasks.workunit.client.0.smithi116.stdout:1/688: creat d5/d9/d21/d27/d94/db6/d42/fe3 x:0 0 0 2024-03-04T16:04:21.003 INFO:tasks.workunit.client.0.smithi116.stdout:3/860: dread dc/df/d14/f5d [0,4194304] 0 2024-03-04T16:04:21.004 INFO:tasks.workunit.client.0.smithi116.stdout:3/861: chown dc/d23/d84/d7f/d8b/fc7 99144 1 2024-03-04T16:04:21.005 INFO:tasks.workunit.client.0.smithi116.stdout:3/862: chown dc/d23/d84/fd1 727935987 1 2024-03-04T16:04:21.008 INFO:tasks.workunit.client.0.smithi116.stdout:9/753: fsync d2/d19/d27/dbd/fdb 0 2024-03-04T16:04:21.012 INFO:tasks.workunit.client.0.smithi116.stdout:7/799: dwrite d3/d95/ff6 [0,4194304] 0 2024-03-04T16:04:21.015 INFO:tasks.workunit.client.0.smithi116.stdout:0/681: creat da/d16/d28/d40/d48/fd9 x:0 0 0 2024-03-04T16:04:21.016 INFO:tasks.workunit.client.0.smithi116.stdout:9/754: fsync d2/d19/f2a 0 2024-03-04T16:04:21.016 INFO:tasks.workunit.client.0.smithi116.stdout:1/689: dwrite d5/d9/d21/d27/d32/d35/f53 [0,4194304] 0 2024-03-04T16:04:21.017 INFO:tasks.workunit.client.0.smithi116.stdout:1/690: chown d5/d9/d21/d27/d32/d35/dbf 2057 1 2024-03-04T16:04:21.019 INFO:tasks.workunit.client.0.smithi116.stdout:1/691: write d5/d86/fab [559206,45039] 0 2024-03-04T16:04:21.022 INFO:tasks.workunit.client.0.smithi116.stdout:3/863: mkdir dc/df/dbc/d101/d11c 0 2024-03-04T16:04:21.030 INFO:tasks.workunit.client.0.smithi116.stdout:4/835: dwrite d1/d3/d15/d20/d1c/f27 [0,4194304] 0 2024-03-04T16:04:21.049 INFO:tasks.workunit.client.0.smithi116.stdout:1/692: symlink d5/d9/d21/d27/d32/d35/le4 0 2024-03-04T16:04:21.051 INFO:tasks.workunit.client.0.smithi116.stdout:3/864: creat dc/d34/de7/d110/f11d x:0 0 0 2024-03-04T16:04:21.052 INFO:tasks.workunit.client.0.smithi116.stdout:0/682: mknod da/d16/d28/d40/d5d/cda 0 2024-03-04T16:04:21.054 INFO:tasks.workunit.client.0.smithi116.stdout:1/693: dwrite d5/d9/d21/d27/d32/d35/d55/d75/d91/f9e [0,4194304] 0 2024-03-04T16:04:21.055 INFO:tasks.workunit.client.0.smithi116.stdout:9/755: write d2/d33/d65/d6c/f66 [293398,117795] 0 2024-03-04T16:04:21.055 INFO:tasks.workunit.client.0.smithi116.stdout:1/694: chown d5/d9/d21/d27/d32/d35/d55/fc0 690251228 1 2024-03-04T16:04:21.056 INFO:tasks.workunit.client.0.smithi116.stdout:3/865: mkdir dc/d23/d84/d11e 0 2024-03-04T16:04:21.062 INFO:tasks.workunit.client.0.smithi116.stdout:1/695: read d5/d9/d21/d27/d32/fa3 [1676282,64413] 0 2024-03-04T16:04:21.063 INFO:tasks.workunit.client.0.smithi116.stdout:7/800: truncate d3/d4/d34/f74 528106 0 2024-03-04T16:04:21.063 INFO:tasks.workunit.client.0.smithi116.stdout:1/696: readlink d5/d9/d21/d27/l1f 0 2024-03-04T16:04:21.064 INFO:tasks.workunit.client.0.smithi116.stdout:7/801: dread - d3/d4/d8/d4a/d65/fff zero size 2024-03-04T16:04:21.064 INFO:tasks.workunit.client.0.smithi116.stdout:7/802: chown d3/de9/ff9 31471 1 2024-03-04T16:04:21.068 INFO:tasks.workunit.client.0.smithi116.stdout:3/866: rename dc/d23/d84/d7f/d93/db6/dc5/de0/da3/cad to dc/d23/d84/d7f/d93/c11f 0 2024-03-04T16:04:21.069 INFO:tasks.workunit.client.0.smithi116.stdout:1/697: mkdir d5/d86/dac/de5 0 2024-03-04T16:04:21.072 INFO:tasks.workunit.client.0.smithi116.stdout:4/836: dwrite d1/d3/d15/d20/d1c/d62/d81/fb0 [0,4194304] 0 2024-03-04T16:04:21.080 INFO:tasks.workunit.client.0.smithi116.stdout:1/698: dread d5/d9/d21/d27/d32/d35/d55/db4/f70 [0,4194304] 0 2024-03-04T16:04:21.082 INFO:tasks.workunit.client.0.smithi116.stdout:9/756: link d2/d26/d28/c5a d2/d26/dd4/dab/d3c/cf1 0 2024-03-04T16:04:21.082 INFO:tasks.workunit.client.0.smithi116.stdout:4/837: mknod d1/d3/c127 0 2024-03-04T16:04:21.083 INFO:tasks.workunit.client.0.smithi116.stdout:1/699: dread d5/d9/d21/d24/f44 [0,4194304] 0 2024-03-04T16:04:21.084 INFO:tasks.workunit.client.0.smithi116.stdout:7/803: mkdir d3/d4/d8/d52/d5e/d9a/d105 0 2024-03-04T16:04:21.088 INFO:tasks.workunit.client.0.smithi116.stdout:0/683: read - da/d16/d28/d40/d5d/d7e/fc5 zero size 2024-03-04T16:04:21.091 INFO:tasks.workunit.client.0.smithi116.stdout:1/700: rmdir d5 39 2024-03-04T16:04:21.093 INFO:tasks.workunit.client.0.smithi116.stdout:4/838: creat d1/d3/dd/d8a/f128 x:0 0 0 2024-03-04T16:04:21.095 INFO:tasks.workunit.client.0.smithi116.stdout:4/839: write d1/d6e/d74/f117 [93622,7062] 0 2024-03-04T16:04:21.096 INFO:tasks.workunit.client.0.smithi116.stdout:4/840: write d1/d3/d15/fea [3561049,14909] 0 2024-03-04T16:04:21.096 INFO:tasks.workunit.client.0.smithi116.stdout:7/804: mknod d3/d4/d9e/dd7/dd1/c106 0 2024-03-04T16:04:21.097 INFO:tasks.workunit.client.0.smithi116.stdout:0/684: stat da/d16/d20/dcf/cd2 0 2024-03-04T16:04:21.098 INFO:tasks.workunit.client.0.smithi116.stdout:3/867: dread dc/d23/f22 [0,4194304] 0 2024-03-04T16:04:21.099 INFO:tasks.workunit.client.0.smithi116.stdout:3/868: chown dc/d23/l32 364209363 1 2024-03-04T16:04:21.102 INFO:tasks.workunit.client.0.smithi116.stdout:3/869: dread dc/d23/d84/d7f/d8b/fc9 [0,4194304] 0 2024-03-04T16:04:21.105 INFO:tasks.workunit.client.0.smithi116.stdout:9/757: dread d2/f4f [0,4194304] 0 2024-03-04T16:04:21.112 INFO:tasks.workunit.client.0.smithi116.stdout:3/870: mkdir dc/d34/d44/dab/d120 0 2024-03-04T16:04:21.115 INFO:tasks.workunit.client.0.smithi116.stdout:7/805: write d3/d4/d11/f51 [1777086,122740] 0 2024-03-04T16:04:21.117 INFO:tasks.workunit.client.0.smithi116.stdout:9/758: creat d2/d26/dd4/d4a/d6a/d94/dbe/ff2 x:0 0 0 2024-03-04T16:04:21.119 INFO:tasks.workunit.client.0.smithi116.stdout:9/759: write d2/f1a [1880871,45916] 0 2024-03-04T16:04:21.120 INFO:tasks.workunit.client.0.smithi116.stdout:3/871: creat dc/d34/d44/dab/f121 x:0 0 0 2024-03-04T16:04:21.122 INFO:tasks.workunit.client.0.smithi116.stdout:7/806: rmdir d3/d95 39 2024-03-04T16:04:21.123 INFO:tasks.workunit.client.0.smithi116.stdout:0/685: dwrite da/d16/d28/d3d/d41/f59 [0,4194304] 0 2024-03-04T16:04:21.124 INFO:tasks.workunit.client.0.smithi116.stdout:3/872: write dc/df/f92 [965160,36121] 0 2024-03-04T16:04:21.139 INFO:tasks.workunit.client.0.smithi116.stdout:1/701: dwrite d5/d9/d21/d24/de0/f6d [0,4194304] 0 2024-03-04T16:04:21.141 INFO:tasks.workunit.client.0.smithi116.stdout:9/760: creat d2/d26/dd4/d4a/d6a/d94/d95/db7/ff3 x:0 0 0 2024-03-04T16:04:21.141 INFO:tasks.workunit.client.0.smithi116.stdout:1/702: write d5/d9/f23 [1333515,20057] 0 2024-03-04T16:04:21.144 INFO:tasks.workunit.client.0.smithi116.stdout:9/761: stat d2/d19/d5f/ca2 0 2024-03-04T16:04:21.144 INFO:tasks.workunit.client.0.smithi116.stdout:9/762: dread - d2/d33/fec zero size 2024-03-04T16:04:21.146 INFO:tasks.workunit.client.0.smithi116.stdout:0/686: dread da/d16/d19/d27/f78 [0,4194304] 0 2024-03-04T16:04:21.146 INFO:tasks.workunit.client.0.smithi116.stdout:9/763: chown d2/d19/l85 813730 1 2024-03-04T16:04:21.148 INFO:tasks.workunit.client.0.smithi116.stdout:3/873: dread dc/df/f92 [0,4194304] 0 2024-03-04T16:04:21.149 INFO:tasks.workunit.client.0.smithi116.stdout:1/703: fsync d5/f38 0 2024-03-04T16:04:21.150 INFO:tasks.workunit.client.0.smithi116.stdout:3/874: write dc/df/d47/f61 [5217919,126140] 0 2024-03-04T16:04:21.151 INFO:tasks.workunit.client.0.smithi116.stdout:1/704: read - d5/d9/d21/d24/de0/d4c/d67/f73 zero size 2024-03-04T16:04:21.152 INFO:tasks.workunit.client.0.smithi116.stdout:7/807: creat d3/d4/f107 x:0 0 0 2024-03-04T16:04:21.160 INFO:tasks.workunit.client.0.smithi116.stdout:8/850: sync 2024-03-04T16:04:21.160 INFO:tasks.workunit.client.0.smithi116.stdout:2/843: sync 2024-03-04T16:04:21.160 INFO:tasks.workunit.client.0.smithi116.stdout:6/736: sync 2024-03-04T16:04:21.160 INFO:tasks.workunit.client.0.smithi116.stdout:5/820: sync 2024-03-04T16:04:21.161 INFO:tasks.workunit.client.0.smithi116.stdout:5/821: chown dd/d1f/l3d 541 1 2024-03-04T16:04:21.162 INFO:tasks.workunit.client.0.smithi116.stdout:8/851: stat d2/d13/d3c/d46/d6e/cda 0 2024-03-04T16:04:21.171 INFO:tasks.workunit.client.0.smithi116.stdout:6/737: creat d6/de/d25/d4a/d63/d95/ff9 x:0 0 0 2024-03-04T16:04:21.175 INFO:tasks.workunit.client.0.smithi116.stdout:2/844: rename d7/d13/d3e/f5c to d7/d13/d3e/d71/dad/f138 0 2024-03-04T16:04:21.179 INFO:tasks.workunit.client.0.smithi116.stdout:9/764: truncate d2/f48 1479312 0 2024-03-04T16:04:21.181 INFO:tasks.workunit.client.0.smithi116.stdout:0/687: getdents da/d16/d28/d40/d48/d57 0 2024-03-04T16:04:21.182 INFO:tasks.workunit.client.0.smithi116.stdout:6/738: rmdir d6/d56/d58 39 2024-03-04T16:04:21.184 INFO:tasks.workunit.client.0.smithi116.stdout:5/822: write dd/d6d/f88 [1371063,83769] 0 2024-03-04T16:04:21.184 INFO:tasks.workunit.client.0.smithi116.stdout:1/705: dwrite d5/d9/d21/d27/d32/d35/f76 [4194304,4194304] 0 2024-03-04T16:04:21.188 INFO:tasks.workunit.client.0.smithi116.stdout:3/875: getdents dc/d23/d84/d7f/d93/db6/dc5/de0/da3 0 2024-03-04T16:04:21.188 INFO:tasks.workunit.client.0.smithi116.stdout:8/852: dwrite d2/d13/d3c/d77/fd9 [0,4194304] 0 2024-03-04T16:04:21.190 INFO:tasks.workunit.client.0.smithi116.stdout:0/688: mknod da/d16/d20/d53/cdb 0 2024-03-04T16:04:21.193 INFO:tasks.workunit.client.0.smithi116.stdout:2/845: creat d7/d15/d5e/d9d/df1/d125/f139 x:0 0 0 2024-03-04T16:04:21.193 INFO:tasks.workunit.client.0.smithi116.stdout:0/689: read da/d16/fb1 [2987204,63598] 0 2024-03-04T16:04:21.194 INFO:tasks.workunit.client.0.smithi116.stdout:3/876: chown dc/df/d47/d91/f98 302705 1 2024-03-04T16:04:21.194 INFO:tasks.workunit.client.0.smithi116.stdout:6/739: unlink d6/de/d25/d4a/l82 0 2024-03-04T16:04:21.197 INFO:tasks.workunit.client.0.smithi116.stdout:8/853: stat d2/de/fa 0 2024-03-04T16:04:21.199 INFO:tasks.workunit.client.0.smithi116.stdout:2/846: chown d7/d13/d3e/ded/d6c/l8a 996598 1 2024-03-04T16:04:21.201 INFO:tasks.workunit.client.0.smithi116.stdout:7/808: dwrite d3/d4/d8/d1a/f20 [0,4194304] 0 2024-03-04T16:04:21.202 INFO:tasks.workunit.client.0.smithi116.stdout:0/690: fsync da/d16/d28/d40/d48/fd9 0 2024-03-04T16:04:21.203 INFO:tasks.workunit.client.0.smithi116.stdout:5/823: truncate dd/d1f/d55/d73/d2d/d59/fac 889551 0 2024-03-04T16:04:21.206 INFO:tasks.workunit.client.0.smithi116.stdout:6/740: truncate d6/de/d25/d36/d4e/fc1 972760 0 2024-03-04T16:04:21.208 INFO:tasks.workunit.client.0.smithi116.stdout:3/877: dread dc/df/faf [0,4194304] 0 2024-03-04T16:04:21.212 INFO:tasks.workunit.client.0.smithi116.stdout:3/878: read - dc/df/db8/f106 zero size 2024-03-04T16:04:21.212 INFO:tasks.workunit.client.0.smithi116.stdout:1/706: link d5/d9/d21/d27/d32/d35/d55/d75/d91/fb2 d5/d9/d21/d27/d94/db6/d42/d8b/fe6 0 2024-03-04T16:04:21.214 INFO:tasks.workunit.client.0.smithi116.stdout:2/847: dwrite d7/d13/d3e/d98/f129 [0,4194304] 0 2024-03-04T16:04:21.217 INFO:tasks.workunit.client.0.smithi116.stdout:0/691: dwrite da/d16/f4c [4194304,4194304] 0 2024-03-04T16:04:21.217 INFO:tasks.workunit.client.0.smithi116.stdout:7/809: rename d3/de9 to d3/d4/d34/d5d/ded/d108 0 2024-03-04T16:04:21.221 INFO:tasks.workunit.client.0.smithi116.stdout:1/707: mknod d5/d9/d21/dc7/ce7 0 2024-03-04T16:04:21.222 INFO:tasks.workunit.client.0.smithi116.stdout:2/848: creat d7/d15/d5e/d9d/df1/d113/f13a x:0 0 0 2024-03-04T16:04:21.225 INFO:tasks.workunit.client.0.smithi116.stdout:0/692: fdatasync da/d16/d28/d40/d48/d57/db3/db2/fb6 0 2024-03-04T16:04:21.227 INFO:tasks.workunit.client.0.smithi116.stdout:3/879: symlink dc/d34/l122 0 2024-03-04T16:04:21.228 INFO:tasks.workunit.client.0.smithi116.stdout:5/824: dread dd/d1f/d55/d73/d2d/f42 [0,4194304] 0 2024-03-04T16:04:21.229 INFO:tasks.workunit.client.0.smithi116.stdout:9/765: truncate d2/d26/fa0 414721 0 2024-03-04T16:04:21.229 INFO:tasks.workunit.client.0.smithi116.stdout:3/880: stat dc/d23/d84/d7f/d93/db6/cd9 0 2024-03-04T16:04:21.233 INFO:tasks.workunit.client.0.smithi116.stdout:5/825: rmdir dd/d1f/d55/d73/d2d/d59 39 2024-03-04T16:04:21.235 INFO:tasks.workunit.client.0.smithi116.stdout:0/693: unlink da/d16/d28/la0 0 2024-03-04T16:04:21.237 INFO:tasks.workunit.client.0.smithi116.stdout:9/766: creat d2/d26/dd4/d4a/d6a/d94/d95/ff4 x:0 0 0 2024-03-04T16:04:21.237 INFO:tasks.workunit.client.0.smithi116.stdout:3/881: truncate dc/d34/d44/fb4 339116 0 2024-03-04T16:04:21.241 INFO:tasks.workunit.client.0.smithi116.stdout:2/849: dread d7/d13/d3e/d78/f84 [0,4194304] 0 2024-03-04T16:04:21.241 INFO:tasks.workunit.client.0.smithi116.stdout:3/882: chown dc/df/d14/c29 53 1 2024-03-04T16:04:21.242 INFO:tasks.workunit.client.0.smithi116.stdout:5/826: creat dd/d1f/f10a x:0 0 0 2024-03-04T16:04:21.246 INFO:tasks.workunit.client.0.smithi116.stdout:0/694: dwrite da/fa4 [0,4194304] 0 2024-03-04T16:04:21.246 INFO:tasks.workunit.client.0.smithi116.stdout:3/883: creat dc/d23/d84/d7f/d93/f123 x:0 0 0 2024-03-04T16:04:21.250 INFO:tasks.workunit.client.0.smithi116.stdout:0/695: chown da/d16/d28/d40/d48/d57/dd6 1493940 1 2024-03-04T16:04:21.251 INFO:tasks.workunit.client.0.smithi116.stdout:3/884: chown dc/d23/d84/d7f/d93/db6/dee 9 1 2024-03-04T16:04:21.251 INFO:tasks.workunit.client.0.smithi116.stdout:4/841: sync 2024-03-04T16:04:21.252 INFO:tasks.workunit.client.0.smithi116.stdout:6/741: dread d6/de/d29/fe5 [0,4194304] 0 2024-03-04T16:04:21.254 INFO:tasks.workunit.client.0.smithi116.stdout:3/885: fdatasync dc/df/f79 0 2024-03-04T16:04:21.254 INFO:tasks.workunit.client.0.smithi116.stdout:0/696: truncate da/ff 4427038 0 2024-03-04T16:04:21.256 INFO:tasks.workunit.client.0.smithi116.stdout:2/850: dwrite d7/d15/d5e/d9d/df1/d125/f139 [0,4194304] 0 2024-03-04T16:04:21.258 INFO:tasks.workunit.client.0.smithi116.stdout:7/810: dread d3/d4/d8/d52/f53 [0,4194304] 0 2024-03-04T16:04:21.260 INFO:tasks.workunit.client.0.smithi116.stdout:7/811: fsync d3/d4/d8/f3e 0 2024-03-04T16:04:21.261 INFO:tasks.workunit.client.0.smithi116.stdout:6/742: mkdir d6/de/d25/d36/d4e/dfa 0 2024-03-04T16:04:21.261 INFO:tasks.workunit.client.0.smithi116.stdout:0/697: symlink da/d16/d19/d27/d82/ldc 0 2024-03-04T16:04:21.262 INFO:tasks.workunit.client.0.smithi116.stdout:2/851: rmdir d7/d15/d5e/d9d/dd4 39 2024-03-04T16:04:21.263 INFO:tasks.workunit.client.0.smithi116.stdout:7/812: dread - d3/fbb zero size 2024-03-04T16:04:21.265 INFO:tasks.workunit.client.0.smithi116.stdout:4/842: getdents d1/d3/d15/d20/d1c/d62/d81/d97 0 2024-03-04T16:04:21.265 INFO:tasks.workunit.client.0.smithi116.stdout:0/698: chown da/d16/d28/d40/d5d/d7e/fc5 1206 1 2024-03-04T16:04:21.266 INFO:tasks.workunit.client.0.smithi116.stdout:2/852: read f6 [2304016,3747] 0 2024-03-04T16:04:21.267 INFO:tasks.workunit.client.0.smithi116.stdout:6/743: symlink d6/de/d25/d4a/lfb 0 2024-03-04T16:04:21.268 INFO:tasks.workunit.client.0.smithi116.stdout:9/767: dread d2/f48 [0,4194304] 0 2024-03-04T16:04:21.270 INFO:tasks.workunit.client.0.smithi116.stdout:6/744: mkdir d6/de/d15/dfc 0 2024-03-04T16:04:21.271 INFO:tasks.workunit.client.0.smithi116.stdout:9/768: creat d2/d33/dc5/ff5 x:0 0 0 2024-03-04T16:04:21.271 INFO:tasks.workunit.client.0.smithi116.stdout:6/745: stat d6/de/d15/f26 0 2024-03-04T16:04:21.274 INFO:tasks.workunit.client.0.smithi116.stdout:2/853: dwrite d7/d13/d3e/ded/f50 [4194304,4194304] 0 2024-03-04T16:04:21.274 INFO:tasks.workunit.client.0.smithi116.stdout:4/843: dread d1/d3/dd/dd9/d11a/f99 [0,4194304] 0 2024-03-04T16:04:21.276 INFO:tasks.workunit.client.0.smithi116.stdout:9/769: creat d2/d19/d27/dbd/ff6 x:0 0 0 2024-03-04T16:04:21.280 INFO:tasks.workunit.client.0.smithi116.stdout:4/844: symlink d1/d3/dd/d24/l129 0 2024-03-04T16:04:21.280 INFO:tasks.workunit.client.0.smithi116.stdout:0/699: dread da/d16/d20/f3b [0,4194304] 0 2024-03-04T16:04:21.284 INFO:tasks.workunit.client.0.smithi116.stdout:0/700: stat da/d16/d28/d40/d48/da7/db7 0 2024-03-04T16:04:21.284 INFO:tasks.workunit.client.0.smithi116.stdout:3/886: dwrite dc/df/d47/d91/fd7 [0,4194304] 0 2024-03-04T16:04:21.286 INFO:tasks.workunit.client.0.smithi116.stdout:4/845: creat d1/d3/dd/d45/f12a x:0 0 0 2024-03-04T16:04:21.286 INFO:tasks.workunit.client.0.smithi116.stdout:0/701: fdatasync da/d16/d28/d3d/d41/f94 0 2024-03-04T16:04:21.289 INFO:tasks.workunit.client.0.smithi116.stdout:4/846: write d1/f75 [688805,1347] 0 2024-03-04T16:04:21.289 INFO:tasks.workunit.client.0.smithi116.stdout:3/887: write dc/d23/d84/d7f/d8b/f113 [486690,105207] 0 2024-03-04T16:04:21.290 INFO:tasks.workunit.client.0.smithi116.stdout:7/813: dread d3/d4/d34/d5d/fa7 [4194304,4194304] 0 2024-03-04T16:04:21.291 INFO:tasks.workunit.client.0.smithi116.stdout:0/702: dread - da/d16/d28/d40/d48/fd7 zero size 2024-03-04T16:04:21.292 INFO:tasks.workunit.client.0.smithi116.stdout:2/854: write d7/d15/f93 [4413766,91896] 0 2024-03-04T16:04:21.293 INFO:tasks.workunit.client.0.smithi116.stdout:9/770: getdents d2/d26/dd4/d4a/d6a/d94/dbe 0 2024-03-04T16:04:21.297 INFO:tasks.workunit.client.0.smithi116.stdout:6/746: dwrite d6/de/d25/d4a/df8/fc0 [0,4194304] 0 2024-03-04T16:04:21.299 INFO:tasks.workunit.client.0.smithi116.stdout:0/703: mknod da/d16/d28/d40/d48/d57/cdd 0 2024-03-04T16:04:21.299 INFO:tasks.workunit.client.0.smithi116.stdout:9/771: rmdir d2/d33/d65 39 2024-03-04T16:04:21.300 INFO:tasks.workunit.client.0.smithi116.stdout:2/855: mknod d7/d15/d5e/d9d/df1/d113/c13b 0 2024-03-04T16:04:21.301 INFO:tasks.workunit.client.0.smithi116.stdout:3/888: creat dc/df/d47/f124 x:0 0 0 2024-03-04T16:04:21.301 INFO:tasks.workunit.client.0.smithi116.stdout:7/814: mkdir d3/d4/d11/d109 0 2024-03-04T16:04:21.303 INFO:tasks.workunit.client.0.smithi116.stdout:0/704: write da/d16/f6e [98375,114132] 0 2024-03-04T16:04:21.307 INFO:tasks.workunit.client.0.smithi116.stdout:3/889: truncate dc/d23/d84/d7f/d93/db6/fbb 1701041 0 2024-03-04T16:04:21.308 INFO:tasks.workunit.client.0.smithi116.stdout:7/815: write d3/d4/d9e/dd7/daf/dd5/fef [844716,86591] 0 2024-03-04T16:04:21.308 INFO:tasks.workunit.client.0.smithi116.stdout:6/747: getdents d6/de/d25/d4a/df8 0 2024-03-04T16:04:21.309 INFO:tasks.workunit.client.0.smithi116.stdout:9/772: dwrite d2/d26/dd4/dab/fda [0,4194304] 0 2024-03-04T16:04:21.309 INFO:tasks.workunit.client.0.smithi116.stdout:4/847: write d1/d3/dd/dd9/d11a/f99 [434982,91856] 0 2024-03-04T16:04:21.310 INFO:tasks.workunit.client.0.smithi116.stdout:2/856: creat d7/d13/d3e/ded/d6c/f13c x:0 0 0 2024-03-04T16:04:21.312 INFO:tasks.workunit.client.0.smithi116.stdout:7/816: write d3/d4/d11/fb7 [258102,103505] 0 2024-03-04T16:04:21.313 INFO:tasks.workunit.client.0.smithi116.stdout:9/773: fsync d2/d26/dd4/dab/d3c/f8f 0 2024-03-04T16:04:21.314 INFO:tasks.workunit.client.0.smithi116.stdout:7/817: stat d3/d4/d8/d52 0 2024-03-04T16:04:21.314 INFO:tasks.workunit.client.0.smithi116.stdout:3/890: write dc/df/faf [1391721,38619] 0 2024-03-04T16:04:21.314 INFO:tasks.workunit.client.0.smithi116.stdout:2/857: fdatasync d7/d81/f7b 0 2024-03-04T16:04:21.316 INFO:tasks.workunit.client.0.smithi116.stdout:7/818: dread - d3/d4/d8/d1a/feb zero size 2024-03-04T16:04:21.318 INFO:tasks.workunit.client.0.smithi116.stdout:6/748: read d6/d56/f6c [3407303,59404] 0 2024-03-04T16:04:21.319 INFO:tasks.workunit.client.0.smithi116.stdout:9/774: read d2/d33/d65/d6c/db8/fd1 [4112520,26828] 0 2024-03-04T16:04:21.320 INFO:tasks.workunit.client.0.smithi116.stdout:6/749: write d6/f13 [6239713,8439] 0 2024-03-04T16:04:21.320 INFO:tasks.workunit.client.0.smithi116.stdout:6/750: stat d6/de/d25/d36/d44/c5a 0 2024-03-04T16:04:21.321 INFO:tasks.workunit.client.0.smithi116.stdout:2/858: mkdir d7/d13/dec/d13d 0 2024-03-04T16:04:21.324 INFO:tasks.workunit.client.0.smithi116.stdout:6/751: chown d6/de/d25/d4a/df8/da5/fc4 2 1 2024-03-04T16:04:21.328 INFO:tasks.workunit.client.0.smithi116.stdout:4/848: getdents d1/d3/d15/d20/d1c 0 2024-03-04T16:04:21.329 INFO:tasks.workunit.client.0.smithi116.stdout:6/752: symlink d6/de/d29/d3a/lfd 0 2024-03-04T16:04:21.329 INFO:tasks.workunit.client.0.smithi116.stdout:0/705: link da/d16/d20/d42/c58 da/d16/d28/d40/d5d/cde 0 2024-03-04T16:04:21.329 INFO:tasks.workunit.client.0.smithi116.stdout:4/849: write d1/d3/dd/d24/db4/f123 [72591,125968] 0 2024-03-04T16:04:21.331 INFO:tasks.workunit.client.0.smithi116.stdout:4/850: chown d1/d3/d15/d20/d1c/f120 27897 1 2024-03-04T16:04:21.331 INFO:tasks.workunit.client.0.smithi116.stdout:0/706: fdatasync da/d16/d28/d40/d48/d57/d8a/fc1 0 2024-03-04T16:04:21.332 INFO:tasks.workunit.client.0.smithi116.stdout:6/753: mkdir d6/de/d25/d4a/df8/dec/dfe 0 2024-03-04T16:04:21.334 INFO:tasks.workunit.client.0.smithi116.stdout:0/707: unlink da/d16/d20/d53/f9e 0 2024-03-04T16:04:21.334 INFO:tasks.workunit.client.0.smithi116.stdout:2/859: write d7/d15/d5e/d9d/ffc [2332843,60382] 0 2024-03-04T16:04:21.336 INFO:tasks.workunit.client.0.smithi116.stdout:7/819: truncate d3/d4/d9e/dd7/fab 670154 0 2024-03-04T16:04:21.341 INFO:tasks.workunit.client.0.smithi116.stdout:6/754: rename d6/de/d25/c39 to d6/de/d15/dfc/cff 0 2024-03-04T16:04:21.342 INFO:tasks.workunit.client.0.smithi116.stdout:6/755: write d6/de/d25/d36/f1a [210991,130261] 0 2024-03-04T16:04:21.342 INFO:tasks.workunit.client.0.smithi116.stdout:7/820: read d3/d4/d8/f13 [2262531,25076] 0 2024-03-04T16:04:21.345 INFO:tasks.workunit.client.0.smithi116.stdout:3/891: dread dc/dd3/f117 [0,4194304] 0 2024-03-04T16:04:21.350 INFO:tasks.workunit.client.0.smithi116.stdout:6/756: write d6/d56/f6c [4608359,126692] 0 2024-03-04T16:04:21.353 INFO:tasks.workunit.client.0.smithi116.stdout:0/708: truncate da/d16/d19/d27/f9d 2875666 0 2024-03-04T16:04:21.354 INFO:tasks.workunit.client.0.smithi116.stdout:3/892: dwrite dc/f9e [0,4194304] 0 2024-03-04T16:04:21.355 INFO:tasks.workunit.client.0.smithi116.stdout:6/757: mkdir d6/de/d15/dfc/d100 0 2024-03-04T16:04:21.356 INFO:tasks.workunit.client.0.smithi116.stdout:2/860: dwrite d7/d13/d3e/d98/dce/d47/f5b [0,4194304] 0 2024-03-04T16:04:21.356 INFO:tasks.workunit.client.0.smithi116.stdout:7/821: mkdir d3/d4/dbf/de6/dd3/ddf/dee/d10a 0 2024-03-04T16:04:21.361 INFO:tasks.workunit.client.0.smithi116.stdout:7/822: truncate d3/d4/d9e/ff8 171003 0 2024-03-04T16:04:21.364 INFO:tasks.workunit.client.0.smithi116.stdout:6/758: creat d6/de/d25/d4a/df8/da5/f101 x:0 0 0 2024-03-04T16:04:21.367 INFO:tasks.workunit.client.0.smithi116.stdout:6/759: dread d6/de/d25/d4a/f9d [0,4194304] 0 2024-03-04T16:04:21.369 INFO:tasks.workunit.client.0.smithi116.stdout:7/823: write d3/d4/d8/f1c [219757,109791] 0 2024-03-04T16:04:21.372 INFO:tasks.workunit.client.0.smithi116.stdout:6/760: truncate d6/f9 3101236 0 2024-03-04T16:04:21.373 INFO:tasks.workunit.client.0.smithi116.stdout:7/824: mknod d3/d4/d8/d4a/d87/c10b 0 2024-03-04T16:04:21.375 INFO:tasks.workunit.client.0.smithi116.stdout:6/761: creat d6/de/d25/d36/d44/dbb/ddd/d7b/f102 x:0 0 0 2024-03-04T16:04:21.376 INFO:tasks.workunit.client.0.smithi116.stdout:2/861: write d7/f4d [1304154,108473] 0 2024-03-04T16:04:21.377 INFO:tasks.workunit.client.0.smithi116.stdout:0/709: truncate da/d16/d28/d3d/d41/f59 1751254 0 2024-03-04T16:04:21.380 INFO:tasks.workunit.client.0.smithi116.stdout:3/893: dwrite dc/d34/f9a [0,4194304] 0 2024-03-04T16:04:21.382 INFO:tasks.workunit.client.0.smithi116.stdout:0/710: fsync da/d16/d28/d40/d48/d57/db3/db2/fb6 0 2024-03-04T16:04:21.383 INFO:tasks.workunit.client.0.smithi116.stdout:2/862: getdents d7/d13/d3e/ded/d6c/da4/da1 0 2024-03-04T16:04:21.384 INFO:tasks.workunit.client.0.smithi116.stdout:3/894: symlink dc/d23/d84/d7f/d93/df1/l125 0 2024-03-04T16:04:21.387 INFO:tasks.workunit.client.0.smithi116.stdout:0/711: creat da/d16/d20/d79/fdf x:0 0 0 2024-03-04T16:04:21.388 INFO:tasks.workunit.client.0.smithi116.stdout:3/895: symlink dc/dd3/l126 0 2024-03-04T16:04:21.389 INFO:tasks.workunit.client.0.smithi116.stdout:2/863: truncate d7/d15/d5e/d9d/d11f/f103 83279 0 2024-03-04T16:04:21.389 INFO:tasks.workunit.client.0.smithi116.stdout:6/762: dwrite d6/de/d25/d36/d44/dbb/ddd/fbf [0,4194304] 0 2024-03-04T16:04:21.393 INFO:tasks.workunit.client.0.smithi116.stdout:2/864: creat d7/d13/dec/f13e x:0 0 0 2024-03-04T16:04:21.396 INFO:tasks.workunit.client.0.smithi116.stdout:2/865: readlink d7/d13/d3e/d78/l95 0 2024-03-04T16:04:21.399 INFO:tasks.workunit.client.0.smithi116.stdout:0/712: dwrite da/d16/d20/dcf/fd3 [0,4194304] 0 2024-03-04T16:04:21.401 INFO:tasks.workunit.client.0.smithi116.stdout:2/866: dread - d7/d13/d3e/ded/d76/fda zero size 2024-03-04T16:04:21.404 INFO:tasks.workunit.client.0.smithi116.stdout:6/763: dwrite d6/de/d84/f8f [0,4194304] 0 2024-03-04T16:04:21.404 INFO:tasks.workunit.client.0.smithi116.stdout:3/896: dread dc/d23/d84/d7c/f69 [4194304,4194304] 0 2024-03-04T16:04:21.407 INFO:tasks.workunit.client.0.smithi116.stdout:2/867: fdatasync d7/d81/dc2/f10f 0 2024-03-04T16:04:21.408 INFO:tasks.workunit.client.0.smithi116.stdout:2/868: chown d7/d13/dec/l132 1041600164 1 2024-03-04T16:04:21.414 INFO:tasks.workunit.client.0.smithi116.stdout:7/825: dread d3/fd [0,4194304] 0 2024-03-04T16:04:21.416 INFO:tasks.workunit.client.0.smithi116.stdout:0/713: rename da/d16/d28/d40/d5d to da/d16/d20/d42/de0 0 2024-03-04T16:04:21.419 INFO:tasks.workunit.client.0.smithi116.stdout:2/869: link d7/d13/d3e/ded/d6c/da4/faa d7/d13/d3e/d98/dce/d47/d64/d75/d77/d119/f13f 0 2024-03-04T16:04:21.420 INFO:tasks.workunit.client.0.smithi116.stdout:0/714: creat da/d16/d28/d40/d48/da7/db7/fe1 x:0 0 0 2024-03-04T16:04:21.421 INFO:tasks.workunit.client.0.smithi116.stdout:0/715: stat da/d16/d20/dcf/fd3 0 2024-03-04T16:04:21.421 INFO:tasks.workunit.client.0.smithi116.stdout:6/764: rename d6/de/lbc to d6/de/d1e/d80/l103 0 2024-03-04T16:04:21.422 INFO:tasks.workunit.client.0.smithi116.stdout:7/826: creat d3/d4/d8/f10c x:0 0 0 2024-03-04T16:04:21.423 INFO:tasks.workunit.client.0.smithi116.stdout:6/765: chown d6/de/d84/db4/fe2 17366 1 2024-03-04T16:04:21.425 INFO:tasks.workunit.client.0.smithi116.stdout:3/897: rename dc/df/d14/f72 to dc/d34/ddd/f127 0 2024-03-04T16:04:21.425 INFO:tasks.workunit.client.0.smithi116.stdout:2/870: link d7/d13/d3e/d78/d107/f127 d7/d15/d5e/f140 0 2024-03-04T16:04:21.426 INFO:tasks.workunit.client.0.smithi116.stdout:0/716: getdents da/d16/d28/d7b 0 2024-03-04T16:04:21.430 INFO:tasks.workunit.client.0.smithi116.stdout:7/827: rename d3/f1e to d3/d4/d9e/dd7/daf/dd5/f10d 0 2024-03-04T16:04:21.431 INFO:tasks.workunit.client.0.smithi116.stdout:7/828: write d3/d4/f107 [403384,39620] 0 2024-03-04T16:04:21.434 INFO:tasks.workunit.client.0.smithi116.stdout:2/871: dwrite d7/d13/d3e/ded/d76/d7c/f7d [4194304,4194304] 0 2024-03-04T16:04:21.436 INFO:tasks.workunit.client.0.smithi116.stdout:0/717: write da/d16/d19/d27/f78 [2171020,17960] 0 2024-03-04T16:04:21.437 INFO:tasks.workunit.client.0.smithi116.stdout:7/829: creat d3/d4/d34/f10e x:0 0 0 2024-03-04T16:04:21.442 INFO:tasks.workunit.client.0.smithi116.stdout:0/718: read da/d16/d28/d40/d48/d57/db3/db2/fb6 [3627113,94678] 0 2024-03-04T16:04:21.444 INFO:tasks.workunit.client.0.smithi116.stdout:7/830: mkdir d3/d4/dbf/de6/dd3/ddf/de0/d10f 0 2024-03-04T16:04:21.446 INFO:tasks.workunit.client.0.smithi116.stdout:0/719: mknod da/d16/d20/ce2 0 2024-03-04T16:04:21.446 INFO:tasks.workunit.client.0.smithi116.stdout:6/766: dwrite d6/de/d25/d4a/df8/da5/fc4 [0,4194304] 0 2024-03-04T16:04:21.451 INFO:tasks.workunit.client.0.smithi116.stdout:0/720: fdatasync da/d16/d28/d40/f5c 0 2024-03-04T16:04:21.463 INFO:tasks.workunit.client.0.smithi116.stdout:0/721: chown da/d16/d20/d42/cc4 262 1 2024-03-04T16:04:21.464 INFO:tasks.workunit.client.0.smithi116.stdout:3/898: dwrite dc/d34/d44/fac [0,4194304] 0 2024-03-04T16:04:21.468 INFO:tasks.workunit.client.0.smithi116.stdout:2/872: dwrite d7/d13/d3e/d98/dce/d47/d64/f6f [0,4194304] 0 2024-03-04T16:04:21.468 INFO:tasks.workunit.client.0.smithi116.stdout:7/831: dwrite d3/d4/d8/d4a/d87/f9d [4194304,4194304] 0 2024-03-04T16:04:21.471 INFO:tasks.workunit.client.0.smithi116.stdout:6/767: dwrite d6/de/d25/d36/f27 [0,4194304] 0 2024-03-04T16:04:21.474 INFO:tasks.workunit.client.0.smithi116.stdout:6/768: fsync d6/de/d25/d36/d44/f4b 0 2024-03-04T16:04:21.484 INFO:tasks.workunit.client.0.smithi116.stdout:6/769: dwrite d6/de/d25/d36/d44/dbb/ddd/d7b/dc6/fe1 [0,4194304] 0 2024-03-04T16:04:21.484 INFO:tasks.workunit.client.0.smithi116.stdout:2/873: fsync d7/d13/f1e 0 2024-03-04T16:04:21.486 INFO:tasks.workunit.client.0.smithi116.stdout:3/899: creat dc/d23/d84/d7f/d93/db6/f128 x:0 0 0 2024-03-04T16:04:21.486 INFO:tasks.workunit.client.0.smithi116.stdout:3/900: readlink dc/l52 0 2024-03-04T16:04:21.486 INFO:tasks.workunit.client.0.smithi116.stdout:7/832: stat d3/d4/d8/c19 0 2024-03-04T16:04:21.489 INFO:tasks.workunit.client.0.smithi116.stdout:6/770: rmdir d6/de/d84 39 2024-03-04T16:04:21.492 INFO:tasks.workunit.client.0.smithi116.stdout:3/901: write dc/df/f92 [8487058,88565] 0 2024-03-04T16:04:21.494 INFO:tasks.workunit.client.0.smithi116.stdout:6/771: mkdir d6/de/d25/d4a/d63/d95/d104 0 2024-03-04T16:04:21.495 INFO:tasks.workunit.client.0.smithi116.stdout:6/772: readlink d6/de/d15/l69 0 2024-03-04T16:04:21.497 INFO:tasks.workunit.client.0.smithi116.stdout:0/722: rename da/d16/d20/d42/de0/d7e/fac to da/d16/d28/d40/fe3 0 2024-03-04T16:04:21.499 INFO:tasks.workunit.client.0.smithi116.stdout:3/902: creat dc/d34/de7/f129 x:0 0 0 2024-03-04T16:04:21.501 INFO:tasks.workunit.client.0.smithi116.stdout:0/723: mknod da/d16/d20/d42/ce4 0 2024-03-04T16:04:21.501 INFO:tasks.workunit.client.0.smithi116.stdout:0/724: chown da/d16/d28/d40/d48/d57/d8a 64578 1 2024-03-04T16:04:21.501 INFO:tasks.workunit.client.0.smithi116.stdout:3/903: creat dc/d23/d84/d7f/d8b/f12a x:0 0 0 2024-03-04T16:04:21.502 INFO:tasks.workunit.client.0.smithi116.stdout:0/725: chown da/d16/d19/d27/d82 6837 1 2024-03-04T16:04:21.508 INFO:tasks.workunit.client.0.smithi116.stdout:3/904: dwrite dc/df/f92 [0,4194304] 0 2024-03-04T16:04:21.509 INFO:tasks.workunit.client.0.smithi116.stdout:2/874: dread d7/d81/dc2/fd0 [0,4194304] 0 2024-03-04T16:04:21.509 INFO:tasks.workunit.client.0.smithi116.stdout:0/726: dread da/d16/d28/d40/d48/d57/f8c [0,4194304] 0 2024-03-04T16:04:21.510 INFO:tasks.workunit.client.0.smithi116.stdout:8/854: sync 2024-03-04T16:04:21.514 INFO:tasks.workunit.client.0.smithi116.stdout:8/855: rename d2/de/d16/d78/d29/d7a/cef to d2/de/d16/dd1/c11c 0 2024-03-04T16:04:21.516 INFO:tasks.workunit.client.0.smithi116.stdout:8/856: mknod d2/de/d3e/d7b/c11d 0 2024-03-04T16:04:21.518 INFO:tasks.workunit.client.0.smithi116.stdout:2/875: creat d7/d13/f141 x:0 0 0 2024-03-04T16:04:21.520 INFO:tasks.workunit.client.0.smithi116.stdout:7/833: truncate d3/ffa 191067 0 2024-03-04T16:04:21.521 INFO:tasks.workunit.client.0.smithi116.stdout:8/857: creat d2/d38/d41/dab/de7/f11e x:0 0 0 2024-03-04T16:04:21.523 INFO:tasks.workunit.client.0.smithi116.stdout:2/876: rename d7/f4d to d7/d15/d5e/d9d/df1/d125/f142 0 2024-03-04T16:04:21.523 INFO:tasks.workunit.client.0.smithi116.stdout:7/834: readlink d3/d4/l57 0 2024-03-04T16:04:21.524 INFO:tasks.workunit.client.0.smithi116.stdout:8/858: dread - d2/d13/d5e/fc4 zero size 2024-03-04T16:04:21.527 INFO:tasks.workunit.client.0.smithi116.stdout:8/859: mknod d2/d13/c11f 0 2024-03-04T16:04:21.528 INFO:tasks.workunit.client.0.smithi116.stdout:2/877: creat d7/d13/d3e/ded/d6c/f143 x:0 0 0 2024-03-04T16:04:21.530 INFO:tasks.workunit.client.0.smithi116.stdout:2/878: creat d7/d13/d3e/ded/d6c/ddb/f144 x:0 0 0 2024-03-04T16:04:21.534 INFO:tasks.workunit.client.0.smithi116.stdout:8/860: dwrite d2/de/d3e/f105 [0,4194304] 0 2024-03-04T16:04:21.535 INFO:tasks.workunit.client.0.smithi116.stdout:3/905: dread dc/d34/f9d [0,4194304] 0 2024-03-04T16:04:21.537 INFO:tasks.workunit.client.0.smithi116.stdout:2/879: rename d7/d13/d3e/d78/fb1 to d7/d13/d3e/ded/d76/d7c/f145 0 2024-03-04T16:04:21.538 INFO:tasks.workunit.client.0.smithi116.stdout:8/861: rmdir d2/d13/d3c/d77/d97/d42 39 2024-03-04T16:04:21.539 INFO:tasks.workunit.client.0.smithi116.stdout:8/862: chown d2/d4a/d8a 1362 1 2024-03-04T16:04:21.540 INFO:tasks.workunit.client.0.smithi116.stdout:3/906: rename dc/df/d47/fb3 to dc/dd3/f12b 0 2024-03-04T16:04:21.542 INFO:tasks.workunit.client.0.smithi116.stdout:7/835: write d3/d4/f92 [2944648,83700] 0 2024-03-04T16:04:21.543 INFO:tasks.workunit.client.0.smithi116.stdout:7/836: fsync d3/d4/d8/f71 0 2024-03-04T16:04:21.544 INFO:tasks.workunit.client.0.smithi116.stdout:3/907: stat dc/d34/de7/l103 0 2024-03-04T16:04:21.545 INFO:tasks.workunit.client.0.smithi116.stdout:7/837: write d3/d4/f92 [2541561,127873] 0 2024-03-04T16:04:21.545 INFO:tasks.workunit.client.0.smithi116.stdout:8/863: dread d2/d13/d3c/d77/d97/d5a/f10d [0,4194304] 0 2024-03-04T16:04:21.547 INFO:tasks.workunit.client.0.smithi116.stdout:3/908: mkdir dc/d23/d84/d7c/d12c 0 2024-03-04T16:04:21.548 INFO:tasks.workunit.client.0.smithi116.stdout:3/909: write dc/d23/d84/d7f/d93/db6/dee/ff6 [1246825,56030] 0 2024-03-04T16:04:21.549 INFO:tasks.workunit.client.0.smithi116.stdout:7/838: mknod d3/d4/dbf/de6/dd3/ddf/dee/c110 0 2024-03-04T16:04:21.553 INFO:tasks.workunit.client.0.smithi116.stdout:2/880: write d7/d15/d5e/d9d/d11f/fb3 [4608799,24006] 0 2024-03-04T16:04:21.554 INFO:tasks.workunit.client.0.smithi116.stdout:2/881: fsync d7/d81/f7b 0 2024-03-04T16:04:21.556 INFO:tasks.workunit.client.0.smithi116.stdout:2/882: chown d7/d13/d3e/ded/d76 4770127 1 2024-03-04T16:04:21.557 INFO:tasks.workunit.client.0.smithi116.stdout:7/839: dwrite d3/d4/d8/d1a/ff7 [0,4194304] 0 2024-03-04T16:04:21.558 INFO:tasks.workunit.client.0.smithi116.stdout:2/883: truncate d7/d13/d3e/ded/d6c/f97 1450797 0 2024-03-04T16:04:21.559 INFO:tasks.workunit.client.0.smithi116.stdout:7/840: fsync d3/d4/d8/d1a/f32 0 2024-03-04T16:04:21.561 INFO:tasks.workunit.client.0.smithi116.stdout:3/910: rename dc/d23/d84 to dc/df/d47/d12d 0 2024-03-04T16:04:21.562 INFO:tasks.workunit.client.0.smithi116.stdout:2/884: symlink d7/d13/d3e/d98/dce/d47/d64/da5/l146 0 2024-03-04T16:04:21.564 INFO:tasks.workunit.client.0.smithi116.stdout:2/885: chown d7/d13/d3e/d98/dce/ffa 28817271 1 2024-03-04T16:04:21.565 INFO:tasks.workunit.client.0.smithi116.stdout:7/841: rename d3/d4/d8/c4b to d3/d4/d9e/c111 0 2024-03-04T16:04:21.566 INFO:tasks.workunit.client.0.smithi116.stdout:2/886: mkdir d7/d13/d3e/d71/dad/d147 0 2024-03-04T16:04:21.569 INFO:tasks.workunit.client.0.smithi116.stdout:7/842: rename d3/d4/d8/fae to d3/d4/d9e/dd7/dfd/f112 0 2024-03-04T16:04:21.570 INFO:tasks.workunit.client.0.smithi116.stdout:2/887: dread - d7/d13/d3e/ded/d6c/da4/faa zero size 2024-03-04T16:04:21.570 INFO:tasks.workunit.client.0.smithi116.stdout:2/888: chown d7/d13 6237 1 2024-03-04T16:04:21.573 INFO:tasks.workunit.client.0.smithi116.stdout:7/843: rename d3/d4/d8/d4a/db3 to d3/d4/d8/d4a/d113 0 2024-03-04T16:04:21.575 INFO:tasks.workunit.client.0.smithi116.stdout:8/864: dread d2/de/fa5 [4194304,4194304] 0 2024-03-04T16:04:21.575 INFO:tasks.workunit.client.0.smithi116.stdout:3/911: write dc/df/d14/f58 [4758772,5395] 0 2024-03-04T16:04:21.579 INFO:tasks.workunit.client.0.smithi116.stdout:3/912: creat dc/dd3/f12e x:0 0 0 2024-03-04T16:04:21.580 INFO:tasks.workunit.client.0.smithi116.stdout:2/889: write d7/d25/fa2 [15536,1193] 0 2024-03-04T16:04:21.580 INFO:tasks.workunit.client.0.smithi116.stdout:7/844: rmdir d3/d4/dbf/d101 0 2024-03-04T16:04:21.581 INFO:tasks.workunit.client.0.smithi116.stdout:3/913: write dc/df/f79 [1502209,125166] 0 2024-03-04T16:04:21.582 INFO:tasks.workunit.client.0.smithi116.stdout:3/914: write dc/d34/f7a [933414,25176] 0 2024-03-04T16:04:21.587 INFO:tasks.workunit.client.0.smithi116.stdout:8/865: dwrite d2/de/d3e/f4e [0,4194304] 0 2024-03-04T16:04:21.589 INFO:tasks.workunit.client.0.smithi116.stdout:7/845: link d3/d4/d8/d4a/l4f d3/d4/dbf/de6/dd3/ddf/de0/l114 0 2024-03-04T16:04:21.589 INFO:tasks.workunit.client.0.smithi116.stdout:2/890: link d7/d81/dc2/ldf d7/d15/d5e/l148 0 2024-03-04T16:04:21.590 INFO:tasks.workunit.client.0.smithi116.stdout:3/915: mkdir dc/df/d47/d12d/d7f/d93/db6/dc5/de0/d12f 0 2024-03-04T16:04:21.596 INFO:tasks.workunit.client.0.smithi116.stdout:3/916: mkdir dc/df/d47/d12d/d7f/d93/db6/dc5/d130 0 2024-03-04T16:04:21.598 INFO:tasks.workunit.client.0.smithi116.stdout:2/891: mkdir d7/d13/d3e/d71/dad/d147/d149 0 2024-03-04T16:04:21.598 INFO:tasks.workunit.client.0.smithi116.stdout:8/866: write d2/de/d3e/f50 [2049835,92544] 0 2024-03-04T16:04:21.599 INFO:tasks.workunit.client.0.smithi116.stdout:8/867: fdatasync d2/f32 0 2024-03-04T16:04:21.602 INFO:tasks.workunit.client.0.smithi116.stdout:3/917: read dc/df/d47/d12d/d7c/f65 [7976355,78112] 0 2024-03-04T16:04:21.602 INFO:tasks.workunit.client.0.smithi116.stdout:2/892: readlink d7/d13/d3e/d98/dce/d47/d64/da5/lcc 0 2024-03-04T16:04:21.602 INFO:tasks.workunit.client.0.smithi116.stdout:8/868: symlink d2/d13/d3c/d77/d97/deb/l120 0 2024-03-04T16:04:21.603 INFO:tasks.workunit.client.0.smithi116.stdout:2/893: fsync d7/d13/d3e/d98/dce/d47/d64/f87 0 2024-03-04T16:04:21.604 INFO:tasks.workunit.client.0.smithi116.stdout:3/918: symlink dc/df/d47/d12d/d7f/d93/db6/dc5/l131 0 2024-03-04T16:04:21.605 INFO:tasks.workunit.client.0.smithi116.stdout:7/846: dwrite d3/f90 [0,4194304] 0 2024-03-04T16:04:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:21 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:21 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:21 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:04:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:21 smithi152 ceph-mon[36876]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:04:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:21 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:21 smithi152 ceph-mon[36876]: Upgrade: Need to upgrade myself (mgr.smithi152.aawieo) 2024-03-04T16:04:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:21 smithi152 ceph-mon[36876]: Upgrade: Need to upgrade myself (mgr.smithi152.aawieo) 2024-03-04T16:04:21.608 INFO:tasks.workunit.client.0.smithi116.stdout:7/847: dread f2 [4194304,4194304] 0 2024-03-04T16:04:21.608 INFO:tasks.workunit.client.0.smithi116.stdout:2/894: symlink d7/d13/d3e/d78/d107/l14a 0 2024-03-04T16:04:21.609 INFO:tasks.workunit.client.0.smithi116.stdout:3/919: creat dc/d34/dc1/f132 x:0 0 0 2024-03-04T16:04:21.612 INFO:tasks.workunit.client.0.smithi116.stdout:2/895: creat d7/d15/d5e/d9d/df1/f14b x:0 0 0 2024-03-04T16:04:21.613 INFO:tasks.workunit.client.0.smithi116.stdout:3/920: creat dc/df/dbc/d101/f133 x:0 0 0 2024-03-04T16:04:21.613 INFO:tasks.workunit.client.0.smithi116.stdout:8/869: link d2/d13/d5e/da7/lec d2/d13/d3c/d46/d6e/l121 0 2024-03-04T16:04:21.614 INFO:tasks.workunit.client.0.smithi116.stdout:2/896: mknod d7/d15/d5e/c14c 0 2024-03-04T16:04:21.614 INFO:tasks.workunit.client.0.smithi116.stdout:8/870: fdatasync d2/de/d3e/d7b/fdd 0 2024-03-04T16:04:21.615 INFO:tasks.workunit.client.0.smithi116.stdout:7/848: getdents d3/d4/dbf/de6 0 2024-03-04T16:04:21.615 INFO:tasks.workunit.client.0.smithi116.stdout:2/897: stat d7/l72 0 2024-03-04T16:04:21.616 INFO:tasks.workunit.client.0.smithi116.stdout:3/921: chown dc/d23/caa 51 1 2024-03-04T16:04:21.616 INFO:tasks.workunit.client.0.smithi116.stdout:8/871: dread - d2/de/d16/d78/d29/d7a/db1/ff6 zero size 2024-03-04T16:04:21.619 INFO:tasks.workunit.client.0.smithi116.stdout:3/922: creat dc/df/d47/d12d/d7c/d12c/f134 x:0 0 0 2024-03-04T16:04:21.619 INFO:tasks.workunit.client.0.smithi116.stdout:8/872: mkdir d2/d4a/d114/d122 0 2024-03-04T16:04:21.622 INFO:tasks.workunit.client.0.smithi116.stdout:7/849: dwrite d3/d4/d11/f51 [0,4194304] 0 2024-03-04T16:04:21.623 INFO:tasks.workunit.client.0.smithi116.stdout:8/873: mknod d2/d13/c123 0 2024-03-04T16:04:21.625 INFO:tasks.workunit.client.0.smithi116.stdout:7/850: mknod d3/d4/d34/c115 0 2024-03-04T16:04:21.625 INFO:tasks.workunit.client.0.smithi116.stdout:3/923: dwrite dc/d34/d44/fbd [0,4194304] 0 2024-03-04T16:04:21.627 INFO:tasks.workunit.client.0.smithi116.stdout:3/924: read - dc/df/d47/d12d/d7f/de8/f11a zero size 2024-03-04T16:04:21.627 INFO:tasks.workunit.client.0.smithi116.stdout:8/874: rename d2/d13/d5e/da7 to d2/d38/d41/dab/d124 0 2024-03-04T16:04:21.633 INFO:tasks.workunit.client.0.smithi116.stdout:3/925: rename dc/d23 to dc/df/dbc/d135 0 2024-03-04T16:04:21.634 INFO:tasks.workunit.client.0.smithi116.stdout:7/851: mkdir d3/d4/d8/d52/d116 0 2024-03-04T16:04:21.636 INFO:tasks.workunit.client.0.smithi116.stdout:3/926: symlink dc/d34/de7/deb/de2/l136 0 2024-03-04T16:04:21.639 INFO:tasks.workunit.client.0.smithi116.stdout:3/927: creat dc/df/d47/d12d/d7f/de8/f137 x:0 0 0 2024-03-04T16:04:21.640 INFO:tasks.workunit.client.0.smithi116.stdout:8/875: creat d2/d13/d3c/d77/d97/f125 x:0 0 0 2024-03-04T16:04:21.642 INFO:tasks.workunit.client.0.smithi116.stdout:7/852: unlink d3/d4/d8/d1a/l63 0 2024-03-04T16:04:21.643 INFO:tasks.workunit.client.0.smithi116.stdout:3/928: mknod dc/c138 0 2024-03-04T16:04:21.648 INFO:tasks.workunit.client.0.smithi116.stdout:3/929: dread dc/df/d47/d12d/d7c/fa1 [0,4194304] 0 2024-03-04T16:04:21.651 INFO:tasks.workunit.client.0.smithi116.stdout:3/930: symlink dc/df/d47/d12d/d7f/d93/l139 0 2024-03-04T16:04:21.651 INFO:tasks.workunit.client.0.smithi116.stdout:8/876: write d2/de/d16/d78/fbd [3768563,12936] 0 2024-03-04T16:04:21.655 INFO:tasks.workunit.client.0.smithi116.stdout:8/877: rename d2/d13/d3c/d77/fb3 to d2/d38/d55/f126 0 2024-03-04T16:04:21.655 INFO:tasks.workunit.client.0.smithi116.stdout:7/853: dwrite d3/f68 [0,4194304] 0 2024-03-04T16:04:21.658 INFO:tasks.workunit.client.0.smithi116.stdout:3/931: truncate dc/df/d47/d12d/d7c/fa1 1075179 0 2024-03-04T16:04:21.659 INFO:tasks.workunit.client.0.smithi116.stdout:3/932: truncate dc/d34/de7/deb/ff5 888477 0 2024-03-04T16:04:21.660 INFO:tasks.workunit.client.0.smithi116.stdout:7/854: creat d3/d4/d8/d4a/d65/df2/f117 x:0 0 0 2024-03-04T16:04:21.662 INFO:tasks.workunit.client.0.smithi116.stdout:5/827: sync 2024-03-04T16:04:21.663 INFO:tasks.workunit.client.0.smithi116.stdout:9/775: sync 2024-03-04T16:04:21.663 INFO:tasks.workunit.client.0.smithi116.stdout:1/708: sync 2024-03-04T16:04:21.663 INFO:tasks.workunit.client.0.smithi116.stdout:4/851: sync 2024-03-04T16:04:21.664 INFO:tasks.workunit.client.0.smithi116.stdout:3/933: creat dc/d34/de7/f13a x:0 0 0 2024-03-04T16:04:21.665 INFO:tasks.workunit.client.0.smithi116.stdout:7/855: mknod d3/d4/d34/d5d/ded/d108/c118 0 2024-03-04T16:04:21.665 INFO:tasks.workunit.client.0.smithi116.stdout:3/934: fdatasync dc/df/f92 0 2024-03-04T16:04:21.666 INFO:tasks.workunit.client.0.smithi116.stdout:9/776: chown d2/d33/d65/cd9 5837811 1 2024-03-04T16:04:21.667 INFO:tasks.workunit.client.0.smithi116.stdout:7/856: fdatasync d3/d4/d8/d1a/f42 0 2024-03-04T16:04:21.667 INFO:tasks.workunit.client.0.smithi116.stdout:3/935: readlink dc/df/d47/ld5 0 2024-03-04T16:04:21.667 INFO:tasks.workunit.client.0.smithi116.stdout:5/828: truncate dd/d1f/d55/d73/f3f 3758517 0 2024-03-04T16:04:21.668 INFO:tasks.workunit.client.0.smithi116.stdout:8/878: truncate d2/d4a/d8a/fd4 4645515 0 2024-03-04T16:04:21.669 INFO:tasks.workunit.client.0.smithi116.stdout:1/709: creat d5/d9/d21/d27/d32/d35/d55/db4/d50/fe8 x:0 0 0 2024-03-04T16:04:21.670 INFO:tasks.workunit.client.0.smithi116.stdout:1/710: fsync d5/da1/fc3 0 2024-03-04T16:04:21.671 INFO:tasks.workunit.client.0.smithi116.stdout:7/857: dread f2 [0,4194304] 0 2024-03-04T16:04:21.672 INFO:tasks.workunit.client.0.smithi116.stdout:4/852: rename d1/d3/d40/d52/d84/cfb to d1/d3/d15/d20/d1c/d62/d81/da6/c12b 0 2024-03-04T16:04:21.674 INFO:tasks.workunit.client.0.smithi116.stdout:4/853: fdatasync d1/d3/d40/d52/d84/fa0 0 2024-03-04T16:04:21.674 INFO:tasks.workunit.client.0.smithi116.stdout:7/858: read d3/d4/f92 [3083141,90083] 0 2024-03-04T16:04:21.678 INFO:tasks.workunit.client.0.smithi116.stdout:1/711: dwrite d5/d9/d21/d27/d32/d35/d55/db4/fc5 [0,4194304] 0 2024-03-04T16:04:21.686 INFO:tasks.workunit.client.0.smithi116.stdout:4/854: creat d1/d3/dd/d8a/f12c x:0 0 0 2024-03-04T16:04:21.690 INFO:tasks.workunit.client.0.smithi116.stdout:1/712: unlink d5/d9/d21/d27/d32/d35/d55/db4/fa0 0 2024-03-04T16:04:21.692 INFO:tasks.workunit.client.0.smithi116.stdout:9/777: getdents d2/d26/dd4/de5 0 2024-03-04T16:04:21.694 INFO:tasks.workunit.client.0.smithi116.stdout:4/855: dwrite d1/d3/d15/d20/f4e [0,4194304] 0 2024-03-04T16:04:21.695 INFO:tasks.workunit.client.0.smithi116.stdout:5/829: creat dd/d1f/d55/d73/d2d/d59/f10b x:0 0 0 2024-03-04T16:04:21.695 INFO:tasks.workunit.client.0.smithi116.stdout:1/713: write d5/d9/d21/d27/d32/d35/f5c [782797,29294] 0 2024-03-04T16:04:21.699 INFO:tasks.workunit.client.0.smithi116.stdout:7/859: write d3/f1f [416414,20106] 0 2024-03-04T16:04:21.699 INFO:tasks.workunit.client.0.smithi116.stdout:5/830: chown dd/d1f/d55/d73/d2d/f42 0 1 2024-03-04T16:04:21.700 INFO:tasks.workunit.client.0.smithi116.stdout:8/879: dwrite d2/d13/d5e/f87 [0,4194304] 0 2024-03-04T16:04:21.703 INFO:tasks.workunit.client.0.smithi116.stdout:8/880: chown d2/d13/d3c/fae 205378302 1 2024-03-04T16:04:21.703 INFO:tasks.workunit.client.0.smithi116.stdout:1/714: write d5/d9/d21/d27/d32/d35/d55/db4/f63 [1091600,94529] 0 2024-03-04T16:04:21.704 INFO:tasks.workunit.client.0.smithi116.stdout:1/715: write d5/d9/fb [8534243,92128] 0 2024-03-04T16:04:21.705 INFO:tasks.workunit.client.0.smithi116.stdout:5/831: mkdir dd/d45/d10c 0 2024-03-04T16:04:21.707 INFO:tasks.workunit.client.0.smithi116.stdout:4/856: link d1/d3/dd/dd9/d11a/ff3 d1/d3/d40/f12d 0 2024-03-04T16:04:21.709 INFO:tasks.workunit.client.0.smithi116.stdout:8/881: dread - d2/d13/db9/fc1 zero size 2024-03-04T16:04:21.712 INFO:tasks.workunit.client.0.smithi116.stdout:9/778: write d2/d33/f46 [1969071,114751] 0 2024-03-04T16:04:21.716 INFO:tasks.workunit.client.0.smithi116.stdout:9/779: chown d2/d26/dd4/dab/d3c/l7a 2 1 2024-03-04T16:04:21.716 INFO:tasks.workunit.client.0.smithi116.stdout:9/780: chown d2/d33/d65/l8d 1179255701 1 2024-03-04T16:04:21.716 INFO:tasks.workunit.client.0.smithi116.stdout:1/716: rename d5/d9/d21/d24/de0/dda to d5/d9/d21/d84/db0/de9 0 2024-03-04T16:04:21.716 INFO:tasks.workunit.client.0.smithi116.stdout:9/781: dread - d2/d33/dc5/ff5 zero size 2024-03-04T16:04:21.716 INFO:tasks.workunit.client.0.smithi116.stdout:4/857: mkdir d1/d3/dd/dd9/d11a/d12e 0 2024-03-04T16:04:21.716 INFO:tasks.workunit.client.0.smithi116.stdout:9/782: dread - d2/d19/d27/dbd/ff6 zero size 2024-03-04T16:04:21.719 INFO:tasks.workunit.client.0.smithi116.stdout:1/717: dread d5/d9/d21/d24/de0/f66 [0,4194304] 0 2024-03-04T16:04:21.719 INFO:tasks.workunit.client.0.smithi116.stdout:4/858: mknod d1/d3/d15/d20/d1c/ddd/c12f 0 2024-03-04T16:04:21.720 INFO:tasks.workunit.client.0.smithi116.stdout:9/783: creat d2/d33/d65/d6c/ff7 x:0 0 0 2024-03-04T16:04:21.722 INFO:tasks.workunit.client.0.smithi116.stdout:8/882: rmdir d2/d4a/dc7/d116 0 2024-03-04T16:04:21.723 INFO:tasks.workunit.client.0.smithi116.stdout:5/832: getdents dd/d1f/d55/d73/d2d/d8d 0 2024-03-04T16:04:21.723 INFO:tasks.workunit.client.0.smithi116.stdout:9/784: mknod d2/d26/dd4/d4a/d6a/d94/dbe/cf8 0 2024-03-04T16:04:21.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:21 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:21 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:21 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:04:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:21 smithi116 ceph-mon[29120]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:04:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:21 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:21 smithi116 ceph-mon[29120]: Upgrade: Need to upgrade myself (mgr.smithi152.aawieo) 2024-03-04T16:04:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:21 smithi116 ceph-mon[29120]: Upgrade: Need to upgrade myself (mgr.smithi152.aawieo) 2024-03-04T16:04:21.726 INFO:tasks.workunit.client.0.smithi116.stdout:4/859: getdents d1/d3/de/d47 0 2024-03-04T16:04:21.726 INFO:tasks.workunit.client.0.smithi116.stdout:1/718: dwrite d5/d86/fab [0,4194304] 0 2024-03-04T16:04:21.727 INFO:tasks.workunit.client.0.smithi116.stdout:9/785: chown d2/d26/dd4/de5/fa8 959200672 1 2024-03-04T16:04:21.727 INFO:tasks.workunit.client.0.smithi116.stdout:4/860: stat d1/d3/d40/f12d 0 2024-03-04T16:04:21.728 INFO:tasks.workunit.client.0.smithi116.stdout:4/861: read - d1/d3/dd/d8a/ddc/ff6 zero size 2024-03-04T16:04:21.728 INFO:tasks.workunit.client.0.smithi116.stdout:5/833: creat dd/d1f/d55/d73/d2d/dc0/f10d x:0 0 0 2024-03-04T16:04:21.731 INFO:tasks.workunit.client.0.smithi116.stdout:8/883: getdents d2/d13/d3c/d77 0 2024-03-04T16:04:21.733 INFO:tasks.workunit.client.0.smithi116.stdout:5/834: rmdir dd/d1f/d55/d73/d2d/d59/d71/dff 39 2024-03-04T16:04:21.735 INFO:tasks.workunit.client.0.smithi116.stdout:4/862: creat d1/d3/dd/d8a/ddc/f130 x:0 0 0 2024-03-04T16:04:21.736 INFO:tasks.workunit.client.0.smithi116.stdout:4/863: write d1/d3/d15/d20/f1e [1204000,115064] 0 2024-03-04T16:04:21.736 INFO:tasks.workunit.client.0.smithi116.stdout:4/864: stat d1/d3/dd/dd9 0 2024-03-04T16:04:21.739 INFO:tasks.workunit.client.0.smithi116.stdout:8/884: rename d2/d13/db9 to d2/d4a/d114/d122/d127 0 2024-03-04T16:04:21.739 INFO:tasks.workunit.client.0.smithi116.stdout:7/860: truncate d3/d4/d8/d52/d5e/fc5 2007346 0 2024-03-04T16:04:21.739 INFO:tasks.workunit.client.0.smithi116.stdout:8/885: dread - d2/d38/d41/dab/de7/f11e zero size 2024-03-04T16:04:21.740 INFO:tasks.workunit.client.0.smithi116.stdout:5/835: unlink dd/d1f/d55/le2 0 2024-03-04T16:04:21.745 INFO:tasks.workunit.client.0.smithi116.stdout:7/861: creat d3/d4/d8/d52/d5e/d9a/f119 x:0 0 0 2024-03-04T16:04:21.746 INFO:tasks.workunit.client.0.smithi116.stdout:8/886: mknod d2/d13/d3c/d46/d4f/c128 0 2024-03-04T16:04:21.747 INFO:tasks.workunit.client.0.smithi116.stdout:8/887: chown d2/d13/d3c/d77/d97/d5a/dcf/cd3 147 1 2024-03-04T16:04:21.748 INFO:tasks.workunit.client.0.smithi116.stdout:9/786: getdents d2/d19/d5f 0 2024-03-04T16:04:21.751 INFO:tasks.workunit.client.0.smithi116.stdout:7/862: symlink d3/d4/d34/d5d/ded/l11a 0 2024-03-04T16:04:21.753 INFO:tasks.workunit.client.0.smithi116.stdout:5/836: unlink dd/d1f/d5c/d5e/faa 0 2024-03-04T16:04:21.753 INFO:tasks.workunit.client.0.smithi116.stdout:8/888: dwrite d2/de/d16/d78/d29/ff0 [4194304,4194304] 0 2024-03-04T16:04:21.755 INFO:tasks.workunit.client.0.smithi116.stdout:8/889: chown d2/d13/d3c/d77/d97/d5a/f91 3501717 1 2024-03-04T16:04:21.755 INFO:tasks.workunit.client.0.smithi116.stdout:9/787: creat d2/d26/dd4/d4a/d6a/d94/d95/db7/ff9 x:0 0 0 2024-03-04T16:04:21.756 INFO:tasks.workunit.client.0.smithi116.stdout:9/788: chown d2/d19/d5f/cdc 53011 1 2024-03-04T16:04:21.756 INFO:tasks.workunit.client.0.smithi116.stdout:4/865: truncate d1/f5 4189060 0 2024-03-04T16:04:21.760 INFO:tasks.workunit.client.0.smithi116.stdout:5/837: mknod dd/d1f/d55/d73/d2d/d59/db0/dca/c10e 0 2024-03-04T16:04:21.761 INFO:tasks.workunit.client.0.smithi116.stdout:7/863: creat d3/d4/d8/d52/f11b x:0 0 0 2024-03-04T16:04:21.764 INFO:tasks.workunit.client.0.smithi116.stdout:5/838: chown dd/c27 272 1 2024-03-04T16:04:21.764 INFO:tasks.workunit.client.0.smithi116.stdout:8/890: dwrite d2/de/d16/d78/d29/d7a/db1/ff6 [0,4194304] 0 2024-03-04T16:04:21.768 INFO:tasks.workunit.client.0.smithi116.stdout:4/866: dwrite d1/d3/dd/d8a/f128 [0,4194304] 0 2024-03-04T16:04:21.768 INFO:tasks.workunit.client.0.smithi116.stdout:8/891: dread - d2/de/f101 zero size 2024-03-04T16:04:21.769 INFO:tasks.workunit.client.0.smithi116.stdout:4/867: creat d1/f131 x:0 0 0 2024-03-04T16:04:21.770 INFO:tasks.workunit.client.0.smithi116.stdout:7/864: getdents d3/d4/d9e/dd7/dfd/d100 0 2024-03-04T16:04:21.771 INFO:tasks.workunit.client.0.smithi116.stdout:7/865: dread - d3/d4/d8/d4a/d113/dc3/fc6 zero size 2024-03-04T16:04:21.771 INFO:tasks.workunit.client.0.smithi116.stdout:5/839: truncate dd/d1f/d55/d73/d2d/d59/fac 318466 0 2024-03-04T16:04:21.771 INFO:tasks.workunit.client.0.smithi116.stdout:4/868: stat d1/d3/d15/d20/d1c/d62/d81/fb0 0 2024-03-04T16:04:21.772 INFO:tasks.workunit.client.0.smithi116.stdout:7/866: write d3/f6 [523375,33547] 0 2024-03-04T16:04:21.773 INFO:tasks.workunit.client.0.smithi116.stdout:8/892: mknod d2/de/d16/d78/d29/de5/d118/c129 0 2024-03-04T16:04:21.778 INFO:tasks.workunit.client.0.smithi116.stdout:5/840: rmdir dd/d1f/d55/d73/d2d/d59/db0/dca 39 2024-03-04T16:04:21.781 INFO:tasks.workunit.client.0.smithi116.stdout:7/867: dwrite d3/d4/d8/d4a/d65/df2/f117 [0,4194304] 0 2024-03-04T16:04:21.785 INFO:tasks.workunit.client.0.smithi116.stdout:8/893: mknod d2/d13/d3c/d77/d97/d75/c12a 0 2024-03-04T16:04:21.787 INFO:tasks.workunit.client.0.smithi116.stdout:7/868: mknod d3/d4/d9e/dd7/daf/dfc/c11c 0 2024-03-04T16:04:21.789 INFO:tasks.workunit.client.0.smithi116.stdout:5/841: creat dd/d1f/d55/d73/d2d/d59/d71/dff/f10f x:0 0 0 2024-03-04T16:04:21.792 INFO:tasks.workunit.client.0.smithi116.stdout:8/894: creat d2/d38/f12b x:0 0 0 2024-03-04T16:04:21.805 INFO:tasks.workunit.client.0.smithi116.stdout:4/869: dwrite d1/d3/d40/d52/d84/d88/d8d/fc8 [0,4194304] 0 2024-03-04T16:04:21.806 INFO:tasks.workunit.client.0.smithi116.stdout:8/895: truncate d2/de/f15 684285 0 2024-03-04T16:04:21.811 INFO:tasks.workunit.client.0.smithi116.stdout:5/842: dwrite dd/d1f/d55/d73/d2d/d31/f7b [0,4194304] 0 2024-03-04T16:04:21.814 INFO:tasks.workunit.client.0.smithi116.stdout:4/870: rename d1/d3/d15/d20/d1c/d62/d81/d97/dc6 to d1/d3/de/d47/df9/d132 0 2024-03-04T16:04:21.816 INFO:tasks.workunit.client.0.smithi116.stdout:4/871: symlink d1/d6c/dbf/l133 0 2024-03-04T16:04:21.817 INFO:tasks.workunit.client.0.smithi116.stdout:8/896: dwrite d2/d13/d5e/f87 [0,4194304] 0 2024-03-04T16:04:21.824 INFO:tasks.workunit.client.0.smithi116.stdout:4/872: getdents d1/d3/de/d124 0 2024-03-04T16:04:21.826 INFO:tasks.workunit.client.0.smithi116.stdout:8/897: dwrite d2/d13/d3c/d77/d97/d5a/f91 [0,4194304] 0 2024-03-04T16:04:21.826 INFO:tasks.workunit.client.0.smithi116.stdout:4/873: chown d1/d3/dd/d8a/f121 16536717 1 2024-03-04T16:04:21.828 INFO:tasks.workunit.client.0.smithi116.stdout:5/843: dread dd/d1f/d5c/fbb [0,4194304] 0 2024-03-04T16:04:21.829 INFO:tasks.workunit.client.0.smithi116.stdout:4/874: mkdir d1/d134 0 2024-03-04T16:04:21.831 INFO:tasks.workunit.client.0.smithi116.stdout:5/844: read dd/d1f/d72/d8e/f98 [3425611,18109] 0 2024-03-04T16:04:21.832 INFO:tasks.workunit.client.0.smithi116.stdout:8/898: rename d2/d4a/c8b to d2/de/d16/d78/d29/de5/df2/d37/c12c 0 2024-03-04T16:04:21.833 INFO:tasks.workunit.client.0.smithi116.stdout:4/875: mknod d1/d3/dd/dd9/d11a/dfe/c135 0 2024-03-04T16:04:21.833 INFO:tasks.workunit.client.0.smithi116.stdout:8/899: write d2/de/d3e/f4e [2481739,72100] 0 2024-03-04T16:04:21.835 INFO:tasks.workunit.client.0.smithi116.stdout:5/845: creat dd/d1f/d55/d73/f110 x:0 0 0 2024-03-04T16:04:21.838 INFO:tasks.workunit.client.0.smithi116.stdout:4/876: creat d1/d3/dd/d24/db4/f136 x:0 0 0 2024-03-04T16:04:21.839 INFO:tasks.workunit.client.0.smithi116.stdout:5/846: read dd/d1f/d55/d73/d2d/d59/db0/fbc [548666,114407] 0 2024-03-04T16:04:21.840 INFO:tasks.workunit.client.0.smithi116.stdout:5/847: chown dd/d1f/de5/cf5 101943 1 2024-03-04T16:04:21.841 INFO:tasks.workunit.client.0.smithi116.stdout:4/877: link d1/d3/dd/d8a/f121 d1/d3/f137 0 2024-03-04T16:04:21.842 INFO:tasks.workunit.client.0.smithi116.stdout:5/848: mkdir dd/d6d/df0/d111 0 2024-03-04T16:04:21.843 INFO:tasks.workunit.client.0.smithi116.stdout:5/849: stat dd/d6d/ld6 0 2024-03-04T16:04:21.845 INFO:tasks.workunit.client.0.smithi116.stdout:4/878: rename d1/d6c/l89 to d1/d3/dd/d45/d119/l138 0 2024-03-04T16:04:21.848 INFO:tasks.workunit.client.0.smithi116.stdout:4/879: creat d1/d3/de/d47/df9/d132/f139 x:0 0 0 2024-03-04T16:04:21.849 INFO:tasks.workunit.client.0.smithi116.stdout:4/880: dread - d1/d3/d15/d20/d1c/d62/d81/f122 zero size 2024-03-04T16:04:21.850 INFO:tasks.workunit.client.0.smithi116.stdout:5/850: rename dd/d1f/d55/d73/d2d/d59/d71/la2 to dd/d1f/d55/d73/d2d/d7c/l112 0 2024-03-04T16:04:21.851 INFO:tasks.workunit.client.0.smithi116.stdout:5/851: fdatasync dd/d1f/d55/d73/d2d/d7c/fd9 0 2024-03-04T16:04:21.853 INFO:tasks.workunit.client.0.smithi116.stdout:8/900: dread d2/d13/d5e/fc6 [0,4194304] 0 2024-03-04T16:04:21.864 INFO:tasks.workunit.client.0.smithi116.stdout:5/852: dread dd/d1f/d55/d73/d2d/f3b [0,4194304] 0 2024-03-04T16:04:21.864 INFO:tasks.workunit.client.0.smithi116.stdout:8/901: dread d2/d13/d3c/d77/fd9 [0,4194304] 0 2024-03-04T16:04:21.866 INFO:tasks.workunit.client.0.smithi116.stdout:5/853: mknod dd/d1f/dcd/c113 0 2024-03-04T16:04:21.944 INFO:tasks.workunit.client.0.smithi116.stdout:8/902: write d2/de/d16/d78/d29/de5/ff4 [10910,129635] 0 2024-03-04T16:04:21.944 INFO:tasks.workunit.client.0.smithi116.stdout:5/854: dread dd/d1f/d55/d73/d2d/d31/f7b [0,4194304] 0 2024-03-04T16:04:21.945 INFO:tasks.workunit.client.0.smithi116.stdout:8/903: symlink d2/d13/d3c/d46/d96/l12d 0 2024-03-04T16:04:21.947 INFO:tasks.workunit.client.0.smithi116.stdout:5/855: symlink dd/d1f/d55/d73/d2d/d59/db0/dca/l114 0 2024-03-04T16:04:21.947 INFO:tasks.workunit.client.0.smithi116.stdout:8/904: symlink d2/d13/d3c/d46/d4f/l12e 0 2024-03-04T16:04:21.948 INFO:tasks.workunit.client.0.smithi116.stdout:5/856: write dd/d1f/d55/d73/d2d/d31/f3c [4650178,83510] 0 2024-03-04T16:04:21.968 INFO:tasks.workunit.client.0.smithi116.stdout:6/773: sync 2024-03-04T16:04:21.970 INFO:tasks.workunit.client.0.smithi116.stdout:8/905: mknod d2/d13/d3c/d46/d6e/c12f 0 2024-03-04T16:04:21.970 INFO:tasks.workunit.client.0.smithi116.stdout:5/857: mknod dd/d1f/dc3/c115 0 2024-03-04T16:04:21.973 INFO:tasks.workunit.client.0.smithi116.stdout:5/858: creat dd/d1f/dc3/f116 x:0 0 0 2024-03-04T16:04:21.976 INFO:tasks.workunit.client.0.smithi116.stdout:8/906: truncate d2/de/d3e/d7b/fdd 566231 0 2024-03-04T16:04:21.977 INFO:tasks.workunit.client.0.smithi116.stdout:5/859: rmdir dd/d1f/d72 39 2024-03-04T16:04:21.979 INFO:tasks.workunit.client.0.smithi116.stdout:8/907: chown d2/d13/d3c/d77/d97/d42/la2 950818 1 2024-03-04T16:04:21.980 INFO:tasks.workunit.client.0.smithi116.stdout:5/860: creat dd/d1f/d51/f117 x:0 0 0 2024-03-04T16:04:21.983 INFO:tasks.workunit.client.0.smithi116.stdout:8/908: symlink d2/d13/d3c/d77/d97/d75/l130 0 2024-03-04T16:04:21.984 INFO:tasks.workunit.client.0.smithi116.stdout:6/774: dwrite d6/de/f1b [4194304,4194304] 0 2024-03-04T16:04:21.986 INFO:tasks.workunit.client.0.smithi116.stdout:8/909: write d2/d13/d3c/d77/d97/d5a/f9d [2334939,10684] 0 2024-03-04T16:04:21.987 INFO:tasks.workunit.client.0.smithi116.stdout:6/775: read d6/de/d15/f26 [1189500,28064] 0 2024-03-04T16:04:21.990 INFO:tasks.workunit.client.0.smithi116.stdout:5/861: dwrite dd/d6d/fc7 [0,4194304] 0 2024-03-04T16:04:21.992 INFO:tasks.workunit.client.0.smithi116.stdout:8/910: truncate d2/de/d16/d78/d29/d7a/db1/fe2 40740 0 2024-03-04T16:04:21.994 INFO:tasks.workunit.client.0.smithi116.stdout:6/776: rename d6/de/d1e/f88 to d6/de/d25/d36/d44/f105 0 2024-03-04T16:04:21.995 INFO:tasks.workunit.client.0.smithi116.stdout:8/911: fsync d2/d13/d3c/d77/d97/d75/f79 0 2024-03-04T16:04:21.999 INFO:tasks.workunit.client.0.smithi116.stdout:6/777: unlink d6/d56/d58/l67 0 2024-03-04T16:04:22.001 INFO:tasks.workunit.client.0.smithi116.stdout:6/778: mknod d6/de/d25/d4a/d63/d8c/c106 0 2024-03-04T16:04:22.002 INFO:tasks.workunit.client.0.smithi116.stdout:8/912: dwrite d2/de/d16/d78/d29/d7a/fb8 [0,4194304] 0 2024-03-04T16:04:22.006 INFO:tasks.workunit.client.0.smithi116.stdout:8/913: rename d2/d13/d3c/d77/d97/c58 to d2/d4a/d114/d122/c131 0 2024-03-04T16:04:22.008 INFO:tasks.workunit.client.0.smithi116.stdout:6/779: link d6/de/d25/d4a/df8/fc0 d6/de/d15/dfc/f107 0 2024-03-04T16:04:22.013 INFO:tasks.workunit.client.0.smithi116.stdout:5/862: dwrite dd/d1f/d55/f97 [0,4194304] 0 2024-03-04T16:04:22.015 INFO:tasks.workunit.client.0.smithi116.stdout:6/780: link d6/fd5 d6/f108 0 2024-03-04T16:04:22.018 INFO:tasks.workunit.client.0.smithi116.stdout:0/727: sync 2024-03-04T16:04:22.021 INFO:tasks.workunit.client.0.smithi116.stdout:5/863: dwrite dd/d1f/d51/f117 [0,4194304] 0 2024-03-04T16:04:22.022 INFO:tasks.workunit.client.0.smithi116.stdout:5/864: chown dd/d1f/d55/d73/d2d/d59/db0/dca 35686143 1 2024-03-04T16:04:22.022 INFO:tasks.workunit.client.0.smithi116.stdout:6/781: creat d6/de/d84/db4/f109 x:0 0 0 2024-03-04T16:04:22.023 INFO:tasks.workunit.client.0.smithi116.stdout:5/865: readlink dd/d1f/d55/d73/d2d/d59/lc5 0 2024-03-04T16:04:22.025 INFO:tasks.workunit.client.0.smithi116.stdout:0/728: symlink da/d16/d19/d27/d82/le5 0 2024-03-04T16:04:22.031 INFO:tasks.workunit.client.0.smithi116.stdout:6/782: mknod d6/d56/c10a 0 2024-03-04T16:04:22.031 INFO:tasks.workunit.client.0.smithi116.stdout:0/729: dwrite da/d16/d28/d3d/f66 [4194304,4194304] 0 2024-03-04T16:04:22.037 INFO:tasks.workunit.client.0.smithi116.stdout:5/866: fsync dd/d1f/d55/d73/d2d/dae/fbe 0 2024-03-04T16:04:22.038 INFO:tasks.workunit.client.0.smithi116.stdout:5/867: chown dd/d1f/ccf 25139875 1 2024-03-04T16:04:22.040 INFO:tasks.workunit.client.0.smithi116.stdout:8/914: dread d2/de/f8 [4194304,4194304] 0 2024-03-04T16:04:22.040 INFO:tasks.workunit.client.0.smithi116.stdout:6/783: symlink d6/de/d25/d36/d44/l10b 0 2024-03-04T16:04:22.042 INFO:tasks.workunit.client.0.smithi116.stdout:0/730: truncate da/f1a 3633744 0 2024-03-04T16:04:22.051 INFO:tasks.workunit.client.0.smithi116.stdout:5/868: truncate dd/f15 1865990 0 2024-03-04T16:04:22.052 INFO:tasks.workunit.client.0.smithi116.stdout:6/784: symlink d6/de/d1e/d7c/l10c 0 2024-03-04T16:04:22.057 INFO:tasks.workunit.client.0.smithi116.stdout:6/785: creat d6/de/d29/d3a/f10d x:0 0 0 2024-03-04T16:04:22.058 INFO:tasks.workunit.client.0.smithi116.stdout:0/731: dread da/d16/d20/d42/de0/f6d [0,4194304] 0 2024-03-04T16:04:22.062 INFO:tasks.workunit.client.0.smithi116.stdout:8/915: getdents d2/de/d16/d78/d29 0 2024-03-04T16:04:22.062 INFO:tasks.workunit.client.0.smithi116.stdout:8/916: chown d2/d13/d3c/d77/fb2 28304754 1 2024-03-04T16:04:22.067 INFO:tasks.workunit.client.0.smithi116.stdout:0/732: creat da/d16/d20/d42/de0/d7e/fe6 x:0 0 0 2024-03-04T16:04:22.070 INFO:tasks.workunit.client.0.smithi116.stdout:8/917: creat d2/de/d16/d78/d29/de5/df2/da4/f132 x:0 0 0 2024-03-04T16:04:22.070 INFO:tasks.workunit.client.0.smithi116.stdout:3/936: sync 2024-03-04T16:04:22.071 INFO:tasks.workunit.client.0.smithi116.stdout:4/881: sync 2024-03-04T16:04:22.071 INFO:tasks.workunit.client.0.smithi116.stdout:2/898: sync 2024-03-04T16:04:22.071 INFO:tasks.workunit.client.0.smithi116.stdout:7/869: sync 2024-03-04T16:04:22.071 INFO:tasks.workunit.client.0.smithi116.stdout:9/789: sync 2024-03-04T16:04:22.071 INFO:tasks.workunit.client.0.smithi116.stdout:1/719: sync 2024-03-04T16:04:22.073 INFO:tasks.workunit.client.0.smithi116.stdout:0/733: dwrite da/d16/d28/d40/d48/d57/d8a/daa/fc0 [0,4194304] 0 2024-03-04T16:04:22.074 INFO:tasks.workunit.client.0.smithi116.stdout:7/870: read d3/f1f [263009,7170] 0 2024-03-04T16:04:22.075 INFO:tasks.workunit.client.0.smithi116.stdout:2/899: truncate d7/d15/d5e/d9d/df1/f14b 512998 0 2024-03-04T16:04:22.075 INFO:tasks.workunit.client.0.smithi116.stdout:0/734: read f7 [31381,9932] 0 2024-03-04T16:04:22.083 INFO:tasks.workunit.client.0.smithi116.stdout:1/720: dwrite d5/d9/d21/d27/d32/d35/f53 [0,4194304] 0 2024-03-04T16:04:22.086 INFO:tasks.workunit.client.0.smithi116.stdout:8/918: dwrite d2/fd [0,4194304] 0 2024-03-04T16:04:22.087 INFO:tasks.workunit.client.0.smithi116.stdout:0/735: dwrite da/d16/d28/d40/f5c [0,4194304] 0 2024-03-04T16:04:22.090 INFO:tasks.workunit.client.0.smithi116.stdout:4/882: chown d1/d3/d15/d20/d1c/d62/d81/daf/f10b 59 1 2024-03-04T16:04:22.095 INFO:tasks.workunit.client.0.smithi116.stdout:0/736: dwrite da/fe [0,4194304] 0 2024-03-04T16:04:22.100 INFO:tasks.workunit.client.0.smithi116.stdout:2/900: truncate d7/d13/dec/f109 1241459 0 2024-03-04T16:04:22.105 INFO:tasks.workunit.client.0.smithi116.stdout:6/786: getdents d6/de/d25/d4a/d63 0 2024-03-04T16:04:22.106 INFO:tasks.workunit.client.0.smithi116.stdout:5/869: write dd/d1f/d55/d73/d2d/d59/fac [196818,125676] 0 2024-03-04T16:04:22.108 INFO:tasks.workunit.client.0.smithi116.stdout:3/937: mknod dc/d34/c13b 0 2024-03-04T16:04:22.108 INFO:tasks.workunit.client.0.smithi116.stdout:5/870: read dd/d1f/d55/d73/d2d/d59/db0/fbc [537359,129048] 0 2024-03-04T16:04:22.109 INFO:tasks.workunit.client.0.smithi116.stdout:5/871: stat dd/l4f 0 2024-03-04T16:04:22.112 INFO:tasks.workunit.client.0.smithi116.stdout:1/721: mkdir d5/d9/d21/d84/db0/de9/dea 0 2024-03-04T16:04:22.116 INFO:tasks.workunit.client.0.smithi116.stdout:8/919: symlink d2/d13/d3c/d77/d97/d5a/dcf/l133 0 2024-03-04T16:04:22.122 INFO:tasks.workunit.client.0.smithi116.stdout:0/737: mkdir da/d16/d19/d27/d45/de7 0 2024-03-04T16:04:22.123 INFO:tasks.workunit.client.0.smithi116.stdout:0/738: read - da/d16/d20/d42/de0/d7e/fb9 zero size 2024-03-04T16:04:22.127 INFO:tasks.workunit.client.0.smithi116.stdout:6/787: chown d6/de/d15/l49 65936517 1 2024-03-04T16:04:22.129 INFO:tasks.workunit.client.0.smithi116.stdout:3/938: unlink dc/df/d47/d12d/f76 0 2024-03-04T16:04:22.130 INFO:tasks.workunit.client.0.smithi116.stdout:7/871: dwrite d3/d4/d8/d4a/d113/db6/f7c [0,4194304] 0 2024-03-04T16:04:22.135 INFO:tasks.workunit.client.0.smithi116.stdout:8/920: creat d2/de/d16/d78/d29/f134 x:0 0 0 2024-03-04T16:04:22.139 INFO:tasks.workunit.client.0.smithi116.stdout:0/739: readlink da/d16/d28/d7b/lbd 0 2024-03-04T16:04:22.199 INFO:tasks.workunit.client.0.smithi116.stdout:4/883: write d1/d3/d15/d20/d1c/fce [395715,103031] 0 2024-03-04T16:04:22.203 INFO:tasks.workunit.client.0.smithi116.stdout:7/872: mknod d3/d4/d9e/dd7/daf/dfc/c11d 0 2024-03-04T16:04:22.205 INFO:tasks.workunit.client.0.smithi116.stdout:0/740: mkdir da/d16/d28/d7b/de8 0 2024-03-04T16:04:22.206 INFO:tasks.workunit.client.0.smithi116.stdout:0/741: write da/d16/d28/d3d/f66 [1123679,5991] 0 2024-03-04T16:04:22.206 INFO:tasks.workunit.client.0.smithi116.stdout:2/901: rmdir d7/d13/dec/d13d 0 2024-03-04T16:04:22.208 INFO:tasks.workunit.client.0.smithi116.stdout:6/788: mknod d6/de/d25/d4a/c10e 0 2024-03-04T16:04:22.209 INFO:tasks.workunit.client.0.smithi116.stdout:4/884: symlink d1/d3/dd/d24/l13a 0 2024-03-04T16:04:22.210 INFO:tasks.workunit.client.0.smithi116.stdout:6/789: fsync d6/de/d84/f8f 0 2024-03-04T16:04:22.211 INFO:tasks.workunit.client.0.smithi116.stdout:4/885: dread - d1/d3/dd/d8a/ff1 zero size 2024-03-04T16:04:22.212 INFO:tasks.workunit.client.0.smithi116.stdout:7/873: dread d3/f6 [0,4194304] 0 2024-03-04T16:04:22.212 INFO:tasks.workunit.client.0.smithi116.stdout:2/902: dwrite d7/d13/dec/f101 [0,4194304] 0 2024-03-04T16:04:22.213 INFO:tasks.workunit.client.0.smithi116.stdout:4/886: fdatasync d1/d3/dd/d8a/f128 0 2024-03-04T16:04:22.218 INFO:tasks.workunit.client.0.smithi116.stdout:7/874: dwrite d3/d4/d11/f38 [0,4194304] 0 2024-03-04T16:04:22.220 INFO:tasks.workunit.client.0.smithi116.stdout:6/790: dread d6/d56/d58/f98 [0,4194304] 0 2024-03-04T16:04:22.225 INFO:tasks.workunit.client.0.smithi116.stdout:2/903: fsync d7/d15/f3f 0 2024-03-04T16:04:22.226 INFO:tasks.workunit.client.0.smithi116.stdout:6/791: dread d6/de/d25/d36/d44/dbb/ddd/d7b/dc6/fe1 [0,4194304] 0 2024-03-04T16:04:22.228 INFO:tasks.workunit.client.0.smithi116.stdout:0/742: fdatasync da/d16/d19/d27/f9d 0 2024-03-04T16:04:22.232 INFO:tasks.workunit.client.0.smithi116.stdout:6/792: dwrite d6/de/d29/d3a/f9f [0,4194304] 0 2024-03-04T16:04:22.234 INFO:tasks.workunit.client.0.smithi116.stdout:6/793: read - d6/de/d25/d4a/df8/da5/db3/fda zero size 2024-03-04T16:04:22.240 INFO:tasks.workunit.client.0.smithi116.stdout:6/794: dwrite d6/de/d84/f8f [0,4194304] 0 2024-03-04T16:04:22.242 INFO:tasks.workunit.client.0.smithi116.stdout:6/795: stat d6/de/d84/db4/d92/le6 0 2024-03-04T16:04:22.242 INFO:tasks.workunit.client.0.smithi116.stdout:6/796: chown d6/de/d15/l75 1904227 1 2024-03-04T16:04:22.245 INFO:tasks.workunit.client.0.smithi116.stdout:9/790: sync 2024-03-04T16:04:22.245 INFO:tasks.workunit.client.0.smithi116.stdout:5/872: sync 2024-03-04T16:04:22.248 INFO:tasks.workunit.client.0.smithi116.stdout:2/904: symlink d7/d13/d3e/d78/d107/l14d 0 2024-03-04T16:04:22.278 INFO:tasks.workunit.client.0.smithi116.stdout:6/797: creat d6/de/d84/f10f x:0 0 0 2024-03-04T16:04:22.281 INFO:tasks.workunit.client.0.smithi116.stdout:5/873: fdatasync dd/d1f/f29 0 2024-03-04T16:04:22.285 INFO:tasks.workunit.client.0.smithi116.stdout:9/791: mknod d2/d19/d27/dbd/cfa 0 2024-03-04T16:04:22.288 INFO:tasks.workunit.client.0.smithi116.stdout:2/905: mknod d7/d15/dd7/c14e 0 2024-03-04T16:04:22.288 INFO:tasks.workunit.client.0.smithi116.stdout:2/906: chown d7/d13/dec/f13e 0 1 2024-03-04T16:04:22.291 INFO:tasks.workunit.client.0.smithi116.stdout:6/798: symlink d6/de/d25/d4a/d63/d95/l110 0 2024-03-04T16:04:22.299 INFO:tasks.workunit.client.0.smithi116.stdout:9/792: creat d2/d26/dd4/d4a/d6a/d94/dbe/ffb x:0 0 0 2024-03-04T16:04:22.302 INFO:tasks.workunit.client.0.smithi116.stdout:2/907: rename d7/d15/dd7/f88 to d7/d25/d12f/f14f 0 2024-03-04T16:04:22.305 INFO:tasks.workunit.client.0.smithi116.stdout:6/799: write d6/de/d25/d4a/d63/d95/feb [2715013,85672] 0 2024-03-04T16:04:22.312 INFO:tasks.workunit.client.0.smithi116.stdout:5/874: dread dd/d1f/d55/d73/f3f [0,4194304] 0 2024-03-04T16:04:22.331 INFO:tasks.workunit.client.0.smithi116.stdout:6/800: mkdir d6/de/d25/d36/d4e/dfa/d111 0 2024-03-04T16:04:22.336 INFO:tasks.workunit.client.0.smithi116.stdout:6/801: dwrite d6/f45 [0,4194304] 0 2024-03-04T16:04:22.336 INFO:tasks.workunit.client.0.smithi116.stdout:0/743: rmdir da/d16/d28/d40/d48/da7 39 2024-03-04T16:04:22.345 INFO:tasks.workunit.client.0.smithi116.stdout:2/908: link d7/fb5 d7/d13/d3e/d98/dce/d47/d64/d75/f150 0 2024-03-04T16:04:22.355 INFO:tasks.workunit.client.0.smithi116.stdout:0/744: fsync da/f1a 0 2024-03-04T16:04:22.356 INFO:tasks.workunit.client.0.smithi116.stdout:0/745: stat da/d16/d19/d27/d45/l76 0 2024-03-04T16:04:22.357 INFO:tasks.workunit.client.0.smithi116.stdout:2/909: dread d7/d15/dd7/fef [0,4194304] 0 2024-03-04T16:04:22.357 INFO:tasks.workunit.client.0.smithi116.stdout:0/746: dread - da/d16/d20/d79/fdf zero size 2024-03-04T16:04:22.364 INFO:tasks.workunit.client.0.smithi116.stdout:2/910: creat d7/d13/d3e/d78/d107/f151 x:0 0 0 2024-03-04T16:04:22.367 INFO:tasks.workunit.client.0.smithi116.stdout:2/911: symlink d7/d15/d5e/d9d/d11f/l152 0 2024-03-04T16:04:22.379 INFO:tasks.workunit.client.0.smithi116.stdout:1/722: write f2 [1441902,118799] 0 2024-03-04T16:04:22.382 INFO:tasks.workunit.client.0.smithi116.stdout:1/723: unlink d5/d9/d21/d27/d94/db6/d42/d8b/fe6 0 2024-03-04T16:04:22.384 INFO:tasks.workunit.client.0.smithi116.stdout:1/724: creat d5/d9/d21/d84/db0/feb x:0 0 0 2024-03-04T16:04:22.387 INFO:tasks.workunit.client.0.smithi116.stdout:8/921: truncate d2/d13/d3c/d77/fd9 909094 0 2024-03-04T16:04:22.390 INFO:tasks.workunit.client.0.smithi116.stdout:8/922: rename d2/d13/d3c/d46/d4f/c9c to d2/d13/d3c/d77/d97/deb/c135 0 2024-03-04T16:04:22.396 INFO:tasks.workunit.client.0.smithi116.stdout:7/875: write d3/f56 [1013226,80343] 0 2024-03-04T16:04:22.411 INFO:tasks.workunit.client.0.smithi116.stdout:4/887: truncate d1/d3/de/d47/f60 1267174 0 2024-03-04T16:04:22.412 INFO:tasks.workunit.client.0.smithi116.stdout:4/888: chown d1/d3/d15/d20/d1c/d62/d81/daf/lcf 59 1 2024-03-04T16:04:22.414 INFO:tasks.workunit.client.0.smithi116.stdout:4/889: symlink d1/d3/d15/d20/d1c/d62/d112/l13b 0 2024-03-04T16:04:22.417 INFO:tasks.workunit.client.0.smithi116.stdout:4/890: symlink d1/d3/dd/d24/db4/d10c/l13c 0 2024-03-04T16:04:22.424 INFO:tasks.workunit.client.0.smithi116.stdout:4/891: dread d1/d3/f91 [0,4194304] 0 2024-03-04T16:04:22.427 INFO:tasks.workunit.client.0.smithi116.stdout:0/747: creat da/d16/d19/fe9 x:0 0 0 2024-03-04T16:04:22.427 INFO:tasks.workunit.client.0.smithi116.stdout:0/748: chown da/d16/d20/d79/f7c 477398 1 2024-03-04T16:04:22.428 INFO:tasks.workunit.client.0.smithi116.stdout:4/892: creat d1/d3/d15/d20/d1c/d62/f13d x:0 0 0 2024-03-04T16:04:22.431 INFO:tasks.workunit.client.0.smithi116.stdout:4/893: mknod d1/d3/d15/d20/d1c/ddd/c13e 0 2024-03-04T16:04:22.433 INFO:tasks.workunit.client.0.smithi116.stdout:4/894: unlink d1/d3/de/d47/c7a 0 2024-03-04T16:04:22.434 INFO:tasks.workunit.client.0.smithi116.stdout:4/895: dread - d1/d3/d15/d20/d1c/d62/f13d zero size 2024-03-04T16:04:22.435 INFO:tasks.workunit.client.0.smithi116.stdout:4/896: mknod d1/d3/d40/d52/d84/d88/d8d/c13f 0 2024-03-04T16:04:22.437 INFO:tasks.workunit.client.0.smithi116.stdout:4/897: read d1/f85 [449204,118768] 0 2024-03-04T16:04:22.439 INFO:tasks.workunit.client.0.smithi116.stdout:4/898: creat d1/d3/dd/f140 x:0 0 0 2024-03-04T16:04:22.439 INFO:tasks.workunit.client.0.smithi116.stdout:4/899: write d1/d3/dd/d24/fef [1028613,87900] 0 2024-03-04T16:04:22.440 INFO:tasks.workunit.client.0.smithi116.stdout:9/793: truncate d2/f87 2520022 0 2024-03-04T16:04:22.442 INFO:tasks.workunit.client.0.smithi116.stdout:4/900: rmdir d1/d3/d40/d52/d84/d88/d8d 39 2024-03-04T16:04:22.445 INFO:tasks.workunit.client.0.smithi116.stdout:4/901: mkdir d1/d3/d15/d20/d1c/d62/d81/d97/d141 0 2024-03-04T16:04:22.445 INFO:tasks.workunit.client.0.smithi116.stdout:9/794: link d2/d26/d28/l4b d2/d33/d65/dbf/lfc 0 2024-03-04T16:04:22.446 INFO:tasks.workunit.client.0.smithi116.stdout:4/902: fsync d1/d3/d15/d20/d1c/fce 0 2024-03-04T16:04:22.448 INFO:tasks.workunit.client.0.smithi116.stdout:7/876: rmdir d3/d4/dbf 39 2024-03-04T16:04:22.453 INFO:tasks.workunit.client.0.smithi116.stdout:5/875: write dd/d1f/fb5 [2332498,75124] 0 2024-03-04T16:04:22.453 INFO:tasks.workunit.client.0.smithi116.stdout:4/903: link d1/d3/d15/d20/d1c/d62/d81/daf/fca d1/d3/d40/d52/d84/d88/f142 0 2024-03-04T16:04:22.455 INFO:tasks.workunit.client.0.smithi116.stdout:7/877: creat d3/d4/dbf/de6/dd3/ddf/f11e x:0 0 0 2024-03-04T16:04:22.456 INFO:tasks.workunit.client.0.smithi116.stdout:4/904: mknod d1/d3/dd/d24/db4/c143 0 2024-03-04T16:04:22.458 INFO:tasks.workunit.client.0.smithi116.stdout:5/876: creat dd/d1f/d55/d73/d2d/d59/db0/f118 x:0 0 0 2024-03-04T16:04:22.459 INFO:tasks.workunit.client.0.smithi116.stdout:7/878: mkdir d3/d4/dbf/de6/dd3/ddf/dee/d10a/d11f 0 2024-03-04T16:04:22.459 INFO:tasks.workunit.client.0.smithi116.stdout:7/879: stat d3/d4/d34 0 2024-03-04T16:04:22.460 INFO:tasks.workunit.client.0.smithi116.stdout:4/905: rename d1/d3/d15/d20/d1c/d62/d81/d97/d141 to d1/d134/d144 0 2024-03-04T16:04:22.469 INFO:tasks.workunit.client.0.smithi116.stdout:4/906: write d1/d3/d15/d20/d1c/d62/d81/fe6 [2337693,23489] 0 2024-03-04T16:04:22.473 INFO:tasks.workunit.client.0.smithi116.stdout:6/802: dwrite d6/de/d5b/f8e [0,4194304] 0 2024-03-04T16:04:22.475 INFO:tasks.workunit.client.0.smithi116.stdout:2/912: write d7/d13/d3e/ded/dfd/f124 [1303148,67703] 0 2024-03-04T16:04:22.479 INFO:tasks.workunit.client.0.smithi116.stdout:4/907: dread d1/d3/d15/d20/d1c/f6a [0,4194304] 0 2024-03-04T16:04:22.484 INFO:tasks.workunit.client.0.smithi116.stdout:1/725: write d5/d9/d21/d27/d94/f95 [545608,97634] 0 2024-03-04T16:04:22.486 INFO:tasks.workunit.client.0.smithi116.stdout:4/908: dwrite d1/d3/de/d47/df9/d132/f139 [0,4194304] 0 2024-03-04T16:04:22.488 INFO:tasks.workunit.client.0.smithi116.stdout:8/923: truncate d2/de/d16/fdc 3458112 0 2024-03-04T16:04:22.491 INFO:tasks.workunit.client.0.smithi116.stdout:0/749: mkdir da/dea 0 2024-03-04T16:04:22.503 INFO:tasks.workunit.client.0.smithi116.stdout:3/939: mknod dc/df/d47/d12d/d7f/d93/db6/c13c 0 2024-03-04T16:04:22.505 INFO:tasks.workunit.client.0.smithi116.stdout:7/880: fdatasync d3/ffa 0 2024-03-04T16:04:22.509 INFO:tasks.workunit.client.0.smithi116.stdout:3/940: dwrite dc/df/d47/d12d/d7f/d93/db6/fbb [0,4194304] 0 2024-03-04T16:04:22.511 INFO:tasks.workunit.client.0.smithi116.stdout:6/803: creat d6/de/d25/d36/d4e/dfa/f112 x:0 0 0 2024-03-04T16:04:22.513 INFO:tasks.workunit.client.0.smithi116.stdout:8/924: symlink d2/d13/d5e/l136 0 2024-03-04T16:04:22.515 INFO:tasks.workunit.client.0.smithi116.stdout:4/909: mkdir d1/d6c/d145 0 2024-03-04T16:04:22.516 INFO:tasks.workunit.client.0.smithi116.stdout:4/910: chown d1/d6c/f38 4314614 1 2024-03-04T16:04:22.521 INFO:tasks.workunit.client.0.smithi116.stdout:5/877: rename dd/d1f/d55/d73/d2d/ld4 to dd/d1f/l119 0 2024-03-04T16:04:22.526 INFO:tasks.workunit.client.0.smithi116.stdout:4/911: truncate d1/d3/d15/d20/d1c/d6f/fe5 849320 0 2024-03-04T16:04:22.528 INFO:tasks.workunit.client.0.smithi116.stdout:9/795: dwrite d2/d26/dd4/f68 [0,4194304] 0 2024-03-04T16:04:22.528 INFO:tasks.workunit.client.0.smithi116.stdout:2/913: rename d7/d13/d3e/d98/dce/f4b to d7/d13/d3e/ded/dfd/f153 0 2024-03-04T16:04:22.532 INFO:tasks.workunit.client.0.smithi116.stdout:7/881: symlink d3/d4/d8/d4a/l120 0 2024-03-04T16:04:22.535 INFO:tasks.workunit.client.0.smithi116.stdout:5/878: dwrite dd/d1f/d55/d73/d2d/dc0/f10d [0,4194304] 0 2024-03-04T16:04:22.535 INFO:tasks.workunit.client.0.smithi116.stdout:7/882: write d3/d4/d11/f51 [2414224,14384] 0 2024-03-04T16:04:22.536 INFO:tasks.workunit.client.0.smithi116.stdout:8/925: fdatasync d2/de/d16/fdc 0 2024-03-04T16:04:22.540 INFO:tasks.workunit.client.0.smithi116.stdout:8/926: fdatasync d2/d13/d3c/d77/d97/d5a/f91 0 2024-03-04T16:04:22.540 INFO:tasks.workunit.client.0.smithi116.stdout:1/726: getdents d5/d9/d21/d84/db0 0 2024-03-04T16:04:22.543 INFO:tasks.workunit.client.0.smithi116.stdout:9/796: dwrite d2/d33/d65/d6c/db8/fee [0,4194304] 0 2024-03-04T16:04:22.543 INFO:tasks.workunit.client.0.smithi116.stdout:2/914: creat d7/d81/f154 x:0 0 0 2024-03-04T16:04:22.547 INFO:tasks.workunit.client.0.smithi116.stdout:0/750: rename da/d16/d28/d7b/lbd to da/d16/d19/d27/d82/d9f/leb 0 2024-03-04T16:04:22.553 INFO:tasks.workunit.client.1.smithi152.stderr:++ readlink -f fsstress 2024-03-04T16:04:22.556 INFO:tasks.workunit.client.1.smithi152.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-04T16:04:22.556 INFO:tasks.workunit.client.1.smithi152.stderr:+ popd 2024-03-04T16:04:22.557 INFO:tasks.workunit.client.0.smithi116.stdout:8/927: chown l0 411 1 2024-03-04T16:04:22.558 INFO:tasks.workunit.client.1.smithi152.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-04T16:04:22.558 INFO:tasks.workunit.client.0.smithi116.stdout:7/883: dwrite d3/d4/d9e/dd7/daf/dfc/f104 [0,4194304] 0 2024-03-04T16:04:22.558 INFO:tasks.workunit.client.1.smithi152.stderr:+ popd 2024-03-04T16:04:22.558 INFO:tasks.workunit.client.1.smithi152.stdout:~/cephtest/mnt.1/client.1/tmp 2024-03-04T16:04:22.559 INFO:tasks.workunit.client.1.smithi152.stderr:++ mktemp -d -p . 2024-03-04T16:04:22.559 INFO:tasks.workunit.client.0.smithi116.stdout:9/797: creat d2/d26/dd4/de5/ffd x:0 0 0 2024-03-04T16:04:22.560 INFO:tasks.workunit.client.0.smithi116.stdout:2/915: fsync d7/d13/f1e 0 2024-03-04T16:04:22.560 INFO:tasks.workunit.client.0.smithi116.stdout:9/798: dread - d2/d26/dd4/d4a/d6a/d94/d95/ff4 zero size 2024-03-04T16:04:22.563 INFO:tasks.workunit.client.0.smithi116.stdout:6/804: getdents d6/de/d29 0 2024-03-04T16:04:22.563 INFO:tasks.workunit.client.0.smithi116.stdout:6/805: readlink d6/de/d25/d36/l41 0 2024-03-04T16:04:22.565 INFO:tasks.workunit.client.1.smithi152.stderr:+ T=./tmp.a3eCHG0wLq 2024-03-04T16:04:22.565 INFO:tasks.workunit.client.1.smithi152.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.a3eCHG0wLq -l 1 -n 1000 -p 10 -v 2024-03-04T16:04:22.566 INFO:tasks.workunit.client.0.smithi116.stdout:0/751: fsync da/d16/d19/f68 0 2024-03-04T16:04:22.568 INFO:tasks.workunit.client.0.smithi116.stdout:5/879: mkdir dd/d45/d10c/d11a 0 2024-03-04T16:04:22.568 INFO:tasks.workunit.client.0.smithi116.stdout:0/752: write da/d16/d28/d40/d48/d57/db3/db2/dc2/fd0 [445003,73990] 0 2024-03-04T16:04:22.570 INFO:tasks.workunit.client.0.smithi116.stdout:7/884: mknod d3/d4/d9e/dd7/dfd/c121 0 2024-03-04T16:04:22.573 INFO:tasks.workunit.client.0.smithi116.stdout:3/941: rename dc/d34/l38 to dc/df/d47/d12d/d7f/d93/l13d 0 2024-03-04T16:04:22.574 INFO:tasks.workunit.client.1.smithi152.stdout:seed = 1710044728 2024-03-04T16:04:22.580 INFO:tasks.workunit.client.0.smithi116.stdout:6/806: mkdir d6/de/d25/d36/d44/dbb/ddd/d7b/d113 0 2024-03-04T16:04:22.582 INFO:tasks.workunit.client.0.smithi116.stdout:6/807: truncate d6/de/d25/d36/d44/dbb/ff7 792328 0 2024-03-04T16:04:22.583 INFO:tasks.workunit.client.0.smithi116.stdout:6/808: stat d6/de/d29/d3a/c6a 0 2024-03-04T16:04:22.584 INFO:tasks.workunit.client.1.smithi152.stdout:0/0: dwrite - no filename 2024-03-04T16:04:22.589 INFO:tasks.workunit.client.1.smithi152.stdout:1/0: dwrite - no filename 2024-03-04T16:04:22.589 INFO:tasks.workunit.client.0.smithi116.stdout:5/880: dread dd/d1f/d55/d73/d2d/f50 [0,4194304] 0 2024-03-04T16:04:22.590 INFO:tasks.workunit.client.0.smithi116.stdout:7/885: chown d3/d4/d8/d4a/l4f 7591 1 2024-03-04T16:04:22.593 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:22 smithi116 ceph-mon[29120]: pgmap v27: 65 pgs: 65 active+clean; 1.7 GiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 71 MiB/s rd, 202 MiB/s wr, 558 op/s 2024-03-04T16:04:22.596 INFO:tasks.workunit.client.1.smithi152.stdout:0/1: symlink l0 0 2024-03-04T16:04:22.596 INFO:tasks.workunit.client.1.smithi152.stdout:0/2: dwrite - no filename 2024-03-04T16:04:22.597 INFO:tasks.workunit.client.0.smithi116.stdout:2/916: symlink d7/d15/dd7/d104/l155 0 2024-03-04T16:04:22.598 INFO:tasks.workunit.client.1.smithi152.stdout:3/0: dread - no filename 2024-03-04T16:04:22.600 INFO:tasks.workunit.client.1.smithi152.stdout:1/1: symlink l0 0 2024-03-04T16:04:22.600 INFO:tasks.workunit.client.1.smithi152.stdout:1/2: readlink l0 0 2024-03-04T16:04:22.600 INFO:tasks.workunit.client.1.smithi152.stdout:1/3: readlink l0 0 2024-03-04T16:04:22.602 INFO:tasks.workunit.client.1.smithi152.stdout:0/3: creat f1 x:0 0 0 2024-03-04T16:04:22.605 INFO:tasks.workunit.client.1.smithi152.stdout:4/0: dread - no filename 2024-03-04T16:04:22.605 INFO:tasks.workunit.client.1.smithi152.stdout:2/0: creat f0 x:0 0 0 2024-03-04T16:04:22.606 INFO:tasks.workunit.client.0.smithi116.stdout:5/881: mknod dd/d1f/de5/c11b 0 2024-03-04T16:04:22.606 INFO:tasks.workunit.client.0.smithi116.stdout:5/882: stat dd/d45/c89 0 2024-03-04T16:04:22.608 INFO:tasks.workunit.client.1.smithi152.stdout:1/4: creat f1 x:0 0 0 2024-03-04T16:04:22.610 INFO:tasks.workunit.client.1.smithi152.stdout:0/4: mknod c2 0 2024-03-04T16:04:22.611 INFO:tasks.workunit.client.1.smithi152.stdout:0/5: write f1 [898011,101502] 0 2024-03-04T16:04:22.617 INFO:tasks.workunit.client.0.smithi116.stdout:6/809: mknod d6/de/d25/d4a/df8/da5/c114 0 2024-03-04T16:04:22.618 INFO:tasks.workunit.client.1.smithi152.stdout:3/1: creat f0 x:0 0 0 2024-03-04T16:04:22.618 INFO:tasks.workunit.client.0.smithi116.stdout:4/912: write d1/d3/d40/d52/d84/d88/d8d/fb3 [603140,88081] 0 2024-03-04T16:04:22.619 INFO:tasks.workunit.client.1.smithi152.stdout:3/2: write f0 [366859,122047] 0 2024-03-04T16:04:22.619 INFO:tasks.workunit.client.0.smithi116.stdout:0/753: rmdir da/dea 0 2024-03-04T16:04:22.620 INFO:tasks.workunit.client.1.smithi152.stdout:3/3: write f0 [122058,107849] 0 2024-03-04T16:04:22.621 INFO:tasks.workunit.client.1.smithi152.stdout:1/5: creat f2 x:0 0 0 2024-03-04T16:04:22.621 INFO:tasks.workunit.client.1.smithi152.stdout:1/6: chown f2 14863937 1 2024-03-04T16:04:22.621 INFO:tasks.workunit.client.1.smithi152.stdout:1/7: rmdir - no directory 2024-03-04T16:04:22.623 INFO:tasks.workunit.client.1.smithi152.stdout:0/6: mknod c3 0 2024-03-04T16:04:22.627 INFO:tasks.workunit.client.1.smithi152.stdout:0/7: dread f1 [0,4194304] 0 2024-03-04T16:04:22.627 INFO:tasks.workunit.client.1.smithi152.stdout:6/0: chown . 1 1 2024-03-04T16:04:22.630 INFO:tasks.workunit.client.0.smithi116.stdout:1/727: rename d5/d9/d21/d27/d32/f3e to d5/d9/d21/fec 0 2024-03-04T16:04:22.630 INFO:tasks.workunit.client.1.smithi152.stdout:4/1: creat f0 x:0 0 0 2024-03-04T16:04:22.630 INFO:tasks.workunit.client.1.smithi152.stdout:4/2: rmdir - no directory 2024-03-04T16:04:22.633 INFO:tasks.workunit.client.0.smithi116.stdout:9/799: write d2/d26/dd4/de5/f43 [1148971,109291] 0 2024-03-04T16:04:22.636 INFO:tasks.workunit.client.0.smithi116.stdout:4/913: creat d1/d3/dd/dd9/d11a/f146 x:0 0 0 2024-03-04T16:04:22.636 INFO:tasks.workunit.client.1.smithi152.stdout:3/4: creat f1 x:0 0 0 2024-03-04T16:04:22.638 INFO:tasks.workunit.client.0.smithi116.stdout:0/754: chown da/d16/d20/d42/c4b 215262042 1 2024-03-04T16:04:22.639 INFO:tasks.workunit.client.0.smithi116.stdout:7/886: link d3/d4/d8/d1a/f29 d3/d4/d9e/dd7/dfd/f122 0 2024-03-04T16:04:22.643 INFO:tasks.workunit.client.1.smithi152.stdout:7/0: dwrite - no filename 2024-03-04T16:04:22.643 INFO:tasks.workunit.client.1.smithi152.stdout:3/5: dwrite f1 [0,4194304] 0 2024-03-04T16:04:22.645 INFO:tasks.workunit.client.0.smithi116.stdout:1/728: creat d5/d9/d21/d24/de0/fed x:0 0 0 2024-03-04T16:04:22.648 INFO:tasks.workunit.client.1.smithi152.stdout:5/0: mknod c0 0 2024-03-04T16:04:22.649 INFO:tasks.workunit.client.0.smithi116.stdout:1/729: truncate d5/d9/f23 2558358 0 2024-03-04T16:04:22.649 INFO:tasks.workunit.client.0.smithi116.stdout:4/914: dread d1/d3/d15/d20/d1c/f27 [0,4194304] 0 2024-03-04T16:04:22.649 INFO:tasks.workunit.client.0.smithi116.stdout:3/942: dwrite dc/df/d47/d12d/d7c/f69 [0,4194304] 0 2024-03-04T16:04:22.650 INFO:tasks.workunit.client.0.smithi116.stdout:1/730: fsync d5/d9/d21/d27/d94/f95 0 2024-03-04T16:04:22.650 INFO:tasks.workunit.client.1.smithi152.stdout:4/3: symlink l1 0 2024-03-04T16:04:22.653 INFO:tasks.workunit.client.0.smithi116.stdout:9/800: stat d2/d26/dd4/c58 0 2024-03-04T16:04:22.653 INFO:tasks.workunit.client.0.smithi116.stdout:1/731: dread - d5/d9/d21/d27/d32/d35/d55/db4/d50/f8e zero size 2024-03-04T16:04:22.656 INFO:tasks.workunit.client.0.smithi116.stdout:4/915: dwrite d1/d3/d15/fb6 [0,4194304] 0 2024-03-04T16:04:22.656 INFO:tasks.workunit.client.0.smithi116.stdout:1/732: chown d5/d9/d21/d27/d94/c99 319394 1 2024-03-04T16:04:22.658 INFO:tasks.workunit.client.0.smithi116.stdout:1/733: fdatasync d5/d9/d21/d27/d94/db6/f69 0 2024-03-04T16:04:22.659 INFO:tasks.workunit.client.0.smithi116.stdout:2/917: getdents d7/d13/d3e 0 2024-03-04T16:04:22.661 INFO:tasks.workunit.client.0.smithi116.stdout:1/734: read d5/d9/d21/d27/d32/d35/d55/fb8 [806955,70155] 0 2024-03-04T16:04:22.663 INFO:tasks.workunit.client.1.smithi152.stdout:3/6: mknod c2 0 2024-03-04T16:04:22.664 INFO:tasks.workunit.client.0.smithi116.stdout:6/810: link d6/de/d15/l75 d6/de/d25/d36/d4e/dfa/l115 0 2024-03-04T16:04:22.665 INFO:tasks.workunit.client.1.smithi152.stdout:6/1: creat f0 x:0 0 0 2024-03-04T16:04:22.665 INFO:tasks.workunit.client.1.smithi152.stdout:6/2: stat f0 0 2024-03-04T16:04:22.666 INFO:tasks.workunit.client.0.smithi116.stdout:6/811: write d6/de/d25/d36/d44/fba [2079420,82403] 0 2024-03-04T16:04:22.667 INFO:tasks.workunit.client.0.smithi116.stdout:1/735: dwrite d5/d9/d21/d24/de0/f6d [0,4194304] 0 2024-03-04T16:04:22.667 INFO:tasks.workunit.client.0.smithi116.stdout:3/943: unlink dc/df/faf 0 2024-03-04T16:04:22.668 INFO:tasks.workunit.client.0.smithi116.stdout:3/944: fsync dc/d34/de7/f13a 0 2024-03-04T16:04:22.669 INFO:tasks.workunit.client.1.smithi152.stdout:5/1: mknod c1 0 2024-03-04T16:04:22.669 INFO:tasks.workunit.client.1.smithi152.stdout:5/2: write - no filename 2024-03-04T16:04:22.670 INFO:tasks.workunit.client.0.smithi116.stdout:3/945: truncate dc/d34/dc1/fcb 2293838 0 2024-03-04T16:04:22.671 INFO:tasks.workunit.client.1.smithi152.stdout:4/4: mknod c2 0 2024-03-04T16:04:22.672 INFO:tasks.workunit.client.1.smithi152.stdout:4/5: readlink l1 0 2024-03-04T16:04:22.674 INFO:tasks.workunit.client.1.smithi152.stdout:4/6: write f0 [622392,85731] 0 2024-03-04T16:04:22.674 INFO:tasks.workunit.client.0.smithi116.stdout:9/801: mkdir d2/d26/dd4/d4a/d6a/d94/d95/dfe 0 2024-03-04T16:04:22.675 INFO:tasks.workunit.client.1.smithi152.stdout:6/3: dwrite f0 [0,4194304] 0 2024-03-04T16:04:22.676 INFO:tasks.workunit.client.0.smithi116.stdout:9/802: chown d2/d26/dd4/d4a/d6a/d94/dbe/cf8 1560 1 2024-03-04T16:04:22.676 INFO:tasks.workunit.client.1.smithi152.stdout:2/1: getdents . 0 2024-03-04T16:04:22.677 INFO:tasks.workunit.client.0.smithi116.stdout:0/755: mkdir da/d16/d28/d7b/de8/dec 0 2024-03-04T16:04:22.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/0: unlink - no file 2024-03-04T16:04:22.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/1: dwrite - no filename 2024-03-04T16:04:22.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/2: read - no filename 2024-03-04T16:04:22.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/3: link - no file 2024-03-04T16:04:22.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/4: read - no filename 2024-03-04T16:04:22.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/5: dwrite - no filename 2024-03-04T16:04:22.682 INFO:tasks.workunit.client.0.smithi116.stdout:4/916: creat d1/d6e/d74/f147 x:0 0 0 2024-03-04T16:04:22.684 INFO:tasks.workunit.client.1.smithi152.stdout:3/7: creat f3 x:0 0 0 2024-03-04T16:04:22.685 INFO:tasks.workunit.client.1.smithi152.stdout:3/8: chown f1 338293 1 2024-03-04T16:04:22.687 INFO:tasks.workunit.client.0.smithi116.stdout:2/918: creat d7/d13/d3e/d98/f156 x:0 0 0 2024-03-04T16:04:22.688 INFO:tasks.workunit.client.1.smithi152.stdout:5/3: mknod c2 0 2024-03-04T16:04:22.696 INFO:tasks.workunit.client.1.smithi152.stdout:3/9: dwrite f1 [4194304,4194304] 0 2024-03-04T16:04:22.697 INFO:tasks.workunit.client.1.smithi152.stdout:3/10: chown f3 21429998 1 2024-03-04T16:04:22.698 INFO:tasks.workunit.client.1.smithi152.stdout:2/2: creat f1 x:0 0 0 2024-03-04T16:04:22.698 INFO:tasks.workunit.client.0.smithi116.stdout:6/812: dread d6/de/d25/d4a/d63/fe0 [0,4194304] 0 2024-03-04T16:04:22.698 INFO:tasks.workunit.client.0.smithi116.stdout:3/946: fdatasync dc/df/d47/d12d/d7f/d93/fb1 0 2024-03-04T16:04:22.700 INFO:tasks.workunit.client.0.smithi116.stdout:3/947: chown dc/df/d47/d12d/d7f/d93/db6/f128 2144782901 1 2024-03-04T16:04:22.700 INFO:tasks.workunit.client.1.smithi152.stdout:1/8: getdents . 0 2024-03-04T16:04:22.703 INFO:tasks.workunit.client.0.smithi116.stdout:0/756: rmdir da/d16/d20 39 2024-03-04T16:04:22.705 INFO:tasks.workunit.client.1.smithi152.stdout:2/3: dwrite f1 [0,4194304] 0 2024-03-04T16:04:22.705 INFO:tasks.workunit.client.1.smithi152.stdout:8/0: symlink l0 0 2024-03-04T16:04:22.705 INFO:tasks.workunit.client.1.smithi152.stdout:8/1: read - no filename 2024-03-04T16:04:22.708 INFO:tasks.workunit.client.0.smithi116.stdout:5/883: rename dd/d1f/d51/lbd to dd/l11c 0 2024-03-04T16:04:22.713 INFO:tasks.workunit.client.1.smithi152.stdout:5/4: mknod c3 0 2024-03-04T16:04:22.713 INFO:tasks.workunit.client.1.smithi152.stdout:5/5: readlink - no filename 2024-03-04T16:04:22.714 INFO:tasks.workunit.client.1.smithi152.stdout:5/6: write - no filename 2024-03-04T16:04:22.714 INFO:tasks.workunit.client.0.smithi116.stdout:3/948: creat dc/df/d47/d12d/d7f/de8/f13e x:0 0 0 2024-03-04T16:04:22.716 INFO:tasks.workunit.client.1.smithi152.stdout:3/11: mknod c4 0 2024-03-04T16:04:22.718 INFO:tasks.workunit.client.0.smithi116.stdout:0/757: unlink da/d16/d28/l8b 0 2024-03-04T16:04:22.719 INFO:tasks.workunit.client.1.smithi152.stdout:3/12: dread f0 [0,4194304] 0 2024-03-04T16:04:22.720 INFO:tasks.workunit.client.0.smithi116.stdout:7/887: rename d3/d4/d8/d52/d5e to d3/d4/d9e/dd7/daf/dd5/d123 0 2024-03-04T16:04:22.721 INFO:tasks.workunit.client.1.smithi152.stdout:9/6: mknod c0 0 2024-03-04T16:04:22.722 INFO:tasks.workunit.client.0.smithi116.stdout:5/884: mkdir dd/d1f/d51/d11d 0 2024-03-04T16:04:22.723 INFO:tasks.workunit.client.0.smithi116.stdout:4/917: mknod d1/d3/d15/c148 0 2024-03-04T16:04:22.724 INFO:tasks.workunit.client.1.smithi152.stdout:8/2: symlink l1 0 2024-03-04T16:04:22.724 INFO:tasks.workunit.client.1.smithi152.stdout:8/3: chown l0 3375199 1 2024-03-04T16:04:22.725 INFO:tasks.workunit.client.0.smithi116.stdout:3/949: symlink dc/df/d47/d12d/d7c/d12c/l13f 0 2024-03-04T16:04:22.725 INFO:tasks.workunit.client.0.smithi116.stdout:0/758: dread da/d16/d19/f1d [0,4194304] 0 2024-03-04T16:04:22.726 INFO:tasks.workunit.client.1.smithi152.stdout:5/7: rename c1 to c4 0 2024-03-04T16:04:22.726 INFO:tasks.workunit.client.0.smithi116.stdout:4/918: stat d1/d3/d15/d20/d1c/d62/d81/da6/le3 0 2024-03-04T16:04:22.727 INFO:tasks.workunit.client.0.smithi116.stdout:1/736: rename d5/d9/d21/d27/d32/d35/l51 to d5/d9/d21/d27/d32/d35/dbf/lee 0 2024-03-04T16:04:22.729 INFO:tasks.workunit.client.0.smithi116.stdout:5/885: dwrite dd/d1f/d51/f117 [0,4194304] 0 2024-03-04T16:04:22.729 INFO:tasks.workunit.client.0.smithi116.stdout:7/888: truncate d3/d4/d8/d4a/f84 66974 0 2024-03-04T16:04:22.732 INFO:tasks.workunit.client.0.smithi116.stdout:7/889: write d3/d4/d8/f10c [737048,75123] 0 2024-03-04T16:04:22.735 INFO:tasks.workunit.client.0.smithi116.stdout:3/950: dwrite dc/df/d47/d12d/d7f/d8b/f12a [0,4194304] 0 2024-03-04T16:04:22.736 INFO:tasks.workunit.client.0.smithi116.stdout:0/759: readlink da/d16/d20/d79/l80 0 2024-03-04T16:04:22.740 INFO:tasks.workunit.client.0.smithi116.stdout:4/919: mkdir d1/d3/dd/d8a/d149 0 2024-03-04T16:04:22.740 INFO:tasks.workunit.client.0.smithi116.stdout:9/803: truncate d2/d26/dd4/d4a/f73 946962 0 2024-03-04T16:04:22.740 INFO:tasks.workunit.client.1.smithi152.stdout:9/7: mknod c1 0 2024-03-04T16:04:22.741 INFO:tasks.workunit.client.1.smithi152.stdout:8/4: mkdir d2 0 2024-03-04T16:04:22.741 INFO:tasks.workunit.client.1.smithi152.stdout:8/5: dread - no filename 2024-03-04T16:04:22.746 INFO:tasks.workunit.client.1.smithi152.stdout:5/8: symlink l5 0 2024-03-04T16:04:22.749 INFO:tasks.workunit.client.0.smithi116.stdout:1/737: creat d5/d9/d21/d27/d32/d35/d55/d75/fef x:0 0 0 2024-03-04T16:04:22.749 INFO:tasks.workunit.client.1.smithi152.stdout:4/7: getdents . 0 2024-03-04T16:04:22.750 INFO:tasks.workunit.client.0.smithi116.stdout:1/738: fsync d5/d9/d21/d27/d32/d35/d55/dae/fd0 0 2024-03-04T16:04:22.752 INFO:tasks.workunit.client.0.smithi116.stdout:8/928: sync 2024-03-04T16:04:22.752 INFO:tasks.workunit.client.1.smithi152.stdout:6/4: getdents . 0 2024-03-04T16:04:22.752 INFO:tasks.workunit.client.1.smithi152.stdout:6/5: truncate f0 4247135 0 2024-03-04T16:04:22.753 INFO:tasks.workunit.client.0.smithi116.stdout:7/890: rmdir d3/d4/d8/d4a/d88/dc2 39 2024-03-04T16:04:22.756 INFO:tasks.workunit.client.1.smithi152.stdout:1/9: link l0 l3 0 2024-03-04T16:04:22.758 INFO:tasks.workunit.client.0.smithi116.stdout:0/760: creat da/d16/d20/d42/de0/d7e/fed x:0 0 0 2024-03-04T16:04:22.758 INFO:tasks.workunit.client.0.smithi116.stdout:0/761: chown da/d16/d28/d40/d48 202828799 1 2024-03-04T16:04:22.759 INFO:tasks.workunit.client.1.smithi152.stdout:5/9: mkdir d6 0 2024-03-04T16:04:22.759 INFO:tasks.workunit.client.1.smithi152.stdout:5/10: dwrite - no filename 2024-03-04T16:04:22.759 INFO:tasks.workunit.client.1.smithi152.stdout:5/11: truncate - no filename 2024-03-04T16:04:22.760 INFO:tasks.workunit.client.0.smithi116.stdout:9/804: readlink d2/d26/d28/l4b 0 2024-03-04T16:04:22.761 INFO:tasks.workunit.client.1.smithi152.stdout:4/8: creat f3 x:0 0 0 2024-03-04T16:04:22.762 INFO:tasks.workunit.client.0.smithi116.stdout:2/919: rename d7/fb5 to d7/d13/d3e/ded/d6c/da4/da1/f157 0 2024-03-04T16:04:22.763 INFO:tasks.workunit.client.1.smithi152.stdout:6/6: creat f1 x:0 0 0 2024-03-04T16:04:22.764 INFO:tasks.workunit.client.0.smithi116.stdout:2/920: fsync d7/d13/d3e/d71/f136 0 2024-03-04T16:04:22.765 INFO:tasks.workunit.client.1.smithi152.stdout:1/10: mknod c4 0 2024-03-04T16:04:22.765 INFO:tasks.workunit.client.1.smithi152.stdout:1/11: chown f1 0 1 2024-03-04T16:04:22.766 INFO:tasks.workunit.client.1.smithi152.stdout:1/12: dread - f2 zero size 2024-03-04T16:04:22.767 INFO:tasks.workunit.client.0.smithi116.stdout:1/739: mknod d5/d9/d21/d27/d32/d35/d55/db4/d50/cf0 0 2024-03-04T16:04:22.767 INFO:tasks.workunit.client.1.smithi152.stdout:8/6: getdents d2 0 2024-03-04T16:04:22.768 INFO:tasks.workunit.client.1.smithi152.stdout:8/7: chown l1 165 1 2024-03-04T16:04:22.768 INFO:tasks.workunit.client.1.smithi152.stdout:8/8: chown d2 5121742 1 2024-03-04T16:04:22.768 INFO:tasks.workunit.client.1.smithi152.stdout:8/9: chown l1 1915 1 2024-03-04T16:04:22.769 INFO:tasks.workunit.client.0.smithi116.stdout:0/762: dread da/f1a [0,4194304] 0 2024-03-04T16:04:22.770 INFO:tasks.workunit.client.0.smithi116.stdout:7/891: creat d3/d4/d9e/dd7/f124 x:0 0 0 2024-03-04T16:04:22.772 INFO:tasks.workunit.client.1.smithi152.stdout:4/9: creat f4 x:0 0 0 2024-03-04T16:04:22.773 INFO:tasks.workunit.client.0.smithi116.stdout:0/763: dread da/d16/d28/d40/d48/d57/db3/db2/dc2/fd0 [0,4194304] 0 2024-03-04T16:04:22.773 INFO:tasks.workunit.client.0.smithi116.stdout:0/764: dread - da/d16/d20/d42/de0/d7e/fb9 zero size 2024-03-04T16:04:22.774 INFO:tasks.workunit.client.0.smithi116.stdout:4/920: fdatasync d1/d3/d15/d20/d1c/d6f/fe5 0 2024-03-04T16:04:22.774 INFO:tasks.workunit.client.0.smithi116.stdout:4/921: chown d1/d3/d15/d20/l64 0 1 2024-03-04T16:04:22.774 INFO:tasks.workunit.client.0.smithi116.stdout:0/765: chown da/d16/d20/d42/de0/f6d 20616388 1 2024-03-04T16:04:22.775 INFO:tasks.workunit.client.1.smithi152.stdout:6/7: mknod c2 0 2024-03-04T16:04:22.775 INFO:tasks.workunit.client.0.smithi116.stdout:4/922: read d1/d3/dd/d45/f90 [972777,36807] 0 2024-03-04T16:04:22.776 INFO:tasks.workunit.client.0.smithi116.stdout:9/805: mknod d2/d33/d65/d6c/cff 0 2024-03-04T16:04:22.777 INFO:tasks.workunit.client.0.smithi116.stdout:9/806: read - d2/d33/d65/d6c/db8/de3/fe7 zero size 2024-03-04T16:04:22.778 INFO:tasks.workunit.client.0.smithi116.stdout:9/807: truncate d2/d26/f71 2222277 0 2024-03-04T16:04:22.778 INFO:tasks.workunit.client.0.smithi116.stdout:9/808: stat d2/d33/d65/fea 0 2024-03-04T16:04:22.779 INFO:tasks.workunit.client.0.smithi116.stdout:6/813: rename d6/de/d25/d36/d44/dbb/ddd/d7b/f102 to d6/de/d25/d36/d4e/dfa/d111/f116 0 2024-03-04T16:04:22.780 INFO:tasks.workunit.client.0.smithi116.stdout:9/809: write d2/d26/dd4/d4a/d6a/d94/d95/ff4 [26124,78245] 0 2024-03-04T16:04:22.780 INFO:tasks.workunit.client.0.smithi116.stdout:4/923: dread d1/d3/dd/dd9/d11a/f99 [0,4194304] 0 2024-03-04T16:04:22.781 INFO:tasks.workunit.client.1.smithi152.stdout:4/10: dwrite f3 [0,4194304] 0 2024-03-04T16:04:22.781 INFO:tasks.workunit.client.0.smithi116.stdout:5/886: creat dd/f11e x:0 0 0 2024-03-04T16:04:22.782 INFO:tasks.workunit.client.1.smithi152.stdout:4/11: write f4 [718240,28663] 0 2024-03-04T16:04:22.782 INFO:tasks.workunit.client.1.smithi152.stdout:4/12: rmdir - no directory 2024-03-04T16:04:22.783 INFO:tasks.workunit.client.1.smithi152.stdout:5/12: creat d6/f7 x:0 0 0 2024-03-04T16:04:22.785 INFO:tasks.workunit.client.0.smithi116.stdout:2/921: truncate d7/d13/d3e/d98/dce/d47/f49 2893398 0 2024-03-04T16:04:22.785 INFO:tasks.workunit.client.1.smithi152.stdout:5/13: fdatasync d6/f7 0 2024-03-04T16:04:22.786 INFO:tasks.workunit.client.0.smithi116.stdout:6/814: dwrite d6/de/d84/f8f [0,4194304] 0 2024-03-04T16:04:22.786 INFO:tasks.workunit.client.0.smithi116.stdout:2/922: write d7/d13/d3e/d98/f156 [558332,17622] 0 2024-03-04T16:04:22.787 INFO:tasks.workunit.client.1.smithi152.stdout:6/8: creat f3 x:0 0 0 2024-03-04T16:04:22.790 INFO:tasks.workunit.client.1.smithi152.stdout:1/13: link f2 f5 0 2024-03-04T16:04:22.791 INFO:tasks.workunit.client.1.smithi152.stdout:1/14: dread - f2 zero size 2024-03-04T16:04:22.792 INFO:tasks.workunit.client.1.smithi152.stdout:4/13: dwrite f4 [0,4194304] 0 2024-03-04T16:04:22.798 INFO:tasks.workunit.client.1.smithi152.stdout:8/10: mkdir d2/d3 0 2024-03-04T16:04:22.798 INFO:tasks.workunit.client.1.smithi152.stdout:8/11: fdatasync - no filename 2024-03-04T16:04:22.801 INFO:tasks.workunit.client.1.smithi152.stdout:5/14: mkdir d6/d8 0 2024-03-04T16:04:22.802 INFO:tasks.workunit.client.1.smithi152.stdout:6/9: symlink l4 0 2024-03-04T16:04:22.803 INFO:tasks.workunit.client.1.smithi152.stdout:6/10: stat c2 0 2024-03-04T16:04:22.803 INFO:tasks.workunit.client.0.smithi116.stdout:0/766: mkdir da/d16/d28/d40/d48/d57/db3/db2/dc2/dee 0 2024-03-04T16:04:22.809 INFO:tasks.workunit.client.1.smithi152.stdout:4/14: creat f5 x:0 0 0 2024-03-04T16:04:22.810 INFO:tasks.workunit.client.1.smithi152.stdout:1/15: write f2 [198947,13192] 0 2024-03-04T16:04:22.810 INFO:tasks.workunit.client.1.smithi152.stdout:8/12: symlink d2/l4 0 2024-03-04T16:04:22.810 INFO:tasks.workunit.client.1.smithi152.stdout:8/13: truncate - no filename 2024-03-04T16:04:22.810 INFO:tasks.workunit.client.1.smithi152.stdout:8/14: dwrite - no filename 2024-03-04T16:04:22.810 INFO:tasks.workunit.client.1.smithi152.stdout:8/15: write - no filename 2024-03-04T16:04:22.812 INFO:tasks.workunit.client.0.smithi116.stdout:8/929: rename d2/d13/d5e/f87 to d2/d13/d3c/d77/f137 0 2024-03-04T16:04:22.814 INFO:tasks.workunit.client.1.smithi152.stdout:8/16: chown l0 467146 1 2024-03-04T16:04:22.814 INFO:tasks.workunit.client.1.smithi152.stdout:5/15: dwrite d6/f7 [0,4194304] 0 2024-03-04T16:04:22.815 INFO:tasks.workunit.client.1.smithi152.stdout:4/15: write f0 [1608354,109269] 0 2024-03-04T16:04:22.816 INFO:tasks.workunit.client.1.smithi152.stdout:8/17: stat l0 0 2024-03-04T16:04:22.819 INFO:tasks.workunit.client.0.smithi116.stdout:5/887: dread dd/d1f/d72/fa0 [0,4194304] 0 2024-03-04T16:04:22.819 INFO:tasks.workunit.client.1.smithi152.stdout:4/16: chown f4 6691 1 2024-03-04T16:04:22.820 INFO:tasks.workunit.client.0.smithi116.stdout:4/924: symlink d1/d3/d15/d20/d1c/d6f/l14a 0 2024-03-04T16:04:22.821 INFO:tasks.workunit.client.1.smithi152.stdout:6/11: dwrite f3 [0,4194304] 0 2024-03-04T16:04:22.827 INFO:tasks.workunit.client.1.smithi152.stdout:1/16: dwrite f1 [0,4194304] 0 2024-03-04T16:04:22.828 INFO:tasks.workunit.client.0.smithi116.stdout:9/810: truncate d2/d33/d40/f8b 370229 0 2024-03-04T16:04:22.828 INFO:tasks.workunit.client.0.smithi116.stdout:2/923: mkdir d7/d13/d3e/d71/dad/d147/d158 0 2024-03-04T16:04:22.829 INFO:tasks.workunit.client.1.smithi152.stdout:5/16: symlink d6/l9 0 2024-03-04T16:04:22.829 INFO:tasks.workunit.client.1.smithi152.stdout:1/17: truncate f5 482664 0 2024-03-04T16:04:22.832 INFO:tasks.workunit.client.0.smithi116.stdout:1/740: mknod d5/d9/d21/d24/de0/d4a/cf1 0 2024-03-04T16:04:22.834 INFO:tasks.workunit.client.0.smithi116.stdout:9/811: dread d2/d26/d28/f81 [0,4194304] 0 2024-03-04T16:04:22.834 INFO:tasks.workunit.client.0.smithi116.stdout:5/888: dread dd/d1f/d72/d8e/f98 [0,4194304] 0 2024-03-04T16:04:22.834 INFO:tasks.workunit.client.1.smithi152.stdout:4/17: mknod c6 0 2024-03-04T16:04:22.837 INFO:tasks.workunit.client.1.smithi152.stdout:6/12: symlink l5 0 2024-03-04T16:04:22.841 INFO:tasks.workunit.client.0.smithi116.stdout:3/951: getdents dc/d34/de7 0 2024-03-04T16:04:22.842 INFO:tasks.workunit.client.1.smithi152.stdout:8/18: creat d2/d3/f5 x:0 0 0 2024-03-04T16:04:22.843 INFO:tasks.workunit.client.1.smithi152.stdout:8/19: dread - d2/d3/f5 zero size 2024-03-04T16:04:22.844 INFO:tasks.workunit.client.1.smithi152.stdout:8/20: write d2/d3/f5 [431345,38836] 0 2024-03-04T16:04:22.846 INFO:tasks.workunit.client.1.smithi152.stdout:1/18: creat f6 x:0 0 0 2024-03-04T16:04:22.849 INFO:tasks.workunit.client.1.smithi152.stdout:5/17: creat d6/fa x:0 0 0 2024-03-04T16:04:22.855 INFO:tasks.workunit.client.1.smithi152.stdout:5/18: dread d6/f7 [0,4194304] 0 2024-03-04T16:04:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:22 smithi152 ceph-mon[36876]: pgmap v27: 65 pgs: 65 active+clean; 1.7 GiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 71 MiB/s rd, 202 MiB/s wr, 558 op/s 2024-03-04T16:04:22.858 INFO:tasks.workunit.client.1.smithi152.stdout:8/21: unlink d2/d3/f5 0 2024-03-04T16:04:22.858 INFO:tasks.workunit.client.1.smithi152.stdout:1/19: dwrite f5 [0,4194304] 0 2024-03-04T16:04:22.859 INFO:tasks.workunit.client.1.smithi152.stdout:1/20: chown f2 597632 1 2024-03-04T16:04:22.860 INFO:tasks.workunit.client.0.smithi116.stdout:4/925: creat d1/d3/d40/f14b x:0 0 0 2024-03-04T16:04:22.862 INFO:tasks.workunit.client.0.smithi116.stdout:2/924: mkdir d7/d13/d3e/d98/dce/d47/d64/d75/d77/d12e/d159 0 2024-03-04T16:04:22.865 INFO:tasks.workunit.client.0.smithi116.stdout:1/741: mkdir d5/d9/d21/d27/d94/df2 0 2024-03-04T16:04:22.865 INFO:tasks.workunit.client.0.smithi116.stdout:1/742: stat d5/d9/d21/d24/de0/d4a/l6e 0 2024-03-04T16:04:22.870 INFO:tasks.workunit.client.0.smithi116.stdout:3/952: mkdir dc/df/d47/d12d/d7f/d93/db6/dc5/de0/da3/d140 0 2024-03-04T16:04:22.876 INFO:tasks.workunit.client.1.smithi152.stdout:8/22: creat d2/d3/f6 x:0 0 0 2024-03-04T16:04:22.877 INFO:tasks.workunit.client.1.smithi152.stdout:8/23: fdatasync d2/d3/f6 0 2024-03-04T16:04:22.878 INFO:tasks.workunit.client.1.smithi152.stdout:8/24: write d2/d3/f6 [592985,112688] 0 2024-03-04T16:04:22.879 INFO:tasks.workunit.client.0.smithi116.stdout:1/743: creat d5/d9/d21/d24/de0/d4c/d67/ff3 x:0 0 0 2024-03-04T16:04:22.881 INFO:tasks.workunit.client.0.smithi116.stdout:5/889: mkdir dd/d1f/d11f 0 2024-03-04T16:04:22.883 INFO:tasks.workunit.client.0.smithi116.stdout:0/767: write da/d16/d28/d40/d48/d57/db3/db2/fb6 [1758127,47145] 0 2024-03-04T16:04:22.884 INFO:tasks.workunit.client.1.smithi152.stdout:5/19: mknod d6/d8/cb 0 2024-03-04T16:04:22.884 INFO:tasks.workunit.client.1.smithi152.stdout:5/20: fdatasync d6/fa 0 2024-03-04T16:04:22.885 INFO:tasks.workunit.client.0.smithi116.stdout:0/768: read - da/d16/f93 zero size 2024-03-04T16:04:22.885 INFO:tasks.workunit.client.1.smithi152.stdout:5/21: read - d6/fa zero size 2024-03-04T16:04:22.886 INFO:tasks.workunit.client.1.smithi152.stdout:5/22: write d6/fa [252945,94287] 0 2024-03-04T16:04:22.887 INFO:tasks.workunit.client.0.smithi116.stdout:1/744: dwrite d5/d9/d21/d27/d32/d35/d55/fc0 [0,4194304] 0 2024-03-04T16:04:22.888 INFO:tasks.workunit.client.0.smithi116.stdout:0/769: chown da/d16/d28/d40/d48/d57/d8a/daa/dba 21381 1 2024-03-04T16:04:22.891 INFO:tasks.workunit.client.1.smithi152.stdout:6/13: getdents . 0 2024-03-04T16:04:22.894 INFO:tasks.workunit.client.0.smithi116.stdout:9/812: link d2/d26/dd4/d4a/f7b d2/d26/dd4/f100 0 2024-03-04T16:04:22.898 INFO:tasks.workunit.client.1.smithi152.stdout:4/18: truncate f0 1177986 0 2024-03-04T16:04:22.901 INFO:tasks.workunit.client.1.smithi152.stdout:1/21: getdents . 0 2024-03-04T16:04:22.903 INFO:tasks.workunit.client.0.smithi116.stdout:2/925: dread d7/d13/d3e/d78/f7e [0,4194304] 0 2024-03-04T16:04:22.903 INFO:tasks.workunit.client.0.smithi116.stdout:3/953: rename dc/df/c26 to dc/d34/de7/deb/c141 0 2024-03-04T16:04:22.903 INFO:tasks.workunit.client.1.smithi152.stdout:5/23: dwrite d6/f7 [0,4194304] 0 2024-03-04T16:04:22.905 INFO:tasks.workunit.client.0.smithi116.stdout:2/926: write d7/d15/d35/f8f [506753,113432] 0 2024-03-04T16:04:22.906 INFO:tasks.workunit.client.0.smithi116.stdout:2/927: dread - d7/d13/f141 zero size 2024-03-04T16:04:22.908 INFO:tasks.workunit.client.0.smithi116.stdout:2/928: dread - d7/d13/dec/f13e zero size 2024-03-04T16:04:22.908 INFO:tasks.workunit.client.1.smithi152.stdout:5/24: dread d6/f7 [0,4194304] 0 2024-03-04T16:04:22.909 INFO:tasks.workunit.client.0.smithi116.stdout:1/745: creat d5/d86/dac/ff4 x:0 0 0 2024-03-04T16:04:22.910 INFO:tasks.workunit.client.1.smithi152.stdout:5/25: rename d6/d8 to d6/d8/dc 22 2024-03-04T16:04:22.910 INFO:tasks.workunit.client.0.smithi116.stdout:4/926: dwrite d1/d3/dd/d45/f90 [0,4194304] 0 2024-03-04T16:04:22.911 INFO:tasks.workunit.client.0.smithi116.stdout:1/746: fsync d5/da1/fc3 0 2024-03-04T16:04:22.913 INFO:tasks.workunit.client.1.smithi152.stdout:1/22: readlink l0 0 2024-03-04T16:04:22.914 INFO:tasks.workunit.client.0.smithi116.stdout:9/813: mkdir d2/d33/d90/d101 0 2024-03-04T16:04:22.915 INFO:tasks.workunit.client.0.smithi116.stdout:9/814: chown d2/d26/dd4/dab 205194791 1 2024-03-04T16:04:22.916 INFO:tasks.workunit.client.0.smithi116.stdout:9/815: write d2/d26/f47 [3470738,67456] 0 2024-03-04T16:04:22.919 INFO:tasks.workunit.client.1.smithi152.stdout:6/14: link f1 f6 0 2024-03-04T16:04:22.919 INFO:tasks.workunit.client.1.smithi152.stdout:6/15: chown f6 2 1 2024-03-04T16:04:22.922 INFO:tasks.workunit.client.1.smithi152.stdout:4/19: fsync f0 0 2024-03-04T16:04:22.923 INFO:tasks.workunit.client.1.smithi152.stdout:8/25: getdents d2 0 2024-03-04T16:04:22.926 INFO:tasks.workunit.client.1.smithi152.stdout:1/23: unlink f6 0 2024-03-04T16:04:22.927 INFO:tasks.workunit.client.0.smithi116.stdout:2/929: symlink d7/d13/d3e/d71/dad/l15a 0 2024-03-04T16:04:22.927 INFO:tasks.workunit.client.0.smithi116.stdout:4/927: symlink d1/d3/d15/d20/d1c/l14c 0 2024-03-04T16:04:22.929 INFO:tasks.workunit.client.0.smithi116.stdout:4/928: write d1/d6c/f38 [2291952,23663] 0 2024-03-04T16:04:22.929 INFO:tasks.workunit.client.0.smithi116.stdout:1/747: rmdir d5/d9 39 2024-03-04T16:04:22.929 INFO:tasks.workunit.client.0.smithi116.stdout:4/929: dread - d1/d3/f137 zero size 2024-03-04T16:04:22.931 INFO:tasks.workunit.client.1.smithi152.stdout:4/20: creat f7 x:0 0 0 2024-03-04T16:04:22.933 INFO:tasks.workunit.client.1.smithi152.stdout:8/26: mknod d2/c7 0 2024-03-04T16:04:22.935 INFO:tasks.workunit.client.0.smithi116.stdout:9/816: creat d2/d26/dd4/d4a/d6a/d94/d95/f102 x:0 0 0 2024-03-04T16:04:22.936 INFO:tasks.workunit.client.0.smithi116.stdout:8/930: getdents d2/de/d16/d78/d29/de5/df2 0 2024-03-04T16:04:22.938 INFO:tasks.workunit.client.0.smithi116.stdout:2/930: mknod d7/d25/d12f/c15b 0 2024-03-04T16:04:22.940 INFO:tasks.workunit.client.1.smithi152.stdout:6/16: link f6 f7 0 2024-03-04T16:04:22.940 INFO:tasks.workunit.client.0.smithi116.stdout:3/954: dread dc/f4b [0,4194304] 0 2024-03-04T16:04:22.941 INFO:tasks.workunit.client.1.smithi152.stdout:6/17: read - f1 zero size 2024-03-04T16:04:22.941 INFO:tasks.workunit.client.0.smithi116.stdout:3/955: write dc/df/d47/d12d/d7f/d93/db6/dc5/de0/f6b [759006,68984] 0 2024-03-04T16:04:22.942 INFO:tasks.workunit.client.1.smithi152.stdout:8/27: dwrite d2/d3/f6 [0,4194304] 0 2024-03-04T16:04:22.942 INFO:tasks.workunit.client.0.smithi116.stdout:3/956: fdatasync dc/df/d47/f61 0 2024-03-04T16:04:22.943 INFO:tasks.workunit.client.1.smithi152.stdout:8/28: truncate d2/d3/f6 4823954 0 2024-03-04T16:04:22.943 INFO:tasks.workunit.client.0.smithi116.stdout:3/957: write dc/df/d47/d12d/d7f/d8b/f113 [1055027,78131] 0 2024-03-04T16:04:22.944 INFO:tasks.workunit.client.0.smithi116.stdout:3/958: readlink dc/dd3/lfe 0 2024-03-04T16:04:22.951 INFO:tasks.workunit.client.0.smithi116.stdout:3/959: dwrite dc/d34/d44/dab/f121 [0,4194304] 0 2024-03-04T16:04:22.954 INFO:tasks.workunit.client.0.smithi116.stdout:5/890: link dd/c27 dd/d1f/d55/d73/d2d/d59/c120 0 2024-03-04T16:04:22.958 INFO:tasks.workunit.client.0.smithi116.stdout:0/770: dwrite da/d16/d28/d3d/d41/f59 [0,4194304] 0 2024-03-04T16:04:22.960 INFO:tasks.workunit.client.1.smithi152.stdout:8/29: unlink d2/c7 0 2024-03-04T16:04:22.962 INFO:tasks.workunit.client.0.smithi116.stdout:2/931: creat d7/d13/d3e/d98/dce/d47/d64/d75/d77/d12e/f15c x:0 0 0 2024-03-04T16:04:22.963 INFO:tasks.workunit.client.1.smithi152.stdout:5/26: getdents d6/d8 0 2024-03-04T16:04:22.965 INFO:tasks.workunit.client.0.smithi116.stdout:0/771: dwrite da/d16/d19/d27/f78 [0,4194304] 0 2024-03-04T16:04:22.969 INFO:tasks.workunit.client.1.smithi152.stdout:8/30: mkdir d2/d8 0 2024-03-04T16:04:22.973 INFO:tasks.workunit.client.1.smithi152.stdout:4/21: read f0 [675265,87469] 0 2024-03-04T16:04:22.974 INFO:tasks.workunit.client.1.smithi152.stdout:5/27: dwrite d6/fa [0,4194304] 0 2024-03-04T16:04:22.974 INFO:tasks.workunit.client.1.smithi152.stdout:4/22: chown c6 245 1 2024-03-04T16:04:22.975 INFO:tasks.workunit.client.0.smithi116.stdout:1/748: unlink d5/d9/d21/d24/de0/d4c/d67/c92 0 2024-03-04T16:04:22.976 INFO:tasks.workunit.client.1.smithi152.stdout:1/24: truncate f2 795047 0 2024-03-04T16:04:22.979 INFO:tasks.workunit.client.0.smithi116.stdout:8/931: write d2/de/d16/d78/d29/de5/df2/d37/fad [127038,57559] 0 2024-03-04T16:04:22.980 INFO:tasks.workunit.client.1.smithi152.stdout:8/31: mkdir d2/d3/d9 0 2024-03-04T16:04:22.981 INFO:tasks.workunit.client.0.smithi116.stdout:4/930: symlink d1/d3/d15/l14d 0 2024-03-04T16:04:22.982 INFO:tasks.workunit.client.1.smithi152.stdout:8/32: read d2/d3/f6 [4807230,61108] 0 2024-03-04T16:04:22.983 INFO:tasks.workunit.client.0.smithi116.stdout:3/960: rename dc/df/d14/f49 to dc/df/d47/d12d/d7f/dfd/f142 0 2024-03-04T16:04:22.985 INFO:tasks.workunit.client.0.smithi116.stdout:5/891: chown dd/f13 7555870 1 2024-03-04T16:04:22.985 INFO:tasks.workunit.client.1.smithi152.stdout:4/23: mkdir d8 0 2024-03-04T16:04:22.986 INFO:tasks.workunit.client.0.smithi116.stdout:5/892: chown dd/d6d/fc7 219 1 2024-03-04T16:04:22.986 INFO:tasks.workunit.client.1.smithi152.stdout:4/24: write f4 [1669242,42267] 0 2024-03-04T16:04:22.986 INFO:tasks.workunit.client.0.smithi116.stdout:9/817: mknod d2/d19/c103 0 2024-03-04T16:04:22.988 INFO:tasks.workunit.client.1.smithi152.stdout:5/28: creat d6/fd x:0 0 0 2024-03-04T16:04:22.988 INFO:tasks.workunit.client.1.smithi152.stdout:5/29: stat d6/l9 0 2024-03-04T16:04:22.990 INFO:tasks.workunit.client.1.smithi152.stdout:6/18: getdents . 0 2024-03-04T16:04:22.994 INFO:tasks.workunit.client.1.smithi152.stdout:1/25: mknod c7 0 2024-03-04T16:04:22.996 INFO:tasks.workunit.client.1.smithi152.stdout:5/30: dwrite d6/fd [0,4194304] 0 2024-03-04T16:04:22.998 INFO:tasks.workunit.client.0.smithi116.stdout:2/932: mknod d7/d13/d3e/d71/c15d 0 2024-03-04T16:04:22.999 INFO:tasks.workunit.client.1.smithi152.stdout:8/33: symlink d2/d3/la 0 2024-03-04T16:04:23.000 INFO:tasks.workunit.client.1.smithi152.stdout:8/34: rename d2/d3 to d2/d3/db 22 2024-03-04T16:04:23.006 INFO:tasks.workunit.client.0.smithi116.stdout:8/932: fdatasync d2/de/d16/d78/d29/d7a/fd7 0 2024-03-04T16:04:23.007 INFO:tasks.workunit.client.1.smithi152.stdout:6/19: mkdir d8 0 2024-03-04T16:04:23.009 INFO:tasks.workunit.client.1.smithi152.stdout:1/26: creat f8 x:0 0 0 2024-03-04T16:04:23.009 INFO:tasks.workunit.client.0.smithi116.stdout:3/961: symlink dc/df/d47/d91/l143 0 2024-03-04T16:04:23.012 INFO:tasks.workunit.client.0.smithi116.stdout:9/818: unlink d2/d26/dd4/dab/d3c/l7a 0 2024-03-04T16:04:23.013 INFO:tasks.workunit.client.0.smithi116.stdout:5/893: read dd/f22 [1107545,62218] 0 2024-03-04T16:04:23.013 INFO:tasks.workunit.client.1.smithi152.stdout:8/35: fdatasync d2/d3/f6 0 2024-03-04T16:04:23.014 INFO:tasks.workunit.client.0.smithi116.stdout:5/894: chown dd/d6d/dc6/lcc 32174577 1 2024-03-04T16:04:23.023 INFO:tasks.workunit.client.1.smithi152.stdout:4/25: mkdir d8/d9 0 2024-03-04T16:04:23.023 INFO:tasks.workunit.client.1.smithi152.stdout:1/27: mknod c9 0 2024-03-04T16:04:23.023 INFO:tasks.workunit.client.1.smithi152.stdout:4/26: truncate f5 236963 0 2024-03-04T16:04:23.024 INFO:tasks.workunit.client.0.smithi116.stdout:1/749: dwrite d5/d9/d21/d27/d32/d35/d55/db4/d50/f5e [0,4194304] 0 2024-03-04T16:04:23.025 INFO:tasks.workunit.client.0.smithi116.stdout:8/933: dread d2/de/d3e/d7b/f8e [0,4194304] 0 2024-03-04T16:04:23.025 INFO:tasks.workunit.client.0.smithi116.stdout:1/750: stat d5/d9/d21/d27/d8c 0 2024-03-04T16:04:23.025 INFO:tasks.workunit.client.1.smithi152.stdout:4/27: write f7 [167733,61054] 0 2024-03-04T16:04:23.029 INFO:tasks.workunit.client.0.smithi116.stdout:9/819: symlink d2/d33/d40/l104 0 2024-03-04T16:04:23.029 INFO:tasks.workunit.client.1.smithi152.stdout:4/28: dread f7 [0,4194304] 0 2024-03-04T16:04:23.032 INFO:tasks.workunit.client.0.smithi116.stdout:5/895: fdatasync dd/d1f/d55/d73/d2d/d59/db0/fbc 0 2024-03-04T16:04:23.033 INFO:tasks.workunit.client.0.smithi116.stdout:2/933: dwrite d7/d13/d3e/ded/d76/fda [0,4194304] 0 2024-03-04T16:04:23.034 INFO:tasks.workunit.client.1.smithi152.stdout:8/36: dwrite d2/d3/f6 [0,4194304] 0 2024-03-04T16:04:23.039 INFO:tasks.workunit.client.1.smithi152.stdout:5/31: truncate d6/f7 3863771 0 2024-03-04T16:04:23.039 INFO:tasks.workunit.client.1.smithi152.stdout:4/29: dwrite f3 [0,4194304] 0 2024-03-04T16:04:23.040 INFO:tasks.workunit.client.1.smithi152.stdout:5/32: read d6/fd [1075732,125263] 0 2024-03-04T16:04:23.041 INFO:tasks.workunit.client.1.smithi152.stdout:5/33: fdatasync d6/fa 0 2024-03-04T16:04:23.042 INFO:tasks.workunit.client.1.smithi152.stdout:4/30: write f4 [1725387,107278] 0 2024-03-04T16:04:23.049 INFO:tasks.workunit.client.1.smithi152.stdout:5/34: dread d6/fd [0,4194304] 0 2024-03-04T16:04:23.050 INFO:tasks.workunit.client.1.smithi152.stdout:5/35: write d6/fa [2296007,57540] 0 2024-03-04T16:04:23.059 INFO:tasks.workunit.client.0.smithi116.stdout:5/896: mkdir dd/d1f/d55/d73/d2d/d7c/d121 0 2024-03-04T16:04:23.060 INFO:tasks.workunit.client.0.smithi116.stdout:0/772: link da/d16/d20/d42/d4e/l50 da/d16/d28/d40/d48/d57/lef 0 2024-03-04T16:04:23.061 INFO:tasks.workunit.client.0.smithi116.stdout:3/962: link dc/df/d47/d12d/d7f/d93/df1/l125 dc/df/d47/d12d/d7f/d93/db6/dc5/de0/da3/l144 0 2024-03-04T16:04:23.065 INFO:tasks.workunit.client.0.smithi116.stdout:3/963: read dc/df/d47/d12d/fd1 [854525,101906] 0 2024-03-04T16:04:23.068 INFO:tasks.workunit.client.0.smithi116.stdout:5/897: mkdir dd/d1f/dcd/d122 0 2024-03-04T16:04:23.075 INFO:tasks.workunit.client.0.smithi116.stdout:8/934: dread d2/d13/d3c/d77/d97/deb/f109 [0,4194304] 0 2024-03-04T16:04:23.075 INFO:tasks.workunit.client.0.smithi116.stdout:3/964: creat dc/df/d47/d12d/f145 x:0 0 0 2024-03-04T16:04:23.081 INFO:tasks.workunit.client.0.smithi116.stdout:1/751: truncate d5/d9/d21/d27/d32/d35/d55/db4/d50/f5e 1479786 0 2024-03-04T16:04:23.082 INFO:tasks.workunit.client.0.smithi116.stdout:9/820: getdents d2/d33/d65/d6c/db8/de3 0 2024-03-04T16:04:23.085 INFO:tasks.workunit.client.0.smithi116.stdout:0/773: dread da/f1e [0,4194304] 0 2024-03-04T16:04:23.087 INFO:tasks.workunit.client.0.smithi116.stdout:8/935: dread d2/d13/d3c/d46/d6e/fbb [0,4194304] 0 2024-03-04T16:04:23.089 INFO:tasks.workunit.client.0.smithi116.stdout:1/752: symlink d5/d9/d21/d24/de0/d4c/d67/lf5 0 2024-03-04T16:04:23.090 INFO:tasks.workunit.client.0.smithi116.stdout:8/936: dread d2/d13/d3c/d77/d97/deb/f109 [0,4194304] 0 2024-03-04T16:04:23.096 INFO:tasks.workunit.client.0.smithi116.stdout:5/898: dwrite dd/d1f/d55/d73/d2d/d31/f6a [0,4194304] 0 2024-03-04T16:04:23.099 INFO:tasks.workunit.client.0.smithi116.stdout:8/937: rename d2/f32 to d2/d13/d3c/d46/d4f/f138 0 2024-03-04T16:04:23.100 INFO:tasks.workunit.client.0.smithi116.stdout:3/965: write dc/df/d14/fe5 [291637,18907] 0 2024-03-04T16:04:23.100 INFO:tasks.workunit.client.0.smithi116.stdout:8/938: chown d2/d13/d3c/f83 22775109 1 2024-03-04T16:04:23.101 INFO:tasks.workunit.client.0.smithi116.stdout:3/966: dread - dc/df/d47/d12d/d7f/d93/df1/f118 zero size 2024-03-04T16:04:23.103 INFO:tasks.workunit.client.0.smithi116.stdout:1/753: dread d5/d9/d21/d27/d94/db6/d42/d8b/dc9/fd4 [0,4194304] 0 2024-03-04T16:04:23.103 INFO:tasks.workunit.client.0.smithi116.stdout:9/821: creat d2/d19/f105 x:0 0 0 2024-03-04T16:04:23.111 INFO:tasks.workunit.client.0.smithi116.stdout:3/967: read - dc/df/db8/fe4 zero size 2024-03-04T16:04:23.113 INFO:tasks.workunit.client.0.smithi116.stdout:0/774: dwrite da/f35 [0,4194304] 0 2024-03-04T16:04:23.114 INFO:tasks.workunit.client.0.smithi116.stdout:1/754: symlink d5/d9/d21/d27/d94/db6/d42/lf6 0 2024-03-04T16:04:23.114 INFO:tasks.workunit.client.0.smithi116.stdout:7/892: sync 2024-03-04T16:04:23.114 INFO:tasks.workunit.client.0.smithi116.stdout:6/815: sync 2024-03-04T16:04:23.117 INFO:tasks.workunit.client.0.smithi116.stdout:5/899: write dd/d1f/d55/d73/d2d/d59/f81 [404148,49598] 0 2024-03-04T16:04:23.119 INFO:tasks.workunit.client.0.smithi116.stdout:8/939: dwrite d2/de/d3e/fa8 [0,4194304] 0 2024-03-04T16:04:23.122 INFO:tasks.workunit.client.0.smithi116.stdout:5/900: write dd/d1f/d55/d73/d2d/dc0/f10d [2702023,21865] 0 2024-03-04T16:04:23.122 INFO:tasks.workunit.client.0.smithi116.stdout:0/775: unlink da/d16/lc9 0 2024-03-04T16:04:23.126 INFO:tasks.workunit.client.0.smithi116.stdout:0/776: dread da/d16/d19/f68 [0,4194304] 0 2024-03-04T16:04:23.135 INFO:tasks.workunit.client.0.smithi116.stdout:3/968: write dc/df/d47/d12d/d7c/f62 [2994126,238] 0 2024-03-04T16:04:23.135 INFO:tasks.workunit.client.0.smithi116.stdout:0/777: fdatasync da/f35 0 2024-03-04T16:04:23.136 INFO:tasks.workunit.client.0.smithi116.stdout:5/901: fdatasync dd/d1f/d55/d73/d2d/f50 0 2024-03-04T16:04:23.136 INFO:tasks.workunit.client.0.smithi116.stdout:3/969: write dc/df/dbc/ffa [222739,7244] 0 2024-03-04T16:04:23.137 INFO:tasks.workunit.client.0.smithi116.stdout:0/778: chown da/d16/d20/d79/fce 97 1 2024-03-04T16:04:23.137 INFO:tasks.workunit.client.0.smithi116.stdout:9/822: rename d2/d26/dd4/f68 to d2/d19/d27/f106 0 2024-03-04T16:04:23.138 INFO:tasks.workunit.client.0.smithi116.stdout:9/823: chown d2/d33/d90/d101 1093 1 2024-03-04T16:04:23.139 INFO:tasks.workunit.client.0.smithi116.stdout:1/755: symlink d5/d9/d21/d27/d94/df2/lf7 0 2024-03-04T16:04:23.140 INFO:tasks.workunit.client.0.smithi116.stdout:9/824: chown d2/d33/d65/fea 5557094 1 2024-03-04T16:04:23.141 INFO:tasks.workunit.client.0.smithi116.stdout:6/816: dread f2 [0,4194304] 0 2024-03-04T16:04:23.141 INFO:tasks.workunit.client.0.smithi116.stdout:9/825: write d2/d33/fec [250742,104972] 0 2024-03-04T16:04:23.142 INFO:tasks.workunit.client.0.smithi116.stdout:6/817: fdatasync d6/de/d25/d36/d44/dbb/ff7 0 2024-03-04T16:04:23.143 INFO:tasks.workunit.client.0.smithi116.stdout:6/818: fdatasync d6/de/d84/db4/fe2 0 2024-03-04T16:04:23.148 INFO:tasks.workunit.client.0.smithi116.stdout:3/970: symlink dc/d34/de7/deb/de2/l146 0 2024-03-04T16:04:23.149 INFO:tasks.workunit.client.0.smithi116.stdout:6/819: dwrite d6/de/d84/db4/d92/fed [0,4194304] 0 2024-03-04T16:04:23.151 INFO:tasks.workunit.client.0.smithi116.stdout:8/940: dwrite d2/de/fa5 [0,4194304] 0 2024-03-04T16:04:23.151 INFO:tasks.workunit.client.0.smithi116.stdout:5/902: rename dd/d1f/d55/d73/d2d/d59/fac to dd/d45/d10c/f123 0 2024-03-04T16:04:23.157 INFO:tasks.workunit.client.0.smithi116.stdout:7/893: link d3/d4/d9e/dd7/fab d3/d4/d9e/dd7/daf/f125 0 2024-03-04T16:04:23.158 INFO:tasks.workunit.client.0.smithi116.stdout:2/934: sync 2024-03-04T16:04:23.158 INFO:tasks.workunit.client.0.smithi116.stdout:4/931: sync 2024-03-04T16:04:23.165 INFO:tasks.workunit.client.0.smithi116.stdout:3/971: fdatasync dc/df/d47/d12d/d7c/f80 0 2024-03-04T16:04:23.167 INFO:tasks.workunit.client.0.smithi116.stdout:3/972: chown dc/df/d47/d12d/d7f/d93/db6/dc5/de0/f6b 1 1 2024-03-04T16:04:23.168 INFO:tasks.workunit.client.0.smithi116.stdout:5/903: rmdir dd/d1f/dc3 39 2024-03-04T16:04:23.170 INFO:tasks.workunit.client.0.smithi116.stdout:1/756: unlink d5/d9/d21/d27/d32/d35/d55/d75/d91/cd6 0 2024-03-04T16:04:23.172 INFO:tasks.workunit.client.0.smithi116.stdout:7/894: mkdir d3/d4/d8/d1a/d126 0 2024-03-04T16:04:23.172 INFO:tasks.workunit.client.0.smithi116.stdout:8/941: dread d2/d13/d3c/fae [0,4194304] 0 2024-03-04T16:04:23.173 INFO:tasks.workunit.client.0.smithi116.stdout:2/935: rmdir d7/dd3 39 2024-03-04T16:04:23.182 INFO:tasks.workunit.client.0.smithi116.stdout:0/779: rename da/d16/d28/d40/d48/da7 to da/d16/d28/d40/d48/d57/d8a/daa/dba/df0 0 2024-03-04T16:04:23.185 INFO:tasks.workunit.client.0.smithi116.stdout:3/973: dread dc/df/d47/d12d/d7f/d93/db6/dc5/de0/f6b [0,4194304] 0 2024-03-04T16:04:23.185 INFO:tasks.workunit.client.0.smithi116.stdout:5/904: mknod dd/d1f/de5/c124 0 2024-03-04T16:04:23.187 INFO:tasks.workunit.client.0.smithi116.stdout:3/974: write dc/df/d47/d12d/d7f/d93/db6/dc5/de0/f114 [516121,90805] 0 2024-03-04T16:04:23.187 INFO:tasks.workunit.client.0.smithi116.stdout:1/757: mknod d5/d9/d21/d27/d32/d35/d55/db4/d50/cf8 0 2024-03-04T16:04:23.188 INFO:tasks.workunit.client.0.smithi116.stdout:1/758: write d5/d9/d21/d27/d94/fd7 [278854,64129] 0 2024-03-04T16:04:23.189 INFO:tasks.workunit.client.0.smithi116.stdout:7/895: rmdir d3/d4/dbf/de6/dd3/ddf/dee 39 2024-03-04T16:04:23.192 INFO:tasks.workunit.client.0.smithi116.stdout:8/942: creat d2/d13/d3c/d46/f139 x:0 0 0 2024-03-04T16:04:23.198 INFO:tasks.workunit.client.0.smithi116.stdout:6/820: dwrite d6/f9 [0,4194304] 0 2024-03-04T16:04:23.200 INFO:tasks.workunit.client.0.smithi116.stdout:9/826: rename d2/d26/d28/c29 to d2/d26/dd4/de5/c107 0 2024-03-04T16:04:23.201 INFO:tasks.workunit.client.0.smithi116.stdout:0/780: rmdir da/d16/d28/d7b 39 2024-03-04T16:04:23.204 INFO:tasks.workunit.client.0.smithi116.stdout:3/975: dread dc/f16 [0,4194304] 0 2024-03-04T16:04:23.207 INFO:tasks.workunit.client.0.smithi116.stdout:5/905: truncate dd/d1f/d55/f77 1725300 0 2024-03-04T16:04:23.209 INFO:tasks.workunit.client.0.smithi116.stdout:1/759: mkdir d5/d9/d21/d27/d94/df9 0 2024-03-04T16:04:23.210 INFO:tasks.workunit.client.0.smithi116.stdout:1/760: read d5/d9/d21/d27/d32/d35/f87 [402082,67005] 0 2024-03-04T16:04:23.211 INFO:tasks.workunit.client.0.smithi116.stdout:4/932: truncate d1/fae 2285098 0 2024-03-04T16:04:23.212 INFO:tasks.workunit.client.0.smithi116.stdout:7/896: mknod d3/d4/d9e/dd7/dfd/c127 0 2024-03-04T16:04:23.214 INFO:tasks.workunit.client.0.smithi116.stdout:8/943: creat d2/de/d16/dc3/f13a x:0 0 0 2024-03-04T16:04:23.219 INFO:tasks.workunit.client.0.smithi116.stdout:9/827: creat d2/d26/dd4/dab/d3c/f108 x:0 0 0 2024-03-04T16:04:23.220 INFO:tasks.workunit.client.0.smithi116.stdout:9/828: readlink d2/d33/d90/la5 0 2024-03-04T16:04:23.222 INFO:tasks.workunit.client.0.smithi116.stdout:0/781: read da/d16/d20/d42/f88 [1161345,70795] 0 2024-03-04T16:04:23.226 INFO:tasks.workunit.client.0.smithi116.stdout:1/761: creat d5/d9/d21/d27/d32/d35/d55/db4/d50/ffa x:0 0 0 2024-03-04T16:04:23.229 INFO:tasks.workunit.client.0.smithi116.stdout:0/782: dwrite da/d16/d28/d40/d48/fd7 [0,4194304] 0 2024-03-04T16:04:23.229 INFO:tasks.workunit.client.0.smithi116.stdout:1/762: fsync d5/d9/d21/d27/d94/db6/d42/fe3 0 2024-03-04T16:04:23.230 INFO:tasks.workunit.client.0.smithi116.stdout:8/944: dread d2/de3/f108 [0,4194304] 0 2024-03-04T16:04:23.232 INFO:tasks.workunit.client.0.smithi116.stdout:8/945: chown d2/d38/d41/dab/d124 72 1 2024-03-04T16:04:23.235 INFO:tasks.workunit.client.0.smithi116.stdout:4/933: dwrite d1/d3/d40/d52/d84/d88/fc4 [0,4194304] 0 2024-03-04T16:04:23.236 INFO:tasks.workunit.client.0.smithi116.stdout:4/934: chown d1/d3/de/d47/df9/c104 16 1 2024-03-04T16:04:23.238 INFO:tasks.workunit.client.0.smithi116.stdout:9/829: mknod d2/d19/d5f/c109 0 2024-03-04T16:04:23.238 INFO:tasks.workunit.client.0.smithi116.stdout:3/976: write dc/df/d47/d91/f98 [373762,15578] 0 2024-03-04T16:04:23.242 INFO:tasks.workunit.client.0.smithi116.stdout:5/906: getdents dd/d1f/dcd/d122 0 2024-03-04T16:04:23.244 INFO:tasks.workunit.client.0.smithi116.stdout:3/977: dwrite dc/df/f7e [0,4194304] 0 2024-03-04T16:04:23.247 INFO:tasks.workunit.client.0.smithi116.stdout:7/897: creat d3/d4/d8/d1a/dcb/f128 x:0 0 0 2024-03-04T16:04:23.247 INFO:tasks.workunit.client.0.smithi116.stdout:7/898: chown d3/d4/d8 440175 1 2024-03-04T16:04:23.249 INFO:tasks.workunit.client.0.smithi116.stdout:7/899: stat d3/d4/d9e/dd7/dfd/f112 0 2024-03-04T16:04:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:6/821: truncate d6/de/f52 955521 0 2024-03-04T16:04:23.252 INFO:tasks.workunit.client.0.smithi116.stdout:1/763: mkdir d5/d9/d21/d84/db0/de9/dfb 0 2024-03-04T16:04:23.254 INFO:tasks.workunit.client.0.smithi116.stdout:2/936: link d7/d13/d3e/d98/dce/d47/d64/d75/f150 d7/d13/d3e/ded/d76/d7c/f15e 0 2024-03-04T16:04:23.255 INFO:tasks.workunit.client.0.smithi116.stdout:2/937: chown d7/d81/f10d 672112 1 2024-03-04T16:04:23.259 INFO:tasks.workunit.client.0.smithi116.stdout:8/946: mkdir d2/de/d16/d78/d29/d7a/db1/d13b 0 2024-03-04T16:04:23.260 INFO:tasks.workunit.client.0.smithi116.stdout:4/935: symlink d1/d3/dd/d45/d50/dad/l14e 0 2024-03-04T16:04:23.267 INFO:tasks.workunit.client.0.smithi116.stdout:9/830: dread d2/d26/dd4/dab/d3c/f8f [0,4194304] 0 2024-03-04T16:04:23.267 INFO:tasks.workunit.client.0.smithi116.stdout:3/978: dread dc/df/d47/d12d/d7f/d8b/fc7 [0,4194304] 0 2024-03-04T16:04:23.268 INFO:tasks.workunit.client.0.smithi116.stdout:3/979: fdatasync dc/df/db8/fcc 0 2024-03-04T16:04:23.269 INFO:tasks.workunit.client.0.smithi116.stdout:0/783: mknod da/d16/d28/cf1 0 2024-03-04T16:04:23.271 INFO:tasks.workunit.client.0.smithi116.stdout:0/784: write da/d16/d20/d53/f63 [4769555,47504] 0 2024-03-04T16:04:23.272 INFO:tasks.workunit.client.0.smithi116.stdout:9/831: dread d2/d19/d27/fae [0,4194304] 0 2024-03-04T16:04:23.273 INFO:tasks.workunit.client.1.smithi152.stdout:6/20: dwrite f7 [0,4194304] 0 2024-03-04T16:04:23.278 INFO:tasks.workunit.client.0.smithi116.stdout:2/938: creat d7/d13/d3e/d98/dce/d47/d64/d75/d77/f15f x:0 0 0 2024-03-04T16:04:23.281 INFO:tasks.workunit.client.0.smithi116.stdout:8/947: rename d2/de/d16/dc3/f13a to d2/de/d16/d78/d29/de5/df2/d37/df8/f13c 0 2024-03-04T16:04:23.282 INFO:tasks.workunit.client.0.smithi116.stdout:7/900: dread d3/d4/d8/f3e [0,4194304] 0 2024-03-04T16:04:23.285 INFO:tasks.workunit.client.0.smithi116.stdout:5/907: link dd/d1f/d51/fbf dd/d1f/d55/d73/d2d/d59/db0/dca/f125 0 2024-03-04T16:04:23.286 INFO:tasks.workunit.client.0.smithi116.stdout:5/908: write dd/d1f/f10a [974647,121836] 0 2024-03-04T16:04:23.290 INFO:tasks.workunit.client.1.smithi152.stdout:1/28: link f8 fa 0 2024-03-04T16:04:23.291 INFO:tasks.workunit.client.0.smithi116.stdout:6/822: write d6/f108 [230411,114159] 0 2024-03-04T16:04:23.291 INFO:tasks.workunit.client.0.smithi116.stdout:3/980: symlink dc/d34/de7/deb/de2/l147 0 2024-03-04T16:04:23.292 INFO:tasks.workunit.client.0.smithi116.stdout:3/981: readlink dc/l52 0 2024-03-04T16:04:23.295 INFO:tasks.workunit.client.0.smithi116.stdout:5/909: dwrite dd/d1f/d55/d73/d2d/d7c/fd9 [0,4194304] 0 2024-03-04T16:04:23.295 INFO:tasks.workunit.client.0.smithi116.stdout:3/982: write dc/d34/dc1/ffc [338825,81670] 0 2024-03-04T16:04:23.295 INFO:tasks.workunit.client.1.smithi152.stdout:6/21: write f6 [229381,12062] 0 2024-03-04T16:04:23.297 INFO:tasks.workunit.client.0.smithi116.stdout:0/785: creat da/d16/d19/d27/d45/ff2 x:0 0 0 2024-03-04T16:04:23.297 INFO:tasks.workunit.client.0.smithi116.stdout:0/786: chown da/d16/d20/lc7 0 1 2024-03-04T16:04:23.300 INFO:tasks.workunit.client.0.smithi116.stdout:7/901: dread d3/d4/d8/d4a/d65/df2/fcd [0,4194304] 0 2024-03-04T16:04:23.301 INFO:tasks.workunit.client.0.smithi116.stdout:7/902: dread - d3/d4/d8/d4a/d113/db6/fda zero size 2024-03-04T16:04:23.302 INFO:tasks.workunit.client.0.smithi116.stdout:1/764: creat d5/d9/d21/d24/de0/d4c/d67/dcf/ffc x:0 0 0 2024-03-04T16:04:23.304 INFO:tasks.workunit.client.1.smithi152.stdout:1/29: symlink lb 0 2024-03-04T16:04:23.307 INFO:tasks.workunit.client.1.smithi152.stdout:6/22: stat f3 0 2024-03-04T16:04:23.310 INFO:tasks.workunit.client.0.smithi116.stdout:4/936: mkdir d1/d3/d40/d52/d84/dc1/d14f 0 2024-03-04T16:04:23.310 INFO:tasks.workunit.client.1.smithi152.stdout:1/30: symlink lc 0 2024-03-04T16:04:23.314 INFO:tasks.workunit.client.0.smithi116.stdout:8/948: rename d2/d13/d3c/d77/f137 to d2/d4a/d114/d122/d127/f13d 0 2024-03-04T16:04:23.314 INFO:tasks.workunit.client.1.smithi152.stdout:5/36: link d6/f7 d6/d8/fe 0 2024-03-04T16:04:23.315 INFO:tasks.workunit.client.0.smithi116.stdout:8/949: chown d2/d13/d3c/d77/d97/d5a/f6f 1842 1 2024-03-04T16:04:23.318 INFO:tasks.workunit.client.1.smithi152.stdout:6/23: creat d8/f9 x:0 0 0 2024-03-04T16:04:23.319 INFO:tasks.workunit.client.1.smithi152.stdout:6/24: fsync f7 0 2024-03-04T16:04:23.322 INFO:tasks.workunit.client.1.smithi152.stdout:5/37: write d6/fd [4550466,31413] 0 2024-03-04T16:04:23.323 INFO:tasks.workunit.client.0.smithi116.stdout:5/910: dread dd/d1f/f54 [0,4194304] 0 2024-03-04T16:04:23.324 INFO:tasks.workunit.client.1.smithi152.stdout:1/31: fsync f5 0 2024-03-04T16:04:23.324 INFO:tasks.workunit.client.1.smithi152.stdout:1/32: dread - f8 zero size 2024-03-04T16:04:23.328 INFO:tasks.workunit.client.1.smithi152.stdout:6/25: symlink d8/la 0 2024-03-04T16:04:23.329 INFO:tasks.workunit.client.0.smithi116.stdout:0/787: mknod da/d16/d19/d27/d82/d9f/cf3 0 2024-03-04T16:04:23.331 INFO:tasks.workunit.client.0.smithi116.stdout:9/832: creat d2/d33/d90/d101/f10a x:0 0 0 2024-03-04T16:04:23.332 INFO:tasks.workunit.client.0.smithi116.stdout:1/765: stat d5/d9/d21/d24/de0/d4c/d67/c96 0 2024-03-04T16:04:23.334 INFO:tasks.workunit.client.0.smithi116.stdout:2/939: fdatasync d7/d81/f22 0 2024-03-04T16:04:23.335 INFO:tasks.workunit.client.0.smithi116.stdout:2/940: chown d7/d81/dc2/fd0 1336 1 2024-03-04T16:04:23.335 INFO:tasks.workunit.client.1.smithi152.stdout:1/33: mknod cd 0 2024-03-04T16:04:23.338 INFO:tasks.workunit.client.0.smithi116.stdout:1/766: dwrite d5/d9/d21/d27/d32/d35/d55/d75/d91/f9e [4194304,4194304] 0 2024-03-04T16:04:23.339 INFO:tasks.workunit.client.1.smithi152.stdout:5/38: truncate d6/f7 3806299 0 2024-03-04T16:04:23.339 INFO:tasks.workunit.client.0.smithi116.stdout:1/767: dread - d5/d9/d21/d24/de0/fed zero size 2024-03-04T16:04:23.340 INFO:tasks.workunit.client.0.smithi116.stdout:8/950: symlink d2/d13/d3c/d46/d104/l13e 0 2024-03-04T16:04:23.342 INFO:tasks.workunit.client.0.smithi116.stdout:6/823: symlink d6/de/d25/d4a/l117 0 2024-03-04T16:04:23.343 INFO:tasks.workunit.client.0.smithi116.stdout:6/824: chown d6/d56/f60 6799 1 2024-03-04T16:04:23.343 INFO:tasks.workunit.client.1.smithi152.stdout:5/39: dread d6/fa [0,4194304] 0 2024-03-04T16:04:23.343 INFO:tasks.workunit.client.1.smithi152.stdout:5/40: fsync d6/fa 0 2024-03-04T16:04:23.343 INFO:tasks.workunit.client.1.smithi152.stdout:1/34: creat fe x:0 0 0 2024-03-04T16:04:23.344 INFO:tasks.workunit.client.0.smithi116.stdout:5/911: creat dd/d1f/d51/f126 x:0 0 0 2024-03-04T16:04:23.351 INFO:tasks.workunit.client.1.smithi152.stdout:1/35: mknod cf 0 2024-03-04T16:04:23.351 INFO:tasks.workunit.client.0.smithi116.stdout:9/833: fsync d2/d33/d65/d6c/db8/fd1 0 2024-03-04T16:04:23.352 INFO:tasks.workunit.client.0.smithi116.stdout:9/834: dread - d2/d19/d5f/fe4 zero size 2024-03-04T16:04:23.355 INFO:tasks.workunit.client.1.smithi152.stdout:6/26: truncate f3 2956008 0 2024-03-04T16:04:23.361 INFO:tasks.workunit.client.0.smithi116.stdout:1/768: fsync d5/d9/d21/fec 0 2024-03-04T16:04:23.362 INFO:tasks.workunit.client.0.smithi116.stdout:8/951: unlink d2/d38/d55/c115 0 2024-03-04T16:04:23.364 INFO:tasks.workunit.client.1.smithi152.stdout:5/41: dwrite d6/f7 [0,4194304] 0 2024-03-04T16:04:23.366 INFO:tasks.workunit.client.0.smithi116.stdout:5/912: rmdir dd/d1f/d55/d73/d2d/d59/dea 39 2024-03-04T16:04:23.366 INFO:tasks.workunit.client.0.smithi116.stdout:4/937: write d1/d3/d40/d52/d84/d88/f142 [1369803,101029] 0 2024-03-04T16:04:23.370 INFO:tasks.workunit.client.1.smithi152.stdout:5/42: creat d6/d8/ff x:0 0 0 2024-03-04T16:04:23.371 INFO:tasks.workunit.client.1.smithi152.stdout:6/27: write f3 [943843,98320] 0 2024-03-04T16:04:23.378 INFO:tasks.workunit.client.0.smithi116.stdout:6/825: symlink d6/de/l118 0 2024-03-04T16:04:23.378 INFO:tasks.workunit.client.1.smithi152.stdout:5/43: dread d6/fa [0,4194304] 0 2024-03-04T16:04:23.379 INFO:tasks.workunit.client.1.smithi152.stdout:6/28: getdents d8 0 2024-03-04T16:04:23.379 INFO:tasks.workunit.client.1.smithi152.stdout:5/44: write d6/d8/ff [295782,958] 0 2024-03-04T16:04:23.380 INFO:tasks.workunit.client.0.smithi116.stdout:4/938: mknod d1/d6c/c150 0 2024-03-04T16:04:23.381 INFO:tasks.workunit.client.0.smithi116.stdout:3/983: getdents dc/df/d47/d12d/d7c 0 2024-03-04T16:04:23.381 INFO:tasks.workunit.client.1.smithi152.stdout:6/29: mknod d8/cb 0 2024-03-04T16:04:23.383 INFO:tasks.workunit.client.1.smithi152.stdout:5/45: mkdir d6/d8/d10 0 2024-03-04T16:04:23.384 INFO:tasks.workunit.client.0.smithi116.stdout:7/903: getdents d3/d4/d8/d4a/d113 0 2024-03-04T16:04:23.385 INFO:tasks.workunit.client.0.smithi116.stdout:9/835: write d2/d19/f84 [1353364,49516] 0 2024-03-04T16:04:23.386 INFO:tasks.workunit.client.1.smithi152.stdout:5/46: creat d6/f11 x:0 0 0 2024-03-04T16:04:23.389 INFO:tasks.workunit.client.1.smithi152.stdout:6/30: truncate f7 1795917 0 2024-03-04T16:04:23.392 INFO:tasks.workunit.client.1.smithi152.stdout:6/31: symlink d8/lc 0 2024-03-04T16:04:23.394 INFO:tasks.workunit.client.0.smithi116.stdout:0/788: getdents da/d16/d28/d40 0 2024-03-04T16:04:23.395 INFO:tasks.workunit.client.0.smithi116.stdout:0/789: chown da/d16/d28/d40/d48/d57/d8a/daa 238591 1 2024-03-04T16:04:23.396 INFO:tasks.workunit.client.0.smithi116.stdout:4/939: write d1/d3/d40/f4f [3999050,80301] 0 2024-03-04T16:04:23.397 INFO:tasks.workunit.client.0.smithi116.stdout:7/904: symlink d3/d4/d34/d5d/db9/l129 0 2024-03-04T16:04:23.399 INFO:tasks.workunit.client.0.smithi116.stdout:9/836: unlink d2/d33/d40/c6b 0 2024-03-04T16:04:23.400 INFO:tasks.workunit.client.1.smithi152.stdout:5/47: getdents d6/d8 0 2024-03-04T16:04:23.401 INFO:tasks.workunit.client.0.smithi116.stdout:2/941: rename d7/d13/d3e/d71/dad/f138 to d7/f160 0 2024-03-04T16:04:23.402 INFO:tasks.workunit.client.0.smithi116.stdout:1/769: creat d5/d9/d21/d27/d32/ffd x:0 0 0 2024-03-04T16:04:23.402 INFO:tasks.workunit.client.0.smithi116.stdout:2/942: truncate d7/d81/f154 688821 0 2024-03-04T16:04:23.404 INFO:tasks.workunit.client.0.smithi116.stdout:1/770: chown d5/d9/d21/d27/d32/d35/d55/db4/d50/c6b 29976158 1 2024-03-04T16:04:23.404 INFO:tasks.workunit.client.0.smithi116.stdout:2/943: read d7/d81/f7b [3182650,34346] 0 2024-03-04T16:04:23.405 INFO:tasks.workunit.client.1.smithi152.stdout:5/48: rename d6/f7 to d6/d8/d10/f12 0 2024-03-04T16:04:23.409 INFO:tasks.workunit.client.1.smithi152.stdout:5/49: dread d6/d8/fe [0,4194304] 0 2024-03-04T16:04:23.410 INFO:tasks.workunit.client.1.smithi152.stdout:6/32: write f1 [1094751,49203] 0 2024-03-04T16:04:23.411 INFO:tasks.workunit.client.0.smithi116.stdout:3/984: creat dc/df/d47/d12d/d11e/f148 x:0 0 0 2024-03-04T16:04:23.412 INFO:tasks.workunit.client.1.smithi152.stdout:5/50: dread d6/d8/ff [0,4194304] 0 2024-03-04T16:04:23.413 INFO:tasks.workunit.client.1.smithi152.stdout:5/51: truncate d6/f11 380911 0 2024-03-04T16:04:23.413 INFO:tasks.workunit.client.0.smithi116.stdout:6/826: truncate d6/de/d25/d36/f27 3411277 0 2024-03-04T16:04:23.414 INFO:tasks.workunit.client.0.smithi116.stdout:0/790: write da/d16/d19/d27/f55 [68913,26952] 0 2024-03-04T16:04:23.415 INFO:tasks.workunit.client.1.smithi152.stdout:6/33: creat d8/fd x:0 0 0 2024-03-04T16:04:23.416 INFO:tasks.workunit.client.0.smithi116.stdout:4/940: mknod d1/d3/dd/d24/db4/d10c/c151 0 2024-03-04T16:04:23.418 INFO:tasks.workunit.client.0.smithi116.stdout:1/771: dread d5/d9/d21/d27/d94/db6/f69 [0,4194304] 0 2024-03-04T16:04:23.418 INFO:tasks.workunit.client.1.smithi152.stdout:5/52: truncate d6/fa 1126228 0 2024-03-04T16:04:23.419 INFO:tasks.workunit.client.0.smithi116.stdout:9/837: symlink d2/d33/d65/d6c/db8/de3/l10b 0 2024-03-04T16:04:23.421 INFO:tasks.workunit.client.0.smithi116.stdout:5/913: getdents dd/d1f/de5 0 2024-03-04T16:04:23.421 INFO:tasks.workunit.client.1.smithi152.stdout:5/53: mknod d6/d8/d10/c13 0 2024-03-04T16:04:23.421 INFO:tasks.workunit.client.1.smithi152.stdout:5/54: chown d6/d8/d10 5 1 2024-03-04T16:04:23.425 INFO:tasks.workunit.client.1.smithi152.stdout:6/34: unlink l4 0 2024-03-04T16:04:23.428 INFO:tasks.workunit.client.0.smithi116.stdout:3/985: mknod dc/c149 0 2024-03-04T16:04:23.429 INFO:tasks.workunit.client.0.smithi116.stdout:3/986: fsync dc/df/db8/f106 0 2024-03-04T16:04:23.429 INFO:tasks.workunit.client.1.smithi152.stdout:5/55: creat d6/d8/d10/f14 x:0 0 0 2024-03-04T16:04:23.430 INFO:tasks.workunit.client.0.smithi116.stdout:3/987: stat dc/df/d47/d12d/d7f/de8 0 2024-03-04T16:04:23.430 INFO:tasks.workunit.client.0.smithi116.stdout:0/791: dread da/d16/d28/d3d/f64 [0,4194304] 0 2024-03-04T16:04:23.432 INFO:tasks.workunit.client.0.smithi116.stdout:7/905: write d3/d4/d9e/dd7/daf/dd5/d123/f6e [3903559,128618] 0 2024-03-04T16:04:23.432 INFO:tasks.workunit.client.0.smithi116.stdout:3/988: read dc/df/d47/d12d/d7f/d93/db6/dc5/de0/f6b [1121562,89946] 0 2024-03-04T16:04:23.434 INFO:tasks.workunit.client.0.smithi116.stdout:7/906: stat d3/d4/dbf/de6/dd3/ddf/df3 0 2024-03-04T16:04:23.437 INFO:tasks.workunit.client.1.smithi152.stdout:5/56: dwrite d6/d8/d10/f14 [0,4194304] 0 2024-03-04T16:04:23.438 INFO:tasks.workunit.client.1.smithi152.stdout:6/35: rename d8/fd to d8/fe 0 2024-03-04T16:04:23.439 INFO:tasks.workunit.client.1.smithi152.stdout:6/36: dread - d8/f9 zero size 2024-03-04T16:04:23.442 INFO:tasks.workunit.client.0.smithi116.stdout:5/914: dread dd/d1f/d55/d73/d2d/d31/f3c [0,4194304] 0 2024-03-04T16:04:23.443 INFO:tasks.workunit.client.0.smithi116.stdout:1/772: truncate d5/d9/d21/d27/d94/db6/d42/f62 1249406 0 2024-03-04T16:04:23.445 INFO:tasks.workunit.client.0.smithi116.stdout:2/944: dwrite d7/d13/d3e/d98/dce/d47/d64/d75/d77/d119/f13f [0,4194304] 0 2024-03-04T16:04:23.449 INFO:tasks.workunit.client.0.smithi116.stdout:0/792: fdatasync da/d16/d28/d40/fe3 0 2024-03-04T16:04:23.451 INFO:tasks.workunit.client.1.smithi152.stdout:6/37: rmdir d8 39 2024-03-04T16:04:23.453 INFO:tasks.workunit.client.0.smithi116.stdout:3/989: rename dc/df/dbc/d101/d11c to dc/d34/de7/d14a 0 2024-03-04T16:04:23.454 INFO:tasks.workunit.client.0.smithi116.stdout:6/827: dwrite d6/de/d5b/fc8 [0,4194304] 0 2024-03-04T16:04:23.456 INFO:tasks.workunit.client.0.smithi116.stdout:7/907: mknod d3/d4/d8/d4a/d65/c12a 0 2024-03-04T16:04:23.457 INFO:tasks.workunit.client.0.smithi116.stdout:6/828: dread - d6/de/d25/d4a/d63/fe4 zero size 2024-03-04T16:04:23.457 INFO:tasks.workunit.client.1.smithi152.stdout:6/38: chown d8/cb 3 1 2024-03-04T16:04:23.460 INFO:tasks.workunit.client.0.smithi116.stdout:6/829: chown d6/de/d25/cd8 53 1 2024-03-04T16:04:23.461 INFO:tasks.workunit.client.1.smithi152.stdout:6/39: symlink d8/lf 0 2024-03-04T16:04:23.462 INFO:tasks.workunit.client.0.smithi116.stdout:9/838: dwrite d2/d19/d27/fae [0,4194304] 0 2024-03-04T16:04:23.463 INFO:tasks.workunit.client.0.smithi116.stdout:1/773: chown d5/d9/d21/d27/lbd 308152 1 2024-03-04T16:04:23.465 INFO:tasks.workunit.client.0.smithi116.stdout:3/990: dread dc/df/d47/d12d/d7f/d93/fb1 [0,4194304] 0 2024-03-04T16:04:23.466 INFO:tasks.workunit.client.1.smithi152.stdout:5/57: dwrite d6/d8/ff [0,4194304] 0 2024-03-04T16:04:23.472 INFO:tasks.workunit.client.1.smithi152.stdout:6/40: dread f0 [0,4194304] 0 2024-03-04T16:04:23.473 INFO:tasks.workunit.client.1.smithi152.stdout:6/41: readlink d8/lc 0 2024-03-04T16:04:23.476 INFO:tasks.workunit.client.1.smithi152.stdout:5/58: dread d6/d8/ff [0,4194304] 0 2024-03-04T16:04:23.476 INFO:tasks.workunit.client.0.smithi116.stdout:0/793: unlink da/d16/d20/d42/d4e/f8f 0 2024-03-04T16:04:23.477 INFO:tasks.workunit.client.0.smithi116.stdout:0/794: chown da/d16/d28/d40/fe3 1636 1 2024-03-04T16:04:23.480 INFO:tasks.workunit.client.0.smithi116.stdout:2/945: rename f4 to d7/d81/dc2/d12a/f161 0 2024-03-04T16:04:23.481 INFO:tasks.workunit.client.1.smithi152.stdout:6/42: dread f0 [0,4194304] 0 2024-03-04T16:04:23.483 INFO:tasks.workunit.client.1.smithi152.stdout:5/59: dread d6/d8/fe [0,4194304] 0 2024-03-04T16:04:23.483 INFO:tasks.workunit.client.0.smithi116.stdout:5/915: symlink dd/d1f/dcd/d122/l127 0 2024-03-04T16:04:23.484 INFO:tasks.workunit.client.0.smithi116.stdout:0/795: dwrite da/d16/d20/d42/de0/d7e/fc5 [0,4194304] 0 2024-03-04T16:04:23.486 INFO:tasks.workunit.client.0.smithi116.stdout:1/774: truncate d5/d9/d21/d27/d94/db6/f6a 140565 0 2024-03-04T16:04:23.486 INFO:tasks.workunit.client.0.smithi116.stdout:8/952: sync 2024-03-04T16:04:23.488 INFO:tasks.workunit.client.0.smithi116.stdout:3/991: mkdir dc/dd3/d14b 0 2024-03-04T16:04:23.489 INFO:tasks.workunit.client.1.smithi152.stdout:5/60: creat d6/d8/f15 x:0 0 0 2024-03-04T16:04:23.490 INFO:tasks.workunit.client.0.smithi116.stdout:3/992: chown dc/df/dbc/d135/c10d 2121822557 1 2024-03-04T16:04:23.490 INFO:tasks.workunit.client.0.smithi116.stdout:5/916: dread dd/d1f/d55/d73/d2d/d31/f6a [0,4194304] 0 2024-03-04T16:04:23.491 INFO:tasks.workunit.client.0.smithi116.stdout:3/993: stat dc/df/d47/d12d/f2d 0 2024-03-04T16:04:23.492 INFO:tasks.workunit.client.0.smithi116.stdout:0/796: dwrite da/d16/d28/d3d/f66 [0,4194304] 0 2024-03-04T16:04:23.498 INFO:tasks.workunit.client.0.smithi116.stdout:3/994: dread dc/df/d47/d12d/d7f/d93/db6/dc5/de0/f77 [0,4194304] 0 2024-03-04T16:04:23.498 INFO:tasks.workunit.client.1.smithi152.stdout:5/61: link d6/d8/d10/c13 d6/c16 0 2024-03-04T16:04:23.504 INFO:tasks.workunit.client.1.smithi152.stdout:5/62: dread d6/d8/d10/f14 [0,4194304] 0 2024-03-04T16:04:23.505 INFO:tasks.workunit.client.1.smithi152.stdout:5/63: write d6/d8/f15 [345644,93645] 0 2024-03-04T16:04:23.506 INFO:tasks.workunit.client.0.smithi116.stdout:4/941: dread d1/d3/d15/d20/d1c/d62/f11d [0,4194304] 0 2024-03-04T16:04:23.509 INFO:tasks.workunit.client.1.smithi152.stdout:5/64: dread d6/d8/fe [0,4194304] 0 2024-03-04T16:04:23.512 INFO:tasks.workunit.client.1.smithi152.stdout:5/65: mknod d6/d8/d10/c17 0 2024-03-04T16:04:23.514 INFO:tasks.workunit.client.1.smithi152.stdout:5/66: creat d6/f18 x:0 0 0 2024-03-04T16:04:23.532 INFO:tasks.workunit.client.1.smithi152.stdout:5/67: getdents d6/d8 0 2024-03-04T16:04:23.534 INFO:tasks.workunit.client.1.smithi152.stdout:6/43: dwrite d8/fe [0,4194304] 0 2024-03-04T16:04:23.536 INFO:tasks.workunit.client.1.smithi152.stdout:5/68: mkdir d6/d19 0 2024-03-04T16:04:23.537 INFO:tasks.workunit.client.1.smithi152.stdout:6/44: dread f7 [0,4194304] 0 2024-03-04T16:04:23.541 INFO:tasks.workunit.client.1.smithi152.stdout:6/45: dread d8/fe [0,4194304] 0 2024-03-04T16:04:23.546 INFO:tasks.workunit.client.1.smithi152.stdout:5/69: dwrite d6/d8/d10/f12 [0,4194304] 0 2024-03-04T16:04:23.546 INFO:tasks.workunit.client.1.smithi152.stdout:6/46: mkdir d8/d10 0 2024-03-04T16:04:23.547 INFO:tasks.workunit.client.1.smithi152.stdout:6/47: chown f7 8976 1 2024-03-04T16:04:23.548 INFO:tasks.workunit.client.0.smithi116.stdout:2/946: stat d7/d15/f20 0 2024-03-04T16:04:23.554 INFO:tasks.workunit.client.1.smithi152.stdout:6/48: fdatasync f0 0 2024-03-04T16:04:23.554 INFO:tasks.workunit.client.1.smithi152.stdout:5/70: dwrite d6/fd [0,4194304] 0 2024-03-04T16:04:23.558 INFO:tasks.workunit.client.1.smithi152.stdout:5/71: write d6/d8/d10/f12 [240415,14860] 0 2024-03-04T16:04:23.562 INFO:tasks.workunit.client.1.smithi152.stdout:6/49: unlink f3 0 2024-03-04T16:04:23.567 INFO:tasks.workunit.client.0.smithi116.stdout:1/775: mknod d5/d9/d21/d24/de0/cfe 0 2024-03-04T16:04:23.568 INFO:tasks.workunit.client.0.smithi116.stdout:1/776: write d5/d9/d21/d27/d94/fd7 [1214345,18464] 0 2024-03-04T16:04:23.569 INFO:tasks.workunit.client.0.smithi116.stdout:8/953: creat d2/d38/d41/dab/d124/f13f x:0 0 0 2024-03-04T16:04:23.572 INFO:tasks.workunit.client.1.smithi152.stdout:6/50: mknod d8/c11 0 2024-03-04T16:04:23.572 INFO:tasks.workunit.client.1.smithi152.stdout:5/72: fdatasync d6/d8/d10/f12 0 2024-03-04T16:04:23.573 INFO:tasks.workunit.client.0.smithi116.stdout:0/797: mkdir da/d16/d19/d27/d82/df4 0 2024-03-04T16:04:23.574 INFO:tasks.workunit.client.0.smithi116.stdout:0/798: write da/d16/d20/f65 [6255977,14027] 0 2024-03-04T16:04:23.577 INFO:tasks.workunit.client.1.smithi152.stdout:5/73: mknod d6/d8/c1a 0 2024-03-04T16:04:23.581 INFO:tasks.workunit.client.1.smithi152.stdout:6/51: dwrite f1 [0,4194304] 0 2024-03-04T16:04:23.582 INFO:tasks.workunit.client.0.smithi116.stdout:6/830: creat d6/de/d25/d36/f119 x:0 0 0 2024-03-04T16:04:23.584 INFO:tasks.workunit.client.1.smithi152.stdout:5/74: dread d6/d8/fe [0,4194304] 0 2024-03-04T16:04:23.585 INFO:tasks.workunit.client.0.smithi116.stdout:6/831: dread d6/de/d25/d36/f99 [0,4194304] 0 2024-03-04T16:04:23.586 INFO:tasks.workunit.client.1.smithi152.stdout:6/52: creat d8/f12 x:0 0 0 2024-03-04T16:04:23.586 INFO:tasks.workunit.client.1.smithi152.stdout:6/53: readlink d8/lc 0 2024-03-04T16:04:23.588 INFO:tasks.workunit.client.1.smithi152.stdout:5/75: rename c3 to d6/d8/c1b 0 2024-03-04T16:04:23.590 INFO:tasks.workunit.client.0.smithi116.stdout:8/954: truncate d2/d13/d5e/f6b 2302822 0 2024-03-04T16:04:23.591 INFO:tasks.workunit.client.0.smithi116.stdout:8/955: read d2/de3/f108 [137613,122591] 0 2024-03-04T16:04:23.591 INFO:tasks.workunit.client.0.smithi116.stdout:8/956: readlink d2/d13/d3c/d77/d97/d42/la2 0 2024-03-04T16:04:23.592 INFO:tasks.workunit.client.1.smithi152.stdout:5/76: truncate d6/d8/d10/f12 4860173 0 2024-03-04T16:04:23.594 INFO:tasks.workunit.client.1.smithi152.stdout:6/54: dwrite f6 [0,4194304] 0 2024-03-04T16:04:23.595 INFO:tasks.workunit.client.1.smithi152.stdout:5/77: truncate d6/d8/d10/f14 4900145 0 2024-03-04T16:04:23.598 INFO:tasks.workunit.client.0.smithi116.stdout:0/799: creat da/d16/d28/d39/ff5 x:0 0 0 2024-03-04T16:04:23.600 INFO:tasks.workunit.client.1.smithi152.stdout:5/78: mknod d6/d8/d10/c1c 0 2024-03-04T16:04:23.602 INFO:tasks.workunit.client.0.smithi116.stdout:4/942: rmdir d1/d3/d15/d20/d1c/ddd 39 2024-03-04T16:04:23.603 INFO:tasks.workunit.client.1.smithi152.stdout:6/55: dwrite d8/f12 [0,4194304] 0 2024-03-04T16:04:23.605 INFO:tasks.workunit.client.0.smithi116.stdout:7/908: rename d3/d4/d8/d4a/fdd to d3/d4/f12b 0 2024-03-04T16:04:23.610 INFO:tasks.workunit.client.0.smithi116.stdout:2/947: fdatasync d7/d13/f56 0 2024-03-04T16:04:23.612 INFO:tasks.workunit.client.0.smithi116.stdout:9/839: getdents d2/d26/dd4/d4a 0 2024-03-04T16:04:23.613 INFO:tasks.workunit.client.0.smithi116.stdout:7/909: dwrite d3/d4/d9e/dd7/f124 [0,4194304] 0 2024-03-04T16:04:23.614 INFO:tasks.workunit.client.0.smithi116.stdout:6/832: dread d6/de/d1e/f48 [4194304,4194304] 0 2024-03-04T16:04:23.614 INFO:tasks.workunit.client.0.smithi116.stdout:9/840: dread - d2/d19/fed zero size 2024-03-04T16:04:23.615 INFO:tasks.workunit.client.0.smithi116.stdout:7/910: read d3/f56 [742909,104618] 0 2024-03-04T16:04:23.616 INFO:tasks.workunit.client.0.smithi116.stdout:9/841: read - d2/d26/dd4/dab/d3c/f108 zero size 2024-03-04T16:04:23.618 INFO:tasks.workunit.client.1.smithi152.stdout:6/56: symlink d8/d10/l13 0 2024-03-04T16:04:23.619 INFO:tasks.workunit.client.0.smithi116.stdout:5/917: link dd/d1f/d55/d73/la8 dd/d6d/dc6/l128 0 2024-03-04T16:04:23.619 INFO:tasks.workunit.client.1.smithi152.stdout:6/57: chown d8/d10 104 1 2024-03-04T16:04:23.622 INFO:tasks.workunit.client.0.smithi116.stdout:8/957: dwrite d2/d13/d3c/d77/d97/f44 [0,4194304] 0 2024-03-04T16:04:23.624 INFO:tasks.workunit.client.0.smithi116.stdout:3/995: creat dc/d34/f14c x:0 0 0 2024-03-04T16:04:23.626 INFO:tasks.workunit.client.0.smithi116.stdout:4/943: creat d1/d3/dd/d45/f152 x:0 0 0 2024-03-04T16:04:23.627 INFO:tasks.workunit.client.0.smithi116.stdout:6/833: dread d6/de/f3b [0,4194304] 0 2024-03-04T16:04:23.628 INFO:tasks.workunit.client.0.smithi116.stdout:4/944: dread - d1/d3/d15/d20/d1c/d62/d81/f122 zero size 2024-03-04T16:04:23.628 INFO:tasks.workunit.client.0.smithi116.stdout:4/945: chown d1/d3/d15/d20/l8e 104105 1 2024-03-04T16:04:23.640 INFO:tasks.workunit.client.0.smithi116.stdout:2/948: read d7/d13/d3e/ded/d76/d7c/f145 [3889381,94760] 0 2024-03-04T16:04:23.640 INFO:tasks.workunit.client.0.smithi116.stdout:9/842: write d2/d19/fed [146796,99265] 0 2024-03-04T16:04:23.643 INFO:tasks.workunit.client.0.smithi116.stdout:3/996: creat dc/dd3/f14d x:0 0 0 2024-03-04T16:04:23.644 INFO:tasks.workunit.client.0.smithi116.stdout:6/834: fsync d6/de/d84/fa7 0 2024-03-04T16:04:23.646 INFO:tasks.workunit.client.0.smithi116.stdout:4/946: creat d1/d6c/dbf/f153 x:0 0 0 2024-03-04T16:04:23.648 INFO:tasks.workunit.client.1.smithi152.stdout:5/79: truncate d6/d8/fe 4754493 0 2024-03-04T16:04:23.649 INFO:tasks.workunit.client.1.smithi152.stdout:5/80: dread d6/f11 [0,4194304] 0 2024-03-04T16:04:23.651 INFO:tasks.workunit.client.1.smithi152.stdout:6/58: truncate f1 2606911 0 2024-03-04T16:04:23.652 INFO:tasks.workunit.client.0.smithi116.stdout:2/949: dwrite d7/d15/d5e/d9d/d11f/f9f [0,4194304] 0 2024-03-04T16:04:23.656 INFO:tasks.workunit.client.1.smithi152.stdout:6/59: creat d8/f14 x:0 0 0 2024-03-04T16:04:23.657 INFO:tasks.workunit.client.0.smithi116.stdout:0/800: rmdir da/d16/d28/dd5 0 2024-03-04T16:04:23.659 INFO:tasks.workunit.client.1.smithi152.stdout:6/60: creat d8/d10/f15 x:0 0 0 2024-03-04T16:04:23.659 INFO:tasks.workunit.client.1.smithi152.stdout:5/81: dwrite d6/d8/d10/f14 [0,4194304] 0 2024-03-04T16:04:23.660 INFO:tasks.workunit.client.1.smithi152.stdout:6/61: stat d8/d10/f15 0 2024-03-04T16:04:23.662 INFO:tasks.workunit.client.1.smithi152.stdout:6/62: write d8/d10/f15 [531430,119630] 0 2024-03-04T16:04:23.662 INFO:tasks.workunit.client.0.smithi116.stdout:2/950: dwrite d7/d81/dc2/f10f [0,4194304] 0 2024-03-04T16:04:23.662 INFO:tasks.workunit.client.0.smithi116.stdout:3/997: chown dc/df/d47/d12d/d7f/d93/db6/dc5/de0/da3/fb2 3 1 2024-03-04T16:04:23.664 INFO:tasks.workunit.client.0.smithi116.stdout:6/835: write d6/de/d25/d4a/d63/fe0 [4206233,20128] 0 2024-03-04T16:04:23.665 INFO:tasks.workunit.client.1.smithi152.stdout:5/82: creat d6/d8/f1d x:0 0 0 2024-03-04T16:04:23.668 INFO:tasks.workunit.client.1.smithi152.stdout:5/83: truncate d6/f18 679777 0 2024-03-04T16:04:23.668 INFO:tasks.workunit.client.0.smithi116.stdout:2/951: chown d7/d13/d3e/l11d 586632981 1 2024-03-04T16:04:23.669 INFO:tasks.workunit.client.0.smithi116.stdout:3/998: chown dc/df/d47/d12d/d7f/f8a 2 1 2024-03-04T16:04:23.670 INFO:tasks.workunit.client.1.smithi152.stdout:6/63: dwrite f0 [4194304,4194304] 0 2024-03-04T16:04:23.672 INFO:tasks.workunit.client.0.smithi116.stdout:1/777: rename d5/d9/d21/d27/d8c/lc6 to d5/d9/d21/d27/lff 0 2024-03-04T16:04:23.672 INFO:tasks.workunit.client.0.smithi116.stdout:3/999: write dc/d34/de7/deb/f116 [468969,36744] 0 2024-03-04T16:04:23.673 INFO:tasks.workunit.client.0.smithi116.stdout:8/958: dwrite d2/d13/d3c/d77/d97/fc2 [0,4194304] 0 2024-03-04T16:04:23.673 INFO:tasks.workunit.client.0.smithi116.stdout:7/911: dwrite d3/d4/d8/f13 [0,4194304] 0 2024-03-04T16:04:23.676 INFO:tasks.workunit.client.0.smithi116.stdout:1/778: fsync d5/d9/d21/d27/d32/d35/f52 0 2024-03-04T16:04:23.677 INFO:tasks.workunit.client.0.smithi116.stdout:5/918: write dd/d1f/d55/d73/d2d/d7c/f86 [224666,119925] 0 2024-03-04T16:04:23.677 INFO:tasks.workunit.client.0.smithi116.stdout:7/912: chown d3/d4/d8/d4a/d65/df2/l97 3081112 1 2024-03-04T16:04:23.678 INFO:tasks.workunit.client.0.smithi116.stdout:0/801: creat da/d16/d19/d27/d45/ff6 x:0 0 0 2024-03-04T16:04:23.682 INFO:tasks.workunit.client.0.smithi116.stdout:1/779: write d5/d9/d21/d27/d94/f95 [183494,46593] 0 2024-03-04T16:04:23.685 INFO:tasks.workunit.client.0.smithi116.stdout:6/836: creat d6/de/d25/d36/d44/dbb/ddd/d7b/f11a x:0 0 0 2024-03-04T16:04:23.686 INFO:tasks.workunit.client.0.smithi116.stdout:6/837: stat d6/de/d15/f26 0 2024-03-04T16:04:23.686 INFO:tasks.workunit.client.0.smithi116.stdout:9/843: creat d2/d33/f10c x:0 0 0 2024-03-04T16:04:23.688 INFO:tasks.workunit.client.0.smithi116.stdout:0/802: dwrite da/d16/d20/d42/fc6 [0,4194304] 0 2024-03-04T16:04:23.688 INFO:tasks.workunit.client.1.smithi152.stdout:5/84: rmdir d6/d19 0 2024-03-04T16:04:23.690 INFO:tasks.workunit.client.0.smithi116.stdout:0/803: chown da/d16/d28/d40/d48/d57/d8a/daa/dba/df0/db7/fe1 7708 1 2024-03-04T16:04:23.692 INFO:tasks.workunit.client.0.smithi116.stdout:2/952: unlink d7/d13/d3e/d98/dce/d47/d64/d75/fb6 0 2024-03-04T16:04:23.695 INFO:tasks.workunit.client.1.smithi152.stdout:5/85: mknod d6/d8/c1e 0 2024-03-04T16:04:23.699 INFO:tasks.workunit.client.1.smithi152.stdout:5/86: mkdir d6/d8/d1f 0 2024-03-04T16:04:23.701 INFO:tasks.workunit.client.0.smithi116.stdout:6/838: unlink d6/de/d25/d4a/l117 0 2024-03-04T16:04:23.704 INFO:tasks.workunit.client.1.smithi152.stdout:5/87: dread d6/fa [0,4194304] 0 2024-03-04T16:04:23.704 INFO:tasks.workunit.client.0.smithi116.stdout:0/804: unlink da/d16/d20/d42/c4b 0 2024-03-04T16:04:23.705 INFO:tasks.workunit.client.0.smithi116.stdout:9/844: dread d2/d26/fa0 [0,4194304] 0 2024-03-04T16:04:23.707 INFO:tasks.workunit.client.1.smithi152.stdout:5/88: symlink d6/d8/d1f/l20 0 2024-03-04T16:04:23.709 INFO:tasks.workunit.client.0.smithi116.stdout:6/839: fdatasync d6/de/f3b 0 2024-03-04T16:04:23.709 INFO:tasks.workunit.client.1.smithi152.stdout:5/89: creat d6/d8/d10/f21 x:0 0 0 2024-03-04T16:04:23.710 INFO:tasks.workunit.client.0.smithi116.stdout:4/947: link d1/d3/dd/l10d d1/d6c/l154 0 2024-03-04T16:04:23.712 INFO:tasks.workunit.client.0.smithi116.stdout:9/845: dwrite d2/d19/d5f/fe4 [0,4194304] 0 2024-03-04T16:04:23.713 INFO:tasks.workunit.client.0.smithi116.stdout:0/805: getdents da/d16/d28/d3d 0 2024-03-04T16:04:23.715 INFO:tasks.workunit.client.0.smithi116.stdout:1/780: dread d5/d9/d21/d27/d94/db6/d42/f88 [0,4194304] 0 2024-03-04T16:04:23.717 INFO:tasks.workunit.client.1.smithi152.stdout:5/90: dwrite d6/d8/d10/f21 [0,4194304] 0 2024-03-04T16:04:23.719 INFO:tasks.workunit.client.0.smithi116.stdout:6/840: rename d6/f13 to d6/de/d25/d36/d44/dbb/ddd/d7b/dc6/f11b 0 2024-03-04T16:04:23.720 INFO:tasks.workunit.client.0.smithi116.stdout:6/841: write d6/de/d84/f10f [788355,82032] 0 2024-03-04T16:04:23.723 INFO:tasks.workunit.client.0.smithi116.stdout:6/842: write d6/d56/f6c [1249438,69208] 0 2024-03-04T16:04:23.724 INFO:tasks.workunit.client.0.smithi116.stdout:0/806: dwrite da/d16/d19/d27/d70/f8e [4194304,4194304] 0 2024-03-04T16:04:23.725 INFO:tasks.workunit.client.0.smithi116.stdout:8/959: dread d2/d13/f90 [0,4194304] 0 2024-03-04T16:04:23.729 INFO:tasks.workunit.client.0.smithi116.stdout:4/948: link d1/d3/de/f23 d1/d3/dd/d45/d50/dad/f155 0 2024-03-04T16:04:23.730 INFO:tasks.workunit.client.0.smithi116.stdout:9/846: mkdir d2/d26/dd4/de5/dcc/d10d 0 2024-03-04T16:04:23.732 INFO:tasks.workunit.client.0.smithi116.stdout:6/843: dwrite d6/de/d5b/f8e [0,4194304] 0 2024-03-04T16:04:23.735 INFO:tasks.workunit.client.0.smithi116.stdout:8/960: rmdir d2/d13/d3c/d46/d6e 39 2024-03-04T16:04:23.737 INFO:tasks.workunit.client.0.smithi116.stdout:2/953: rmdir d7/d13/d3e/d98/dce/d47/d64 39 2024-03-04T16:04:23.738 INFO:tasks.workunit.client.0.smithi116.stdout:2/954: fsync d7/d13/dec/f13e 0 2024-03-04T16:04:23.742 INFO:tasks.workunit.client.0.smithi116.stdout:7/913: write d3/d4/d34/d5d/db9/fd4 [315291,40040] 0 2024-03-04T16:04:23.745 INFO:tasks.workunit.client.0.smithi116.stdout:8/961: dread d2/de/d3e/fa8 [0,4194304] 0 2024-03-04T16:04:23.748 INFO:tasks.workunit.client.0.smithi116.stdout:5/919: dwrite dd/d1f/d55/d73/d2d/dae/fbe [0,4194304] 0 2024-03-04T16:04:23.750 INFO:tasks.workunit.client.0.smithi116.stdout:5/920: write dd/d1f/d55/d73/d2d/d59/f81 [1221686,111427] 0 2024-03-04T16:04:23.751 INFO:tasks.workunit.client.0.smithi116.stdout:7/914: dwrite d3/d4/d8/d1a/ff7 [0,4194304] 0 2024-03-04T16:04:23.753 INFO:tasks.workunit.client.0.smithi116.stdout:0/807: creat da/d16/d20/d42/de0/ff7 x:0 0 0 2024-03-04T16:04:23.760 INFO:tasks.workunit.client.0.smithi116.stdout:5/921: unlink dd/d1f/d5c/d5e/dc2/f107 0 2024-03-04T16:04:23.761 INFO:tasks.workunit.client.0.smithi116.stdout:4/949: getdents d1/d6e/d74/de0 0 2024-03-04T16:04:23.763 INFO:tasks.workunit.client.0.smithi116.stdout:7/915: creat d3/d4/d8/d1a/d126/f12c x:0 0 0 2024-03-04T16:04:23.763 INFO:tasks.workunit.client.0.smithi116.stdout:0/808: dwrite da/d16/d19/d27/d45/ff2 [0,4194304] 0 2024-03-04T16:04:23.764 INFO:tasks.workunit.client.0.smithi116.stdout:1/781: dwrite d5/d9/f41 [0,4194304] 0 2024-03-04T16:04:23.768 INFO:tasks.workunit.client.0.smithi116.stdout:4/950: unlink d1/d3/d15/d20/d1c/d62/f13d 0 2024-03-04T16:04:23.768 INFO:tasks.workunit.client.0.smithi116.stdout:7/916: read d3/d4/d9e/dd7/daf/dd5/f10d [4646126,24770] 0 2024-03-04T16:04:23.769 INFO:tasks.workunit.client.0.smithi116.stdout:0/809: symlink da/d16/d19/d27/d70/lf8 0 2024-03-04T16:04:23.770 INFO:tasks.workunit.client.0.smithi116.stdout:0/810: stat da/d16/d19/d27/f34 0 2024-03-04T16:04:23.771 INFO:tasks.workunit.client.0.smithi116.stdout:7/917: truncate d3/d4/d8/d52/f11b 122896 0 2024-03-04T16:04:23.771 INFO:tasks.workunit.client.0.smithi116.stdout:1/782: creat d5/d9/d21/d27/d32/d35/d55/d75/d91/f100 x:0 0 0 2024-03-04T16:04:23.776 INFO:tasks.workunit.client.0.smithi116.stdout:7/918: symlink d3/d4/d8/d4a/d65/l12d 0 2024-03-04T16:04:23.776 INFO:tasks.workunit.client.0.smithi116.stdout:0/811: chown da/d16/d19/c31 5981079 1 2024-03-04T16:04:23.778 INFO:tasks.workunit.client.0.smithi116.stdout:0/812: rmdir da/d16/d28/d40 39 2024-03-04T16:04:23.778 INFO:tasks.workunit.client.0.smithi116.stdout:0/813: dread - da/d16/d19/d27/d45/ff6 zero size 2024-03-04T16:04:23.779 INFO:tasks.workunit.client.0.smithi116.stdout:7/919: creat d3/d4/d9e/dd7/daf/dd5/d123/d9a/f12e x:0 0 0 2024-03-04T16:04:23.780 INFO:tasks.workunit.client.0.smithi116.stdout:0/814: read - da/d16/d19/d27/d45/ff6 zero size 2024-03-04T16:04:23.781 INFO:tasks.workunit.client.0.smithi116.stdout:7/920: chown d3/d4/d8/d4a/c78 3429 1 2024-03-04T16:04:23.782 INFO:tasks.workunit.client.0.smithi116.stdout:7/921: chown d3/d4/d9e/ff8 11882 1 2024-03-04T16:04:23.784 INFO:tasks.workunit.client.0.smithi116.stdout:7/922: symlink d3/d4/d8/d4a/d65/df2/l12f 0 2024-03-04T16:04:23.785 INFO:tasks.workunit.client.0.smithi116.stdout:0/815: getdents da/d16/d20/d42/d4e 0 2024-03-04T16:04:23.786 INFO:tasks.workunit.client.0.smithi116.stdout:2/955: write d7/d81/f39 [779581,83306] 0 2024-03-04T16:04:23.786 INFO:tasks.workunit.client.0.smithi116.stdout:9/847: write d2/d33/d40/f8b [41515,128628] 0 2024-03-04T16:04:23.789 INFO:tasks.workunit.client.0.smithi116.stdout:0/816: fsync da/d16/d28/d40/d48/d57/db3/db2/fb6 0 2024-03-04T16:04:23.790 INFO:tasks.workunit.client.0.smithi116.stdout:0/817: write da/d16/d19/fe9 [882715,94465] 0 2024-03-04T16:04:23.790 INFO:tasks.workunit.client.1.smithi152.stdout:5/91: dwrite d6/d8/d10/f12 [0,4194304] 0 2024-03-04T16:04:23.792 INFO:tasks.workunit.client.0.smithi116.stdout:6/844: write d6/de/d25/faa [446418,100377] 0 2024-03-04T16:04:23.794 INFO:tasks.workunit.client.0.smithi116.stdout:6/845: write d6/de/d1e/d80/fce [3820297,749] 0 2024-03-04T16:04:23.797 INFO:tasks.workunit.client.1.smithi152.stdout:5/92: symlink d6/d8/l22 0 2024-03-04T16:04:23.798 INFO:tasks.workunit.client.1.smithi152.stdout:5/93: chown d6/f18 66472148 1 2024-03-04T16:04:23.798 INFO:tasks.workunit.client.1.smithi152.stdout:5/94: chown d6/fa 49687 1 2024-03-04T16:04:23.799 INFO:tasks.workunit.client.1.smithi152.stdout:5/95: stat d6/l9 0 2024-03-04T16:04:23.800 INFO:tasks.workunit.client.0.smithi116.stdout:2/956: dwrite d7/d13/d3e/d98/dce/d47/d64/f87 [0,4194304] 0 2024-03-04T16:04:23.800 INFO:tasks.workunit.client.0.smithi116.stdout:4/951: dread d1/d3/d15/d20/f4e [0,4194304] 0 2024-03-04T16:04:23.804 INFO:tasks.workunit.client.1.smithi152.stdout:5/96: creat d6/d8/d1f/f23 x:0 0 0 2024-03-04T16:04:23.804 INFO:tasks.workunit.client.1.smithi152.stdout:5/97: readlink d6/d8/d1f/l20 0 2024-03-04T16:04:23.805 INFO:tasks.workunit.client.1.smithi152.stdout:5/98: fdatasync d6/d8/f15 0 2024-03-04T16:04:23.808 INFO:tasks.workunit.client.0.smithi116.stdout:2/957: dwrite d7/d13/d3e/d98/dce/d47/d64/f87 [0,4194304] 0 2024-03-04T16:04:23.813 INFO:tasks.workunit.client.0.smithi116.stdout:6/846: link d6/d56/f8b d6/f11c 0 2024-03-04T16:04:23.814 INFO:tasks.workunit.client.0.smithi116.stdout:0/818: dread da/d16/d20/d42/de0/fad [0,4194304] 0 2024-03-04T16:04:23.817 INFO:tasks.workunit.client.0.smithi116.stdout:5/922: truncate dd/d1f/d55/d73/d2d/d7c/f86 725734 0 2024-03-04T16:04:23.821 INFO:tasks.workunit.client.0.smithi116.stdout:7/923: rename d3/d4/d8 to d3/d4/dbf/de6/dd3/ddf/dee/d130 0 2024-03-04T16:04:23.822 INFO:tasks.workunit.client.0.smithi116.stdout:6/847: creat d6/de/d1e/d80/f11d x:0 0 0 2024-03-04T16:04:23.824 INFO:tasks.workunit.client.0.smithi116.stdout:2/958: fdatasync d7/d13/d3e/d98/dce/d47/f49 0 2024-03-04T16:04:23.827 INFO:tasks.workunit.client.0.smithi116.stdout:4/952: rename d1/d6c/dbf/ff4 to d1/d3/dd/d24/db4/f156 0 2024-03-04T16:04:23.828 INFO:tasks.workunit.client.0.smithi116.stdout:6/848: symlink d6/de/d84/db4/d92/l11e 0 2024-03-04T16:04:23.829 INFO:tasks.workunit.client.0.smithi116.stdout:9/848: dwrite d2/f87 [0,4194304] 0 2024-03-04T16:04:23.831 INFO:tasks.workunit.client.0.smithi116.stdout:7/924: unlink d3/d4/dbf/de6/dd3/ddf/dee/d130/d1a/ff7 0 2024-03-04T16:04:23.834 INFO:tasks.workunit.client.0.smithi116.stdout:2/959: dwrite d7/d13/d3e/ded/d76/fda [0,4194304] 0 2024-03-04T16:04:23.835 INFO:tasks.workunit.client.1.smithi152.stdout:5/99: write d6/f11 [808265,38745] 0 2024-03-04T16:04:23.837 INFO:tasks.workunit.client.1.smithi152.stdout:5/100: truncate d6/d8/d10/f21 5212284 0 2024-03-04T16:04:23.838 INFO:tasks.workunit.client.0.smithi116.stdout:0/819: link da/d16/d19/d27/d82/d9f/cf3 da/d16/d20/d42/de0/d7e/cf9 0 2024-03-04T16:04:23.839 INFO:tasks.workunit.client.0.smithi116.stdout:0/820: write da/d16/d20/d42/de0/d7e/fb9 [293799,104633] 0 2024-03-04T16:04:23.840 INFO:tasks.workunit.client.0.smithi116.stdout:0/821: fsync da/d16/f6e 0 2024-03-04T16:04:23.841 INFO:tasks.workunit.client.0.smithi116.stdout:9/849: write d2/f48 [2245801,69582] 0 2024-03-04T16:04:23.841 INFO:tasks.workunit.client.1.smithi152.stdout:5/101: creat d6/d8/f24 x:0 0 0 2024-03-04T16:04:23.842 INFO:tasks.workunit.client.0.smithi116.stdout:6/849: truncate d6/de/d1e/f3f 1148564 0 2024-03-04T16:04:23.843 INFO:tasks.workunit.client.0.smithi116.stdout:6/850: fdatasync d6/de/d84/db4/fe2 0 2024-03-04T16:04:23.846 INFO:tasks.workunit.client.0.smithi116.stdout:7/925: rename d3/d4/dbf/de6/dd3/ddf/de0/fea to d3/d4/d9e/dd7/dd1/f131 0 2024-03-04T16:04:23.848 INFO:tasks.workunit.client.1.smithi152.stdout:5/102: rename d6/f11 to d6/d8/d1f/f25 0 2024-03-04T16:04:23.849 INFO:tasks.workunit.client.1.smithi152.stdout:5/103: write d6/d8/f1d [457425,8266] 0 2024-03-04T16:04:23.849 INFO:tasks.workunit.client.0.smithi116.stdout:2/960: stat d7/dd3/fe2 0 2024-03-04T16:04:23.854 INFO:tasks.workunit.client.1.smithi152.stdout:5/104: creat d6/d8/d10/f26 x:0 0 0 2024-03-04T16:04:23.857 INFO:tasks.workunit.client.0.smithi116.stdout:5/923: getdents dd/d1f/d51 0 2024-03-04T16:04:23.858 INFO:tasks.workunit.client.0.smithi116.stdout:6/851: creat d6/de/d25/d4a/d63/d8c/f11f x:0 0 0 2024-03-04T16:04:23.859 INFO:tasks.workunit.client.0.smithi116.stdout:7/926: creat d3/d4/d34/d5d/ded/f132 x:0 0 0 2024-03-04T16:04:23.861 INFO:tasks.workunit.client.0.smithi116.stdout:2/961: rename d7/d13/d3e/d98/dce/d47/d64/d75 to d7/d13/d3e/d71/dad/d147/d162 0 2024-03-04T16:04:23.862 INFO:tasks.workunit.client.0.smithi116.stdout:0/822: readlink da/d16/d28/d40/d48/d77/lbc 0 2024-03-04T16:04:23.863 INFO:tasks.workunit.client.0.smithi116.stdout:5/924: truncate dd/d45/f49 4029239 0 2024-03-04T16:04:23.864 INFO:tasks.workunit.client.0.smithi116.stdout:6/852: dwrite d6/de/d25/d36/d4e/dfa/f112 [0,4194304] 0 2024-03-04T16:04:23.869 INFO:tasks.workunit.client.0.smithi116.stdout:7/927: write d3/d95/ff6 [3297276,49655] 0 2024-03-04T16:04:23.870 INFO:tasks.workunit.client.0.smithi116.stdout:9/850: read d2/d26/d28/f55 [951,122361] 0 2024-03-04T16:04:23.871 INFO:tasks.workunit.client.0.smithi116.stdout:0/823: creat da/d16/d28/d40/da8/ffa x:0 0 0 2024-03-04T16:04:23.876 INFO:tasks.workunit.client.0.smithi116.stdout:6/853: mknod d6/d56/d58/c120 0 2024-03-04T16:04:23.877 INFO:tasks.workunit.client.0.smithi116.stdout:0/824: fdatasync da/d16/d20/f3b 0 2024-03-04T16:04:23.881 INFO:tasks.workunit.client.0.smithi116.stdout:7/928: dwrite d3/d4/dbf/de6/dd3/ddf/dee/d130/d1a/f20 [4194304,4194304] 0 2024-03-04T16:04:23.886 INFO:tasks.workunit.client.0.smithi116.stdout:4/953: dwrite d1/d3/de/f26 [0,4194304] 0 2024-03-04T16:04:23.887 INFO:tasks.workunit.client.0.smithi116.stdout:0/825: mknod da/d16/cfb 0 2024-03-04T16:04:23.893 INFO:tasks.workunit.client.0.smithi116.stdout:2/962: dwrite d7/d81/f22 [4194304,4194304] 0 2024-03-04T16:04:23.899 INFO:tasks.workunit.client.0.smithi116.stdout:4/954: creat d1/d3/d15/d20/d1c/d6f/f157 x:0 0 0 2024-03-04T16:04:23.903 INFO:tasks.workunit.client.0.smithi116.stdout:5/925: dwrite dd/d1f/d5c/f9e [0,4194304] 0 2024-03-04T16:04:23.904 INFO:tasks.workunit.client.0.smithi116.stdout:7/929: dwrite d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/f6b [4194304,4194304] 0 2024-03-04T16:04:23.905 INFO:tasks.workunit.client.0.smithi116.stdout:2/963: rename d7/d13/f141 to d7/d15/dd7/f163 0 2024-03-04T16:04:23.906 INFO:tasks.workunit.client.0.smithi116.stdout:5/926: chown dd/d6d/f80 23061042 1 2024-03-04T16:04:23.906 INFO:tasks.workunit.client.0.smithi116.stdout:4/955: creat d1/d3/dd/d45/f158 x:0 0 0 2024-03-04T16:04:23.906 INFO:tasks.workunit.client.0.smithi116.stdout:0/826: dwrite da/d16/d20/d53/f63 [4194304,4194304] 0 2024-03-04T16:04:23.908 INFO:tasks.workunit.client.0.smithi116.stdout:7/930: chown d3/d4/d9e/dd7/dfd/f122 100 1 2024-03-04T16:04:23.909 INFO:tasks.workunit.client.0.smithi116.stdout:2/964: fdatasync d7/d13/f56 0 2024-03-04T16:04:23.912 INFO:tasks.workunit.client.0.smithi116.stdout:5/927: truncate dd/d1f/d55/d73/dba/fd5 951843 0 2024-03-04T16:04:23.913 INFO:tasks.workunit.client.0.smithi116.stdout:5/928: chown dd/d1f/d55/d73/d2d/d59/db0/dca/c10e 3 1 2024-03-04T16:04:23.913 INFO:tasks.workunit.client.0.smithi116.stdout:4/956: symlink d1/d6e/l159 0 2024-03-04T16:04:23.915 INFO:tasks.workunit.client.0.smithi116.stdout:0/827: mkdir da/d16/d28/d7b/dfc 0 2024-03-04T16:04:23.918 INFO:tasks.workunit.client.0.smithi116.stdout:4/957: fsync d1/d3/d15/d20/fa3 0 2024-03-04T16:04:23.920 INFO:tasks.workunit.client.0.smithi116.stdout:0/828: symlink da/d16/d19/d27/d70/lfd 0 2024-03-04T16:04:23.920 INFO:tasks.workunit.client.0.smithi116.stdout:2/965: dread d7/d13/d3e/d98/dce/f106 [0,4194304] 0 2024-03-04T16:04:23.924 INFO:tasks.workunit.client.0.smithi116.stdout:7/931: rename d3/d4/f16 to d3/d4/dbf/de6/dd3/f133 0 2024-03-04T16:04:23.927 INFO:tasks.workunit.client.0.smithi116.stdout:4/958: mknod d1/d3/dd/dd9/d11a/d12e/c15a 0 2024-03-04T16:04:23.927 INFO:tasks.workunit.client.0.smithi116.stdout:5/929: getdents dd/d1f/d55/d73/d2d/d59/d71 0 2024-03-04T16:04:23.930 INFO:tasks.workunit.client.0.smithi116.stdout:0/829: rename da/d16/d28/d40/d48/d57/d8a/daa/dba/df0/db7/fe1 to da/d16/d28/d40/d48/d57/db3/db2/ffe 0 2024-03-04T16:04:23.931 INFO:tasks.workunit.client.0.smithi116.stdout:7/932: creat d3/d4/d34/d5d/ded/d108/f134 x:0 0 0 2024-03-04T16:04:23.932 INFO:tasks.workunit.client.0.smithi116.stdout:7/933: chown d3/d4/d34/c94 59480 1 2024-03-04T16:04:23.935 INFO:tasks.workunit.client.0.smithi116.stdout:0/830: truncate da/d16/d19/d27/f74 1375676 0 2024-03-04T16:04:23.936 INFO:tasks.workunit.client.0.smithi116.stdout:7/934: readlink d3/d4/dbf/de6/dd3/ddf/dee/d130/lc7 0 2024-03-04T16:04:23.943 INFO:tasks.workunit.client.0.smithi116.stdout:7/935: dwrite d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/d113/db6/fda [0,4194304] 0 2024-03-04T16:04:23.947 INFO:tasks.workunit.client.0.smithi116.stdout:7/936: mknod d3/d95/c135 0 2024-03-04T16:04:23.952 INFO:tasks.workunit.client.0.smithi116.stdout:7/937: mkdir d3/d4/d34/d5d/ded/d136 0 2024-03-04T16:04:23.952 INFO:tasks.workunit.client.0.smithi116.stdout:6/854: dwrite d6/de/d25/f96 [0,4194304] 0 2024-03-04T16:04:23.959 INFO:tasks.workunit.client.0.smithi116.stdout:9/851: dread d2/d19/d27/f9d [0,4194304] 0 2024-03-04T16:04:23.959 INFO:tasks.workunit.client.0.smithi116.stdout:6/855: unlink d6/de/d25/d36/d4e/dfa/d111/f116 0 2024-03-04T16:04:23.962 INFO:tasks.workunit.client.0.smithi116.stdout:9/852: symlink d2/d33/d90/l10e 0 2024-03-04T16:04:23.965 INFO:tasks.workunit.client.0.smithi116.stdout:6/856: dwrite d6/de/d1e/d80/f89 [0,4194304] 0 2024-03-04T16:04:23.967 INFO:tasks.workunit.client.0.smithi116.stdout:2/966: write d7/d81/dc2/ff6 [26996,5418] 0 2024-03-04T16:04:23.969 INFO:tasks.workunit.client.0.smithi116.stdout:6/857: rmdir d6/de/d25/d36/d44/dbb 39 2024-03-04T16:04:23.974 INFO:tasks.workunit.client.0.smithi116.stdout:2/967: mkdir d7/d15/d5e/d9d/dd4/d164 0 2024-03-04T16:04:23.974 INFO:tasks.workunit.client.0.smithi116.stdout:2/968: fsync d7/d25/f12d 0 2024-03-04T16:04:23.976 INFO:tasks.workunit.client.0.smithi116.stdout:6/858: rename d6/d56/d58/l5d to d6/de/d1e/d7c/l121 0 2024-03-04T16:04:23.976 INFO:tasks.workunit.client.0.smithi116.stdout:5/930: write dd/d1f/d55/d73/f3f [1474161,94382] 0 2024-03-04T16:04:23.977 INFO:tasks.workunit.client.0.smithi116.stdout:2/969: rmdir d7/d81 39 2024-03-04T16:04:23.979 INFO:tasks.workunit.client.0.smithi116.stdout:6/859: chown d6/de/d84/db4/d92/fed 749811357 1 2024-03-04T16:04:23.979 INFO:tasks.workunit.client.0.smithi116.stdout:0/831: write da/f1e [514625,92683] 0 2024-03-04T16:04:23.979 INFO:tasks.workunit.client.0.smithi116.stdout:5/931: read dd/d1f/d72/fa0 [229048,14863] 0 2024-03-04T16:04:23.981 INFO:tasks.workunit.client.0.smithi116.stdout:4/959: dwrite d1/d3/dd/f98 [0,4194304] 0 2024-03-04T16:04:23.984 INFO:tasks.workunit.client.0.smithi116.stdout:2/970: creat d7/d13/d3e/d98/dce/d47/d64/da5/f165 x:0 0 0 2024-03-04T16:04:23.985 INFO:tasks.workunit.client.0.smithi116.stdout:7/938: dread d3/d4/dbf/de6/dd3/ddf/dee/d130/f40 [0,4194304] 0 2024-03-04T16:04:23.989 INFO:tasks.workunit.client.0.smithi116.stdout:1/783: sync 2024-03-04T16:04:23.989 INFO:tasks.workunit.client.0.smithi116.stdout:8/962: sync 2024-03-04T16:04:23.990 INFO:tasks.workunit.client.0.smithi116.stdout:6/860: rename d6/de/d1e/d80 to d6/de/d25/d4a/df8/da5/db3/d122 0 2024-03-04T16:04:23.993 INFO:tasks.workunit.client.0.smithi116.stdout:4/960: creat d1/d3/dd/d45/d108/f15b x:0 0 0 2024-03-04T16:04:23.994 INFO:tasks.workunit.client.0.smithi116.stdout:7/939: dread - d3/d4/d9e/dd7/daf/dd5/d123/fa4 zero size 2024-03-04T16:04:23.995 INFO:tasks.workunit.client.0.smithi116.stdout:9/853: dread d2/d26/f47 [0,4194304] 0 2024-03-04T16:04:23.996 INFO:tasks.workunit.client.0.smithi116.stdout:6/861: chown d6/de/c10 701341697 1 2024-03-04T16:04:23.996 INFO:tasks.workunit.client.0.smithi116.stdout:8/963: truncate d2/de3/fe8 464833 0 2024-03-04T16:04:23.998 INFO:tasks.workunit.client.0.smithi116.stdout:1/784: mknod d5/d9/d21/d27/d94/db6/d61/c101 0 2024-03-04T16:04:24.000 INFO:tasks.workunit.client.0.smithi116.stdout:2/971: rename d7/d81/f7b to d7/d15/dd7/f166 0 2024-03-04T16:04:24.001 INFO:tasks.workunit.client.0.smithi116.stdout:4/961: dwrite d1/d3/d15/d20/d1c/d62/d81/daf/f10b [0,4194304] 0 2024-03-04T16:04:24.002 INFO:tasks.workunit.client.0.smithi116.stdout:2/972: stat d7/d81/dc2 0 2024-03-04T16:04:24.007 INFO:tasks.workunit.client.0.smithi116.stdout:0/832: truncate da/d16/d20/d42/de0/f6d 947979 0 2024-03-04T16:04:24.008 INFO:tasks.workunit.client.0.smithi116.stdout:9/854: symlink d2/d26/dd4/l10f 0 2024-03-04T16:04:24.010 INFO:tasks.workunit.client.0.smithi116.stdout:5/932: dwrite dd/d1f/d55/d73/d2d/fc8 [0,4194304] 0 2024-03-04T16:04:24.010 INFO:tasks.workunit.client.0.smithi116.stdout:1/785: dwrite d5/d9/d21/d27/d32/d35/f5c [0,4194304] 0 2024-03-04T16:04:24.012 INFO:tasks.workunit.client.0.smithi116.stdout:8/964: unlink d2/de/d16/d78/d29/ff0 0 2024-03-04T16:04:24.013 INFO:tasks.workunit.client.0.smithi116.stdout:9/855: chown d2/d26/dd4/de5/f97 0 1 2024-03-04T16:04:24.013 INFO:tasks.workunit.client.0.smithi116.stdout:4/962: rename d1/d3/de/c21 to d1/d3/dd/d8a/ddc/c15c 0 2024-03-04T16:04:24.017 INFO:tasks.workunit.client.0.smithi116.stdout:2/973: dwrite d7/d13/d3e/d78/d107/f151 [0,4194304] 0 2024-03-04T16:04:24.017 INFO:tasks.workunit.client.0.smithi116.stdout:9/856: write d2/d19/fed [939512,118080] 0 2024-03-04T16:04:24.020 INFO:tasks.workunit.client.0.smithi116.stdout:4/963: chown d1/d3/de/d47/df9/d132/f139 816576 1 2024-03-04T16:04:24.020 INFO:tasks.workunit.client.0.smithi116.stdout:0/833: write da/d16/d28/d40/d48/d57/f8c [1158524,45568] 0 2024-03-04T16:04:24.022 INFO:tasks.workunit.client.0.smithi116.stdout:0/834: stat da/d16/d20/d42/de0/fad 0 2024-03-04T16:04:24.024 INFO:tasks.workunit.client.0.smithi116.stdout:4/964: write d1/d3/d40/f14b [391767,93904] 0 2024-03-04T16:04:24.024 INFO:tasks.workunit.client.0.smithi116.stdout:5/933: unlink dd/d1f/dcd/d122/l127 0 2024-03-04T16:04:24.024 INFO:tasks.workunit.client.0.smithi116.stdout:7/940: getdents d3/d4/d34/d5d/db9 0 2024-03-04T16:04:24.024 INFO:tasks.workunit.client.0.smithi116.stdout:9/857: truncate d2/d26/dd4/de5/fa8 4955270 0 2024-03-04T16:04:24.026 INFO:tasks.workunit.client.0.smithi116.stdout:8/965: chown d2/d13/d3c/d46/d6e/c12f 490 1 2024-03-04T16:04:24.027 INFO:tasks.workunit.client.0.smithi116.stdout:4/965: fdatasync d1/d3/dd/d45/f158 0 2024-03-04T16:04:24.027 INFO:tasks.workunit.client.0.smithi116.stdout:2/974: chown d7/lb9 11605454 1 2024-03-04T16:04:24.029 INFO:tasks.workunit.client.0.smithi116.stdout:5/934: creat dd/d1f/d5c/d5e/f129 x:0 0 0 2024-03-04T16:04:24.032 INFO:tasks.workunit.client.0.smithi116.stdout:0/835: creat da/d16/d28/d40/d48/d57/dd6/fff x:0 0 0 2024-03-04T16:04:24.033 INFO:tasks.workunit.client.0.smithi116.stdout:8/966: dread d2/d13/d3c/d46/d6e/fbb [0,4194304] 0 2024-03-04T16:04:24.038 INFO:tasks.workunit.client.0.smithi116.stdout:0/836: dread da/d16/d28/fb0 [0,4194304] 0 2024-03-04T16:04:24.041 INFO:tasks.workunit.client.0.smithi116.stdout:8/967: rename d2/d13/d5e/l136 to d2/d13/d3c/d46/d96/l140 0 2024-03-04T16:04:24.041 INFO:tasks.workunit.client.0.smithi116.stdout:8/968: dread - d2/d13/d5e/fc4 zero size 2024-03-04T16:04:24.042 INFO:tasks.workunit.client.0.smithi116.stdout:5/935: dread dd/d1f/d55/d73/d2d/f42 [0,4194304] 0 2024-03-04T16:04:24.053 INFO:tasks.workunit.client.0.smithi116.stdout:8/969: symlink d2/d38/d41/dab/d124/l141 0 2024-03-04T16:04:24.054 INFO:tasks.workunit.client.0.smithi116.stdout:9/858: dread d2/d26/dd4/dab/f89 [0,4194304] 0 2024-03-04T16:04:24.055 INFO:tasks.workunit.client.0.smithi116.stdout:9/859: chown d2/d33/dc5/ff5 7265 1 2024-03-04T16:04:24.056 INFO:tasks.workunit.client.0.smithi116.stdout:9/860: chown d2/d26/dd4/dab/d3c 573 1 2024-03-04T16:04:24.058 INFO:tasks.workunit.client.0.smithi116.stdout:4/966: getdents d1/d3/dd/d45/d108 0 2024-03-04T16:04:24.061 INFO:tasks.workunit.client.0.smithi116.stdout:1/786: dwrite d5/d9/f3f [0,4194304] 0 2024-03-04T16:04:24.062 INFO:tasks.workunit.client.0.smithi116.stdout:1/787: dread - d5/d9/d21/d84/db0/feb zero size 2024-03-04T16:04:24.062 INFO:tasks.workunit.client.0.smithi116.stdout:5/936: fsync dd/d1f/d55/d73/d2d/d7c/f86 0 2024-03-04T16:04:24.066 INFO:tasks.workunit.client.0.smithi116.stdout:9/861: creat d2/d19/d5f/f110 x:0 0 0 2024-03-04T16:04:24.075 INFO:tasks.workunit.client.0.smithi116.stdout:8/970: link d2/d38/d55/f65 d2/d38/d41/dab/f142 0 2024-03-04T16:04:24.079 INFO:tasks.workunit.client.0.smithi116.stdout:5/937: chown dd/c27 78745 1 2024-03-04T16:04:24.080 INFO:tasks.workunit.client.0.smithi116.stdout:5/938: chown dd/d1f/fb2 3227 1 2024-03-04T16:04:24.082 INFO:tasks.workunit.client.0.smithi116.stdout:9/862: mknod d2/d19/d27/dbd/c111 0 2024-03-04T16:04:24.083 INFO:tasks.workunit.client.0.smithi116.stdout:9/863: dread - d2/d26/dd4/d4a/d6a/d94/dbe/feb zero size 2024-03-04T16:04:24.083 INFO:tasks.workunit.client.0.smithi116.stdout:2/975: getdents d7/d13/d3e 0 2024-03-04T16:04:24.084 INFO:tasks.workunit.client.0.smithi116.stdout:7/941: truncate d3/d4/dbf/de6/dd3/ddf/dee/d130/f45 8328671 0 2024-03-04T16:04:24.084 INFO:tasks.workunit.client.0.smithi116.stdout:2/976: dread - d7/d13/d3e/d71/dad/fc4 zero size 2024-03-04T16:04:24.085 INFO:tasks.workunit.client.0.smithi116.stdout:2/977: read - d7/d15/d5e/f140 zero size 2024-03-04T16:04:24.085 INFO:tasks.workunit.client.0.smithi116.stdout:7/942: chown d3/d4/dbf/de6/dd3/ddf/dee/d10a 41 1 2024-03-04T16:04:24.087 INFO:tasks.workunit.client.0.smithi116.stdout:4/967: creat d1/d3/d15/d20/d1c/ddd/f15d x:0 0 0 2024-03-04T16:04:24.089 INFO:tasks.workunit.client.0.smithi116.stdout:0/837: write da/d16/d19/d27/f34 [1730175,127157] 0 2024-03-04T16:04:24.090 INFO:tasks.workunit.client.0.smithi116.stdout:4/968: dread d1/fb [0,4194304] 0 2024-03-04T16:04:24.092 INFO:tasks.workunit.client.0.smithi116.stdout:4/969: write d1/d3/d40/d52/d84/d88/d8d/fc8 [5012141,73106] 0 2024-03-04T16:04:24.094 INFO:tasks.workunit.client.0.smithi116.stdout:7/943: creat d3/d4/d34/f137 x:0 0 0 2024-03-04T16:04:24.095 INFO:tasks.workunit.client.0.smithi116.stdout:7/944: write f1 [673305,19017] 0 2024-03-04T16:04:24.096 INFO:tasks.workunit.client.0.smithi116.stdout:2/978: creat d7/d13/d3e/d98/dce/d47/dd9/f167 x:0 0 0 2024-03-04T16:04:24.098 INFO:tasks.workunit.client.0.smithi116.stdout:4/970: dwrite d1/d3/d40/d52/fbc [0,4194304] 0 2024-03-04T16:04:24.105 INFO:tasks.workunit.client.0.smithi116.stdout:4/971: dwrite d1/d6c/f38 [0,4194304] 0 2024-03-04T16:04:24.119 INFO:tasks.workunit.client.0.smithi116.stdout:9/864: creat d2/d26/f112 x:0 0 0 2024-03-04T16:04:24.120 INFO:tasks.workunit.client.0.smithi116.stdout:9/865: truncate d2/d33/d65/fea 286344 0 2024-03-04T16:04:24.124 INFO:tasks.workunit.client.0.smithi116.stdout:7/945: creat d3/d4/dbf/de6/dd3/ddf/dee/d10a/d11f/f138 x:0 0 0 2024-03-04T16:04:24.131 INFO:tasks.workunit.client.0.smithi116.stdout:4/972: unlink d1/d3/d15/d20/d1c/f27 0 2024-03-04T16:04:24.132 INFO:tasks.workunit.client.0.smithi116.stdout:7/946: readlink d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/d113/dc3/le1 0 2024-03-04T16:04:24.132 INFO:tasks.workunit.client.0.smithi116.stdout:4/973: write d1/d3/d15/d20/d1c/d62/d81/d97/fb9 [2003674,45007] 0 2024-03-04T16:04:24.134 INFO:tasks.workunit.client.0.smithi116.stdout:0/838: link da/d16/d28/d40/d48/d57/db3/db2/dc2/fd0 da/d16/d20/d42/de0/f100 0 2024-03-04T16:04:24.134 INFO:tasks.workunit.client.0.smithi116.stdout:0/839: fsync da/d16/f93 0 2024-03-04T16:04:24.136 INFO:tasks.workunit.client.0.smithi116.stdout:2/979: getdents d7/d25/d12f 0 2024-03-04T16:04:24.137 INFO:tasks.workunit.client.0.smithi116.stdout:2/980: readlink d7/d13/d3e/d71/lb2 0 2024-03-04T16:04:24.138 INFO:tasks.workunit.client.0.smithi116.stdout:4/974: mknod d1/d3/dd/d24/db4/c15e 0 2024-03-04T16:04:24.138 INFO:tasks.workunit.client.0.smithi116.stdout:4/975: chown d1/d3/d15/d20/f82 8142015 1 2024-03-04T16:04:24.139 INFO:tasks.workunit.client.0.smithi116.stdout:9/866: rename d2/d33/fec to d2/d26/f113 0 2024-03-04T16:04:24.141 INFO:tasks.workunit.client.0.smithi116.stdout:2/981: symlink d7/d13/d3e/ded/dfd/l168 0 2024-03-04T16:04:24.142 INFO:tasks.workunit.client.0.smithi116.stdout:2/982: readlink d7/d15/l8c 0 2024-03-04T16:04:24.143 INFO:tasks.workunit.client.0.smithi116.stdout:2/983: dread - d7/d25/f123 zero size 2024-03-04T16:04:24.143 INFO:tasks.workunit.client.0.smithi116.stdout:7/947: rename d3/d4/d34/c103 to d3/d4/dbf/de6/dd3/ddf/dee/d130/d1a/d126/c139 0 2024-03-04T16:04:24.144 INFO:tasks.workunit.client.0.smithi116.stdout:4/976: dwrite d1/d3/d15/d20/d1c/d62/d81/f106 [0,4194304] 0 2024-03-04T16:04:24.144 INFO:tasks.workunit.client.0.smithi116.stdout:0/840: symlink da/d16/d19/d27/d70/da1/l101 0 2024-03-04T16:04:24.146 INFO:tasks.workunit.client.0.smithi116.stdout:0/841: read da/d16/d19/f68 [109356,19452] 0 2024-03-04T16:04:24.146 INFO:tasks.workunit.client.0.smithi116.stdout:9/867: unlink d2/d19/d27/dbd/cfa 0 2024-03-04T16:04:24.148 INFO:tasks.workunit.client.0.smithi116.stdout:7/948: read d3/f55 [208309,78333] 0 2024-03-04T16:04:24.149 INFO:tasks.workunit.client.0.smithi116.stdout:2/984: symlink d7/d13/d3e/ded/d76/d7c/l169 0 2024-03-04T16:04:24.150 INFO:tasks.workunit.client.0.smithi116.stdout:7/949: write d3/d4/dbf/de6/dd3/ddf/dee/d130/d52/f11b [344828,49488] 0 2024-03-04T16:04:24.151 INFO:tasks.workunit.client.0.smithi116.stdout:4/977: creat d1/d6c/f15f x:0 0 0 2024-03-04T16:04:24.153 INFO:tasks.workunit.client.0.smithi116.stdout:7/950: chown d3/d4/dbf/de6/dd3/ddf/dee 2 1 2024-03-04T16:04:24.153 INFO:tasks.workunit.client.0.smithi116.stdout:0/842: creat da/d16/d28/d40/d48/d57/d8a/daa/f102 x:0 0 0 2024-03-04T16:04:24.155 INFO:tasks.workunit.client.0.smithi116.stdout:0/843: write da/d16/d20/d79/fce [845806,36820] 0 2024-03-04T16:04:24.156 INFO:tasks.workunit.client.0.smithi116.stdout:9/868: rename d2/d26/fdf to d2/d26/dd4/d4a/d6a/d94/dbe/f114 0 2024-03-04T16:04:24.157 INFO:tasks.workunit.client.0.smithi116.stdout:0/844: chown da/d16/d28/d40/d48/d57/db3 103900 1 2024-03-04T16:04:24.160 INFO:tasks.workunit.client.0.smithi116.stdout:4/978: mkdir d1/d3/dd/d45/d160 0 2024-03-04T16:04:24.161 INFO:tasks.workunit.client.0.smithi116.stdout:4/979: chown d1/d3/de/f12 5 1 2024-03-04T16:04:24.164 INFO:tasks.workunit.client.0.smithi116.stdout:7/951: symlink d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/d113/db6/l13a 0 2024-03-04T16:04:24.165 INFO:tasks.workunit.client.0.smithi116.stdout:9/869: fsync d2/f4f 0 2024-03-04T16:04:24.168 INFO:tasks.workunit.client.0.smithi116.stdout:0/845: mknod da/d16/d20/d53/c103 0 2024-03-04T16:04:24.170 INFO:tasks.workunit.client.0.smithi116.stdout:2/985: creat d7/d15/d5e/d9d/dd4/d164/f16a x:0 0 0 2024-03-04T16:04:24.183 INFO:tasks.workunit.client.0.smithi116.stdout:0/846: unlink da/d16/d28/d40/d48/d57/d8a/fc1 0 2024-03-04T16:04:24.184 INFO:tasks.workunit.client.0.smithi116.stdout:0/847: read da/d16/d19/d27/f34 [526482,660] 0 2024-03-04T16:04:24.185 INFO:tasks.workunit.client.0.smithi116.stdout:0/848: dread - da/d16/d19/d27/d45/ff6 zero size 2024-03-04T16:04:24.185 INFO:tasks.workunit.client.0.smithi116.stdout:2/986: creat d7/d13/d3e/d71/dad/f16b x:0 0 0 2024-03-04T16:04:24.186 INFO:tasks.workunit.client.0.smithi116.stdout:0/849: stat da/d16/d28/d40/d48/d57/db3/db2/dc2/dee 0 2024-03-04T16:04:24.190 INFO:tasks.workunit.client.0.smithi116.stdout:0/850: dwrite da/fa4 [0,4194304] 0 2024-03-04T16:04:24.204 INFO:tasks.workunit.client.0.smithi116.stdout:0/851: read da/d16/d19/d27/f74 [624347,12650] 0 2024-03-04T16:04:24.206 INFO:tasks.workunit.client.0.smithi116.stdout:2/987: symlink d7/d15/l16c 0 2024-03-04T16:04:24.207 INFO:tasks.workunit.client.0.smithi116.stdout:2/988: chown d7/d15/d5e/d9d/d11f/f68 3 1 2024-03-04T16:04:24.208 INFO:tasks.workunit.client.0.smithi116.stdout:9/870: getdents d2/d33/d65/d7e 0 2024-03-04T16:04:24.211 INFO:tasks.workunit.client.0.smithi116.stdout:0/852: creat da/d16/d20/d79/f104 x:0 0 0 2024-03-04T16:04:24.212 INFO:tasks.workunit.client.0.smithi116.stdout:0/853: dread - da/d16/d20/d79/f104 zero size 2024-03-04T16:04:24.213 INFO:tasks.workunit.client.0.smithi116.stdout:2/989: creat d7/d13/d3e/ded/d6c/da4/da1/f16d x:0 0 0 2024-03-04T16:04:24.215 INFO:tasks.workunit.client.0.smithi116.stdout:9/871: symlink d2/d19/d5f/l115 0 2024-03-04T16:04:24.220 INFO:tasks.workunit.client.0.smithi116.stdout:9/872: mkdir d2/d33/d65/dbf/d116 0 2024-03-04T16:04:24.220 INFO:tasks.workunit.client.0.smithi116.stdout:9/873: chown d2/f48 3 1 2024-03-04T16:04:24.221 INFO:tasks.workunit.client.0.smithi116.stdout:0/854: mkdir da/d16/d28/d40/d48/d57/db3/db2/dc2/dee/d105 0 2024-03-04T16:04:24.224 INFO:tasks.workunit.client.0.smithi116.stdout:9/874: write d2/d33/f46 [732096,55873] 0 2024-03-04T16:04:24.224 INFO:tasks.workunit.client.0.smithi116.stdout:9/875: stat d2/d26/dd4/d4a/d6a/d94/dbe/ff2 0 2024-03-04T16:04:24.227 INFO:tasks.workunit.client.0.smithi116.stdout:0/855: rename da/d16/d28/d40/d48/d57/f8c to da/d16/d28/d40/d48/d57/d8a/f106 0 2024-03-04T16:04:24.227 INFO:tasks.workunit.client.0.smithi116.stdout:4/980: write d1/fb [4717898,78308] 0 2024-03-04T16:04:24.229 INFO:tasks.workunit.client.0.smithi116.stdout:2/990: dread d7/d81/f79 [0,4194304] 0 2024-03-04T16:04:24.229 INFO:tasks.workunit.client.0.smithi116.stdout:7/952: write d3/d95/f8e [413593,57488] 0 2024-03-04T16:04:24.229 INFO:tasks.workunit.client.0.smithi116.stdout:4/981: write d1/d3/dd/dd9/d11a/f146 [592135,30884] 0 2024-03-04T16:04:24.230 INFO:tasks.workunit.client.0.smithi116.stdout:7/953: read - d3/d4/d34/f10e zero size 2024-03-04T16:04:24.231 INFO:tasks.workunit.client.0.smithi116.stdout:2/991: read - d7/d13/d3e/d71/dad/fc4 zero size 2024-03-04T16:04:24.232 INFO:tasks.workunit.client.0.smithi116.stdout:0/856: creat da/d16/d19/d27/d70/da1/f107 x:0 0 0 2024-03-04T16:04:24.233 INFO:tasks.workunit.client.0.smithi116.stdout:7/954: read d3/d4/dbf/de6/dd3/ddf/dee/d130/f1c [862015,32668] 0 2024-03-04T16:04:24.234 INFO:tasks.workunit.client.0.smithi116.stdout:9/876: mknod d2/d33/d65/dbf/d116/c117 0 2024-03-04T16:04:24.236 INFO:tasks.workunit.client.0.smithi116.stdout:4/982: symlink d1/d3/dd/dd9/d11a/dfe/l161 0 2024-03-04T16:04:24.237 INFO:tasks.workunit.client.0.smithi116.stdout:0/857: dwrite da/d16/d28/d40/d48/fd9 [0,4194304] 0 2024-03-04T16:04:24.238 INFO:tasks.workunit.client.0.smithi116.stdout:0/858: readlink da/d16/d19/d27/d70/lf8 0 2024-03-04T16:04:24.275 INFO:tasks.workunit.client.0.smithi116.stdout:6/862: sync 2024-03-04T16:04:24.275 INFO:tasks.workunit.client.0.smithi116.stdout:2/992: rename d7/d13/d3e/ded/d6c/ddb/cdc to d7/d15/d35/c16e 0 2024-03-04T16:04:24.276 INFO:tasks.workunit.client.0.smithi116.stdout:2/993: dread - d7/d13/d3e/d98/dce/d47/dd9/f121 zero size 2024-03-04T16:04:24.278 INFO:tasks.workunit.client.0.smithi116.stdout:0/859: creat da/d16/d28/d3d/d41/f108 x:0 0 0 2024-03-04T16:04:24.280 INFO:tasks.workunit.client.0.smithi116.stdout:9/877: creat d2/d26/dc8/f118 x:0 0 0 2024-03-04T16:04:24.284 INFO:tasks.workunit.client.0.smithi116.stdout:7/955: rename d3/d4/dbf/de6/dd3/ddf/dee/d130/d1a/f42 to d3/d4/d34/d5d/ded/f13b 0 2024-03-04T16:04:24.287 INFO:tasks.workunit.client.0.smithi116.stdout:6/863: mknod d6/de/d25/d4a/d63/d8c/c123 0 2024-03-04T16:04:24.308 INFO:tasks.workunit.client.0.smithi116.stdout:7/956: creat d3/d4/d9e/dd7/dfd/d100/f13c x:0 0 0 2024-03-04T16:04:24.315 INFO:tasks.workunit.client.0.smithi116.stdout:7/957: symlink d3/d4/dbf/de6/dd3/ddf/dee/d130/d1a/d126/l13d 0 2024-03-04T16:04:24.317 INFO:tasks.workunit.client.0.smithi116.stdout:6/864: link d6/de/d25/d4a/lcd d6/de/d15/l124 0 2024-03-04T16:04:24.318 INFO:tasks.workunit.client.0.smithi116.stdout:6/865: write d6/de/d25/d36/d4e/dfa/f112 [1048410,24668] 0 2024-03-04T16:04:24.319 INFO:tasks.workunit.client.0.smithi116.stdout:0/860: link da/d16/d20/d42/de0/ld1 da/d16/d20/d42/de0/d7e/l109 0 2024-03-04T16:04:24.322 INFO:tasks.workunit.client.0.smithi116.stdout:7/958: symlink d3/d4/d9e/dd7/l13e 0 2024-03-04T16:04:24.326 INFO:tasks.workunit.client.0.smithi116.stdout:6/866: rmdir d6/de/d1e/d7c 39 2024-03-04T16:04:24.329 INFO:tasks.workunit.client.0.smithi116.stdout:7/959: truncate d3/d4/dbf/de6/dd3/ddf/dee/d130/d1a/f46 4129704 0 2024-03-04T16:04:24.332 INFO:tasks.workunit.client.0.smithi116.stdout:0/861: link da/d16/d20/f65 da/d16/d28/d40/d48/d57/db3/db2/dc2/f10a 0 2024-03-04T16:04:24.337 INFO:tasks.workunit.client.0.smithi116.stdout:0/862: symlink da/d16/d28/d7b/l10b 0 2024-03-04T16:04:24.338 INFO:tasks.workunit.client.0.smithi116.stdout:2/994: write d7/d25/ff7 [982830,50577] 0 2024-03-04T16:04:24.339 INFO:tasks.workunit.client.0.smithi116.stdout:4/983: dwrite d1/fae [0,4194304] 0 2024-03-04T16:04:24.342 INFO:tasks.workunit.client.0.smithi116.stdout:6/867: rename d6/de/d25/d4a/df8/da5/db3/d122/f89 to d6/d56/d58/f125 0 2024-03-04T16:04:24.347 INFO:tasks.workunit.client.0.smithi116.stdout:9/878: dwrite d2/d26/dd4/dab/d3c/f79 [0,4194304] 0 2024-03-04T16:04:24.348 INFO:tasks.workunit.client.0.smithi116.stdout:4/984: dwrite d1/d3/dd/d8a/ddc/f130 [0,4194304] 0 2024-03-04T16:04:24.354 INFO:tasks.workunit.client.0.smithi116.stdout:2/995: rename d7/d81/dc2/ldf to d7/d81/dc2/l16f 0 2024-03-04T16:04:24.358 INFO:tasks.workunit.client.0.smithi116.stdout:9/879: fdatasync d2/d33/d65/d6c/db8/de3/fc2 0 2024-03-04T16:04:24.362 INFO:tasks.workunit.client.0.smithi116.stdout:4/985: rename d1/d3/dd/d45/d119 to d1/d3/dd/d45/d162 0 2024-03-04T16:04:24.367 INFO:tasks.workunit.client.0.smithi116.stdout:4/986: unlink d1/f113 0 2024-03-04T16:04:24.369 INFO:tasks.workunit.client.0.smithi116.stdout:2/996: symlink d7/d13/d3e/d98/dce/l170 0 2024-03-04T16:04:24.370 INFO:tasks.workunit.client.0.smithi116.stdout:2/997: readlink d7/d25/l11e 0 2024-03-04T16:04:24.370 INFO:tasks.workunit.client.0.smithi116.stdout:9/880: mknod d2/d33/c119 0 2024-03-04T16:04:24.372 INFO:tasks.workunit.client.0.smithi116.stdout:4/987: dread - d1/d3/dd/d24/db4/fc0 zero size 2024-03-04T16:04:24.374 INFO:tasks.workunit.client.0.smithi116.stdout:9/881: rename d2/d33/f10c to d2/d33/d65/d6c/db8/de3/f11a 0 2024-03-04T16:04:24.375 INFO:tasks.workunit.client.0.smithi116.stdout:7/960: write d3/ffa [1123241,109290] 0 2024-03-04T16:04:24.376 INFO:tasks.workunit.client.0.smithi116.stdout:9/882: write d2/d33/d65/d6c/db8/fee [4914507,5307] 0 2024-03-04T16:04:24.377 INFO:tasks.workunit.client.0.smithi116.stdout:2/998: link d7/d81/d115/l130 d7/dd3/l171 0 2024-03-04T16:04:24.384 INFO:tasks.workunit.client.0.smithi116.stdout:4/988: link d1/d3/d15/d20/d1c/ddd/c12f d1/c163 0 2024-03-04T16:04:24.386 INFO:tasks.workunit.client.0.smithi116.stdout:4/989: mknod d1/d134/c164 0 2024-03-04T16:04:24.387 INFO:tasks.workunit.client.0.smithi116.stdout:2/999: creat d7/d13/f172 x:0 0 0 2024-03-04T16:04:24.390 INFO:tasks.workunit.client.0.smithi116.stdout:6/868: dwrite d6/de/f4d [0,4194304] 0 2024-03-04T16:04:24.391 INFO:tasks.workunit.client.0.smithi116.stdout:6/869: fsync d6/de/f4f 0 2024-03-04T16:04:24.393 INFO:tasks.workunit.client.0.smithi116.stdout:7/961: dread d3/f4e [0,4194304] 0 2024-03-04T16:04:24.396 INFO:tasks.workunit.client.0.smithi116.stdout:4/990: getdents d1/d3/d40/d52/d84/dde 0 2024-03-04T16:04:24.397 INFO:tasks.workunit.client.0.smithi116.stdout:4/991: write d1/d3/dd/d24/db4/d10c/f10e [10117,86862] 0 2024-03-04T16:04:24.398 INFO:tasks.workunit.client.0.smithi116.stdout:1/788: sync 2024-03-04T16:04:24.399 INFO:tasks.workunit.client.0.smithi116.stdout:7/962: creat d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/d87/f13f x:0 0 0 2024-03-04T16:04:24.400 INFO:tasks.workunit.client.0.smithi116.stdout:1/789: write d5/d9/d21/d27/d32/d35/d55/d75/fef [28385,35119] 0 2024-03-04T16:04:24.402 INFO:tasks.workunit.client.0.smithi116.stdout:4/992: mkdir d1/d3/d15/d20/d165 0 2024-03-04T16:04:24.405 INFO:tasks.workunit.client.0.smithi116.stdout:7/963: fdatasync d3/f1f 0 2024-03-04T16:04:24.407 INFO:tasks.workunit.client.0.smithi116.stdout:7/964: stat d3/d4/d9e/dd7/dfd/d100/f13c 0 2024-03-04T16:04:24.408 INFO:tasks.workunit.client.0.smithi116.stdout:4/993: dread d1/d3/d15/d20/d1c/d62/d81/daf/f10b [0,4194304] 0 2024-03-04T16:04:24.411 INFO:tasks.workunit.client.0.smithi116.stdout:9/883: write d2/d19/d27/dbd/fcf [1625451,16866] 0 2024-03-04T16:04:24.412 INFO:tasks.workunit.client.0.smithi116.stdout:7/965: mknod d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/d65/c140 0 2024-03-04T16:04:24.413 INFO:tasks.workunit.client.0.smithi116.stdout:9/884: mkdir d2/d19/d5f/d11b 0 2024-03-04T16:04:24.414 INFO:tasks.workunit.client.0.smithi116.stdout:7/966: chown d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/d113/db6/c73 2110896103 1 2024-03-04T16:04:24.414 INFO:tasks.workunit.client.0.smithi116.stdout:6/870: write d6/fb8 [1993072,129592] 0 2024-03-04T16:04:24.416 INFO:tasks.workunit.client.0.smithi116.stdout:9/885: mknod d2/d19/d5f/d11b/c11c 0 2024-03-04T16:04:24.416 INFO:tasks.workunit.client.0.smithi116.stdout:9/886: dread - d2/d33/d65/d7e/f9a zero size 2024-03-04T16:04:24.417 INFO:tasks.workunit.client.0.smithi116.stdout:4/994: dread d1/d3/dd/f5d [0,4194304] 0 2024-03-04T16:04:24.417 INFO:tasks.workunit.client.0.smithi116.stdout:7/967: creat d3/d4/d9e/dd7/dfd/f141 x:0 0 0 2024-03-04T16:04:24.418 INFO:tasks.workunit.client.0.smithi116.stdout:6/871: creat d6/de/d25/d4a/df8/f126 x:0 0 0 2024-03-04T16:04:24.420 INFO:tasks.workunit.client.0.smithi116.stdout:6/872: fsync d6/d56/d58/f98 0 2024-03-04T16:04:24.422 INFO:tasks.workunit.client.0.smithi116.stdout:6/873: rmdir d6/de/d5b 39 2024-03-04T16:04:24.423 INFO:tasks.workunit.client.0.smithi116.stdout:7/968: dread d3/d4/d9e/dd7/daf/f125 [0,4194304] 0 2024-03-04T16:04:24.424 INFO:tasks.workunit.client.0.smithi116.stdout:6/874: chown d6/de/d25/d36/d44/dbb/ddd/d7b 25374 1 2024-03-04T16:04:24.426 INFO:tasks.workunit.client.0.smithi116.stdout:7/969: truncate d3/d4/d9e/dd7/dd1/f131 874513 0 2024-03-04T16:04:24.427 INFO:tasks.workunit.client.0.smithi116.stdout:6/875: rename d6/de/d29/l91 to d6/de/d25/d36/d4e/dfa/l127 0 2024-03-04T16:04:24.427 INFO:tasks.workunit.client.0.smithi116.stdout:1/790: dread d5/d9/d21/d27/d32/d35/f76 [4194304,4194304] 0 2024-03-04T16:04:24.428 INFO:tasks.workunit.client.0.smithi116.stdout:7/970: creat d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/d87/f142 x:0 0 0 2024-03-04T16:04:24.430 INFO:tasks.workunit.client.0.smithi116.stdout:6/876: creat d6/d56/d58/f128 x:0 0 0 2024-03-04T16:04:24.432 INFO:tasks.workunit.client.0.smithi116.stdout:1/791: rename d5/d9/d21/d27/d32/d35/d55/db4/d50/cf8 to d5/d9/d21/d84/db0/de9/dfb/c102 0 2024-03-04T16:04:24.433 INFO:tasks.workunit.client.0.smithi116.stdout:7/971: write d3/d4/dbf/de6/dd3/ddf/dee/d130/f35 [2257241,20561] 0 2024-03-04T16:04:24.436 INFO:tasks.workunit.client.0.smithi116.stdout:6/877: dwrite d6/de/d84/f10f [0,4194304] 0 2024-03-04T16:04:24.444 INFO:tasks.workunit.client.0.smithi116.stdout:6/878: write d6/de/d29/f8d [2405103,81766] 0 2024-03-04T16:04:24.447 INFO:tasks.workunit.client.0.smithi116.stdout:6/879: rename d6/de/d15/dfc/d100 to d6/de/d25/d4a/d63/d95/d104/d129 0 2024-03-04T16:04:24.452 INFO:tasks.workunit.client.0.smithi116.stdout:6/880: dwrite d6/de/d25/d36/fa6 [0,4194304] 0 2024-03-04T16:04:24.453 INFO:tasks.workunit.client.0.smithi116.stdout:1/792: dread d5/d9/d21/f2c [0,4194304] 0 2024-03-04T16:04:24.461 INFO:tasks.workunit.client.0.smithi116.stdout:6/881: dwrite d6/de/d29/fe5 [4194304,4194304] 0 2024-03-04T16:04:24.464 INFO:tasks.workunit.client.0.smithi116.stdout:6/882: rmdir d6/de/d25/d4a/d63/d95 39 2024-03-04T16:04:24.466 INFO:tasks.workunit.client.0.smithi116.stdout:1/793: dwrite d5/f29 [0,4194304] 0 2024-03-04T16:04:24.468 INFO:tasks.workunit.client.0.smithi116.stdout:1/794: write d5/d9/d21/d27/d32/d35/f52 [5017313,99021] 0 2024-03-04T16:04:24.470 INFO:tasks.workunit.client.0.smithi116.stdout:6/883: creat d6/de/d25/d36/d44/dbb/f12a x:0 0 0 2024-03-04T16:04:24.472 INFO:tasks.workunit.client.0.smithi116.stdout:6/884: creat d6/de/d29/d3a/f12b x:0 0 0 2024-03-04T16:04:24.482 INFO:tasks.workunit.client.0.smithi116.stdout:8/971: sync 2024-03-04T16:04:24.482 INFO:tasks.workunit.client.0.smithi116.stdout:5/939: sync 2024-03-04T16:04:24.483 INFO:tasks.workunit.client.0.smithi116.stdout:6/885: creat d6/de/d25/d4a/d63/f12c x:0 0 0 2024-03-04T16:04:24.484 INFO:tasks.workunit.client.0.smithi116.stdout:5/940: chown dd/d45/c53 173398892 1 2024-03-04T16:04:24.484 INFO:tasks.workunit.client.0.smithi116.stdout:8/972: chown d2/d13/d3c/d46/c100 366452536 1 2024-03-04T16:04:24.485 INFO:tasks.workunit.client.0.smithi116.stdout:6/886: dread - d6/de/d25/d4a/df8/da5/db3/d122/f11d zero size 2024-03-04T16:04:24.499 INFO:tasks.workunit.client.0.smithi116.stdout:5/941: mkdir dd/d1f/d55/d73/d2d/d12a 0 2024-03-04T16:04:24.500 INFO:tasks.workunit.client.0.smithi116.stdout:8/973: creat d2/de/d16/d78/d29/d7a/f143 x:0 0 0 2024-03-04T16:04:24.510 INFO:tasks.workunit.client.0.smithi116.stdout:8/974: dread d2/d13/d3c/fae [0,4194304] 0 2024-03-04T16:04:24.510 INFO:tasks.workunit.client.0.smithi116.stdout:8/975: chown d2/de/d16/d78/c3a 3182763 1 2024-03-04T16:04:24.511 INFO:tasks.workunit.client.0.smithi116.stdout:8/976: chown d2/d13/d3c/d46/d6e/fbc 31 1 2024-03-04T16:04:24.512 INFO:tasks.workunit.client.0.smithi116.stdout:8/977: rmdir d2/d4a/d8a 39 2024-03-04T16:04:24.514 INFO:tasks.workunit.client.0.smithi116.stdout:6/887: truncate d6/de/d25/d4a/d63/fe0 535039 0 2024-03-04T16:04:24.515 INFO:tasks.workunit.client.0.smithi116.stdout:6/888: fsync d6/d56/d58/f128 0 2024-03-04T16:04:24.516 INFO:tasks.workunit.client.0.smithi116.stdout:8/978: rename d2/de/d3e/d7b/f8e to d2/de/d16/d78/d29/f144 0 2024-03-04T16:04:24.518 INFO:tasks.workunit.client.0.smithi116.stdout:6/889: creat d6/de/d25/d36/f12d x:0 0 0 2024-03-04T16:04:24.519 INFO:tasks.workunit.client.0.smithi116.stdout:5/942: dwrite dd/d1f/d55/d73/d2d/d7c/f9d [0,4194304] 0 2024-03-04T16:04:24.520 INFO:tasks.workunit.client.0.smithi116.stdout:6/890: unlink d6/de/d15/fb2 0 2024-03-04T16:04:24.522 INFO:tasks.workunit.client.0.smithi116.stdout:6/891: dread - d6/de/d29/d3a/f12b zero size 2024-03-04T16:04:24.528 INFO:tasks.workunit.client.0.smithi116.stdout:6/892: truncate d6/de/d25/d4a/d63/d8c/fa3 4522065 0 2024-03-04T16:04:24.530 INFO:tasks.workunit.client.0.smithi116.stdout:5/943: dwrite dd/f11e [0,4194304] 0 2024-03-04T16:04:24.532 INFO:tasks.workunit.client.0.smithi116.stdout:6/893: mknod d6/de/d25/d4a/df8/dec/dfe/c12e 0 2024-03-04T16:04:24.534 INFO:tasks.workunit.client.0.smithi116.stdout:5/944: creat dd/d1f/dcd/f12b x:0 0 0 2024-03-04T16:04:24.534 INFO:tasks.workunit.client.0.smithi116.stdout:8/979: dwrite d2/de/d3e/d7b/fdd [0,4194304] 0 2024-03-04T16:04:24.537 INFO:tasks.workunit.client.0.smithi116.stdout:6/894: chown d6/de/d25/d4a/df8/da5/db3/d122/cdf 0 1 2024-03-04T16:04:24.538 INFO:tasks.workunit.client.0.smithi116.stdout:6/895: stat d6/de/d25/f40 0 2024-03-04T16:04:24.540 INFO:tasks.workunit.client.0.smithi116.stdout:6/896: mkdir d6/de/d84/d12f 0 2024-03-04T16:04:24.541 INFO:tasks.workunit.client.0.smithi116.stdout:8/980: mkdir d2/de/d16/d78/d29/de5/df2/d37/d145 0 2024-03-04T16:04:24.543 INFO:tasks.workunit.client.0.smithi116.stdout:8/981: write d2/de/d16/d78/d29/d7a/fd7 [1516432,98378] 0 2024-03-04T16:04:24.545 INFO:tasks.workunit.client.0.smithi116.stdout:8/982: truncate d2/d13/d3c/d46/f139 637469 0 2024-03-04T16:04:24.546 INFO:tasks.workunit.client.0.smithi116.stdout:8/983: chown d2/d13/d3c/d46/d96/lfb 11995 1 2024-03-04T16:04:24.562 INFO:tasks.workunit.client.0.smithi116.stdout:5/945: dread dd/d1f/d55/d73/d2d/d31/f74 [0,4194304] 0 2024-03-04T16:04:24.563 INFO:tasks.workunit.client.0.smithi116.stdout:6/897: dread d6/d56/f6c [0,4194304] 0 2024-03-04T16:04:24.564 INFO:tasks.workunit.client.0.smithi116.stdout:5/946: fdatasync dd/d1f/d5c/f75 0 2024-03-04T16:04:24.564 INFO:tasks.workunit.client.0.smithi116.stdout:5/947: chown dd/f103 2234284 1 2024-03-04T16:04:24.565 INFO:tasks.workunit.client.0.smithi116.stdout:6/898: truncate d6/de/d25/d36/d44/dbb/ddd/d7b/f11a 754859 0 2024-03-04T16:04:24.569 INFO:tasks.workunit.client.0.smithi116.stdout:5/948: creat dd/d45/d10c/d11a/f12c x:0 0 0 2024-03-04T16:04:24.572 INFO:tasks.workunit.client.0.smithi116.stdout:5/949: truncate dd/d1f/d55/d73/d2d/d31/f65 2493271 0 2024-03-04T16:04:24.572 INFO:tasks.workunit.client.0.smithi116.stdout:6/899: dwrite d6/de/d84/db4/f5f [0,4194304] 0 2024-03-04T16:04:24.576 INFO:tasks.workunit.client.0.smithi116.stdout:5/950: mknod dd/d1f/d55/d73/d2d/dc0/df7/c12d 0 2024-03-04T16:04:24.577 INFO:tasks.workunit.client.0.smithi116.stdout:6/900: stat d6/de/d5b/c90 0 2024-03-04T16:04:24.579 INFO:tasks.workunit.client.0.smithi116.stdout:0/863: sync 2024-03-04T16:04:24.579 INFO:tasks.workunit.client.0.smithi116.stdout:9/887: sync 2024-03-04T16:04:24.580 INFO:tasks.workunit.client.0.smithi116.stdout:0/864: write da/d16/d28/d40/d48/fd7 [3099228,83880] 0 2024-03-04T16:04:24.582 INFO:tasks.workunit.client.0.smithi116.stdout:5/951: creat dd/d1f/d55/d73/d2d/d59/f12e x:0 0 0 2024-03-04T16:04:24.583 INFO:tasks.workunit.client.0.smithi116.stdout:6/901: mknod d6/de/d25/d4a/d63/d95/d104/d129/c130 0 2024-03-04T16:04:24.584 INFO:tasks.workunit.client.0.smithi116.stdout:5/952: write dd/f103 [979195,83010] 0 2024-03-04T16:04:24.584 INFO:tasks.workunit.client.0.smithi116.stdout:0/865: rmdir da/d16/d28/d40/d48/d57/db3/db2/dc2/dee 39 2024-03-04T16:04:24.586 INFO:tasks.workunit.client.0.smithi116.stdout:6/902: mkdir d6/de/d25/d36/d44/dbb/d131 0 2024-03-04T16:04:24.587 INFO:tasks.workunit.client.0.smithi116.stdout:0/866: creat da/d16/d28/d40/d48/d57/f10c x:0 0 0 2024-03-04T16:04:24.588 INFO:tasks.workunit.client.0.smithi116.stdout:5/953: creat dd/d1f/d55/d73/d2d/d7c/d121/f12f x:0 0 0 2024-03-04T16:04:24.589 INFO:tasks.workunit.client.0.smithi116.stdout:6/903: unlink d6/de/d84/db4/l97 0 2024-03-04T16:04:24.590 INFO:tasks.workunit.client.0.smithi116.stdout:6/904: chown d6/de/d25/d4a/df8/da5/db3/lf4 0 1 2024-03-04T16:04:24.592 INFO:tasks.workunit.client.0.smithi116.stdout:5/954: rename dd/d45 to dd/d1f/d55/d73/d130 0 2024-03-04T16:04:24.594 INFO:tasks.workunit.client.0.smithi116.stdout:5/955: write dd/d6d/f88 [2209842,101760] 0 2024-03-04T16:04:24.594 INFO:tasks.workunit.client.0.smithi116.stdout:6/905: creat d6/de/d84/db4/f132 x:0 0 0 2024-03-04T16:04:24.596 INFO:tasks.workunit.client.0.smithi116.stdout:6/906: fsync d6/de/d84/f8f 0 2024-03-04T16:04:24.596 INFO:tasks.workunit.client.0.smithi116.stdout:5/956: write dd/d1f/f4d [1576285,26976] 0 2024-03-04T16:04:24.597 INFO:tasks.workunit.client.0.smithi116.stdout:0/867: getdents da/d16/d28/d40/d48/d77 0 2024-03-04T16:04:24.600 INFO:tasks.workunit.client.0.smithi116.stdout:5/957: mkdir dd/d1f/d51/d131 0 2024-03-04T16:04:24.601 INFO:tasks.workunit.client.0.smithi116.stdout:0/868: rmdir da/d16/d20 39 2024-03-04T16:04:24.602 INFO:tasks.workunit.client.0.smithi116.stdout:5/958: chown dd/d1f/d55/d73/d2d/d31/f6a 438174 1 2024-03-04T16:04:24.603 INFO:tasks.workunit.client.0.smithi116.stdout:0/869: read da/d16/d19/f1d [1892982,78152] 0 2024-03-04T16:04:24.604 INFO:tasks.workunit.client.0.smithi116.stdout:5/959: read dd/d1f/f54 [641557,36675] 0 2024-03-04T16:04:24.604 INFO:tasks.workunit.client.0.smithi116.stdout:0/870: write da/d16/d28/d3d/d41/f59 [4162852,105403] 0 2024-03-04T16:04:24.607 INFO:tasks.workunit.client.0.smithi116.stdout:0/871: fdatasync da/d16/d28/d40/d48/fd9 0 2024-03-04T16:04:24.607 INFO:tasks.workunit.client.0.smithi116.stdout:5/960: dread dd/fd2 [0,4194304] 0 2024-03-04T16:04:24.610 INFO:tasks.workunit.client.0.smithi116.stdout:0/872: write da/d16/d28/d40/d48/d57/d8a/f106 [1641817,118424] 0 2024-03-04T16:04:24.610 INFO:tasks.workunit.client.0.smithi116.stdout:5/961: mkdir dd/d1f/d51/d11d/d132 0 2024-03-04T16:04:24.612 INFO:tasks.workunit.client.0.smithi116.stdout:5/962: mkdir dd/d1f/d72/d133 0 2024-03-04T16:04:24.613 INFO:tasks.workunit.client.0.smithi116.stdout:6/907: dread d6/de/d1e/d7c/fbd [0,4194304] 0 2024-03-04T16:04:24.613 INFO:tasks.workunit.client.0.smithi116.stdout:0/873: truncate da/d16/d20/d42/de0/ff7 212991 0 2024-03-04T16:04:24.614 INFO:tasks.workunit.client.0.smithi116.stdout:6/908: fsync d6/de/d25/d36/f12d 0 2024-03-04T16:04:24.614 INFO:tasks.workunit.client.0.smithi116.stdout:5/963: chown dd/d1f/d55/d73/d2d/d31/f7b 369 1 2024-03-04T16:04:24.615 INFO:tasks.workunit.client.0.smithi116.stdout:6/909: chown d6/de/d15/l124 54804 1 2024-03-04T16:04:24.617 INFO:tasks.workunit.client.0.smithi116.stdout:0/874: rename da/d16/d19/d27/f78 to da/d16/d19/d27/d82/d9f/f10d 0 2024-03-04T16:04:24.617 INFO:tasks.workunit.client.0.smithi116.stdout:5/964: creat dd/d1f/f134 x:0 0 0 2024-03-04T16:04:24.618 INFO:tasks.workunit.client.0.smithi116.stdout:0/875: write da/d16/d28/d40/d48/d57/f10c [693097,89093] 0 2024-03-04T16:04:24.618 INFO:tasks.workunit.client.0.smithi116.stdout:5/965: readlink dd/d1f/d55/lab 0 2024-03-04T16:04:24.619 INFO:tasks.workunit.client.0.smithi116.stdout:6/910: write d6/d56/f6c [1164956,28424] 0 2024-03-04T16:04:24.620 INFO:tasks.workunit.client.0.smithi116.stdout:0/876: write da/d16/d20/d79/fce [598422,56491] 0 2024-03-04T16:04:24.622 INFO:tasks.workunit.client.0.smithi116.stdout:6/911: fdatasync d6/de/d25/d4a/f9d 0 2024-03-04T16:04:24.623 INFO:tasks.workunit.client.0.smithi116.stdout:5/966: symlink dd/d6d/df0/d111/l135 0 2024-03-04T16:04:24.623 INFO:tasks.workunit.client.0.smithi116.stdout:5/967: chown dd/d1f/d5c/fbb 19055 1 2024-03-04T16:04:24.625 INFO:tasks.workunit.client.0.smithi116.stdout:0/877: mknod da/d16/d28/d40/d48/d57/db3/db2/dc2/c10e 0 2024-03-04T16:04:24.628 INFO:tasks.workunit.client.0.smithi116.stdout:0/878: mknod da/d16/d19/d27/d70/c10f 0 2024-03-04T16:04:24.630 INFO:tasks.workunit.client.0.smithi116.stdout:0/879: mknod da/d16/d28/d40/d48/d57/d8a/daa/c110 0 2024-03-04T16:04:24.633 INFO:tasks.workunit.client.0.smithi116.stdout:0/880: unlink da/d16/d28/d40/d48/d77/la5 0 2024-03-04T16:04:24.634 INFO:tasks.workunit.client.0.smithi116.stdout:6/912: write d6/de/d25/d36/d44/dbb/ddd/d7b/fb5 [628969,126951] 0 2024-03-04T16:04:24.637 INFO:tasks.workunit.client.0.smithi116.stdout:6/913: write d6/de/d25/d4a/df8/da5/db3/d122/f11d [733704,56947] 0 2024-03-04T16:04:24.638 INFO:tasks.workunit.client.0.smithi116.stdout:0/881: unlink da/d16/d28/d40/d48/d57/d8a/daa/lca 0 2024-03-04T16:04:24.639 INFO:tasks.workunit.client.0.smithi116.stdout:5/968: dwrite dd/d1f/d55/f6c [0,4194304] 0 2024-03-04T16:04:24.643 INFO:tasks.workunit.client.0.smithi116.stdout:0/882: mknod da/d16/d28/d40/d48/d57/d8a/c111 0 2024-03-04T16:04:24.644 INFO:tasks.workunit.client.0.smithi116.stdout:6/914: dwrite d6/de/d5b/f8e [0,4194304] 0 2024-03-04T16:04:24.646 INFO:tasks.workunit.client.0.smithi116.stdout:0/883: creat da/d16/d28/d39/f112 x:0 0 0 2024-03-04T16:04:24.649 INFO:tasks.workunit.client.0.smithi116.stdout:0/884: write da/f1e [1949737,27475] 0 2024-03-04T16:04:24.649 INFO:tasks.workunit.client.0.smithi116.stdout:5/969: truncate dd/d1f/f54 987497 0 2024-03-04T16:04:24.651 INFO:tasks.workunit.client.0.smithi116.stdout:0/885: chown da/d16/d19/d27/d82/d9f/f10d 1300 1 2024-03-04T16:04:24.652 INFO:tasks.workunit.client.0.smithi116.stdout:5/970: getdents dd/d6d/df0 0 2024-03-04T16:04:24.654 INFO:tasks.workunit.client.0.smithi116.stdout:5/971: creat dd/d1f/d55/f136 x:0 0 0 2024-03-04T16:04:24.655 INFO:tasks.workunit.client.0.smithi116.stdout:0/886: mknod da/d16/d28/d40/d48/d57/db3/db2/dc2/dee/c113 0 2024-03-04T16:04:24.656 INFO:tasks.workunit.client.0.smithi116.stdout:5/972: symlink dd/d1f/l137 0 2024-03-04T16:04:24.657 INFO:tasks.workunit.client.0.smithi116.stdout:0/887: unlink da/d16/d19/d27/d70/da1/f107 0 2024-03-04T16:04:24.659 INFO:tasks.workunit.client.0.smithi116.stdout:5/973: mknod dd/d1f/d55/d73/d2d/dc0/df7/c138 0 2024-03-04T16:04:24.661 INFO:tasks.workunit.client.0.smithi116.stdout:0/888: symlink da/d16/l114 0 2024-03-04T16:04:24.662 INFO:tasks.workunit.client.0.smithi116.stdout:5/974: mknod dd/d1f/d55/c139 0 2024-03-04T16:04:24.662 INFO:tasks.workunit.client.0.smithi116.stdout:0/889: read da/f35 [2664991,55445] 0 2024-03-04T16:04:24.666 INFO:tasks.workunit.client.0.smithi116.stdout:5/975: rename dd/d1f/d55/d73/d130/d10c to dd/d1f/d51/d11d/d13a 0 2024-03-04T16:04:24.667 INFO:tasks.workunit.client.0.smithi116.stdout:4/995: sync 2024-03-04T16:04:24.667 INFO:tasks.workunit.client.0.smithi116.stdout:7/972: sync 2024-03-04T16:04:24.667 INFO:tasks.workunit.client.0.smithi116.stdout:1/795: sync 2024-03-04T16:04:24.668 INFO:tasks.workunit.client.0.smithi116.stdout:0/890: dwrite da/d16/d20/d79/fdf [0,4194304] 0 2024-03-04T16:04:24.670 INFO:tasks.workunit.client.0.smithi116.stdout:1/796: dread - d5/d9/d21/d27/d32/d35/d55/db4/d50/ffa zero size 2024-03-04T16:04:24.674 INFO:tasks.workunit.client.0.smithi116.stdout:7/973: rename d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/d65/df2/c82 to d3/d4/dbf/c143 0 2024-03-04T16:04:24.678 INFO:tasks.workunit.client.0.smithi116.stdout:1/797: rename d5/d9/d21/d27/d32/d35/f52 to d5/d9/d21/d84/db0/f103 0 2024-03-04T16:04:24.679 INFO:tasks.workunit.client.0.smithi116.stdout:1/798: chown d5/da1/ldd 179914 1 2024-03-04T16:04:24.681 INFO:tasks.workunit.client.0.smithi116.stdout:7/974: fdatasync d3/d4/d34/d5d/fa7 0 2024-03-04T16:04:24.682 INFO:tasks.workunit.client.0.smithi116.stdout:7/975: fdatasync d3/d4/d34/d5d/db9/fd4 0 2024-03-04T16:04:24.683 INFO:tasks.workunit.client.0.smithi116.stdout:0/891: link da/d16/d28/d40/fe3 da/d16/d19/d27/d82/df4/f115 0 2024-03-04T16:04:24.685 INFO:tasks.workunit.client.0.smithi116.stdout:4/996: link d1/d6e/d74/de0/ce4 d1/d3/d15/d20/d1c/d62/d81/c166 0 2024-03-04T16:04:24.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:24 smithi116 ceph-mon[29120]: pgmap v28: 65 pgs: 65 active+clean; 1.7 GiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 59 MiB/s rd, 148 MiB/s wr, 429 op/s 2024-03-04T16:04:24.768 INFO:tasks.workunit.client.0.smithi116.stdout:7/976: write d3/d4/dbf/de6/dd3/ddf/fe4 [468441,108458] 0 2024-03-04T16:04:24.770 INFO:tasks.workunit.client.0.smithi116.stdout:4/997: rename d1/d3/dd/dd9/d11a/dfe/l161 to d1/d3/dd/dd9/d11a/dfe/d111/l167 0 2024-03-04T16:04:24.770 INFO:tasks.workunit.client.0.smithi116.stdout:1/799: link d5/d9/d21/d24/de0/d4c/d67/cd2 d5/d9/d21/d84/db0/c104 0 2024-03-04T16:04:24.771 INFO:tasks.workunit.client.0.smithi116.stdout:0/892: mknod da/d16/d20/d42/de0/c116 0 2024-03-04T16:04:24.772 INFO:tasks.workunit.client.0.smithi116.stdout:7/977: unlink d3/d95/c135 0 2024-03-04T16:04:24.773 INFO:tasks.workunit.client.0.smithi116.stdout:1/800: mkdir d5/d9/d105 0 2024-03-04T16:04:24.775 INFO:tasks.workunit.client.0.smithi116.stdout:4/998: dread d1/d6e/fb7 [0,4194304] 0 2024-03-04T16:04:24.776 INFO:tasks.workunit.client.0.smithi116.stdout:7/978: rename f2 to d3/d4/d34/d5d/ded/d136/f144 0 2024-03-04T16:04:24.780 INFO:tasks.workunit.client.0.smithi116.stdout:4/999: dread d1/d3/d40/d52/d84/d88/d8d/fb3 [0,4194304] 0 2024-03-04T16:04:24.781 INFO:tasks.workunit.client.0.smithi116.stdout:1/801: rmdir d5/d9/d21/d27/d94/db6/d42/d8b/dc9/dd1 39 2024-03-04T16:04:24.783 INFO:tasks.workunit.client.0.smithi116.stdout:0/893: write da/d16/d20/f3b [500591,29139] 0 2024-03-04T16:04:24.784 INFO:tasks.workunit.client.0.smithi116.stdout:7/979: mkdir d3/d4/d9e/dd7/dfd/d145 0 2024-03-04T16:04:24.785 INFO:tasks.workunit.client.0.smithi116.stdout:1/802: fdatasync d5/d9/d21/f2c 0 2024-03-04T16:04:24.791 INFO:tasks.workunit.client.0.smithi116.stdout:0/894: dwrite da/ff [0,4194304] 0 2024-03-04T16:04:24.791 INFO:tasks.workunit.client.0.smithi116.stdout:7/980: dwrite d3/f90 [4194304,4194304] 0 2024-03-04T16:04:24.795 INFO:tasks.workunit.client.0.smithi116.stdout:7/981: write d3/d95/f8e [785972,23673] 0 2024-03-04T16:04:24.796 INFO:tasks.workunit.client.0.smithi116.stdout:7/982: chown d3/d4/d9e/dd7/dd1/lf1 792491409 1 2024-03-04T16:04:24.796 INFO:tasks.workunit.client.0.smithi116.stdout:7/983: chown d3/d4/d9e/dd7/dfd/d100 192783 1 2024-03-04T16:04:24.797 INFO:tasks.workunit.client.0.smithi116.stdout:0/895: creat da/d16/d28/d40/d48/d57/dd6/f117 x:0 0 0 2024-03-04T16:04:24.799 INFO:tasks.workunit.client.0.smithi116.stdout:8/984: sync 2024-03-04T16:04:24.800 INFO:tasks.workunit.client.0.smithi116.stdout:9/888: sync 2024-03-04T16:04:24.801 INFO:tasks.workunit.client.0.smithi116.stdout:9/889: chown d2/d33/f77 5954307 1 2024-03-04T16:04:24.802 INFO:tasks.workunit.client.0.smithi116.stdout:8/985: chown d2/d13/d3c/fae 2684529 1 2024-03-04T16:04:24.802 INFO:tasks.workunit.client.0.smithi116.stdout:0/896: mkdir da/d16/d28/d40/d48/d57/d8a/daa/dba/df0/d118 0 2024-03-04T16:04:24.804 INFO:tasks.workunit.client.0.smithi116.stdout:8/986: rename d2/d4a to d2/d4a/d8a/d146 22 2024-03-04T16:04:24.806 INFO:tasks.workunit.client.0.smithi116.stdout:7/984: dread d3/d4/dbf/de6/dd3/ddf/dee/d130/d1a/f5c [0,4194304] 0 2024-03-04T16:04:24.807 INFO:tasks.workunit.client.0.smithi116.stdout:8/987: unlink d2/de/d16/d78/d29/d7a/l7f 0 2024-03-04T16:04:24.807 INFO:tasks.workunit.client.0.smithi116.stdout:1/803: dread d5/d9/d21/d27/d32/d35/d55/d75/d91/fba [0,4194304] 0 2024-03-04T16:04:24.808 INFO:tasks.workunit.client.0.smithi116.stdout:7/985: mknod d3/d4/d34/d5d/ded/c146 0 2024-03-04T16:04:24.810 INFO:tasks.workunit.client.0.smithi116.stdout:9/890: getdents d2/d26/dd4/de5 0 2024-03-04T16:04:24.810 INFO:tasks.workunit.client.0.smithi116.stdout:8/988: dread - d2/de/d16/dc3/fe6 zero size 2024-03-04T16:04:24.811 INFO:tasks.workunit.client.0.smithi116.stdout:9/891: chown d2/d26/dd4/l41 60280 1 2024-03-04T16:04:24.812 INFO:tasks.workunit.client.0.smithi116.stdout:8/989: write d2/de/d16/d78/d29/d7a/f143 [753079,96801] 0 2024-03-04T16:04:24.813 INFO:tasks.workunit.client.0.smithi116.stdout:1/804: link d5/d9/d21/d27/d32/d35/f74 d5/d9/d21/d84/db0/de9/f106 0 2024-03-04T16:04:24.813 INFO:tasks.workunit.client.0.smithi116.stdout:8/990: chown d2/de/d16/d78/fbd 122890 1 2024-03-04T16:04:24.817 INFO:tasks.workunit.client.0.smithi116.stdout:1/805: creat d5/d9/d21/d24/de0/d4c/d67/f107 x:0 0 0 2024-03-04T16:04:24.817 INFO:tasks.workunit.client.0.smithi116.stdout:9/892: dread d2/d33/d65/f92 [0,4194304] 0 2024-03-04T16:04:24.818 INFO:tasks.workunit.client.0.smithi116.stdout:0/897: write da/d16/d19/f68 [925354,17797] 0 2024-03-04T16:04:24.820 INFO:tasks.workunit.client.0.smithi116.stdout:0/898: rmdir da 39 2024-03-04T16:04:24.822 INFO:tasks.workunit.client.0.smithi116.stdout:9/893: rename d2/d26/dd4/d4a/c69 to d2/d33/dc5/c11d 0 2024-03-04T16:04:24.823 INFO:tasks.workunit.client.0.smithi116.stdout:8/991: dread d2/de/f15 [0,4194304] 0 2024-03-04T16:04:24.823 INFO:tasks.workunit.client.0.smithi116.stdout:1/806: mknod d5/d9/d21/d84/db0/de9/dea/c108 0 2024-03-04T16:04:24.824 INFO:tasks.workunit.client.0.smithi116.stdout:7/986: dwrite d3/d4/dbf/de6/dd3/ddf/dee/d130/f45 [0,4194304] 0 2024-03-04T16:04:24.827 INFO:tasks.workunit.client.0.smithi116.stdout:1/807: chown d5/d9/d21/d27/d94/db6/d61 184050 1 2024-03-04T16:04:24.828 INFO:tasks.workunit.client.0.smithi116.stdout:9/894: mkdir d2/d26/dd4/d11e 0 2024-03-04T16:04:24.831 INFO:tasks.workunit.client.0.smithi116.stdout:7/987: creat d3/f147 x:0 0 0 2024-03-04T16:04:24.832 INFO:tasks.workunit.client.0.smithi116.stdout:0/899: creat da/d16/d20/d42/de0/f119 x:0 0 0 2024-03-04T16:04:24.834 INFO:tasks.workunit.client.0.smithi116.stdout:9/895: creat d2/d33/d65/d6c/db8/de3/df0/f11f x:0 0 0 2024-03-04T16:04:24.836 INFO:tasks.workunit.client.0.smithi116.stdout:0/900: read - da/d16/d20/d42/de0/d7e/fe6 zero size 2024-03-04T16:04:24.837 INFO:tasks.workunit.client.0.smithi116.stdout:8/992: dwrite d2/d13/d3c/d46/f139 [0,4194304] 0 2024-03-04T16:04:24.837 INFO:tasks.workunit.client.0.smithi116.stdout:1/808: mkdir d5/d9/d21/d27/d94/db6/d42/d8b/dc9/dd1/d109 0 2024-03-04T16:04:24.841 INFO:tasks.workunit.client.0.smithi116.stdout:1/809: symlink d5/d9/d21/d84/db0/l10a 0 2024-03-04T16:04:24.841 INFO:tasks.workunit.client.0.smithi116.stdout:8/993: truncate d2/d4a/d8a/fd4 4600422 0 2024-03-04T16:04:24.842 INFO:tasks.workunit.client.0.smithi116.stdout:1/810: read - d5/d9/d21/d27/d32/d35/d55/db4/d50/f8e zero size 2024-03-04T16:04:24.844 INFO:tasks.workunit.client.0.smithi116.stdout:9/896: readlink d2/d26/dd4/d4a/d6a/l9f 0 2024-03-04T16:04:24.845 INFO:tasks.workunit.client.0.smithi116.stdout:9/897: stat d2/d26/dd4/dab/fda 0 2024-03-04T16:04:24.846 INFO:tasks.workunit.client.0.smithi116.stdout:1/811: mkdir d5/d9/d21/d27/d8c/d10b 0 2024-03-04T16:04:24.846 INFO:tasks.workunit.client.0.smithi116.stdout:9/898: read d2/d26/dd4/de5/f97 [40899,63477] 0 2024-03-04T16:04:24.848 INFO:tasks.workunit.client.0.smithi116.stdout:9/899: read d2/d19/d27/f9d [496119,89449] 0 2024-03-04T16:04:24.849 INFO:tasks.workunit.client.0.smithi116.stdout:1/812: fsync d5/d9/d21/d27/d32/f47 0 2024-03-04T16:04:24.850 INFO:tasks.workunit.client.0.smithi116.stdout:9/900: dread d2/d26/d28/f55 [0,4194304] 0 2024-03-04T16:04:24.850 INFO:tasks.workunit.client.0.smithi116.stdout:8/994: link d2/d4a/dc7/le0 d2/d13/d3c/d77/l147 0 2024-03-04T16:04:24.851 INFO:tasks.workunit.client.0.smithi116.stdout:7/988: dread d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/d65/df2/fcd [0,4194304] 0 2024-03-04T16:04:24.854 INFO:tasks.workunit.client.0.smithi116.stdout:1/813: mknod d5/d9/d21/d27/d94/db6/d61/c10c 0 2024-03-04T16:04:24.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:24 smithi152 ceph-mon[36876]: pgmap v28: 65 pgs: 65 active+clean; 1.7 GiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 59 MiB/s rd, 148 MiB/s wr, 429 op/s 2024-03-04T16:04:24.859 INFO:tasks.workunit.client.0.smithi116.stdout:7/989: dwrite d3/d4/dbf/de6/dd3/ddf/dee/d130/d4a/d87/f13f [0,4194304] 0 2024-03-04T16:04:24.860 INFO:tasks.workunit.client.0.smithi116.stdout:1/814: rename d5/f38 to d5/d9/d21/d27/d94/db6/d42/d8b/dc9/dd1/f10d 0 2024-03-04T16:04:24.865 INFO:tasks.workunit.client.0.smithi116.stdout:7/990: creat d3/d4/d9e/dd7/daf/dd5/d123/d9a/d105/f148 x:0 0 0 2024-03-04T16:04:24.867 INFO:tasks.workunit.client.0.smithi116.stdout:7/991: readlink d3/d4/d34/l43 0 2024-03-04T16:04:24.868 INFO:tasks.workunit.client.0.smithi116.stdout:8/995: dwrite d2/d4a/dc7/fc8 [0,4194304] 0 2024-03-04T16:04:24.872 INFO:tasks.workunit.client.0.smithi116.stdout:9/901: dread d2/d26/dd4/d4a/f7b [0,4194304] 0 2024-03-04T16:04:24.872 INFO:tasks.workunit.client.0.smithi116.stdout:8/996: symlink d2/de/d16/d78/d29/d7a/db1/d13b/l148 0 2024-03-04T16:04:24.873 INFO:tasks.workunit.client.0.smithi116.stdout:7/992: getdents d3/d4/d34/d5d/ded/d108 0 2024-03-04T16:04:24.875 INFO:tasks.workunit.client.0.smithi116.stdout:9/902: mknod d2/d26/d28/c120 0 2024-03-04T16:04:24.876 INFO:tasks.workunit.client.0.smithi116.stdout:9/903: chown d2/d19/f9c 1 1 2024-03-04T16:04:24.880 INFO:tasks.workunit.client.0.smithi116.stdout:7/993: dwrite d3/d4/dbf/de6/dd3/ddf/dee/d130/f45 [4194304,4194304] 0 2024-03-04T16:04:24.925 INFO:tasks.workunit.client.0.smithi116.stdout:6/915: sync 2024-03-04T16:04:24.930 INFO:tasks.workunit.client.0.smithi116.stdout:6/916: dwrite d6/de/d84/db4/d92/fed [0,4194304] 0 2024-03-04T16:04:24.931 INFO:tasks.workunit.client.0.smithi116.stdout:6/917: readlink d6/de/d84/db4/d92/l11e 0 2024-03-04T16:04:24.959 INFO:tasks.workunit.client.0.smithi116.stdout:9/904: mkdir d2/d26/d28/d121 0 2024-03-04T16:04:24.965 INFO:tasks.workunit.client.0.smithi116.stdout:6/918: mkdir d6/de/d25/d4a/d63/d8c/d133 0 2024-03-04T16:04:24.966 INFO:tasks.workunit.client.0.smithi116.stdout:6/919: chown d6/de/d15/f26 1377877 1 2024-03-04T16:04:24.969 INFO:tasks.workunit.client.0.smithi116.stdout:7/994: creat d3/d4/dbf/de6/dd3/ddf/dee/d130/d52/d116/f149 x:0 0 0 2024-03-04T16:04:24.971 INFO:tasks.workunit.client.0.smithi116.stdout:9/905: rename d2/d26/dd4/d4a/d6a/d94/lb9 to d2/d19/d5f/d11b/l122 0 2024-03-04T16:04:24.973 INFO:tasks.workunit.client.0.smithi116.stdout:6/920: dwrite d6/d56/f6c [4194304,4194304] 0 2024-03-04T16:04:24.974 INFO:tasks.workunit.client.0.smithi116.stdout:7/995: chown d3/d4/c3d 22895 1 2024-03-04T16:04:24.975 INFO:tasks.workunit.client.0.smithi116.stdout:7/996: chown d3/d4/d11/lb1 25 1 2024-03-04T16:04:24.977 INFO:tasks.workunit.client.0.smithi116.stdout:9/906: dread d2/d33/d65/fea [0,4194304] 0 2024-03-04T16:04:24.979 INFO:tasks.workunit.client.0.smithi116.stdout:7/997: creat d3/d4/d9e/dd7/daf/dd5/d123/f14a x:0 0 0 2024-03-04T16:04:24.979 INFO:tasks.workunit.client.0.smithi116.stdout:9/907: write d2/d26/dd4/d4a/d6a/d94/d95/f102 [911005,88608] 0 2024-03-04T16:04:24.984 INFO:tasks.workunit.client.0.smithi116.stdout:7/998: symlink d3/d4/d34/d5d/ded/d136/l14b 0 2024-03-04T16:04:24.984 INFO:tasks.workunit.client.0.smithi116.stdout:9/908: readlink d2/d19/d27/l2c 0 2024-03-04T16:04:24.985 INFO:tasks.workunit.client.0.smithi116.stdout:7/999: creat d3/d4/dbf/de6/dd3/ddf/de0/f14c x:0 0 0 2024-03-04T16:04:24.986 INFO:tasks.workunit.client.0.smithi116.stdout:9/909: rename d2/d26/dd4/l57 to d2/d26/dd4/de5/dcc/l123 0 2024-03-04T16:04:24.987 INFO:tasks.workunit.client.0.smithi116.stdout:9/910: truncate d2/d33/d65/fea 723107 0 2024-03-04T16:04:24.989 INFO:tasks.workunit.client.0.smithi116.stdout:5/976: sync 2024-03-04T16:04:24.992 INFO:tasks.workunit.client.0.smithi116.stdout:5/977: link dd/ca7 dd/d6d/df0/c13b 0 2024-03-04T16:04:24.994 INFO:tasks.workunit.client.0.smithi116.stdout:8/997: truncate d2/d4a/dc7/fc8 2257497 0 2024-03-04T16:04:24.994 INFO:tasks.workunit.client.0.smithi116.stdout:5/978: write dd/d1f/d55/d73/d2d/dc0/f10d [18726,59065] 0 2024-03-04T16:04:24.995 INFO:tasks.workunit.client.0.smithi116.stdout:8/998: dread - d2/d4a/d114/d122/d127/fc1 zero size 2024-03-04T16:04:24.997 INFO:tasks.workunit.client.0.smithi116.stdout:8/999: symlink d2/de/d16/d78/d29/de5/d118/l149 0 2024-03-04T16:04:24.998 INFO:tasks.workunit.client.0.smithi116.stdout:5/979: creat dd/d1f/dc3/f13c x:0 0 0 2024-03-04T16:04:24.998 INFO:tasks.workunit.client.0.smithi116.stdout:6/921: write d6/de/d25/d4a/df8/da5/db3/d122/f57 [1553170,52856] 0 2024-03-04T16:04:25.000 INFO:tasks.workunit.client.0.smithi116.stdout:5/980: fsync dd/d1f/d72/d8e/f98 0 2024-03-04T16:04:25.001 INFO:tasks.workunit.client.0.smithi116.stdout:9/911: truncate d2/d26/dd4/f100 4041685 0 2024-03-04T16:04:25.003 INFO:tasks.workunit.client.0.smithi116.stdout:9/912: write d2/d26/dd4/d4a/d6a/d94/d95/db7/ff3 [61213,59260] 0 2024-03-04T16:04:25.003 INFO:tasks.workunit.client.0.smithi116.stdout:5/981: creat dd/d1f/d72/d8e/f13d x:0 0 0 2024-03-04T16:04:25.003 INFO:tasks.workunit.client.0.smithi116.stdout:6/922: dread d6/de/d1e/f48 [8388608,4194304] 0 2024-03-04T16:04:25.004 INFO:tasks.workunit.client.0.smithi116.stdout:5/982: dread - dd/d1f/d51/f126 zero size 2024-03-04T16:04:25.006 INFO:tasks.workunit.client.0.smithi116.stdout:5/983: write dd/d1f/d55/d73/d2d/dc0/df7/ffa [716000,123659] 0 2024-03-04T16:04:25.009 INFO:tasks.workunit.client.0.smithi116.stdout:6/923: creat d6/de/d25/d4a/d63/f134 x:0 0 0 2024-03-04T16:04:25.011 INFO:tasks.workunit.client.0.smithi116.stdout:5/984: rename dd/d1f/d51/cdd to dd/d1f/d55/d73/d2d/d8d/c13e 0 2024-03-04T16:04:25.011 INFO:tasks.workunit.client.0.smithi116.stdout:5/985: write dd/d1f/fb5 [3303064,96060] 0 2024-03-04T16:04:25.013 INFO:tasks.workunit.client.0.smithi116.stdout:6/924: dread d6/fd5 [0,4194304] 0 2024-03-04T16:04:25.014 INFO:tasks.workunit.client.0.smithi116.stdout:5/986: creat dd/d1f/d55/d73/d2d/d59/d71/f13f x:0 0 0 2024-03-04T16:04:25.015 INFO:tasks.workunit.client.0.smithi116.stdout:9/913: dread d2/f48 [0,4194304] 0 2024-03-04T16:04:25.016 INFO:tasks.workunit.client.0.smithi116.stdout:9/914: write d2/d26/dd4/de5/f43 [1166204,114691] 0 2024-03-04T16:04:25.019 INFO:tasks.workunit.client.0.smithi116.stdout:9/915: creat d2/d19/d5f/d11b/f124 x:0 0 0 2024-03-04T16:04:25.024 INFO:tasks.workunit.client.0.smithi116.stdout:6/925: dwrite d6/d56/f8b [0,4194304] 0 2024-03-04T16:04:25.025 INFO:tasks.workunit.client.0.smithi116.stdout:9/916: dwrite d2/d26/dd4/dab/d3c/f76 [4194304,4194304] 0 2024-03-04T16:04:25.026 INFO:tasks.workunit.client.0.smithi116.stdout:6/926: dread - d6/de/d25/d4a/df8/da5/f101 zero size 2024-03-04T16:04:25.030 INFO:tasks.workunit.client.0.smithi116.stdout:6/927: read d6/de/d29/d3a/f46 [347302,92637] 0 2024-03-04T16:04:25.031 INFO:tasks.workunit.client.0.smithi116.stdout:6/928: write d6/de/d15/f26 [3742091,8172] 0 2024-03-04T16:04:25.032 INFO:tasks.workunit.client.0.smithi116.stdout:9/917: unlink d2/d26/dd4/d4a/lb6 0 2024-03-04T16:04:25.032 INFO:tasks.workunit.client.0.smithi116.stdout:6/929: fdatasync d6/de/d25/d4a/f71 0 2024-03-04T16:04:25.032 INFO:tasks.workunit.client.0.smithi116.stdout:9/918: write d2/d26/f112 [364836,100734] 0 2024-03-04T16:04:25.034 INFO:tasks.workunit.client.0.smithi116.stdout:6/930: dread - d6/de/d15/fe8 zero size 2024-03-04T16:04:25.038 INFO:tasks.workunit.client.0.smithi116.stdout:9/919: symlink d2/d33/d65/d6c/l125 0 2024-03-04T16:04:25.041 INFO:tasks.workunit.client.0.smithi116.stdout:6/931: creat d6/de/d84/db4/f135 x:0 0 0 2024-03-04T16:04:25.042 INFO:tasks.workunit.client.0.smithi116.stdout:9/920: symlink d2/d33/dc5/l126 0 2024-03-04T16:04:25.043 INFO:tasks.workunit.client.0.smithi116.stdout:9/921: chown d2/d19/d5f/c109 1562 1 2024-03-04T16:04:25.057 INFO:tasks.workunit.client.0.smithi116.stdout:9/922: dread d2/d33/d40/fbc [0,4194304] 0 2024-03-04T16:04:25.058 INFO:tasks.workunit.client.0.smithi116.stdout:9/923: chown d2/d33/l4e 1251276 1 2024-03-04T16:04:25.061 INFO:tasks.workunit.client.0.smithi116.stdout:9/924: mknod d2/d33/d65/dbf/c127 0 2024-03-04T16:04:25.064 INFO:tasks.workunit.client.0.smithi116.stdout:9/925: creat d2/d19/d5f/f128 x:0 0 0 2024-03-04T16:04:25.069 INFO:tasks.workunit.client.0.smithi116.stdout:9/926: mkdir d2/d26/dd4/d4a/d6a/d129 0 2024-03-04T16:04:25.070 INFO:tasks.workunit.client.0.smithi116.stdout:9/927: read d2/d26/dd4/de5/f25 [914047,35601] 0 2024-03-04T16:04:25.070 INFO:tasks.workunit.client.0.smithi116.stdout:9/928: stat d2/d26/dd4/c58 0 2024-03-04T16:04:25.073 INFO:tasks.workunit.client.0.smithi116.stdout:9/929: unlink d2/d33/d65/d6c/l7d 0 2024-03-04T16:04:25.074 INFO:tasks.workunit.client.0.smithi116.stdout:9/930: truncate d2/f48 2981525 0 2024-03-04T16:04:25.077 INFO:tasks.workunit.client.0.smithi116.stdout:9/931: mknod d2/d26/dd4/d4a/d6a/d94/d95/dfe/c12a 0 2024-03-04T16:04:25.080 INFO:tasks.workunit.client.0.smithi116.stdout:9/932: symlink d2/d19/d27/l12b 0 2024-03-04T16:04:25.083 INFO:tasks.workunit.client.0.smithi116.stdout:9/933: creat d2/d33/d65/d6c/db8/f12c x:0 0 0 2024-03-04T16:04:25.083 INFO:tasks.workunit.client.0.smithi116.stdout:9/934: stat d2/d26/dd4/d4a/d6a/d94/dbe/f114 0 2024-03-04T16:04:25.086 INFO:tasks.workunit.client.0.smithi116.stdout:9/935: symlink d2/d33/d65/d6c/db8/de3/l12d 0 2024-03-04T16:04:25.086 INFO:tasks.workunit.client.0.smithi116.stdout:9/936: chown d2/d33/d65/l8d 950044126 1 2024-03-04T16:04:25.088 INFO:tasks.workunit.client.0.smithi116.stdout:9/937: mknod d2/d33/d65/d6c/db8/c12e 0 2024-03-04T16:04:25.089 INFO:tasks.workunit.client.0.smithi116.stdout:9/938: write d2/d19/d5f/f128 [364745,102276] 0 2024-03-04T16:04:25.090 INFO:tasks.workunit.client.0.smithi116.stdout:0/901: sync 2024-03-04T16:04:25.090 INFO:tasks.workunit.client.0.smithi116.stdout:1/815: sync 2024-03-04T16:04:25.091 INFO:tasks.workunit.client.0.smithi116.stdout:9/939: dread - d2/d26/dd4/fd5 zero size 2024-03-04T16:04:25.094 INFO:tasks.workunit.client.0.smithi116.stdout:1/816: rmdir d5/d9/d21/d84 39 2024-03-04T16:04:25.096 INFO:tasks.workunit.client.0.smithi116.stdout:0/902: dwrite da/d16/d20/dcf/fd3 [0,4194304] 0 2024-03-04T16:04:25.098 INFO:tasks.workunit.client.0.smithi116.stdout:9/940: dwrite d2/d19/d27/dbd/ff6 [0,4194304] 0 2024-03-04T16:04:25.101 INFO:tasks.workunit.client.0.smithi116.stdout:1/817: rename d5/f29 to d5/d9/d21/d24/de0/d4c/d67/dcf/f10e 0 2024-03-04T16:04:25.103 INFO:tasks.workunit.client.0.smithi116.stdout:0/903: mkdir da/d16/d28/d40/d48/d57/dd6/d11a 0 2024-03-04T16:04:25.108 INFO:tasks.workunit.client.0.smithi116.stdout:1/818: mkdir d5/d9/d21/d27/d32/d35/dbf/d10f 0 2024-03-04T16:04:25.109 INFO:tasks.workunit.client.0.smithi116.stdout:1/819: readlink d5/d9/d21/d24/l8a 0 2024-03-04T16:04:25.110 INFO:tasks.workunit.client.0.smithi116.stdout:0/904: creat da/d16/d20/d53/f11b x:0 0 0 2024-03-04T16:04:25.110 INFO:tasks.workunit.client.0.smithi116.stdout:0/905: readlink da/d16/d19/d27/d82/le5 0 2024-03-04T16:04:25.111 INFO:tasks.workunit.client.0.smithi116.stdout:9/941: getdents d2/d26/dd4/d11e 0 2024-03-04T16:04:25.120 INFO:tasks.workunit.client.0.smithi116.stdout:9/942: mknod d2/d26/dd4/dab/c12f 0 2024-03-04T16:04:25.122 INFO:tasks.workunit.client.0.smithi116.stdout:0/906: rename da/d16/d28/d7b/la9 to da/d16/d28/d40/d48/d57/db3/l11c 0 2024-03-04T16:04:25.125 INFO:tasks.workunit.client.0.smithi116.stdout:9/943: rename d2/f48 to d2/d19/d5f/f130 0 2024-03-04T16:04:25.126 INFO:tasks.workunit.client.0.smithi116.stdout:9/944: write d2/d19/d5f/fe4 [4708279,109322] 0 2024-03-04T16:04:25.131 INFO:tasks.workunit.client.0.smithi116.stdout:0/907: link da/d16/f4c da/d16/d28/d7b/dfc/f11d 0 2024-03-04T16:04:25.132 INFO:tasks.workunit.client.0.smithi116.stdout:0/908: chown da/d16/d28/d40/d48/d57/db3/db2/dc2/dee 484 1 2024-03-04T16:04:25.133 INFO:tasks.workunit.client.0.smithi116.stdout:0/909: stat da/d16/d28/d40/d48/d57/d8a/daa/dba 0 2024-03-04T16:04:25.138 INFO:tasks.workunit.client.0.smithi116.stdout:0/910: mknod da/d16/d19/d27/c11e 0 2024-03-04T16:04:25.139 INFO:tasks.workunit.client.0.smithi116.stdout:0/911: dread - da/d16/d19/d27/d45/ff6 zero size 2024-03-04T16:04:25.140 INFO:tasks.workunit.client.0.smithi116.stdout:0/912: chown da/d16/d20/f3b 159 1 2024-03-04T16:04:25.156 INFO:tasks.workunit.client.0.smithi116.stdout:5/987: sync 2024-03-04T16:04:25.156 INFO:tasks.workunit.client.0.smithi116.stdout:6/932: sync 2024-03-04T16:04:25.157 INFO:tasks.workunit.client.0.smithi116.stdout:5/988: chown dd/d1f/d55/l8c 0 1 2024-03-04T16:04:25.159 INFO:tasks.workunit.client.0.smithi116.stdout:6/933: rmdir d6/de/d1e 39 2024-03-04T16:04:25.165 INFO:tasks.workunit.client.0.smithi116.stdout:6/934: mkdir d6/de/d25/d4a/d63/d8c/d133/d136 0 2024-03-04T16:04:25.169 INFO:tasks.workunit.client.0.smithi116.stdout:6/935: mknod d6/de/d25/d4a/df8/dec/dfe/c137 0 2024-03-04T16:04:25.171 INFO:tasks.workunit.client.0.smithi116.stdout:5/989: dread dd/d1f/d5c/f9e [0,4194304] 0 2024-03-04T16:04:25.172 INFO:tasks.workunit.client.0.smithi116.stdout:6/936: unlink d6/de/d29/d3a/f46 0 2024-03-04T16:04:25.173 INFO:tasks.workunit.client.0.smithi116.stdout:6/937: chown d6/de/d25/d36/d44/dbb/ddd 1052385 1 2024-03-04T16:04:25.180 INFO:tasks.workunit.client.0.smithi116.stdout:6/938: symlink d6/de/d25/d36/d44/dbb/ddd/l138 0 2024-03-04T16:04:25.181 INFO:tasks.workunit.client.0.smithi116.stdout:6/939: fsync d6/d56/f8b 0 2024-03-04T16:04:25.181 INFO:tasks.workunit.client.0.smithi116.stdout:5/990: getdents dd/d1f/dcd/d122 0 2024-03-04T16:04:25.182 INFO:tasks.workunit.client.0.smithi116.stdout:1/820: dwrite d5/d9/d21/d27/d94/db6/d42/f9c [0,4194304] 0 2024-03-04T16:04:25.186 INFO:tasks.workunit.client.0.smithi116.stdout:5/991: creat dd/d1f/d55/d73/f140 x:0 0 0 2024-03-04T16:04:25.195 INFO:tasks.workunit.client.0.smithi116.stdout:5/992: unlink dd/d1f/d51/f93 0 2024-03-04T16:04:25.195 INFO:tasks.workunit.client.0.smithi116.stdout:5/993: fsync dd/d1f/d55/d73/d2d/dae/fbe 0 2024-03-04T16:04:25.198 INFO:tasks.workunit.client.0.smithi116.stdout:6/940: link d6/de/d25/d4a/df8/fc0 d6/de/d84/db4/d92/f139 0 2024-03-04T16:04:25.203 INFO:tasks.workunit.client.0.smithi116.stdout:5/994: creat dd/d1f/dcd/f141 x:0 0 0 2024-03-04T16:04:25.204 INFO:tasks.workunit.client.0.smithi116.stdout:9/945: dwrite d2/d26/dd4/d4a/f7b [0,4194304] 0 2024-03-04T16:04:25.207 INFO:tasks.workunit.client.0.smithi116.stdout:9/946: mknod d2/d19/d5f/d11b/c131 0 2024-03-04T16:04:25.210 INFO:tasks.workunit.client.0.smithi116.stdout:5/995: symlink dd/d1f/l142 0 2024-03-04T16:04:25.210 INFO:tasks.workunit.client.0.smithi116.stdout:9/947: fsync d2/f1a 0 2024-03-04T16:04:25.214 INFO:tasks.workunit.client.0.smithi116.stdout:6/941: dwrite d6/fd5 [0,4194304] 0 2024-03-04T16:04:25.216 INFO:tasks.workunit.client.0.smithi116.stdout:0/913: dwrite da/d16/d28/d40/fe3 [0,4194304] 0 2024-03-04T16:04:25.223 INFO:tasks.workunit.client.0.smithi116.stdout:6/942: dwrite d6/de/d25/d36/d44/dbb/ddd/d7b/faf [0,4194304] 0 2024-03-04T16:04:25.225 INFO:tasks.workunit.client.0.smithi116.stdout:0/914: unlink da/d16/d28/d39/f112 0 2024-03-04T16:04:25.226 INFO:tasks.workunit.client.0.smithi116.stdout:0/915: dread - da/d16/d28/d40/da8/ffa zero size 2024-03-04T16:04:25.230 INFO:tasks.workunit.client.0.smithi116.stdout:5/996: link dd/d1f/d55/d73/f3f dd/d1f/d55/d73/d2d/dae/f143 0 2024-03-04T16:04:25.241 INFO:tasks.workunit.client.0.smithi116.stdout:0/916: rename da/d16/d20/d79/fdf to da/d16/d28/d40/d48/d57/db3/db2/dc2/dee/d105/f11f 0 2024-03-04T16:04:25.243 INFO:tasks.workunit.client.0.smithi116.stdout:5/997: mkdir dd/d1f/d5c/d5e/dc2/d144 0 2024-03-04T16:04:25.243 INFO:tasks.workunit.client.0.smithi116.stdout:5/998: chown dd/l4f 7 1 2024-03-04T16:04:25.248 INFO:tasks.workunit.client.0.smithi116.stdout:5/999: dwrite dd/d1f/d55/d73/d2d/d59/f10b [0,4194304] 0 2024-03-04T16:04:25.254 INFO:tasks.workunit.client.0.smithi116.stdout:0/917: dread da/d16/d20/d42/de0/f100 [0,4194304] 0 2024-03-04T16:04:25.261 INFO:tasks.workunit.client.0.smithi116.stdout:1/821: write d5/d9/d21/d27/d94/fcd [605402,41249] 0 2024-03-04T16:04:25.261 INFO:tasks.workunit.client.0.smithi116.stdout:6/943: dread d6/de/f3d [0,4194304] 0 2024-03-04T16:04:25.263 INFO:tasks.workunit.client.0.smithi116.stdout:6/944: dread - d6/de/d25/d4a/df8/da5/db3/fda zero size 2024-03-04T16:04:25.270 INFO:tasks.workunit.client.0.smithi116.stdout:6/945: fdatasync d6/de/f3b 0 2024-03-04T16:04:25.272 INFO:tasks.workunit.client.0.smithi116.stdout:1/822: getdents d5/d86/dac/de5 0 2024-03-04T16:04:25.280 INFO:tasks.workunit.client.0.smithi116.stdout:6/946: dwrite d6/de/d1e/f48 [0,4194304] 0 2024-03-04T16:04:25.292 INFO:tasks.workunit.client.0.smithi116.stdout:6/947: creat d6/de/d25/d36/d44/dbb/ddd/f13a x:0 0 0 2024-03-04T16:04:25.293 INFO:tasks.workunit.client.0.smithi116.stdout:6/948: chown d6/de/d25/d4a/df8/da5/db3 18 1 2024-03-04T16:04:25.295 INFO:tasks.workunit.client.0.smithi116.stdout:9/948: truncate d2/d26/dd4/de5/f43 499367 0 2024-03-04T16:04:25.298 INFO:tasks.workunit.client.0.smithi116.stdout:6/949: rename d6/de/d15/f66 to d6/de/d25/d4a/d63/d8c/d133/f13b 0 2024-03-04T16:04:25.303 INFO:tasks.workunit.client.0.smithi116.stdout:9/949: rename d2/d26/dd4/dab/d3c/f93 to d2/d33/d65/dbf/d116/f132 0 2024-03-04T16:04:25.315 INFO:tasks.workunit.client.0.smithi116.stdout:9/950: link d2/d26/dd4/d4a/d6a/d94/d95/db7/ff9 d2/d26/dd4/f133 0 2024-03-04T16:04:25.319 INFO:tasks.workunit.client.0.smithi116.stdout:9/951: rename d2/d26/dd4/de5/fa8 to d2/d33/d65/d6c/f134 0 2024-03-04T16:04:25.321 INFO:tasks.workunit.client.0.smithi116.stdout:0/918: dwrite da/d16/d28/d40/d48/d57/db3/db2/dc2/f10a [0,4194304] 0 2024-03-04T16:04:25.326 INFO:tasks.workunit.client.0.smithi116.stdout:9/952: truncate d2/f15 1010682 0 2024-03-04T16:04:25.327 INFO:tasks.workunit.client.0.smithi116.stdout:9/953: dread - d2/d26/dd4/dab/d3c/f108 zero size 2024-03-04T16:04:25.332 INFO:tasks.workunit.client.0.smithi116.stdout:1/823: write d5/d9/d21/d27/d32/f83 [1005908,68076] 0 2024-03-04T16:04:25.333 INFO:tasks.workunit.client.0.smithi116.stdout:9/954: dwrite d2/d19/d5f/f128 [0,4194304] 0 2024-03-04T16:04:25.341 INFO:tasks.workunit.client.0.smithi116.stdout:9/955: unlink d2/d33/d65/d7e/cba 0 2024-03-04T16:04:25.350 INFO:tasks.workunit.client.0.smithi116.stdout:6/950: rmdir d6/de/d25/d4a/d63/d8c 39 2024-03-04T16:04:25.351 INFO:tasks.workunit.client.0.smithi116.stdout:1/824: getdents d5/d9/d21/d27/d32/d35/d55/d75 0 2024-03-04T16:04:25.351 INFO:tasks.workunit.client.0.smithi116.stdout:6/951: chown d6/de/d25/d36/d4e/dfa/f112 3 1 2024-03-04T16:04:25.352 INFO:tasks.workunit.client.0.smithi116.stdout:6/952: write d6/de/d25/d36/d44/dbb/f12a [646102,89024] 0 2024-03-04T16:04:25.355 INFO:tasks.workunit.client.0.smithi116.stdout:6/953: truncate d6/de/d25/d36/d44/dbb/ddd/f13a 568491 0 2024-03-04T16:04:25.356 INFO:tasks.workunit.client.0.smithi116.stdout:6/954: stat d6/de/d84/db4/d92/l11e 0 2024-03-04T16:04:25.356 INFO:tasks.workunit.client.0.smithi116.stdout:9/956: link d2/d26/dd4/dab/c98 d2/d26/dd4/de5/dcc/c135 0 2024-03-04T16:04:25.357 INFO:tasks.workunit.client.0.smithi116.stdout:9/957: read d2/d26/dd4/dab/f6f [459829,17676] 0 2024-03-04T16:04:25.358 INFO:tasks.workunit.client.0.smithi116.stdout:9/958: dread - d2/d33/d65/dbf/fd7 zero size 2024-03-04T16:04:25.361 INFO:tasks.workunit.client.0.smithi116.stdout:9/959: chown d2/c8a 154 1 2024-03-04T16:04:25.364 INFO:tasks.workunit.client.0.smithi116.stdout:6/955: creat d6/de/d25/d36/d4e/f13c x:0 0 0 2024-03-04T16:04:25.367 INFO:tasks.workunit.client.0.smithi116.stdout:6/956: creat d6/de/d84/db4/d92/f13d x:0 0 0 2024-03-04T16:04:25.368 INFO:tasks.workunit.client.0.smithi116.stdout:9/960: getdents d2/d26/dd4/d4a/d6a/d94/d95/dfe 0 2024-03-04T16:04:25.370 INFO:tasks.workunit.client.0.smithi116.stdout:6/957: truncate d6/de/d25/d36/d44/dbb/ddd/d7b/f11a 774433 0 2024-03-04T16:04:25.371 INFO:tasks.workunit.client.0.smithi116.stdout:9/961: creat d2/d26/d28/d121/f136 x:0 0 0 2024-03-04T16:04:25.371 INFO:tasks.workunit.client.0.smithi116.stdout:0/919: truncate da/d16/d19/fe9 963443 0 2024-03-04T16:04:25.374 INFO:tasks.workunit.client.0.smithi116.stdout:0/920: dread da/d16/f6e [0,4194304] 0 2024-03-04T16:04:25.375 INFO:tasks.workunit.client.0.smithi116.stdout:6/958: truncate d6/d56/f8b 2448751 0 2024-03-04T16:04:25.376 INFO:tasks.workunit.client.0.smithi116.stdout:6/959: write d6/de/d84/db4/d92/f13d [976509,98598] 0 2024-03-04T16:04:25.379 INFO:tasks.workunit.client.0.smithi116.stdout:6/960: read d6/d56/f60 [328104,49607] 0 2024-03-04T16:04:25.382 INFO:tasks.workunit.client.0.smithi116.stdout:6/961: symlink d6/de/d25/d36/l13e 0 2024-03-04T16:04:25.388 INFO:tasks.workunit.client.0.smithi116.stdout:6/962: dwrite d6/de/d29/fe5 [4194304,4194304] 0 2024-03-04T16:04:25.390 INFO:tasks.workunit.client.0.smithi116.stdout:1/825: dwrite d5/d9/d21/f2f [0,4194304] 0 2024-03-04T16:04:25.398 INFO:tasks.workunit.client.0.smithi116.stdout:9/962: dwrite d2/f4f [0,4194304] 0 2024-03-04T16:04:25.402 INFO:tasks.workunit.client.0.smithi116.stdout:9/963: mkdir d2/d26/dc8/d137 0 2024-03-04T16:04:25.402 INFO:tasks.workunit.client.0.smithi116.stdout:6/963: dwrite d6/de/d25/d4a/df8/da5/db3/d122/f57 [0,4194304] 0 2024-03-04T16:04:25.405 INFO:tasks.workunit.client.0.smithi116.stdout:1/826: dread d5/d9/d21/d27/d32/d35/f87 [0,4194304] 0 2024-03-04T16:04:25.407 INFO:tasks.workunit.client.0.smithi116.stdout:6/964: rename f2 to d6/de/d84/db4/d92/f13f 0 2024-03-04T16:04:25.412 INFO:tasks.workunit.client.0.smithi116.stdout:9/964: dwrite d2/d26/f113 [0,4194304] 0 2024-03-04T16:04:25.417 INFO:tasks.workunit.client.0.smithi116.stdout:6/965: dread d6/f108 [0,4194304] 0 2024-03-04T16:04:25.421 INFO:tasks.workunit.client.0.smithi116.stdout:6/966: readlink d6/d56/d58/ld9 0 2024-03-04T16:04:25.423 INFO:tasks.workunit.client.0.smithi116.stdout:9/965: dwrite d2/d26/dd4/d4a/d6a/d94/dbe/ff2 [0,4194304] 0 2024-03-04T16:04:25.425 INFO:tasks.workunit.client.0.smithi116.stdout:9/966: dread - d2/d33/d65/d6c/ff7 zero size 2024-03-04T16:04:25.427 INFO:tasks.workunit.client.0.smithi116.stdout:6/967: dread d6/de/d1e/f2a [0,4194304] 0 2024-03-04T16:04:25.427 INFO:tasks.workunit.client.0.smithi116.stdout:1/827: write d5/f48 [1262938,94294] 0 2024-03-04T16:04:25.433 INFO:tasks.workunit.client.0.smithi116.stdout:6/968: read d6/de/d15/dfc/f107 [2448015,106492] 0 2024-03-04T16:04:25.433 INFO:tasks.workunit.client.0.smithi116.stdout:1/828: creat d5/d9/d21/d27/d94/db6/d42/d8b/dc9/dd1/dd9/f110 x:0 0 0 2024-03-04T16:04:25.436 INFO:tasks.workunit.client.0.smithi116.stdout:1/829: symlink d5/d9/d21/d27/d94/df2/l111 0 2024-03-04T16:04:25.436 INFO:tasks.workunit.client.0.smithi116.stdout:1/830: chown d5/d9/d21/dc7 1088960 1 2024-03-04T16:04:25.437 INFO:tasks.workunit.client.0.smithi116.stdout:1/831: chown d5/d9/d21/d27/d94/db6/d42/d8b/dc9 34485522 1 2024-03-04T16:04:25.439 INFO:tasks.workunit.client.0.smithi116.stdout:1/832: creat d5/d9/d21/d27/d94/db6/d42/f112 x:0 0 0 2024-03-04T16:04:25.441 INFO:tasks.workunit.client.0.smithi116.stdout:9/967: truncate d2/d26/dd4/d4a/f7b 968075 0 2024-03-04T16:04:25.442 INFO:tasks.workunit.client.0.smithi116.stdout:1/833: write d5/d9/d21/d84/db0/f103 [7142328,75957] 0 2024-03-04T16:04:25.445 INFO:tasks.workunit.client.0.smithi116.stdout:1/834: mkdir d5/d86/dac/d113 0 2024-03-04T16:04:25.447 INFO:tasks.workunit.client.0.smithi116.stdout:1/835: creat d5/d9/d21/d27/d8c/d10b/f114 x:0 0 0 2024-03-04T16:04:25.450 INFO:tasks.workunit.client.0.smithi116.stdout:1/836: read d5/d9/d21/d84/fb5 [1609231,112209] 0 2024-03-04T16:04:25.452 INFO:tasks.workunit.client.0.smithi116.stdout:1/837: fdatasync d5/d9/d21/d27/fb9 0 2024-03-04T16:04:25.453 INFO:tasks.workunit.client.0.smithi116.stdout:1/838: dread - d5/d9/d21/d27/d8c/d10b/f114 zero size 2024-03-04T16:04:25.453 INFO:tasks.workunit.client.0.smithi116.stdout:1/839: read d5/d9/d21/f2f [3351651,117405] 0 2024-03-04T16:04:25.455 INFO:tasks.workunit.client.0.smithi116.stdout:6/969: dread d6/de/d25/d4a/d63/d8c/fa3 [0,4194304] 0 2024-03-04T16:04:25.456 INFO:tasks.workunit.client.0.smithi116.stdout:1/840: read d5/f3b [430183,43982] 0 2024-03-04T16:04:25.456 INFO:tasks.workunit.client.0.smithi116.stdout:1/841: stat d5/f6f 0 2024-03-04T16:04:25.461 INFO:tasks.workunit.client.0.smithi116.stdout:1/842: truncate d5/d9/d21/d27/d32/d35/f87 1528606 0 2024-03-04T16:04:25.461 INFO:tasks.workunit.client.0.smithi116.stdout:1/843: dread - d5/d86/dac/ff4 zero size 2024-03-04T16:04:25.462 INFO:tasks.workunit.client.0.smithi116.stdout:6/970: truncate d6/de/d5b/f8e 3342359 0 2024-03-04T16:04:25.464 INFO:tasks.workunit.client.0.smithi116.stdout:1/844: mknod d5/d9/d21/d24/de0/d4c/c115 0 2024-03-04T16:04:25.465 INFO:tasks.workunit.client.0.smithi116.stdout:6/971: creat d6/de/d25/d4a/df8/da5/f140 x:0 0 0 2024-03-04T16:04:25.466 INFO:tasks.workunit.client.0.smithi116.stdout:1/845: creat d5/d9/d105/f116 x:0 0 0 2024-03-04T16:04:25.467 INFO:tasks.workunit.client.0.smithi116.stdout:1/846: read d5/d9/d21/f2c [1555479,83863] 0 2024-03-04T16:04:25.472 INFO:tasks.workunit.client.0.smithi116.stdout:6/972: dwrite d6/d56/d58/f125 [4194304,4194304] 0 2024-03-04T16:04:25.474 INFO:tasks.workunit.client.0.smithi116.stdout:6/973: fdatasync d6/de/d25/d36/d4e/f13c 0 2024-03-04T16:04:25.479 INFO:tasks.workunit.client.0.smithi116.stdout:1/847: dwrite d5/d9/d21/d27/d32/d35/d55/db4/f70 [0,4194304] 0 2024-03-04T16:04:25.488 INFO:tasks.workunit.client.0.smithi116.stdout:1/848: getdents d5/d86/dac/de5 0 2024-03-04T16:04:25.492 INFO:tasks.workunit.client.0.smithi116.stdout:6/974: dwrite d6/de/d25/d36/d44/dbb/ddd/d7b/dc6/fe1 [0,4194304] 0 2024-03-04T16:04:25.505 INFO:tasks.workunit.client.0.smithi116.stdout:6/975: rename d6/de/c10 to d6/de/d25/c141 0 2024-03-04T16:04:25.524 INFO:tasks.workunit.client.0.smithi116.stdout:1/849: dwrite d5/d9/fbe [0,4194304] 0 2024-03-04T16:04:25.526 INFO:tasks.workunit.client.0.smithi116.stdout:6/976: dread d6/de/d25/d36/d44/dbb/fd2 [0,4194304] 0 2024-03-04T16:04:25.529 INFO:tasks.workunit.client.0.smithi116.stdout:1/850: dread d5/d9/d21/d27/d32/d35/f87 [0,4194304] 0 2024-03-04T16:04:25.531 INFO:tasks.workunit.client.0.smithi116.stdout:6/977: creat d6/de/d25/d36/d44/dbb/ddd/f142 x:0 0 0 2024-03-04T16:04:25.532 INFO:tasks.workunit.client.0.smithi116.stdout:1/851: write d5/d9/d21/d27/d32/d35/d55/d75/d91/f100 [1012403,116921] 0 2024-03-04T16:04:25.534 INFO:tasks.workunit.client.0.smithi116.stdout:6/978: truncate d6/de/d25/d36/f27 860678 0 2024-03-04T16:04:25.534 INFO:tasks.workunit.client.0.smithi116.stdout:6/979: stat d6/de/d84/db4/d92/l11e 0 2024-03-04T16:04:25.535 INFO:tasks.workunit.client.0.smithi116.stdout:1/852: mknod d5/d9/d21/d27/d32/d35/d55/db4/c117 0 2024-03-04T16:04:25.536 INFO:tasks.workunit.client.0.smithi116.stdout:0/921: sync 2024-03-04T16:04:25.536 INFO:tasks.workunit.client.0.smithi116.stdout:1/853: chown d5/d9/d21/d84/db0/feb 10495647 1 2024-03-04T16:04:25.539 INFO:tasks.workunit.client.0.smithi116.stdout:0/922: unlink da/l11 0 2024-03-04T16:04:25.539 INFO:tasks.workunit.client.0.smithi116.stdout:6/980: dread d6/de/d84/db4/d92/f13d [0,4194304] 0 2024-03-04T16:04:25.540 INFO:tasks.workunit.client.0.smithi116.stdout:6/981: readlink d6/l55 0 2024-03-04T16:04:25.541 INFO:tasks.workunit.client.0.smithi116.stdout:1/854: rename d5/d9/d21/d27/d32/f47 to d5/d9/d21/d27/d32/d35/f118 0 2024-03-04T16:04:25.542 INFO:tasks.workunit.client.0.smithi116.stdout:1/855: stat d5/d9/d21/d27/d32/d35/d55/db4/d50/ce2 0 2024-03-04T16:04:25.543 INFO:tasks.workunit.client.0.smithi116.stdout:6/982: rename d6/de/d1e/f48 to d6/de/d84/d12f/f143 0 2024-03-04T16:04:25.544 INFO:tasks.workunit.client.0.smithi116.stdout:1/856: mkdir d5/d9/d21/d84/d119 0 2024-03-04T16:04:25.545 INFO:tasks.workunit.client.0.smithi116.stdout:1/857: chown d5/d9/d21/d27/d32/d35 71206805 1 2024-03-04T16:04:25.588 INFO:tasks.workunit.client.0.smithi116.stdout:6/983: rmdir d6/de/d25/d36/d44 39 2024-03-04T16:04:25.589 INFO:tasks.workunit.client.0.smithi116.stdout:6/984: readlink d6/de/d15/l49 0 2024-03-04T16:04:25.594 INFO:tasks.workunit.client.0.smithi116.stdout:1/858: dwrite d5/d9/d21/d27/d94/db6/d42/f62 [0,4194304] 0 2024-03-04T16:04:25.597 INFO:tasks.workunit.client.0.smithi116.stdout:6/985: dwrite d6/de/d84/db4/f135 [0,4194304] 0 2024-03-04T16:04:25.598 INFO:tasks.workunit.client.0.smithi116.stdout:1/859: fsync d5/d9/d21/d27/d32/d35/d55/db4/d50/f8e 0 2024-03-04T16:04:25.600 INFO:tasks.workunit.client.0.smithi116.stdout:1/860: chown d5/d86/ca5 431145041 1 2024-03-04T16:04:25.601 INFO:tasks.workunit.client.0.smithi116.stdout:6/986: write d6/de/d84/f10f [2340190,94723] 0 2024-03-04T16:04:25.603 INFO:tasks.workunit.client.0.smithi116.stdout:6/987: readlink d6/de/d25/d4a/d63/l70 0 2024-03-04T16:04:25.605 INFO:tasks.workunit.client.0.smithi116.stdout:6/988: rmdir d6/de/d25/d4a/df8/da5/db3/d122 39 2024-03-04T16:04:25.606 INFO:tasks.workunit.client.0.smithi116.stdout:6/989: chown d6/de/d29/d3a/lfd 4529 1 2024-03-04T16:04:25.607 INFO:tasks.workunit.client.0.smithi116.stdout:6/990: stat d6/de/d25/d4a/f71 0 2024-03-04T16:04:25.609 INFO:tasks.workunit.client.0.smithi116.stdout:6/991: mknod d6/de/d25/d4a/d63/d8c/d133/c144 0 2024-03-04T16:04:25.611 INFO:tasks.workunit.client.0.smithi116.stdout:6/992: mkdir d6/de/d84/d12f/d145 0 2024-03-04T16:04:25.619 INFO:tasks.workunit.client.0.smithi116.stdout:1/861: dwrite d5/d9/d21/d27/d94/db6/f9d [0,4194304] 0 2024-03-04T16:04:25.623 INFO:tasks.workunit.client.0.smithi116.stdout:1/862: unlink d5/d9/d21/d24/de0/d4c/d67/c6c 0 2024-03-04T16:04:25.626 INFO:tasks.workunit.client.0.smithi116.stdout:1/863: rename d5/d9/d21/d84/db0/c104 to d5/da1/c11a 0 2024-03-04T16:04:25.630 INFO:tasks.workunit.client.0.smithi116.stdout:1/864: link d5/d9/d21/d27/d8c/fb3 d5/d9/d21/d24/f11b 0 2024-03-04T16:04:25.630 INFO:tasks.workunit.client.0.smithi116.stdout:9/968: sync 2024-03-04T16:04:25.632 INFO:tasks.workunit.client.0.smithi116.stdout:1/865: mkdir d5/d9/d21/d27/d32/d11c 0 2024-03-04T16:04:25.633 INFO:tasks.workunit.client.0.smithi116.stdout:9/969: dread d2/d26/f71 [0,4194304] 0 2024-03-04T16:04:25.637 INFO:tasks.workunit.client.0.smithi116.stdout:1/866: dread d5/d9/d21/d27/d32/d35/d55/d75/d91/fb2 [4194304,4194304] 0 2024-03-04T16:04:25.637 INFO:tasks.workunit.client.0.smithi116.stdout:9/970: rmdir d2/d26/dd4/d4a/d6a/d129 0 2024-03-04T16:04:25.639 INFO:tasks.workunit.client.0.smithi116.stdout:1/867: write d5/d9/d21/d27/d32/d35/d55/dae/fd0 [901544,108572] 0 2024-03-04T16:04:25.640 INFO:tasks.workunit.client.0.smithi116.stdout:9/971: dread d2/f4f [0,4194304] 0 2024-03-04T16:04:25.643 INFO:tasks.workunit.client.0.smithi116.stdout:1/868: creat d5/d9/d21/d27/d94/db6/d42/d8b/dc9/dd1/d109/f11d x:0 0 0 2024-03-04T16:04:25.650 INFO:tasks.workunit.client.0.smithi116.stdout:9/972: dwrite d2/d26/fb1 [0,4194304] 0 2024-03-04T16:04:25.656 INFO:tasks.workunit.client.0.smithi116.stdout:9/973: write d2/d19/d5f/f61 [1327255,29723] 0 2024-03-04T16:04:25.658 INFO:tasks.workunit.client.0.smithi116.stdout:1/869: dread d5/d9/d21/d27/d94/db6/d61/fe1 [0,4194304] 0 2024-03-04T16:04:25.660 INFO:tasks.workunit.client.0.smithi116.stdout:9/974: symlink d2/d26/dd4/d4a/d6a/d94/d95/db7/l138 0 2024-03-04T16:04:25.661 INFO:tasks.workunit.client.0.smithi116.stdout:9/975: readlink d2/d19/d27/l2c 0 2024-03-04T16:04:25.662 INFO:tasks.workunit.client.0.smithi116.stdout:1/870: mknod d5/d9/d21/d24/de0/d4a/c11e 0 2024-03-04T16:04:25.662 INFO:tasks.workunit.client.0.smithi116.stdout:9/976: chown d2/d33/d65/d6c/db8 1502 1 2024-03-04T16:04:25.664 INFO:tasks.workunit.client.0.smithi116.stdout:9/977: symlink d2/d19/d5f/d11b/l139 0 2024-03-04T16:04:25.666 INFO:tasks.workunit.client.0.smithi116.stdout:1/871: unlink d5/d9/d21/d27/d94/db6/d42/f112 0 2024-03-04T16:04:25.669 INFO:tasks.workunit.client.0.smithi116.stdout:9/978: getdents d2 0 2024-03-04T16:04:25.670 INFO:tasks.workunit.client.0.smithi116.stdout:9/979: chown d2/d26/dd4/d4a/d6a/d94/d95 1041527932 1 2024-03-04T16:04:25.670 INFO:tasks.workunit.client.0.smithi116.stdout:9/980: stat d2/d26/dd4/d4a/d6a/d94/dbe/cf8 0 2024-03-04T16:04:25.675 INFO:tasks.workunit.client.0.smithi116.stdout:1/872: dwrite d5/d9/d21/d84/fb5 [0,4194304] 0 2024-03-04T16:04:25.678 INFO:tasks.workunit.client.0.smithi116.stdout:9/981: dwrite d2/d26/dd4/d4a/d6a/d94/dbe/ff2 [4194304,4194304] 0 2024-03-04T16:04:25.682 INFO:tasks.workunit.client.0.smithi116.stdout:9/982: symlink d2/d26/dd4/l13a 0 2024-03-04T16:04:25.690 INFO:tasks.workunit.client.0.smithi116.stdout:1/873: truncate d5/d9/fbe 3903430 0 2024-03-04T16:04:25.692 INFO:tasks.workunit.client.0.smithi116.stdout:9/983: write d2/d26/dd4/d4a/d6a/f75 [1490264,10031] 0 2024-03-04T16:04:25.694 INFO:tasks.workunit.client.0.smithi116.stdout:1/874: rename d5/d9/d21/d27/d32/d35/l77 to d5/d9/d21/d24/de0/d4c/d67/dcf/l11f 0 2024-03-04T16:04:25.695 INFO:tasks.workunit.client.0.smithi116.stdout:9/984: truncate d2/d19/f9c 270688 0 2024-03-04T16:04:25.695 INFO:tasks.workunit.client.0.smithi116.stdout:9/985: read d2/d19/fed [1009634,31188] 0 2024-03-04T16:04:25.700 INFO:tasks.workunit.client.0.smithi116.stdout:9/986: dread d2/d26/dd4/d4a/d6a/d94/d95/f102 [0,4194304] 0 2024-03-04T16:04:25.708 INFO:tasks.workunit.client.0.smithi116.stdout:9/987: rename d2/d26/dd4/dab/l11 to d2/d33/d65/l13b 0 2024-03-04T16:04:25.711 INFO:tasks.workunit.client.0.smithi116.stdout:9/988: dread d2/d26/fa0 [0,4194304] 0 2024-03-04T16:04:25.713 INFO:tasks.workunit.client.0.smithi116.stdout:1/875: dwrite d5/d9/d21/d24/f11b [0,4194304] 0 2024-03-04T16:04:25.714 INFO:tasks.workunit.client.0.smithi116.stdout:9/989: fdatasync d2/d19/d27/f106 0 2024-03-04T16:04:25.716 INFO:tasks.workunit.client.0.smithi116.stdout:1/876: symlink d5/d9/d21/d84/db0/de9/dfb/l120 0 2024-03-04T16:04:25.717 INFO:tasks.workunit.client.0.smithi116.stdout:1/877: write d5/d9/d21/d27/d94/db6/f9d [2774774,124948] 0 2024-03-04T16:04:25.719 INFO:tasks.workunit.client.0.smithi116.stdout:1/878: readlink d5/d9/d21/d24/de0/d4c/l7d 0 2024-03-04T16:04:25.720 INFO:tasks.workunit.client.0.smithi116.stdout:9/990: rmdir d2/d26/dd4/d11e 0 2024-03-04T16:04:25.721 INFO:tasks.workunit.client.0.smithi116.stdout:9/991: dread d2/d26/d28/f55 [0,4194304] 0 2024-03-04T16:04:25.722 INFO:tasks.workunit.client.0.smithi116.stdout:1/879: symlink d5/d9/d21/d84/db0/l121 0 2024-03-04T16:04:25.723 INFO:tasks.workunit.client.0.smithi116.stdout:9/992: mknod d2/d33/d40/c13c 0 2024-03-04T16:04:25.724 INFO:tasks.workunit.client.0.smithi116.stdout:1/880: symlink d5/d86/dac/l122 0 2024-03-04T16:04:25.726 INFO:tasks.workunit.client.0.smithi116.stdout:1/881: creat d5/d9/d21/d24/de0/d4c/d67/dcf/f123 x:0 0 0 2024-03-04T16:04:25.727 INFO:tasks.workunit.client.0.smithi116.stdout:6/993: sync 2024-03-04T16:04:25.727 INFO:tasks.workunit.client.0.smithi116.stdout:0/923: sync 2024-03-04T16:04:25.727 INFO:tasks.workunit.client.0.smithi116.stdout:1/882: readlink d5/l33 0 2024-03-04T16:04:25.729 INFO:tasks.workunit.client.0.smithi116.stdout:6/994: mknod d6/de/d84/d12f/c146 0 2024-03-04T16:04:25.730 INFO:tasks.workunit.client.0.smithi116.stdout:0/924: unlink da/d16/d28/d40/d48/d57/ld4 0 2024-03-04T16:04:25.731 INFO:tasks.workunit.client.0.smithi116.stdout:0/925: write da/d16/d20/d79/f104 [62773,50128] 0 2024-03-04T16:04:25.733 INFO:tasks.workunit.client.0.smithi116.stdout:0/926: dread - da/d16/d28/d40/d48/d57/d8a/daa/f102 zero size 2024-03-04T16:04:25.733 INFO:tasks.workunit.client.0.smithi116.stdout:6/995: creat d6/de/d84/db4/f147 x:0 0 0 2024-03-04T16:04:25.733 INFO:tasks.workunit.client.0.smithi116.stdout:0/927: chown f7 5148 1 2024-03-04T16:04:25.734 INFO:tasks.workunit.client.0.smithi116.stdout:1/883: dread d5/d9/d21/d27/d32/d35/fa6 [0,4194304] 0 2024-03-04T16:04:25.738 INFO:tasks.workunit.client.0.smithi116.stdout:9/993: dwrite d2/d33/f77 [0,4194304] 0 2024-03-04T16:04:25.740 INFO:tasks.workunit.client.0.smithi116.stdout:6/996: write d6/de/d84/fa7 [539713,30843] 0 2024-03-04T16:04:25.740 INFO:tasks.workunit.client.0.smithi116.stdout:0/928: write da/d16/fb1 [388866,113863] 0 2024-03-04T16:04:25.741 INFO:tasks.workunit.client.0.smithi116.stdout:0/929: dread - da/d16/d20/d42/de0/d7e/fe6 zero size 2024-03-04T16:04:25.742 INFO:tasks.workunit.client.0.smithi116.stdout:9/994: fdatasync d2/d33/d40/fbc 0 2024-03-04T16:04:25.743 INFO:tasks.workunit.client.0.smithi116.stdout:9/995: fdatasync d2/d26/dd4/d4a/d6a/d94/d95/ff4 0 2024-03-04T16:04:25.744 INFO:tasks.workunit.client.0.smithi116.stdout:6/997: creat d6/de/d25/d4a/d63/d8c/d133/f148 x:0 0 0 2024-03-04T16:04:25.745 INFO:tasks.workunit.client.0.smithi116.stdout:0/930: creat da/d16/d19/d27/d45/f120 x:0 0 0 2024-03-04T16:04:25.747 INFO:tasks.workunit.client.0.smithi116.stdout:0/931: write da/d16/d19/d27/d70/f8e [3553904,31548] 0 2024-03-04T16:04:25.747 INFO:tasks.workunit.client.0.smithi116.stdout:0/932: chown da/d16/d19/d27 7776 1 2024-03-04T16:04:25.748 INFO:tasks.workunit.client.0.smithi116.stdout:6/998: creat d6/de/d25/d4a/df8/dec/f149 x:0 0 0 2024-03-04T16:04:25.750 INFO:tasks.workunit.client.0.smithi116.stdout:0/933: write da/d16/d28/d40/d48/d57/dd6/f117 [942479,108320] 0 2024-03-04T16:04:25.752 INFO:tasks.workunit.client.0.smithi116.stdout:6/999: symlink d6/de/d25/d4a/df8/l14a 0 2024-03-04T16:04:25.754 INFO:tasks.workunit.client.0.smithi116.stdout:0/934: creat da/d16/d28/d7b/de8/dec/f121 x:0 0 0 2024-03-04T16:04:25.756 INFO:tasks.workunit.client.0.smithi116.stdout:0/935: fdatasync da/d16/f4c 0 2024-03-04T16:04:25.758 INFO:tasks.workunit.client.0.smithi116.stdout:9/996: dread d2/d26/dd4/de5/fa9 [0,4194304] 0 2024-03-04T16:04:25.759 INFO:tasks.workunit.client.0.smithi116.stdout:9/997: fdatasync d2/d26/f112 0 2024-03-04T16:04:25.760 INFO:tasks.workunit.client.0.smithi116.stdout:0/936: getdents da/d16/d19/d27/d45 0 2024-03-04T16:04:25.761 INFO:tasks.workunit.client.0.smithi116.stdout:0/937: readlink da/d16/d28/d40/d48/d57/d8a/daa/dba/lc8 0 2024-03-04T16:04:25.763 INFO:tasks.workunit.client.0.smithi116.stdout:9/998: rename d2/d26/fa0 to d2/d33/d65/d6c/db8/de3/f13d 0 2024-03-04T16:04:25.764 INFO:tasks.workunit.client.0.smithi116.stdout:9/999: creat d2/d26/dd4/dab/d3c/f13e x:0 0 0 2024-03-04T16:04:25.765 INFO:tasks.workunit.client.0.smithi116.stdout:0/938: dread da/d16/d20/d79/fce [0,4194304] 0 2024-03-04T16:04:25.768 INFO:tasks.workunit.client.0.smithi116.stdout:0/939: unlink da/d16/d19/d27/d82/d9f/f10d 0 2024-03-04T16:04:25.769 INFO:tasks.workunit.client.0.smithi116.stdout:0/940: creat da/d16/d19/d27/d82/d9f/f122 x:0 0 0 2024-03-04T16:04:25.771 INFO:tasks.workunit.client.0.smithi116.stdout:0/941: mkdir da/d123 0 2024-03-04T16:04:25.772 INFO:tasks.workunit.client.0.smithi116.stdout:0/942: chown da/d16/d28/d40/d48/d57/d8a/daa/c110 384 1 2024-03-04T16:04:25.772 INFO:tasks.workunit.client.0.smithi116.stdout:0/943: write da/f35 [3748714,7267] 0 2024-03-04T16:04:25.773 INFO:tasks.workunit.client.0.smithi116.stdout:0/944: readlink da/d16/d28/d40/d48/d77/l89 0 2024-03-04T16:04:25.774 INFO:tasks.workunit.client.0.smithi116.stdout:0/945: readlink da/d16/d19/d27/d82/le5 0 2024-03-04T16:04:25.775 INFO:tasks.workunit.client.0.smithi116.stdout:0/946: dread - da/d16/d20/d42/de0/d7e/fed zero size 2024-03-04T16:04:25.777 INFO:tasks.workunit.client.0.smithi116.stdout:0/947: creat da/d16/d28/d40/d48/d57/d8a/daa/dba/df0/d118/f124 x:0 0 0 2024-03-04T16:04:25.778 INFO:tasks.workunit.client.0.smithi116.stdout:0/948: chown da/d16/d28/d40/d48/d57/db3/db2/dc2/f10a 60562 1 2024-03-04T16:04:25.789 INFO:tasks.workunit.client.0.smithi116.stdout:0/949: dread da/d16/d28/d40/d48/fd7 [0,4194304] 0 2024-03-04T16:04:25.791 INFO:tasks.workunit.client.0.smithi116.stdout:0/950: symlink da/d16/d19/d27/d82/df4/l125 0 2024-03-04T16:04:25.792 INFO:tasks.workunit.client.0.smithi116.stdout:0/951: dread da/d16/d20/d79/fce [0,4194304] 0 2024-03-04T16:04:25.794 INFO:tasks.workunit.client.0.smithi116.stdout:0/952: mkdir da/d16/d19/d27/d45/de7/d126 0 2024-03-04T16:04:25.796 INFO:tasks.workunit.client.0.smithi116.stdout:0/953: mknod da/d16/d28/d40/d48/d57/d8a/c127 0 2024-03-04T16:04:25.928 INFO:tasks.workunit.client.0.smithi116.stdout:0/954: sync 2024-03-04T16:04:25.929 INFO:tasks.workunit.client.0.smithi116.stdout:1/884: sync 2024-03-04T16:04:25.931 INFO:tasks.workunit.client.0.smithi116.stdout:1/885: mknod d5/d86/dac/c124 0 2024-03-04T16:04:25.945 INFO:tasks.workunit.client.0.smithi116.stdout:0/955: dread da/d16/d19/d27/d45/ff2 [0,4194304] 0 2024-03-04T16:04:25.946 INFO:tasks.workunit.client.0.smithi116.stdout:0/956: chown da/d16/d28/d3d/f64 9180 1 2024-03-04T16:04:25.949 INFO:tasks.workunit.client.0.smithi116.stdout:0/957: dread da/d16/d28/d40/d48/d57/db3/db2/dc2/f10a [0,4194304] 0 2024-03-04T16:04:25.952 INFO:tasks.workunit.client.0.smithi116.stdout:0/958: mknod da/d16/d19/d27/c128 0 2024-03-04T16:04:25.952 INFO:tasks.workunit.client.0.smithi116.stdout:0/959: write da/fa4 [4530417,117400] 0 2024-03-04T16:04:25.958 INFO:tasks.workunit.client.0.smithi116.stdout:0/960: truncate da/d16/d28/d3d/f4a 2570356 0 2024-03-04T16:04:25.960 INFO:tasks.workunit.client.0.smithi116.stdout:0/961: symlink da/d16/d28/d40/d48/d57/db3/l129 0 2024-03-04T16:04:25.967 INFO:tasks.workunit.client.0.smithi116.stdout:0/962: dwrite da/f1a [0,4194304] 0 2024-03-04T16:04:26.116 INFO:tasks.workunit.client.0.smithi116.stdout:0/963: sync 2024-03-04T16:04:26.116 INFO:tasks.workunit.client.0.smithi116.stdout:1/886: sync 2024-03-04T16:04:26.118 INFO:tasks.workunit.client.0.smithi116.stdout:0/964: truncate da/d16/d20/d42/de0/ff7 734957 0 2024-03-04T16:04:26.122 INFO:tasks.workunit.client.0.smithi116.stdout:1/887: dwrite d5/d9/d21/d27/d32/d35/d55/db4/d50/ffa [0,4194304] 0 2024-03-04T16:04:26.130 INFO:tasks.workunit.client.0.smithi116.stdout:1/888: dwrite d5/d9/d21/d24/de0/d4c/d67/f73 [0,4194304] 0 2024-03-04T16:04:26.150 INFO:tasks.workunit.client.0.smithi116.stdout:0/965: dread da/d16/d28/d40/d48/d57/db3/db2/fb6 [0,4194304] 0 2024-03-04T16:04:26.163 INFO:tasks.workunit.client.0.smithi116.stdout:0/966: truncate da/d16/d28/d40/d48/d57/f10c 163050 0 2024-03-04T16:04:26.164 INFO:tasks.workunit.client.0.smithi116.stdout:0/967: write da/d16/d20/dcf/fd3 [479965,31294] 0 2024-03-04T16:04:26.166 INFO:tasks.workunit.client.0.smithi116.stdout:0/968: mknod da/c12a 0 2024-03-04T16:04:26.167 INFO:tasks.workunit.client.0.smithi116.stdout:0/969: stat da/d16/d28 0 2024-03-04T16:04:26.169 INFO:tasks.workunit.client.0.smithi116.stdout:0/970: creat da/d16/d19/d27/d70/da1/f12b x:0 0 0 2024-03-04T16:04:26.171 INFO:tasks.workunit.client.0.smithi116.stdout:0/971: rmdir da/d16/d28/d40 39 2024-03-04T16:04:26.171 INFO:tasks.workunit.client.0.smithi116.stdout:0/972: chown da/d16/d20/d42/de0/c116 3119116 1 2024-03-04T16:04:26.174 INFO:tasks.workunit.client.0.smithi116.stdout:0/973: read - da/d16/d28/d40/d48/d57/dd6/fff zero size 2024-03-04T16:04:26.176 INFO:tasks.workunit.client.0.smithi116.stdout:0/974: creat da/d16/d19/f12c x:0 0 0 2024-03-04T16:04:26.179 INFO:tasks.workunit.client.0.smithi116.stdout:0/975: creat da/d16/d28/d3d/f12d x:0 0 0 2024-03-04T16:04:26.186 INFO:tasks.workunit.client.0.smithi116.stdout:0/976: dwrite da/d16/d28/d40/d48/d57/d8a/daa/f102 [0,4194304] 0 2024-03-04T16:04:26.192 INFO:tasks.workunit.client.0.smithi116.stdout:0/977: dwrite da/d16/d19/f1d [0,4194304] 0 2024-03-04T16:04:26.198 INFO:tasks.workunit.client.0.smithi116.stdout:0/978: rename da/d16/d19/d27/c85 to da/d16/d28/d40/d48/d57/db3/db2/dc2/dee/c12e 0 2024-03-04T16:04:26.199 INFO:tasks.workunit.client.0.smithi116.stdout:0/979: dread - da/d16/d20/fd8 zero size 2024-03-04T16:04:26.204 INFO:tasks.workunit.client.0.smithi116.stdout:0/980: dwrite da/d16/d20/f3b [0,4194304] 0 2024-03-04T16:04:26.208 INFO:tasks.workunit.client.0.smithi116.stdout:0/981: write da/d16/d28/d40/d48/d57/db3/db2/dc2/f10a [2249956,3372] 0 2024-03-04T16:04:26.210 INFO:tasks.workunit.client.0.smithi116.stdout:0/982: mknod da/d16/c12f 0 2024-03-04T16:04:26.213 INFO:tasks.workunit.client.0.smithi116.stdout:0/983: symlink da/d16/d19/d27/d82/df4/l130 0 2024-03-04T16:04:26.216 INFO:tasks.workunit.client.0.smithi116.stdout:0/984: truncate da/d16/d19/fe9 1858059 0 2024-03-04T16:04:26.216 INFO:tasks.workunit.client.0.smithi116.stdout:0/985: chown da/d16/d28/c37 930838 1 2024-03-04T16:04:26.227 INFO:tasks.workunit.client.0.smithi116.stdout:0/986: dwrite da/d16/d19/d27/f74 [0,4194304] 0 2024-03-04T16:04:26.228 INFO:tasks.workunit.client.0.smithi116.stdout:0/987: write da/d16/d28/d40/fe3 [3340824,100932] 0 2024-03-04T16:04:26.354 INFO:tasks.workunit.client.0.smithi116.stdout:1/889: sync 2024-03-04T16:04:26.355 INFO:tasks.workunit.client.0.smithi116.stdout:1/890: fdatasync d5/d9/d21/d27/d32/d35/d55/d75/d91/f9e 0 2024-03-04T16:04:26.356 INFO:tasks.workunit.client.0.smithi116.stdout:1/891: fdatasync d5/d9/d21/d27/d32/d35/fa6 0 2024-03-04T16:04:26.358 INFO:tasks.workunit.client.0.smithi116.stdout:1/892: fdatasync d5/d9/d21/d24/f44 0 2024-03-04T16:04:26.361 INFO:tasks.workunit.client.0.smithi116.stdout:1/893: creat d5/d9/d21/d27/d94/db6/d61/f125 x:0 0 0 2024-03-04T16:04:26.368 INFO:tasks.workunit.client.0.smithi116.stdout:1/894: dwrite d5/d9/d21/f2f [0,4194304] 0 2024-03-04T16:04:26.370 INFO:tasks.workunit.client.0.smithi116.stdout:1/895: symlink d5/d9/d21/d27/d94/db6/d42/l126 0 2024-03-04T16:04:26.377 INFO:tasks.workunit.client.0.smithi116.stdout:1/896: dwrite d5/f48 [0,4194304] 0 2024-03-04T16:04:26.381 INFO:tasks.workunit.client.0.smithi116.stdout:1/897: dread d5/d9/d21/d27/d94/db6/f6a [0,4194304] 0 2024-03-04T16:04:26.382 INFO:tasks.workunit.client.0.smithi116.stdout:1/898: fsync d5/d9/d21/d27/d32/d35/d55/db4/f70 0 2024-03-04T16:04:26.388 INFO:tasks.workunit.client.0.smithi116.stdout:1/899: rename d5/d9/d21/d24/de0/d4c/d67/c9b to d5/d9/d21/d27/d94/db6/d42/d8b/dc9/c127 0 2024-03-04T16:04:26.390 INFO:tasks.workunit.client.0.smithi116.stdout:1/900: symlink d5/d9/d21/d27/d32/d35/l128 0 2024-03-04T16:04:26.399 INFO:tasks.workunit.client.0.smithi116.stdout:1/901: dwrite d5/d9/d21/d27/d32/d35/f118 [0,4194304] 0 2024-03-04T16:04:26.402 INFO:tasks.workunit.client.0.smithi116.stdout:1/902: truncate d5/d9/d21/d84/f90 2701067 0 2024-03-04T16:04:26.407 INFO:tasks.workunit.client.0.smithi116.stdout:1/903: rename d5/d9/d21/d24/de0/d4c/d67/dcf to d5/d86/d129 0 2024-03-04T16:04:26.407 INFO:tasks.workunit.client.0.smithi116.stdout:1/904: readlink d5/d9/d21/d84/db0/l121 0 2024-03-04T16:04:26.411 INFO:tasks.workunit.client.0.smithi116.stdout:0/988: sync 2024-03-04T16:04:26.415 INFO:tasks.workunit.client.0.smithi116.stdout:0/989: rmdir da/d16/d28/d40/d48/d57/db3/db2/dc2/dee 39 2024-03-04T16:04:26.417 INFO:tasks.workunit.client.0.smithi116.stdout:0/990: symlink da/d16/d20/dcf/l131 0 2024-03-04T16:04:26.418 INFO:tasks.workunit.client.0.smithi116.stdout:0/991: fsync da/d16/f93 0 2024-03-04T16:04:26.422 INFO:tasks.workunit.client.0.smithi116.stdout:1/905: dwrite d5/f48 [4194304,4194304] 0 2024-03-04T16:04:26.422 INFO:tasks.workunit.client.0.smithi116.stdout:0/992: dread - da/d16/d28/d3d/d41/f108 zero size 2024-03-04T16:04:26.425 INFO:tasks.workunit.client.0.smithi116.stdout:1/906: dread d5/d9/d21/d27/d32/d35/d55/d75/fef [0,4194304] 0 2024-03-04T16:04:26.426 INFO:tasks.workunit.client.0.smithi116.stdout:0/993: unlink da/d16/d20/d42/d4e/l50 0 2024-03-04T16:04:26.431 INFO:tasks.workunit.client.0.smithi116.stdout:0/994: getdents da/d16/d19/d27/d45/de7 0 2024-03-04T16:04:26.435 INFO:tasks.workunit.client.0.smithi116.stdout:0/995: dread da/d16/d20/d42/de0/ff7 [0,4194304] 0 2024-03-04T16:04:26.435 INFO:tasks.workunit.client.0.smithi116.stdout:0/996: readlink da/d16/d19/d27/d70/lf8 0 2024-03-04T16:04:26.437 INFO:tasks.workunit.client.0.smithi116.stdout:1/907: dread d5/d9/d21/d24/de0/f6d [0,4194304] 0 2024-03-04T16:04:26.439 INFO:tasks.workunit.client.0.smithi116.stdout:0/997: getdents da/d16/d28/d40/d48/d57/db3/db2/dc2 0 2024-03-04T16:04:26.440 INFO:tasks.workunit.client.0.smithi116.stdout:1/908: creat d5/d9/d21/d27/d94/f12a x:0 0 0 2024-03-04T16:04:26.441 INFO:tasks.workunit.client.0.smithi116.stdout:1/909: chown d5/d9/d21/d24/de0/d4a/f93 16 1 2024-03-04T16:04:26.443 INFO:tasks.workunit.client.0.smithi116.stdout:0/998: creat da/d16/d28/d40/d48/d57/db3/f132 x:0 0 0 2024-03-04T16:04:26.444 INFO:tasks.workunit.client.0.smithi116.stdout:0/999: fdatasync da/d16/d19/d27/d45/ff6 0 2024-03-04T16:04:26.445 INFO:tasks.workunit.client.0.smithi116.stdout:1/910: rmdir d5/d9/d21/d27/d94/db6/d42 39 2024-03-04T16:04:26.450 INFO:tasks.workunit.client.0.smithi116.stdout:1/911: creat d5/d9/d21/d27/d32/d35/d55/f12b x:0 0 0 2024-03-04T16:04:26.454 INFO:tasks.workunit.client.0.smithi116.stdout:1/912: rmdir d5/d9/d21/d27/d94/df9 0 2024-03-04T16:04:26.456 INFO:tasks.workunit.client.0.smithi116.stdout:1/913: rmdir d5/da1 39 2024-03-04T16:04:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:26 smithi152 ceph-mon[36876]: Upgrade: Updating mgr.smithi116.jwdhop 2024-03-04T16:04:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:26 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:26 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.jwdhop", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:04:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:26 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.jwdhop", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:04:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:26 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:04:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:26 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:04:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:26 smithi152 ceph-mon[36876]: Deploying daemon mgr.smithi116.jwdhop on smithi116 2024-03-04T16:04:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:26 smithi152 ceph-mon[36876]: pgmap v29: 65 pgs: 65 active+clean; 2.1 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 88 MiB/s rd, 228 MiB/s wr, 628 op/s 2024-03-04T16:04:26.672 INFO:tasks.workunit.client.0.smithi116.stdout:1/914: sync 2024-03-04T16:04:26.676 INFO:tasks.workunit.client.0.smithi116.stdout:1/915: rmdir d5/d9/d21/d27/d32/d35/d55 39 2024-03-04T16:04:26.677 INFO:tasks.workunit.client.0.smithi116.stdout:1/916: mknod d5/d9/d21/d27/d32/d11c/c12c 0 2024-03-04T16:04:26.680 INFO:tasks.workunit.client.0.smithi116.stdout:1/917: creat d5/d9/d21/d27/d32/f12d x:0 0 0 2024-03-04T16:04:26.681 INFO:tasks.workunit.client.0.smithi116.stdout:1/918: mknod d5/d86/c12e 0 2024-03-04T16:04:26.704 INFO:tasks.workunit.client.0.smithi116.stdout:1/919: dwrite d5/d9/d21/d27/d32/fa3 [0,4194304] 0 2024-03-04T16:04:26.715 INFO:tasks.workunit.client.0.smithi116.stdout:1/920: dwrite d5/d9/d21/d27/d32/d35/f74 [0,4194304] 0 2024-03-04T16:04:26.718 INFO:tasks.workunit.client.0.smithi116.stdout:1/921: write d5/d9/d21/d27/d32/d35/d55/db4/d50/f8e [836994,117806] 0 2024-03-04T16:04:26.719 INFO:tasks.workunit.client.0.smithi116.stdout:1/922: chown d5/d9/d21/d27/d32/d35/l128 229232 1 2024-03-04T16:04:26.722 INFO:tasks.workunit.client.0.smithi116.stdout:1/923: fsync d5/d9/d21/d27/d32/d35/d55/d75/d91/fb2 0 2024-03-04T16:04:26.723 INFO:tasks.workunit.client.0.smithi116.stdout:1/924: chown d5/d9/d21/d27/d32/d35/d55/db4/f63 1332621 1 2024-03-04T16:04:26.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:26 smithi116 ceph-mon[29120]: Upgrade: Updating mgr.smithi116.jwdhop 2024-03-04T16:04:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:26 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:26 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.jwdhop", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:04:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:26 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.jwdhop", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:04:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:26 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:04:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:26 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:04:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:26 smithi116 ceph-mon[29120]: Deploying daemon mgr.smithi116.jwdhop on smithi116 2024-03-04T16:04:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:26 smithi116 ceph-mon[29120]: pgmap v29: 65 pgs: 65 active+clean; 2.1 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 88 MiB/s rd, 228 MiB/s wr, 628 op/s 2024-03-04T16:04:26.725 INFO:tasks.workunit.client.0.smithi116.stdout:1/925: truncate d5/da1/fc3 4211084 0 2024-03-04T16:04:26.739 INFO:tasks.workunit.client.0.smithi116.stdout:1/926: dread d5/d9/d21/d27/d94/db6/d42/f9c [0,4194304] 0 2024-03-04T16:04:26.740 INFO:tasks.workunit.client.0.smithi116.stdout:1/927: symlink d5/d9/d21/d24/de0/d4c/l12f 0 2024-03-04T16:04:26.743 INFO:tasks.workunit.client.0.smithi116.stdout:1/928: mkdir d5/d9/d21/d27/d32/d35/d55/db4/d50/d130 0 2024-03-04T16:04:26.748 INFO:tasks.workunit.client.0.smithi116.stdout:1/929: dwrite d5/d9/d21/d27/d32/d35/d55/db4/f70 [0,4194304] 0 2024-03-04T16:04:26.750 INFO:tasks.workunit.client.0.smithi116.stdout:1/930: write d5/d9/d21/f2f [4489285,1525] 0 2024-03-04T16:04:26.754 INFO:tasks.workunit.client.0.smithi116.stdout:1/931: getdents d5/d9/d21/d84/db0 0 2024-03-04T16:04:26.755 INFO:tasks.workunit.client.0.smithi116.stdout:1/932: read d5/d9/d21/d27/d32/d35/d55/db4/f70 [736211,86670] 0 2024-03-04T16:04:26.757 INFO:tasks.workunit.client.0.smithi116.stdout:1/933: mknod d5/d9/d21/d27/d94/db6/d42/c131 0 2024-03-04T16:04:26.759 INFO:tasks.workunit.client.0.smithi116.stdout:1/934: rmdir d5/d9/d21/d27/d94/db6/d42/d8b 39 2024-03-04T16:04:26.767 INFO:tasks.workunit.client.0.smithi116.stdout:1/935: dwrite d5/d9/d21/d27/d32/d35/d55/d75/fef [0,4194304] 0 2024-03-04T16:04:26.772 INFO:tasks.workunit.client.0.smithi116.stdout:1/936: unlink d5/cbb 0 2024-03-04T16:04:26.776 INFO:tasks.workunit.client.0.smithi116.stdout:1/937: creat d5/d9/d21/d24/de0/d4a/f132 x:0 0 0 2024-03-04T16:04:26.777 INFO:tasks.workunit.client.0.smithi116.stdout:1/938: mknod d5/d9/d21/d27/d32/d35/d55/dae/c133 0 2024-03-04T16:04:26.783 INFO:tasks.workunit.client.0.smithi116.stdout:1/939: dwrite d5/d9/d21/d24/de0/d4c/d67/f107 [0,4194304] 0 2024-03-04T16:04:26.788 INFO:tasks.workunit.client.0.smithi116.stdout:1/940: rmdir d5/d9/d21/d27/d94 39 2024-03-04T16:04:26.790 INFO:tasks.workunit.client.0.smithi116.stdout:1/941: symlink d5/d9/d21/d27/d32/d11c/l134 0 2024-03-04T16:04:26.793 INFO:tasks.workunit.client.0.smithi116.stdout:1/942: mkdir d5/d9/d21/d27/d32/d35/d55/d75/d135 0 2024-03-04T16:04:26.799 INFO:tasks.workunit.client.0.smithi116.stdout:1/943: dwrite d5/d9/d21/d27/d32/d35/d55/d75/d91/fba [8388608,4194304] 0 2024-03-04T16:04:26.804 INFO:tasks.workunit.client.0.smithi116.stdout:1/944: rename d5/d9/d21/d27/d32/d35/d55/d75/d91/f100 to d5/d9/d21/d84/db0/de9/dea/f136 0 2024-03-04T16:04:26.806 INFO:tasks.workunit.client.0.smithi116.stdout:1/945: stat d5/d9/d21/d27/d94/db6/d42/d8b/dc9/dd1/dd9/f110 0 2024-03-04T16:04:26.997 INFO:tasks.workunit.client.0.smithi116.stdout:1/946: sync 2024-03-04T16:04:27.007 INFO:tasks.workunit.client.0.smithi116.stdout:1/947: dwrite d5/d9/d21/d24/f37 [0,4194304] 0 2024-03-04T16:04:27.013 INFO:tasks.workunit.client.0.smithi116.stdout:1/948: dwrite d5/d9/d21/d27/d32/ffd [0,4194304] 0 2024-03-04T16:04:27.015 INFO:tasks.workunit.client.0.smithi116.stdout:1/949: read d5/d9/d21/d27/d32/f83 [622455,91918] 0 2024-03-04T16:04:27.016 INFO:tasks.workunit.client.0.smithi116.stdout:1/950: readlink d5/d9/d21/d84/db0/l10a 0 2024-03-04T16:04:27.019 INFO:tasks.workunit.client.0.smithi116.stdout:1/951: mknod d5/c137 0 2024-03-04T16:04:27.020 INFO:tasks.workunit.client.0.smithi116.stdout:1/952: chown d5/d9/d21/d27/d32/d35/d55/d75/cb7 91317 1 2024-03-04T16:04:27.022 INFO:tasks.workunit.client.0.smithi116.stdout:1/953: mkdir d5/d9/d21/d84/db0/de9/dfb/d138 0 2024-03-04T16:04:27.024 INFO:tasks.workunit.client.0.smithi116.stdout:1/954: write d5/d86/d129/f10e [2885160,27163] 0 2024-03-04T16:04:27.025 INFO:tasks.workunit.client.0.smithi116.stdout:1/955: stat d5/d9/d21/d27/d94/db6/d42/c79 0 2024-03-04T16:04:27.027 INFO:tasks.workunit.client.0.smithi116.stdout:1/956: creat d5/d9/d21/d24/de0/d4a/f139 x:0 0 0 2024-03-04T16:04:27.028 INFO:tasks.workunit.client.0.smithi116.stdout:1/957: readlink d5/lc1 0 2024-03-04T16:04:27.030 INFO:tasks.workunit.client.0.smithi116.stdout:1/958: truncate d5/d9/fbe 4788407 0 2024-03-04T16:04:27.041 INFO:tasks.workunit.client.1.smithi152.stdout:6/64: sync 2024-03-04T16:04:27.041 INFO:tasks.workunit.client.1.smithi152.stdout:4/31: sync 2024-03-04T16:04:27.041 INFO:tasks.workunit.client.1.smithi152.stdout:0/8: sync 2024-03-04T16:04:27.041 INFO:tasks.workunit.client.1.smithi152.stdout:1/36: sync 2024-03-04T16:04:27.041 INFO:tasks.workunit.client.1.smithi152.stdout:8/37: sync 2024-03-04T16:04:27.041 INFO:tasks.workunit.client.1.smithi152.stdout:5/105: sync 2024-03-04T16:04:27.042 INFO:tasks.workunit.client.1.smithi152.stdout:9/8: sync 2024-03-04T16:04:27.042 INFO:tasks.workunit.client.1.smithi152.stdout:9/9: dwrite - no filename 2024-03-04T16:04:27.042 INFO:tasks.workunit.client.1.smithi152.stdout:9/10: dwrite - no filename 2024-03-04T16:04:27.042 INFO:tasks.workunit.client.1.smithi152.stdout:7/1: sync 2024-03-04T16:04:27.042 INFO:tasks.workunit.client.1.smithi152.stdout:7/2: write - no filename 2024-03-04T16:04:27.042 INFO:tasks.workunit.client.1.smithi152.stdout:7/3: unlink - no file 2024-03-04T16:04:27.042 INFO:tasks.workunit.client.1.smithi152.stdout:3/13: sync 2024-03-04T16:04:27.042 INFO:tasks.workunit.client.1.smithi152.stdout:2/4: sync 2024-03-04T16:04:27.043 INFO:tasks.workunit.client.1.smithi152.stdout:2/5: stat f1 0 2024-03-04T16:04:27.043 INFO:tasks.workunit.client.1.smithi152.stdout:9/11: chown c0 935 1 2024-03-04T16:04:27.044 INFO:tasks.workunit.client.1.smithi152.stdout:1/37: creat f10 x:0 0 0 2024-03-04T16:04:27.044 INFO:tasks.workunit.client.0.smithi116.stdout:1/959: dread d5/d9/fb [4194304,4194304] 0 2024-03-04T16:04:27.045 INFO:tasks.workunit.client.1.smithi152.stdout:3/14: creat f5 x:0 0 0 2024-03-04T16:04:27.047 INFO:tasks.workunit.client.1.smithi152.stdout:0/9: dread f1 [0,4194304] 0 2024-03-04T16:04:27.048 INFO:tasks.workunit.client.1.smithi152.stdout:8/38: dread d2/d3/f6 [4194304,4194304] 0 2024-03-04T16:04:27.048 INFO:tasks.workunit.client.1.smithi152.stdout:0/10: stat c3 0 2024-03-04T16:04:27.049 INFO:tasks.workunit.client.1.smithi152.stdout:5/106: write d6/d8/ff [1262549,106460] 0 2024-03-04T16:04:27.049 INFO:tasks.workunit.client.1.smithi152.stdout:9/12: mkdir d2 0 2024-03-04T16:04:27.050 INFO:tasks.workunit.client.1.smithi152.stdout:9/13: fsync - no filename 2024-03-04T16:04:27.050 INFO:tasks.workunit.client.1.smithi152.stdout:5/107: stat d6/d8/f1d 0 2024-03-04T16:04:27.051 INFO:tasks.workunit.client.1.smithi152.stdout:2/6: creat f2 x:0 0 0 2024-03-04T16:04:27.053 INFO:tasks.workunit.client.1.smithi152.stdout:2/7: write f0 [258964,5455] 0 2024-03-04T16:04:27.054 INFO:tasks.workunit.client.1.smithi152.stdout:6/65: dwrite f1 [0,4194304] 0 2024-03-04T16:04:27.057 INFO:tasks.workunit.client.1.smithi152.stdout:0/11: write f1 [2022705,130686] 0 2024-03-04T16:04:27.059 INFO:tasks.workunit.client.1.smithi152.stdout:7/4: mknod c0 0 2024-03-04T16:04:27.061 INFO:tasks.workunit.client.1.smithi152.stdout:0/12: truncate f1 2458647 0 2024-03-04T16:04:27.064 INFO:tasks.workunit.client.1.smithi152.stdout:6/66: dwrite f0 [4194304,4194304] 0 2024-03-04T16:04:27.065 INFO:tasks.workunit.client.1.smithi152.stdout:5/108: creat d6/d8/d10/f27 x:0 0 0 2024-03-04T16:04:27.066 INFO:tasks.workunit.client.1.smithi152.stdout:6/67: write f6 [2912961,22271] 0 2024-03-04T16:04:27.066 INFO:tasks.workunit.client.1.smithi152.stdout:1/38: write f5 [1116778,130041] 0 2024-03-04T16:04:27.066 INFO:tasks.workunit.client.0.smithi116.stdout:1/960: dwrite d5/d9/d21/d24/f44 [0,4194304] 0 2024-03-04T16:04:27.067 INFO:tasks.workunit.client.1.smithi152.stdout:3/15: write f0 [192689,40324] 0 2024-03-04T16:04:27.068 INFO:tasks.workunit.client.0.smithi116.stdout:1/961: fsync d5/d9/d21/d27/d32/d35/f76 0 2024-03-04T16:04:27.070 INFO:tasks.workunit.client.0.smithi116.stdout:1/962: write d5/d9/d21/d27/d94/db6/d42/f88 [4786356,12725] 0 2024-03-04T16:04:27.070 INFO:tasks.workunit.client.1.smithi152.stdout:3/16: truncate f5 369296 0 2024-03-04T16:04:27.072 INFO:tasks.workunit.client.1.smithi152.stdout:1/39: truncate fe 301175 0 2024-03-04T16:04:27.073 INFO:tasks.workunit.client.1.smithi152.stdout:4/32: truncate f5 143699 0 2024-03-04T16:04:27.074 INFO:tasks.workunit.client.1.smithi152.stdout:8/39: creat d2/d3/d9/fc x:0 0 0 2024-03-04T16:04:27.074 INFO:tasks.workunit.client.1.smithi152.stdout:4/33: write f3 [3361200,71850] 0 2024-03-04T16:04:27.076 INFO:tasks.workunit.client.1.smithi152.stdout:7/5: mknod c1 0 2024-03-04T16:04:27.077 INFO:tasks.workunit.client.0.smithi116.stdout:1/963: mkdir d5/d86/dac/d13a 0 2024-03-04T16:04:27.078 INFO:tasks.workunit.client.0.smithi116.stdout:1/964: chown d5/d9/d21/d27/d32/d35/d55/db4/cdb 178544758 1 2024-03-04T16:04:27.080 INFO:tasks.workunit.client.1.smithi152.stdout:9/14: creat d2/f3 x:0 0 0 2024-03-04T16:04:27.081 INFO:tasks.workunit.client.1.smithi152.stdout:0/13: rename f1 to f4 0 2024-03-04T16:04:27.081 INFO:tasks.workunit.client.1.smithi152.stdout:9/15: chown c0 94924 1 2024-03-04T16:04:27.083 INFO:tasks.workunit.client.1.smithi152.stdout:2/8: link f1 f3 0 2024-03-04T16:04:27.085 INFO:tasks.workunit.client.1.smithi152.stdout:5/109: rename d6/d8/d10/c1c to d6/c28 0 2024-03-04T16:04:27.085 INFO:tasks.workunit.client.1.smithi152.stdout:6/68: dwrite d8/d10/f15 [0,4194304] 0 2024-03-04T16:04:27.086 INFO:tasks.workunit.client.1.smithi152.stdout:9/16: rename d2 to d2/d4 22 2024-03-04T16:04:27.086 INFO:tasks.workunit.client.1.smithi152.stdout:2/9: write f0 [374980,101892] 0 2024-03-04T16:04:27.088 INFO:tasks.workunit.client.1.smithi152.stdout:3/17: mkdir d6 0 2024-03-04T16:04:27.089 INFO:tasks.workunit.client.1.smithi152.stdout:3/18: chown c4 14 1 2024-03-04T16:04:27.093 INFO:tasks.workunit.client.1.smithi152.stdout:9/17: dread - d2/f3 zero size 2024-03-04T16:04:27.093 INFO:tasks.workunit.client.1.smithi152.stdout:1/40: symlink l11 0 2024-03-04T16:04:27.093 INFO:tasks.workunit.client.1.smithi152.stdout:8/40: dwrite d2/d3/d9/fc [0,4194304] 0 2024-03-04T16:04:27.095 INFO:tasks.workunit.client.1.smithi152.stdout:4/34: symlink d8/la 0 2024-03-04T16:04:27.096 INFO:tasks.workunit.client.1.smithi152.stdout:9/18: fdatasync d2/f3 0 2024-03-04T16:04:27.097 INFO:tasks.workunit.client.1.smithi152.stdout:1/41: chown f1 45135 1 2024-03-04T16:04:27.099 INFO:tasks.workunit.client.1.smithi152.stdout:0/14: creat f5 x:0 0 0 2024-03-04T16:04:27.099 INFO:tasks.workunit.client.1.smithi152.stdout:1/42: read - fa zero size 2024-03-04T16:04:27.100 INFO:tasks.workunit.client.1.smithi152.stdout:5/110: mknod d6/d8/c29 0 2024-03-04T16:04:27.101 INFO:tasks.workunit.client.1.smithi152.stdout:9/19: chown c0 113803 1 2024-03-04T16:04:27.102 INFO:tasks.workunit.client.1.smithi152.stdout:1/43: chown lc 3690425 1 2024-03-04T16:04:27.103 INFO:tasks.workunit.client.1.smithi152.stdout:6/69: write d8/fe [3235092,60130] 0 2024-03-04T16:04:27.106 INFO:tasks.workunit.client.1.smithi152.stdout:3/19: dwrite f5 [0,4194304] 0 2024-03-04T16:04:27.108 INFO:tasks.workunit.client.1.smithi152.stdout:5/111: mknod d6/c2a 0 2024-03-04T16:04:27.109 INFO:tasks.workunit.client.1.smithi152.stdout:5/112: fsync d6/d8/f24 0 2024-03-04T16:04:27.110 INFO:tasks.workunit.client.1.smithi152.stdout:5/113: write d6/d8/f24 [942760,126330] 0 2024-03-04T16:04:27.110 INFO:tasks.workunit.client.1.smithi152.stdout:2/10: link f2 f4 0 2024-03-04T16:04:27.111 INFO:tasks.workunit.client.1.smithi152.stdout:2/11: readlink - no filename 2024-03-04T16:04:27.111 INFO:tasks.workunit.client.1.smithi152.stdout:2/12: chown f0 587565 1 2024-03-04T16:04:27.111 INFO:tasks.workunit.client.1.smithi152.stdout:5/114: truncate d6/d8/d10/f26 1033213 0 2024-03-04T16:04:27.112 INFO:tasks.workunit.client.1.smithi152.stdout:1/44: symlink l12 0 2024-03-04T16:04:27.113 INFO:tasks.workunit.client.1.smithi152.stdout:5/115: write d6/d8/d10/f14 [948492,13704] 0 2024-03-04T16:04:27.114 INFO:tasks.workunit.client.1.smithi152.stdout:9/20: rename c1 to d2/c5 0 2024-03-04T16:04:27.115 INFO:tasks.workunit.client.1.smithi152.stdout:5/116: write d6/d8/d1f/f23 [1033407,70532] 0 2024-03-04T16:04:27.116 INFO:tasks.workunit.client.1.smithi152.stdout:9/21: dread - d2/f3 zero size 2024-03-04T16:04:27.117 INFO:tasks.workunit.client.1.smithi152.stdout:8/41: mkdir d2/d8/dd 0 2024-03-04T16:04:27.119 INFO:tasks.workunit.client.1.smithi152.stdout:4/35: mknod d8/d9/cb 0 2024-03-04T16:04:27.120 INFO:tasks.workunit.client.1.smithi152.stdout:6/70: creat d8/d10/f16 x:0 0 0 2024-03-04T16:04:27.122 INFO:tasks.workunit.client.1.smithi152.stdout:2/13: creat f5 x:0 0 0 2024-03-04T16:04:27.125 INFO:tasks.workunit.client.1.smithi152.stdout:1/45: dwrite fe [0,4194304] 0 2024-03-04T16:04:27.126 INFO:tasks.workunit.client.1.smithi152.stdout:1/46: stat c4 0 2024-03-04T16:04:27.128 INFO:tasks.workunit.client.1.smithi152.stdout:5/117: rmdir d6/d8 39 2024-03-04T16:04:27.129 INFO:tasks.workunit.client.1.smithi152.stdout:5/118: fdatasync d6/fd 0 2024-03-04T16:04:27.131 INFO:tasks.workunit.client.1.smithi152.stdout:9/22: dwrite d2/f3 [0,4194304] 0 2024-03-04T16:04:27.131 INFO:tasks.workunit.client.1.smithi152.stdout:3/20: link f0 d6/f7 0 2024-03-04T16:04:27.131 INFO:tasks.workunit.client.1.smithi152.stdout:3/21: stat f5 0 2024-03-04T16:04:27.133 INFO:tasks.workunit.client.1.smithi152.stdout:4/36: truncate f0 1271957 0 2024-03-04T16:04:27.135 INFO:tasks.workunit.client.1.smithi152.stdout:0/15: truncate f4 879942 0 2024-03-04T16:04:27.137 INFO:tasks.workunit.client.1.smithi152.stdout:2/14: rename f5 to f6 0 2024-03-04T16:04:27.140 INFO:tasks.workunit.client.1.smithi152.stdout:1/47: rename l3 to l13 0 2024-03-04T16:04:27.141 INFO:tasks.workunit.client.1.smithi152.stdout:1/48: write f2 [1621450,27823] 0 2024-03-04T16:04:27.141 INFO:tasks.workunit.client.1.smithi152.stdout:1/49: chown cd 6507643 1 2024-03-04T16:04:27.175 INFO:tasks.workunit.client.0.smithi116.stdout:1/965: sync 2024-03-04T16:04:27.182 INFO:tasks.workunit.client.1.smithi152.stdout:9/23: symlink d2/l6 0 2024-03-04T16:04:27.183 INFO:tasks.workunit.client.1.smithi152.stdout:9/24: fdatasync d2/f3 0 2024-03-04T16:04:27.222 INFO:tasks.workunit.client.1.smithi152.stdout:7/6: sync 2024-03-04T16:04:27.223 INFO:tasks.workunit.client.1.smithi152.stdout:7/7: dwrite - no filename 2024-03-04T16:04:27.223 INFO:tasks.workunit.client.1.smithi152.stdout:7/8: fdatasync - no filename 2024-03-04T16:04:27.253 INFO:tasks.workunit.client.1.smithi152.stdout:4/37: unlink f4 0 2024-03-04T16:04:27.258 INFO:tasks.workunit.client.1.smithi152.stdout:2/15: creat f7 x:0 0 0 2024-03-04T16:04:27.261 INFO:tasks.workunit.client.1.smithi152.stdout:6/71: dwrite d8/d10/f15 [4194304,4194304] 0 2024-03-04T16:04:27.261 INFO:tasks.workunit.client.1.smithi152.stdout:1/50: fdatasync f8 0 2024-03-04T16:04:27.261 INFO:tasks.workunit.client.1.smithi152.stdout:1/51: stat f10 0 2024-03-04T16:04:27.262 INFO:tasks.workunit.client.1.smithi152.stdout:5/119: mkdir d6/d8/d10/d2b 0 2024-03-04T16:04:27.267 INFO:tasks.workunit.client.1.smithi152.stdout:9/25: unlink d2/l6 0 2024-03-04T16:04:27.269 INFO:tasks.workunit.client.1.smithi152.stdout:7/9: unlink c0 0 2024-03-04T16:04:27.269 INFO:tasks.workunit.client.1.smithi152.stdout:7/10: fsync - no filename 2024-03-04T16:04:27.271 INFO:tasks.workunit.client.1.smithi152.stdout:9/26: dread d2/f3 [0,4194304] 0 2024-03-04T16:04:27.272 INFO:tasks.workunit.client.1.smithi152.stdout:9/27: write d2/f3 [274877,28617] 0 2024-03-04T16:04:27.277 INFO:tasks.workunit.client.1.smithi152.stdout:4/38: mknod d8/cc 0 2024-03-04T16:04:27.279 INFO:tasks.workunit.client.1.smithi152.stdout:2/16: mknod c8 0 2024-03-04T16:04:27.285 INFO:tasks.workunit.client.1.smithi152.stdout:1/52: mknod c14 0 2024-03-04T16:04:27.286 INFO:tasks.workunit.client.1.smithi152.stdout:1/53: dread - f8 zero size 2024-03-04T16:04:27.289 INFO:tasks.workunit.client.1.smithi152.stdout:5/120: rename d6/f18 to d6/d8/f2c 0 2024-03-04T16:04:27.291 INFO:tasks.workunit.client.1.smithi152.stdout:7/11: unlink c1 0 2024-03-04T16:04:27.291 INFO:tasks.workunit.client.1.smithi152.stdout:7/12: dread - no filename 2024-03-04T16:04:27.293 INFO:tasks.workunit.client.1.smithi152.stdout:8/42: getdents d2/d8 0 2024-03-04T16:04:27.293 INFO:tasks.workunit.client.1.smithi152.stdout:8/43: readlink l0 0 2024-03-04T16:04:27.293 INFO:tasks.workunit.client.1.smithi152.stdout:8/44: chown d2/l4 674219 1 2024-03-04T16:04:27.294 INFO:tasks.workunit.client.1.smithi152.stdout:8/45: fdatasync d2/d3/d9/fc 0 2024-03-04T16:04:27.299 INFO:tasks.workunit.client.1.smithi152.stdout:2/17: symlink l9 0 2024-03-04T16:04:27.300 INFO:tasks.workunit.client.1.smithi152.stdout:1/54: creat f15 x:0 0 0 2024-03-04T16:04:27.306 INFO:tasks.workunit.client.1.smithi152.stdout:5/121: creat d6/d8/d10/f2d x:0 0 0 2024-03-04T16:04:27.312 INFO:tasks.workunit.client.1.smithi152.stdout:8/46: fdatasync d2/d3/f6 0 2024-03-04T16:04:27.313 INFO:tasks.workunit.client.1.smithi152.stdout:8/47: write d2/d3/f6 [2929460,63133] 0 2024-03-04T16:04:27.314 INFO:tasks.workunit.client.1.smithi152.stdout:8/48: chown d2/d8/dd 10 1 2024-03-04T16:04:27.315 INFO:tasks.workunit.client.1.smithi152.stdout:8/49: chown d2/d8/dd 1979 1 2024-03-04T16:04:27.316 INFO:tasks.workunit.client.1.smithi152.stdout:1/55: fdatasync f8 0 2024-03-04T16:04:27.319 INFO:tasks.workunit.client.1.smithi152.stdout:5/122: creat d6/d8/d1f/f2e x:0 0 0 2024-03-04T16:04:27.320 INFO:tasks.workunit.client.1.smithi152.stdout:8/50: dread d2/d3/d9/fc [0,4194304] 0 2024-03-04T16:04:27.320 INFO:tasks.workunit.client.1.smithi152.stdout:4/39: link d8/cc d8/cd 0 2024-03-04T16:04:27.324 INFO:tasks.workunit.client.1.smithi152.stdout:1/56: creat f16 x:0 0 0 2024-03-04T16:04:27.329 INFO:tasks.workunit.client.1.smithi152.stdout:8/51: creat d2/d3/fe x:0 0 0 2024-03-04T16:04:27.330 INFO:tasks.workunit.client.1.smithi152.stdout:1/57: creat f17 x:0 0 0 2024-03-04T16:04:27.337 INFO:tasks.workunit.client.0.smithi116.stdout:1/966: mkdir d5/d9/d21/d27/d32/d35/d55/db4/d13b 0 2024-03-04T16:04:27.338 INFO:tasks.workunit.client.1.smithi152.stdout:1/58: link f5 f18 0 2024-03-04T16:04:27.341 INFO:tasks.workunit.client.1.smithi152.stdout:8/52: getdents d2/d8 0 2024-03-04T16:04:27.342 INFO:tasks.workunit.client.1.smithi152.stdout:8/53: dread - d2/d3/fe zero size 2024-03-04T16:04:27.343 INFO:tasks.workunit.client.1.smithi152.stdout:8/54: fsync d2/d3/f6 0 2024-03-04T16:04:27.343 INFO:tasks.workunit.client.1.smithi152.stdout:1/59: unlink f15 0 2024-03-04T16:04:27.347 INFO:tasks.workunit.client.1.smithi152.stdout:8/55: mknod d2/d3/d9/cf 0 2024-03-04T16:04:27.349 INFO:tasks.workunit.client.1.smithi152.stdout:8/56: symlink d2/d3/d9/l10 0 2024-03-04T16:04:27.349 INFO:tasks.workunit.client.1.smithi152.stdout:8/57: stat d2/d3/f6 0 2024-03-04T16:04:27.352 INFO:tasks.workunit.client.1.smithi152.stdout:8/58: unlink d2/d3/fe 0 2024-03-04T16:04:27.355 INFO:tasks.workunit.client.1.smithi152.stdout:8/59: rename d2/l4 to d2/d8/dd/l11 0 2024-03-04T16:04:27.355 INFO:tasks.workunit.client.1.smithi152.stdout:8/60: fdatasync d2/d3/f6 0 2024-03-04T16:04:27.356 INFO:tasks.workunit.client.1.smithi152.stdout:8/61: fsync d2/d3/f6 0 2024-03-04T16:04:27.359 INFO:tasks.workunit.client.1.smithi152.stdout:8/62: rename l1 to d2/d8/l12 0 2024-03-04T16:04:27.362 INFO:tasks.workunit.client.1.smithi152.stdout:8/63: rename d2/d3/f6 to d2/d3/f13 0 2024-03-04T16:04:27.365 INFO:tasks.workunit.client.1.smithi152.stdout:8/64: creat d2/d8/f14 x:0 0 0 2024-03-04T16:04:27.365 INFO:tasks.workunit.client.1.smithi152.stdout:8/65: fdatasync d2/d8/f14 0 2024-03-04T16:04:27.366 INFO:tasks.workunit.client.1.smithi152.stdout:8/66: fdatasync d2/d8/f14 0 2024-03-04T16:04:27.370 INFO:tasks.workunit.client.1.smithi152.stdout:8/67: mkdir d2/d3/d15 0 2024-03-04T16:04:27.371 INFO:tasks.workunit.client.1.smithi152.stdout:8/68: symlink d2/d3/l16 0 2024-03-04T16:04:27.373 INFO:tasks.workunit.client.1.smithi152.stdout:8/69: mkdir d2/d3/d17 0 2024-03-04T16:04:27.394 INFO:tasks.workunit.client.1.smithi152.stdout:3/22: write f0 [1492568,106703] 0 2024-03-04T16:04:27.398 INFO:tasks.workunit.client.1.smithi152.stdout:3/23: creat d6/f8 x:0 0 0 2024-03-04T16:04:27.400 INFO:tasks.workunit.client.1.smithi152.stdout:3/24: write d6/f8 [1030210,77505] 0 2024-03-04T16:04:27.404 INFO:tasks.workunit.client.1.smithi152.stdout:0/16: dwrite f4 [0,4194304] 0 2024-03-04T16:04:27.405 INFO:tasks.workunit.client.1.smithi152.stdout:0/17: dread - f5 zero size 2024-03-04T16:04:27.410 INFO:tasks.workunit.client.1.smithi152.stdout:3/25: dwrite f5 [0,4194304] 0 2024-03-04T16:04:27.412 INFO:tasks.workunit.client.1.smithi152.stdout:3/26: write f0 [1738774,76526] 0 2024-03-04T16:04:27.416 INFO:tasks.workunit.client.1.smithi152.stdout:3/27: link d6/f7 d6/f9 0 2024-03-04T16:04:27.421 INFO:tasks.workunit.client.1.smithi152.stdout:7/13: getdents . 0 2024-03-04T16:04:27.421 INFO:tasks.workunit.client.1.smithi152.stdout:7/14: write - no filename 2024-03-04T16:04:27.422 INFO:tasks.workunit.client.1.smithi152.stdout:7/15: creat f2 x:0 0 0 2024-03-04T16:04:27.424 INFO:tasks.workunit.client.1.smithi152.stdout:7/16: symlink l3 0 2024-03-04T16:04:27.427 INFO:tasks.workunit.client.1.smithi152.stdout:7/17: rename f2 to f4 0 2024-03-04T16:04:27.427 INFO:tasks.workunit.client.1.smithi152.stdout:7/18: readlink l3 0 2024-03-04T16:04:27.430 INFO:tasks.workunit.client.1.smithi152.stdout:6/72: truncate d8/d10/f15 6713389 0 2024-03-04T16:04:27.432 INFO:tasks.workunit.client.1.smithi152.stdout:6/73: mknod d8/c17 0 2024-03-04T16:04:27.436 INFO:tasks.workunit.client.1.smithi152.stdout:7/19: dwrite f4 [0,4194304] 0 2024-03-04T16:04:27.436 INFO:tasks.workunit.client.1.smithi152.stdout:7/20: rmdir - no directory 2024-03-04T16:04:27.439 INFO:tasks.workunit.client.1.smithi152.stdout:7/21: mknod c5 0 2024-03-04T16:04:27.440 INFO:tasks.workunit.client.1.smithi152.stdout:2/18: write f2 [138764,46451] 0 2024-03-04T16:04:27.441 INFO:tasks.workunit.client.1.smithi152.stdout:2/19: fdatasync f0 0 2024-03-04T16:04:27.441 INFO:tasks.workunit.client.1.smithi152.stdout:5/123: getdents d6/d8/d1f 0 2024-03-04T16:04:27.442 INFO:tasks.workunit.client.1.smithi152.stdout:6/74: dwrite f7 [0,4194304] 0 2024-03-04T16:04:27.444 INFO:tasks.workunit.client.1.smithi152.stdout:6/75: write f1 [679801,77039] 0 2024-03-04T16:04:27.447 INFO:tasks.workunit.client.1.smithi152.stdout:5/124: rmdir d6/d8/d10 39 2024-03-04T16:04:27.449 INFO:tasks.workunit.client.1.smithi152.stdout:6/76: readlink l5 0 2024-03-04T16:04:27.450 INFO:tasks.workunit.client.1.smithi152.stdout:6/77: write f7 [3963051,89635] 0 2024-03-04T16:04:27.451 INFO:tasks.workunit.client.1.smithi152.stdout:1/60: fsync f17 0 2024-03-04T16:04:27.451 INFO:tasks.workunit.client.1.smithi152.stdout:1/61: rmdir - no directory 2024-03-04T16:04:27.456 INFO:tasks.workunit.client.1.smithi152.stdout:6/78: creat d8/d10/f18 x:0 0 0 2024-03-04T16:04:27.457 INFO:tasks.workunit.client.1.smithi152.stdout:4/40: dwrite f7 [0,4194304] 0 2024-03-04T16:04:27.458 INFO:tasks.workunit.client.1.smithi152.stdout:6/79: write d8/fe [1777276,43046] 0 2024-03-04T16:04:27.460 INFO:tasks.workunit.client.1.smithi152.stdout:6/80: write d8/d10/f18 [762998,112315] 0 2024-03-04T16:04:27.461 INFO:tasks.workunit.client.1.smithi152.stdout:6/81: write d8/f9 [34716,88963] 0 2024-03-04T16:04:27.462 INFO:tasks.workunit.client.1.smithi152.stdout:6/82: write d8/d10/f16 [685740,95416] 0 2024-03-04T16:04:27.463 INFO:tasks.workunit.client.1.smithi152.stdout:5/125: rmdir d6/d8/d10/d2b 0 2024-03-04T16:04:27.466 INFO:tasks.workunit.client.0.smithi116.stdout:1/967: write d5/d9/fce [536654,119899] 0 2024-03-04T16:04:27.468 INFO:tasks.workunit.client.1.smithi152.stdout:4/41: rename c6 to d8/ce 0 2024-03-04T16:04:27.469 INFO:tasks.workunit.client.1.smithi152.stdout:6/83: fdatasync d8/d10/f16 0 2024-03-04T16:04:27.471 INFO:tasks.workunit.client.0.smithi116.stdout:1/968: symlink d5/d9/d21/d27/d94/l13c 0 2024-03-04T16:04:27.473 INFO:tasks.workunit.client.1.smithi152.stdout:6/84: dread f7 [0,4194304] 0 2024-03-04T16:04:27.475 INFO:tasks.workunit.client.0.smithi116.stdout:1/969: mknod d5/d86/d129/c13d 0 2024-03-04T16:04:27.476 INFO:tasks.workunit.client.1.smithi152.stdout:6/85: dread d8/d10/f16 [0,4194304] 0 2024-03-04T16:04:27.481 INFO:tasks.workunit.client.0.smithi116.stdout:1/970: symlink d5/d9/d21/d24/de0/d4a/l13e 0 2024-03-04T16:04:27.482 INFO:tasks.workunit.client.1.smithi152.stdout:6/86: creat d8/f19 x:0 0 0 2024-03-04T16:04:27.483 INFO:tasks.workunit.client.1.smithi152.stdout:5/126: creat d6/d8/f2f x:0 0 0 2024-03-04T16:04:27.484 INFO:tasks.workunit.client.0.smithi116.stdout:1/971: mknod d5/d9/d21/d84/db0/de9/dfb/c13f 0 2024-03-04T16:04:27.486 INFO:tasks.workunit.client.1.smithi152.stdout:6/87: fdatasync f1 0 2024-03-04T16:04:27.487 INFO:tasks.workunit.client.1.smithi152.stdout:5/127: dread d6/d8/d1f/f23 [0,4194304] 0 2024-03-04T16:04:27.488 INFO:tasks.workunit.client.1.smithi152.stdout:8/70: fsync d2/d3/f13 0 2024-03-04T16:04:27.490 INFO:tasks.workunit.client.1.smithi152.stdout:5/128: truncate d6/d8/d10/f2d 707866 0 2024-03-04T16:04:27.494 INFO:tasks.workunit.client.1.smithi152.stdout:8/71: unlink d2/d3/d9/l10 0 2024-03-04T16:04:27.494 INFO:tasks.workunit.client.1.smithi152.stdout:6/88: dwrite d8/f12 [0,4194304] 0 2024-03-04T16:04:27.497 INFO:tasks.workunit.client.0.smithi116.stdout:1/972: rename d5/d9/d21/d27/d32/d35/d55/db4/d50/c8f to d5/d9/d21/c140 0 2024-03-04T16:04:27.498 INFO:tasks.workunit.client.0.smithi116.stdout:1/973: write d5/d9/d21/d27/d32/d35/d55/dae/fd0 [415576,86145] 0 2024-03-04T16:04:27.499 INFO:tasks.workunit.client.1.smithi152.stdout:8/72: creat d2/d3/d9/f18 x:0 0 0 2024-03-04T16:04:27.500 INFO:tasks.workunit.client.1.smithi152.stdout:8/73: write d2/d3/d9/fc [1250865,63753] 0 2024-03-04T16:04:27.502 INFO:tasks.workunit.client.1.smithi152.stdout:8/74: read d2/d3/d9/fc [420708,4669] 0 2024-03-04T16:04:27.503 INFO:tasks.workunit.client.1.smithi152.stdout:6/89: mknod d8/d10/c1a 0 2024-03-04T16:04:27.503 INFO:tasks.workunit.client.1.smithi152.stdout:6/90: read - d8/f14 zero size 2024-03-04T16:04:27.512 INFO:tasks.workunit.client.1.smithi152.stdout:5/129: creat d6/d8/f30 x:0 0 0 2024-03-04T16:04:27.513 INFO:tasks.workunit.client.1.smithi152.stdout:5/130: chown d6/d8/cb 0 1 2024-03-04T16:04:27.513 INFO:tasks.workunit.client.1.smithi152.stdout:5/131: fsync d6/d8/d10/f27 0 2024-03-04T16:04:27.516 INFO:tasks.workunit.client.1.smithi152.stdout:8/75: creat d2/d3/d17/f19 x:0 0 0 2024-03-04T16:04:27.517 INFO:tasks.workunit.client.1.smithi152.stdout:6/91: link d8/f19 d8/f1b 0 2024-03-04T16:04:27.519 INFO:tasks.workunit.client.1.smithi152.stdout:5/132: mknod d6/d8/d10/c31 0 2024-03-04T16:04:27.521 INFO:tasks.workunit.client.1.smithi152.stdout:0/18: truncate f4 3012219 0 2024-03-04T16:04:27.524 INFO:tasks.workunit.client.1.smithi152.stdout:6/92: unlink f0 0 2024-03-04T16:04:27.524 INFO:tasks.workunit.client.1.smithi152.stdout:8/76: dread d2/d3/d9/fc [0,4194304] 0 2024-03-04T16:04:27.525 INFO:tasks.workunit.client.1.smithi152.stdout:8/77: dread - d2/d3/d9/f18 zero size 2024-03-04T16:04:27.526 INFO:tasks.workunit.client.1.smithi152.stdout:3/28: rmdir d6 39 2024-03-04T16:04:27.529 INFO:tasks.workunit.client.1.smithi152.stdout:0/19: rename f5 to f6 0 2024-03-04T16:04:27.532 INFO:tasks.workunit.client.1.smithi152.stdout:8/78: dwrite d2/d8/f14 [0,4194304] 0 2024-03-04T16:04:27.536 INFO:tasks.workunit.client.1.smithi152.stdout:6/93: creat d8/d10/f1c x:0 0 0 2024-03-04T16:04:27.541 INFO:tasks.workunit.client.1.smithi152.stdout:3/29: dread d6/f8 [0,4194304] 0 2024-03-04T16:04:27.542 INFO:tasks.workunit.client.1.smithi152.stdout:3/30: dread - f3 zero size 2024-03-04T16:04:27.543 INFO:tasks.workunit.client.1.smithi152.stdout:2/20: write f1 [4615432,31315] 0 2024-03-04T16:04:27.543 INFO:tasks.workunit.client.1.smithi152.stdout:2/21: readlink l9 0 2024-03-04T16:04:27.548 INFO:tasks.workunit.client.1.smithi152.stdout:8/79: creat d2/d8/f1a x:0 0 0 2024-03-04T16:04:27.550 INFO:tasks.workunit.client.1.smithi152.stdout:6/94: mknod d8/c1d 0 2024-03-04T16:04:27.554 INFO:tasks.workunit.client.1.smithi152.stdout:1/62: truncate f5 907016 0 2024-03-04T16:04:27.555 INFO:tasks.workunit.client.1.smithi152.stdout:5/133: creat d6/d8/f32 x:0 0 0 2024-03-04T16:04:27.556 INFO:tasks.workunit.client.1.smithi152.stdout:3/31: mknod d6/ca 0 2024-03-04T16:04:27.558 INFO:tasks.workunit.client.1.smithi152.stdout:2/22: mkdir da 0 2024-03-04T16:04:27.560 INFO:tasks.workunit.client.1.smithi152.stdout:8/80: creat d2/d3/d17/f1b x:0 0 0 2024-03-04T16:04:27.564 INFO:tasks.workunit.client.1.smithi152.stdout:4/42: dwrite f0 [0,4194304] 0 2024-03-04T16:04:27.578 INFO:tasks.workunit.client.1.smithi152.stdout:6/95: rename d8/f1b to d8/d10/f1e 0 2024-03-04T16:04:27.581 INFO:tasks.workunit.client.1.smithi152.stdout:4/43: dread f5 [0,4194304] 0 2024-03-04T16:04:27.584 INFO:tasks.workunit.client.1.smithi152.stdout:5/134: symlink d6/d8/l33 0 2024-03-04T16:04:27.585 INFO:tasks.workunit.client.1.smithi152.stdout:5/135: write d6/d8/f1d [1496260,89271] 0 2024-03-04T16:04:27.586 INFO:tasks.workunit.client.1.smithi152.stdout:2/23: creat da/fb x:0 0 0 2024-03-04T16:04:27.588 INFO:tasks.workunit.client.1.smithi152.stdout:8/81: mkdir d2/d3/d15/d1c 0 2024-03-04T16:04:27.589 INFO:tasks.workunit.client.1.smithi152.stdout:8/82: dread - d2/d3/d17/f1b zero size 2024-03-04T16:04:27.594 INFO:tasks.workunit.client.0.smithi116.stdout:1/974: sync 2024-03-04T16:04:27.594 INFO:tasks.workunit.client.1.smithi152.stdout:4/44: creat d8/d9/ff x:0 0 0 2024-03-04T16:04:27.597 INFO:tasks.workunit.client.1.smithi152.stdout:8/83: dwrite d2/d3/d9/f18 [0,4194304] 0 2024-03-04T16:04:27.599 INFO:tasks.workunit.client.1.smithi152.stdout:2/24: creat da/fc x:0 0 0 2024-03-04T16:04:27.600 INFO:tasks.workunit.client.1.smithi152.stdout:2/25: truncate f0 1095857 0 2024-03-04T16:04:27.605 INFO:tasks.workunit.client.1.smithi152.stdout:6/96: rename d8/f19 to d8/f1f 0 2024-03-04T16:04:27.607 INFO:tasks.workunit.client.1.smithi152.stdout:2/26: dread f4 [0,4194304] 0 2024-03-04T16:04:27.608 INFO:tasks.workunit.client.1.smithi152.stdout:8/84: dwrite d2/d3/f13 [4194304,4194304] 0 2024-03-04T16:04:27.622 INFO:tasks.workunit.client.1.smithi152.stdout:6/97: mknod d8/c20 0 2024-03-04T16:04:27.625 INFO:tasks.workunit.client.1.smithi152.stdout:2/27: mkdir da/dd 0 2024-03-04T16:04:27.632 INFO:tasks.workunit.client.1.smithi152.stdout:8/85: creat d2/d3/d15/d1c/f1d x:0 0 0 2024-03-04T16:04:27.633 INFO:tasks.workunit.client.1.smithi152.stdout:8/86: dread - d2/d3/d17/f1b zero size 2024-03-04T16:04:27.641 INFO:tasks.workunit.client.1.smithi152.stdout:4/45: dwrite f0 [4194304,4194304] 0 2024-03-04T16:04:27.646 INFO:tasks.workunit.client.1.smithi152.stdout:4/46: creat d8/f10 x:0 0 0 2024-03-04T16:04:27.646 INFO:tasks.workunit.client.1.smithi152.stdout:4/47: stat f7 0 2024-03-04T16:04:27.646 INFO:tasks.workunit.client.1.smithi152.stdout:4/48: rename d8 to d8/d9/d11 22 2024-03-04T16:04:27.650 INFO:tasks.workunit.client.1.smithi152.stdout:9/28: sync 2024-03-04T16:04:27.652 INFO:tasks.workunit.client.1.smithi152.stdout:8/87: fdatasync d2/d3/d9/f18 0 2024-03-04T16:04:27.652 INFO:tasks.workunit.client.1.smithi152.stdout:9/29: read d2/f3 [1481386,411] 0 2024-03-04T16:04:27.653 INFO:tasks.workunit.client.1.smithi152.stdout:4/49: dwrite f3 [0,4194304] 0 2024-03-04T16:04:27.655 INFO:tasks.workunit.client.1.smithi152.stdout:1/63: dread f18 [0,4194304] 0 2024-03-04T16:04:27.655 INFO:tasks.workunit.client.1.smithi152.stdout:8/88: write d2/d3/d17/f19 [357709,79372] 0 2024-03-04T16:04:27.662 INFO:tasks.workunit.client.1.smithi152.stdout:4/50: mknod d8/c12 0 2024-03-04T16:04:27.667 INFO:tasks.workunit.client.1.smithi152.stdout:1/64: write f2 [1762826,15944] 0 2024-03-04T16:04:27.671 INFO:tasks.workunit.client.1.smithi152.stdout:1/65: mkdir d19 0 2024-03-04T16:04:27.671 INFO:tasks.workunit.client.1.smithi152.stdout:1/66: dread - f17 zero size 2024-03-04T16:04:27.673 INFO:tasks.workunit.client.1.smithi152.stdout:6/98: fdatasync d8/f1f 0 2024-03-04T16:04:27.674 INFO:tasks.workunit.client.1.smithi152.stdout:6/99: write d8/f14 [393290,58824] 0 2024-03-04T16:04:27.675 INFO:tasks.workunit.client.1.smithi152.stdout:6/100: write d8/d10/f1c [483661,127947] 0 2024-03-04T16:04:27.679 INFO:tasks.workunit.client.1.smithi152.stdout:5/136: truncate d6/d8/d10/f12 2423352 0 2024-03-04T16:04:27.684 INFO:tasks.workunit.client.1.smithi152.stdout:6/101: creat d8/f21 x:0 0 0 2024-03-04T16:04:27.685 INFO:tasks.workunit.client.1.smithi152.stdout:2/28: getdents da 0 2024-03-04T16:04:27.686 INFO:tasks.workunit.client.1.smithi152.stdout:2/29: readlink l9 0 2024-03-04T16:04:27.687 INFO:tasks.workunit.client.1.smithi152.stdout:5/137: creat d6/d8/d1f/f34 x:0 0 0 2024-03-04T16:04:27.689 INFO:tasks.workunit.client.0.smithi116.stdout:1/975: rmdir d5/d9/d21/d27/d32/d35/d55/db4/d50/d130 0 2024-03-04T16:04:27.691 INFO:tasks.workunit.client.1.smithi152.stdout:2/30: rename f0 to da/fe 0 2024-03-04T16:04:27.692 INFO:tasks.workunit.client.1.smithi152.stdout:0/20: truncate f4 2621806 0 2024-03-04T16:04:27.699 INFO:tasks.workunit.client.1.smithi152.stdout:2/31: rename da/dd to da/df 0 2024-03-04T16:04:27.700 INFO:tasks.workunit.client.0.smithi116.stdout:1/976: rename d5/d9/d21/d24/de0/d4c/l8d to d5/l141 0 2024-03-04T16:04:27.701 INFO:tasks.workunit.client.1.smithi152.stdout:0/21: creat f7 x:0 0 0 2024-03-04T16:04:27.701 INFO:tasks.workunit.client.0.smithi116.stdout:1/977: chown d5/d9/d21/d27/d94/df2/l111 23 1 2024-03-04T16:04:27.702 INFO:tasks.workunit.client.1.smithi152.stdout:0/22: fdatasync f7 0 2024-03-04T16:04:27.702 INFO:tasks.workunit.client.1.smithi152.stdout:2/32: mknod da/c10 0 2024-03-04T16:04:27.703 INFO:tasks.workunit.client.0.smithi116.stdout:1/978: mkdir d5/d9/d21/d27/d8c/d142 0 2024-03-04T16:04:27.704 INFO:tasks.workunit.client.0.smithi116.stdout:1/979: chown d5/da1/fc3 358 1 2024-03-04T16:04:27.704 INFO:tasks.workunit.client.1.smithi152.stdout:0/23: mknod c8 0 2024-03-04T16:04:27.706 INFO:tasks.workunit.client.0.smithi116.stdout:1/980: fsync d5/d9/d21/d24/de0/d4a/f93 0 2024-03-04T16:04:27.709 INFO:tasks.workunit.client.1.smithi152.stdout:0/24: mkdir d9 0 2024-03-04T16:04:27.709 INFO:tasks.workunit.client.1.smithi152.stdout:0/25: read - f7 zero size 2024-03-04T16:04:27.711 INFO:tasks.workunit.client.0.smithi116.stdout:1/981: dwrite d5/d9/d21/d24/f37 [0,4194304] 0 2024-03-04T16:04:27.711 INFO:tasks.workunit.client.1.smithi152.stdout:2/33: link da/fe da/f11 0 2024-03-04T16:04:27.711 INFO:tasks.workunit.client.1.smithi152.stdout:2/34: truncate da/fc 158282 0 2024-03-04T16:04:27.713 INFO:tasks.workunit.client.1.smithi152.stdout:2/35: truncate da/fc 1145496 0 2024-03-04T16:04:27.714 INFO:tasks.workunit.client.1.smithi152.stdout:0/26: creat d9/fa x:0 0 0 2024-03-04T16:04:27.715 INFO:tasks.workunit.client.0.smithi116.stdout:1/982: mknod d5/d9/d21/d27/d32/d35/dbf/d10f/c143 0 2024-03-04T16:04:27.716 INFO:tasks.workunit.client.0.smithi116.stdout:1/983: chown d5/d9/d21/d84/db0/f103 165468324 1 2024-03-04T16:04:27.716 INFO:tasks.workunit.client.1.smithi152.stdout:2/36: fdatasync da/fe 0 2024-03-04T16:04:27.718 INFO:tasks.workunit.client.1.smithi152.stdout:0/27: creat d9/fb x:0 0 0 2024-03-04T16:04:27.718 INFO:tasks.workunit.client.1.smithi152.stdout:0/28: dread - f6 zero size 2024-03-04T16:04:27.719 INFO:tasks.workunit.client.1.smithi152.stdout:0/29: truncate f7 827763 0 2024-03-04T16:04:27.720 INFO:tasks.workunit.client.1.smithi152.stdout:0/30: stat d9/fa 0 2024-03-04T16:04:27.723 INFO:tasks.workunit.client.1.smithi152.stdout:4/51: getdents d8 0 2024-03-04T16:04:27.724 INFO:tasks.workunit.client.1.smithi152.stdout:0/31: symlink d9/lc 0 2024-03-04T16:04:27.725 INFO:tasks.workunit.client.1.smithi152.stdout:0/32: read f7 [550336,55487] 0 2024-03-04T16:04:27.725 INFO:tasks.workunit.client.1.smithi152.stdout:9/30: truncate d2/f3 4225046 0 2024-03-04T16:04:27.726 INFO:tasks.workunit.client.1.smithi152.stdout:9/31: stat c0 0 2024-03-04T16:04:27.726 INFO:tasks.workunit.client.1.smithi152.stdout:0/33: truncate d9/fa 84806 0 2024-03-04T16:04:27.728 INFO:tasks.workunit.client.0.smithi116.stdout:1/984: getdents d5/d9/d21/d27/d94/db6/d42/d8b/dc9/dd1 0 2024-03-04T16:04:27.734 INFO:tasks.workunit.client.1.smithi152.stdout:0/34: dwrite d9/fa [0,4194304] 0 2024-03-04T16:04:27.734 INFO:tasks.workunit.client.0.smithi116.stdout:1/985: creat d5/d9/d21/d27/d32/d35/d55/d75/f144 x:0 0 0 2024-03-04T16:04:27.736 INFO:tasks.workunit.client.1.smithi152.stdout:1/67: write f8 [710467,124152] 0 2024-03-04T16:04:27.742 INFO:tasks.workunit.client.1.smithi152.stdout:4/52: link d8/ce d8/d9/c13 0 2024-03-04T16:04:27.744 INFO:tasks.workunit.client.1.smithi152.stdout:0/35: write f7 [1773959,91752] 0 2024-03-04T16:04:27.744 INFO:tasks.workunit.client.1.smithi152.stdout:0/36: truncate d9/fa 4704635 0 2024-03-04T16:04:27.746 INFO:tasks.workunit.client.1.smithi152.stdout:5/138: rmdir d6/d8/d1f 39 2024-03-04T16:04:27.752 INFO:tasks.workunit.client.1.smithi152.stdout:6/102: truncate d8/fe 1899996 0 2024-03-04T16:04:27.757 INFO:tasks.workunit.client.1.smithi152.stdout:5/139: mknod d6/d8/c35 0 2024-03-04T16:04:27.758 INFO:tasks.workunit.client.1.smithi152.stdout:6/103: creat d8/f22 x:0 0 0 2024-03-04T16:04:27.761 INFO:tasks.workunit.client.1.smithi152.stdout:5/140: fsync d6/d8/d1f/f25 0 2024-03-04T16:04:27.762 INFO:tasks.workunit.client.1.smithi152.stdout:6/104: dread d8/f12 [0,4194304] 0 2024-03-04T16:04:27.762 INFO:tasks.workunit.client.1.smithi152.stdout:6/105: fdatasync f7 0 2024-03-04T16:04:27.764 INFO:tasks.workunit.client.1.smithi152.stdout:2/37: getdents da 0 2024-03-04T16:04:27.765 INFO:tasks.workunit.client.1.smithi152.stdout:2/38: chown da/df 2 1 2024-03-04T16:04:27.765 INFO:tasks.workunit.client.1.smithi152.stdout:2/39: stat f2 0 2024-03-04T16:04:27.766 INFO:tasks.workunit.client.1.smithi152.stdout:6/106: mkdir d8/d23 0 2024-03-04T16:04:27.766 INFO:tasks.workunit.client.1.smithi152.stdout:6/107: stat d8 0 2024-03-04T16:04:27.769 INFO:tasks.workunit.client.1.smithi152.stdout:2/40: mknod da/c12 0 2024-03-04T16:04:27.769 INFO:tasks.workunit.client.1.smithi152.stdout:5/141: dwrite d6/d8/f30 [0,4194304] 0 2024-03-04T16:04:27.769 INFO:tasks.workunit.client.1.smithi152.stdout:2/41: readlink l9 0 2024-03-04T16:04:27.770 INFO:tasks.workunit.client.1.smithi152.stdout:6/108: creat d8/d10/f24 x:0 0 0 2024-03-04T16:04:27.771 INFO:tasks.workunit.client.1.smithi152.stdout:2/42: fdatasync f2 0 2024-03-04T16:04:27.774 INFO:tasks.workunit.client.1.smithi152.stdout:5/142: symlink d6/d8/d1f/l36 0 2024-03-04T16:04:27.776 INFO:tasks.workunit.client.1.smithi152.stdout:6/109: rename d8/d10/f1c to d8/d23/f25 0 2024-03-04T16:04:27.776 INFO:tasks.workunit.client.1.smithi152.stdout:5/143: rename d6/d8/d1f to d6/d8/d1f/d37 22 2024-03-04T16:04:27.780 INFO:tasks.workunit.client.1.smithi152.stdout:2/43: dwrite f3 [0,4194304] 0 2024-03-04T16:04:27.780 INFO:tasks.workunit.client.1.smithi152.stdout:2/44: chown da/fe 2682 1 2024-03-04T16:04:27.788 INFO:tasks.workunit.client.1.smithi152.stdout:1/68: truncate f8 140692 0 2024-03-04T16:04:27.790 INFO:tasks.workunit.client.1.smithi152.stdout:1/69: readlink lb 0 2024-03-04T16:04:27.793 INFO:tasks.workunit.client.1.smithi152.stdout:5/144: link d6/d8/d10/f26 d6/f38 0 2024-03-04T16:04:27.794 INFO:tasks.workunit.client.1.smithi152.stdout:0/37: truncate d9/fa 3448804 0 2024-03-04T16:04:27.795 INFO:tasks.workunit.client.1.smithi152.stdout:6/110: dwrite d8/d23/f25 [0,4194304] 0 2024-03-04T16:04:27.797 INFO:tasks.workunit.client.1.smithi152.stdout:6/111: fsync d8/f21 0 2024-03-04T16:04:27.797 INFO:tasks.workunit.client.1.smithi152.stdout:4/53: dwrite f5 [0,4194304] 0 2024-03-04T16:04:27.798 INFO:tasks.workunit.client.1.smithi152.stdout:4/54: write f0 [95871,923] 0 2024-03-04T16:04:27.801 INFO:tasks.workunit.client.1.smithi152.stdout:1/70: getdents d19 0 2024-03-04T16:04:27.807 INFO:tasks.workunit.client.1.smithi152.stdout:6/112: mknod d8/d10/c26 0 2024-03-04T16:04:27.808 INFO:tasks.workunit.client.1.smithi152.stdout:7/22: sync 2024-03-04T16:04:27.808 INFO:tasks.workunit.client.1.smithi152.stdout:2/45: write f2 [1026159,109645] 0 2024-03-04T16:04:27.811 INFO:tasks.workunit.client.1.smithi152.stdout:7/23: fdatasync f4 0 2024-03-04T16:04:27.811 INFO:tasks.workunit.client.1.smithi152.stdout:5/145: creat d6/f39 x:0 0 0 2024-03-04T16:04:27.822 INFO:tasks.workunit.client.1.smithi152.stdout:5/146: chown d6/fa 2 1 2024-03-04T16:04:27.824 INFO:tasks.workunit.client.1.smithi152.stdout:4/55: dwrite d8/f10 [0,4194304] 0 2024-03-04T16:04:27.826 INFO:tasks.workunit.client.1.smithi152.stdout:1/71: dwrite f17 [0,4194304] 0 2024-03-04T16:04:27.829 INFO:tasks.workunit.client.1.smithi152.stdout:2/46: rename f1 to da/df/f13 0 2024-03-04T16:04:27.835 INFO:tasks.workunit.client.1.smithi152.stdout:5/147: rmdir d6/d8/d10 39 2024-03-04T16:04:27.840 INFO:tasks.workunit.client.0.smithi116.stdout:1/986: sync 2024-03-04T16:04:27.840 INFO:tasks.workunit.client.1.smithi152.stdout:9/32: dwrite d2/f3 [4194304,4194304] 0 2024-03-04T16:04:27.844 INFO:tasks.workunit.client.1.smithi152.stdout:7/24: dread f4 [0,4194304] 0 2024-03-04T16:04:27.844 INFO:tasks.workunit.client.1.smithi152.stdout:2/47: creat da/df/f14 x:0 0 0 2024-03-04T16:04:27.848 INFO:tasks.workunit.client.1.smithi152.stdout:4/56: dwrite d8/f10 [0,4194304] 0 2024-03-04T16:04:27.849 INFO:tasks.workunit.client.1.smithi152.stdout:5/148: creat d6/d8/d1f/f3a x:0 0 0 2024-03-04T16:04:27.849 INFO:tasks.workunit.client.1.smithi152.stdout:1/72: creat d19/f1a x:0 0 0 2024-03-04T16:04:27.853 INFO:tasks.workunit.client.1.smithi152.stdout:9/33: creat d2/f7 x:0 0 0 2024-03-04T16:04:27.854 INFO:tasks.workunit.client.1.smithi152.stdout:7/25: mknod c6 0 2024-03-04T16:04:27.861 INFO:tasks.workunit.client.1.smithi152.stdout:9/34: symlink d2/l8 0 2024-03-04T16:04:27.862 INFO:tasks.workunit.client.0.smithi116.stdout:1/987: write d5/d9/d21/d27/d32/d35/d55/db4/faa [3613864,6898] 0 2024-03-04T16:04:27.863 INFO:tasks.workunit.client.1.smithi152.stdout:7/26: write f4 [2461865,124438] 0 2024-03-04T16:04:27.863 INFO:tasks.workunit.client.0.smithi116.stdout:1/988: chown d5/d9/d21/d24/de0/d4a/f139 3 1 2024-03-04T16:04:27.865 INFO:tasks.workunit.client.1.smithi152.stdout:4/57: unlink d8/ce 0 2024-03-04T16:04:27.866 INFO:tasks.workunit.client.0.smithi116.stdout:1/989: rmdir d5/d9/d21/d27 39 2024-03-04T16:04:27.867 INFO:tasks.workunit.client.1.smithi152.stdout:1/73: dread f18 [0,4194304] 0 2024-03-04T16:04:27.868 INFO:tasks.workunit.client.1.smithi152.stdout:6/113: dwrite d8/f1f [0,4194304] 0 2024-03-04T16:04:27.871 INFO:tasks.workunit.client.0.smithi116.stdout:1/990: creat d5/d9/f145 x:0 0 0 2024-03-04T16:04:27.873 INFO:tasks.workunit.client.1.smithi152.stdout:2/48: dwrite f7 [0,4194304] 0 2024-03-04T16:04:27.875 INFO:tasks.workunit.client.0.smithi116.stdout:1/991: chown d5/d9/d21/d27/d32/d35/d55/db4/d50 10989 1 2024-03-04T16:04:27.876 INFO:tasks.workunit.client.1.smithi152.stdout:9/35: chown d2/c5 94701 1 2024-03-04T16:04:27.880 INFO:tasks.workunit.client.0.smithi116.stdout:1/992: mkdir d5/d9/d21/d27/d32/d35/d55/d75/d146 0 2024-03-04T16:04:27.881 INFO:tasks.workunit.client.0.smithi116.stdout:1/993: write d5/d9/d21/d27/d94/db6/f69 [1607580,102317] 0 2024-03-04T16:04:27.881 INFO:tasks.workunit.client.1.smithi152.stdout:0/38: dwrite d9/fa [0,4194304] 0 2024-03-04T16:04:27.883 INFO:tasks.workunit.client.0.smithi116.stdout:1/994: write d5/d9/d21/d27/d94/f12a [474481,71973] 0 2024-03-04T16:04:27.887 INFO:tasks.workunit.client.1.smithi152.stdout:9/36: creat d2/f9 x:0 0 0 2024-03-04T16:04:27.889 INFO:tasks.workunit.client.1.smithi152.stdout:0/39: truncate d9/fb 288184 0 2024-03-04T16:04:27.889 INFO:tasks.workunit.client.1.smithi152.stdout:6/114: dwrite f1 [0,4194304] 0 2024-03-04T16:04:27.892 INFO:tasks.workunit.client.1.smithi152.stdout:4/58: dread f3 [0,4194304] 0 2024-03-04T16:04:27.892 INFO:tasks.workunit.client.0.smithi116.stdout:1/995: rename d5/d86 to d5/d9/d21/d27/d94/db6/d42/d147 0 2024-03-04T16:04:27.894 INFO:tasks.workunit.client.1.smithi152.stdout:2/49: dwrite f4 [0,4194304] 0 2024-03-04T16:04:27.898 INFO:tasks.workunit.client.0.smithi116.stdout:1/996: creat d5/d9/d21/d27/d32/d35/dbf/f148 x:0 0 0 2024-03-04T16:04:27.899 INFO:tasks.workunit.client.0.smithi116.stdout:1/997: chown d5/d9/d21/lde 269 1 2024-03-04T16:04:27.901 INFO:tasks.workunit.client.1.smithi152.stdout:6/115: rename d8/c1d to d8/c27 0 2024-03-04T16:04:27.901 INFO:tasks.workunit.client.0.smithi116.stdout:1/998: write d5/d9/d21/d24/de0/d4a/f132 [235870,60127] 0 2024-03-04T16:04:27.903 INFO:tasks.workunit.client.1.smithi152.stdout:4/59: mkdir d8/d14 0 2024-03-04T16:04:27.904 INFO:tasks.workunit.client.1.smithi152.stdout:7/27: truncate f4 219239 0 2024-03-04T16:04:27.908 INFO:tasks.workunit.client.1.smithi152.stdout:9/37: rmdir d2 39 2024-03-04T16:04:27.914 INFO:tasks.workunit.client.1.smithi152.stdout:6/116: dread d8/d10/f15 [4194304,4194304] 0 2024-03-04T16:04:27.916 INFO:tasks.workunit.client.1.smithi152.stdout:6/117: fsync d8/d10/f1e 0 2024-03-04T16:04:27.916 INFO:tasks.workunit.client.1.smithi152.stdout:9/38: chown d2 5 1 2024-03-04T16:04:27.917 INFO:tasks.workunit.client.1.smithi152.stdout:9/39: write d2/f7 [752124,1021] 0 2024-03-04T16:04:27.917 INFO:tasks.workunit.client.1.smithi152.stdout:6/118: truncate d8/f9 335617 0 2024-03-04T16:04:27.919 INFO:tasks.workunit.client.1.smithi152.stdout:6/119: chown d8/d23 20 1 2024-03-04T16:04:27.920 INFO:tasks.workunit.client.0.smithi116.stdout:1/999: dwrite d5/d9/d21/f85 [0,4194304] 0 2024-03-04T16:04:27.924 INFO:tasks.workunit.client.0.smithi116.stderr:+ rm -rf -- ./tmp.645vXPrkDJ 2024-03-04T16:04:27.924 INFO:tasks.workunit.client.1.smithi152.stdout:2/50: fsync f4 0 2024-03-04T16:04:27.925 INFO:tasks.workunit.client.1.smithi152.stdout:0/40: getdents d9 0 2024-03-04T16:04:27.928 INFO:tasks.workunit.client.1.smithi152.stdout:7/28: fdatasync f4 0 2024-03-04T16:04:27.930 INFO:tasks.workunit.client.1.smithi152.stdout:4/60: creat d8/d9/f15 x:0 0 0 2024-03-04T16:04:27.930 INFO:tasks.workunit.client.1.smithi152.stdout:2/51: creat da/f15 x:0 0 0 2024-03-04T16:04:27.932 INFO:tasks.workunit.client.1.smithi152.stdout:0/41: mknod d9/cd 0 2024-03-04T16:04:27.933 INFO:tasks.workunit.client.1.smithi152.stdout:7/29: creat f7 x:0 0 0 2024-03-04T16:04:27.934 INFO:tasks.workunit.client.1.smithi152.stdout:2/52: write da/fc [1442905,116927] 0 2024-03-04T16:04:27.936 INFO:tasks.workunit.client.1.smithi152.stdout:0/42: symlink d9/le 0 2024-03-04T16:04:27.937 INFO:tasks.workunit.client.1.smithi152.stdout:7/30: write f7 [24538,49382] 0 2024-03-04T16:04:27.938 INFO:tasks.workunit.client.1.smithi152.stdout:9/40: dwrite d2/f7 [0,4194304] 0 2024-03-04T16:04:27.939 INFO:tasks.workunit.client.1.smithi152.stdout:7/31: chown f4 249 1 2024-03-04T16:04:27.940 INFO:tasks.workunit.client.1.smithi152.stdout:4/61: rmdir d8/d14 0 2024-03-04T16:04:27.940 INFO:tasks.workunit.client.1.smithi152.stdout:4/62: readlink l1 0 2024-03-04T16:04:27.941 INFO:tasks.workunit.client.1.smithi152.stdout:7/32: chown l3 2332 1 2024-03-04T16:04:27.942 INFO:tasks.workunit.client.1.smithi152.stdout:6/120: dwrite d8/d10/f18 [0,4194304] 0 2024-03-04T16:04:27.945 INFO:tasks.workunit.client.1.smithi152.stdout:9/41: rename d2/l8 to d2/la 0 2024-03-04T16:04:27.947 INFO:tasks.workunit.client.1.smithi152.stdout:7/33: write f7 [321840,37585] 0 2024-03-04T16:04:27.947 INFO:tasks.workunit.client.1.smithi152.stdout:4/63: creat d8/f16 x:0 0 0 2024-03-04T16:04:27.954 INFO:tasks.workunit.client.1.smithi152.stdout:0/43: dwrite f7 [0,4194304] 0 2024-03-04T16:04:27.959 INFO:tasks.workunit.client.1.smithi152.stdout:9/42: dread d2/f3 [0,4194304] 0 2024-03-04T16:04:27.959 INFO:tasks.workunit.client.1.smithi152.stdout:6/121: dwrite f7 [0,4194304] 0 2024-03-04T16:04:27.962 INFO:tasks.workunit.client.1.smithi152.stdout:2/53: getdents da 0 2024-03-04T16:04:27.967 INFO:tasks.workunit.client.1.smithi152.stdout:9/43: creat d2/fb x:0 0 0 2024-03-04T16:04:27.969 INFO:tasks.workunit.client.1.smithi152.stdout:0/44: link c8 d9/cf 0 2024-03-04T16:04:27.974 INFO:tasks.workunit.client.1.smithi152.stdout:2/54: dread da/df/f13 [0,4194304] 0 2024-03-04T16:04:27.974 INFO:tasks.workunit.client.1.smithi152.stdout:4/64: truncate f3 3010667 0 2024-03-04T16:04:27.975 INFO:tasks.workunit.client.1.smithi152.stdout:9/44: write d2/f3 [8246681,71321] 0 2024-03-04T16:04:27.976 INFO:tasks.workunit.client.1.smithi152.stdout:6/122: dwrite d8/f9 [0,4194304] 0 2024-03-04T16:04:27.976 INFO:tasks.workunit.client.1.smithi152.stdout:0/45: fsync f4 0 2024-03-04T16:04:27.978 INFO:tasks.workunit.client.1.smithi152.stdout:9/45: dread - d2/fb zero size 2024-03-04T16:04:27.979 INFO:tasks.workunit.client.1.smithi152.stdout:9/46: fdatasync d2/f7 0 2024-03-04T16:04:27.981 INFO:tasks.workunit.client.1.smithi152.stdout:2/55: rename f3 to da/f16 0 2024-03-04T16:04:27.984 INFO:tasks.workunit.client.1.smithi152.stdout:4/65: symlink d8/l17 0 2024-03-04T16:04:27.986 INFO:tasks.workunit.client.1.smithi152.stdout:0/46: creat d9/f10 x:0 0 0 2024-03-04T16:04:27.988 INFO:tasks.workunit.client.1.smithi152.stdout:0/47: dread - f6 zero size 2024-03-04T16:04:27.989 INFO:tasks.workunit.client.1.smithi152.stdout:6/123: creat d8/f28 x:0 0 0 2024-03-04T16:04:27.990 INFO:tasks.workunit.client.1.smithi152.stdout:6/124: stat d8/lc 0 2024-03-04T16:04:27.992 INFO:tasks.workunit.client.1.smithi152.stdout:9/47: dwrite d2/f9 [0,4194304] 0 2024-03-04T16:04:27.994 INFO:tasks.workunit.client.1.smithi152.stdout:2/56: symlink da/df/l17 0 2024-03-04T16:04:27.995 INFO:tasks.workunit.client.1.smithi152.stdout:9/48: write d2/fb [142445,123938] 0 2024-03-04T16:04:27.997 INFO:tasks.workunit.client.1.smithi152.stdout:4/66: dwrite f5 [0,4194304] 0 2024-03-04T16:04:27.998 INFO:tasks.workunit.client.1.smithi152.stdout:9/49: fdatasync d2/f3 0 2024-03-04T16:04:27.999 INFO:tasks.workunit.client.1.smithi152.stdout:0/48: mknod d9/c11 0 2024-03-04T16:04:28.003 INFO:tasks.workunit.client.1.smithi152.stdout:0/49: dread d9/fb [0,4194304] 0 2024-03-04T16:04:28.005 INFO:tasks.workunit.client.1.smithi152.stdout:0/50: write f7 [3424468,79954] 0 2024-03-04T16:04:28.007 INFO:tasks.workunit.client.1.smithi152.stdout:2/57: mknod da/df/c18 0 2024-03-04T16:04:28.012 INFO:tasks.workunit.client.1.smithi152.stdout:2/58: dread f4 [0,4194304] 0 2024-03-04T16:04:28.019 INFO:tasks.workunit.client.1.smithi152.stdout:0/51: symlink d9/l12 0 2024-03-04T16:04:28.019 INFO:tasks.workunit.client.1.smithi152.stdout:0/52: write d9/fa [579796,47433] 0 2024-03-04T16:04:28.020 INFO:tasks.workunit.client.1.smithi152.stdout:4/67: rename d8/cd to d8/c18 0 2024-03-04T16:04:28.021 INFO:tasks.workunit.client.1.smithi152.stdout:8/89: sync 2024-03-04T16:04:28.021 INFO:tasks.workunit.client.1.smithi152.stdout:3/32: sync 2024-03-04T16:04:28.021 INFO:tasks.workunit.client.1.smithi152.stdout:0/53: truncate d9/f10 801491 0 2024-03-04T16:04:28.021 INFO:tasks.workunit.client.1.smithi152.stdout:2/59: write da/f16 [5256789,88312] 0 2024-03-04T16:04:28.026 INFO:tasks.workunit.client.1.smithi152.stdout:3/33: fdatasync d6/f7 0 2024-03-04T16:04:28.030 INFO:tasks.workunit.client.1.smithi152.stdout:6/125: write d8/d10/f15 [6743378,129002] 0 2024-03-04T16:04:28.030 INFO:tasks.workunit.client.1.smithi152.stdout:2/60: rename f7 to da/df/f19 0 2024-03-04T16:04:28.031 INFO:tasks.workunit.client.1.smithi152.stdout:8/90: rename d2/d3/d9/f18 to d2/d3/d9/f1e 0 2024-03-04T16:04:28.036 INFO:tasks.workunit.client.1.smithi152.stdout:0/54: rmdir d9 39 2024-03-04T16:04:28.038 INFO:tasks.workunit.client.1.smithi152.stdout:8/91: dwrite d2/d8/f14 [0,4194304] 0 2024-03-04T16:04:28.040 INFO:tasks.workunit.client.1.smithi152.stdout:2/61: creat da/f1a x:0 0 0 2024-03-04T16:04:28.054 INFO:tasks.workunit.client.1.smithi152.stdout:6/126: symlink d8/d23/l29 0 2024-03-04T16:04:28.055 INFO:tasks.workunit.client.1.smithi152.stdout:2/62: chown da/fb 33188 1 2024-03-04T16:04:28.056 INFO:tasks.workunit.client.1.smithi152.stdout:8/92: creat d2/d3/d17/f1f x:0 0 0 2024-03-04T16:04:28.056 INFO:tasks.workunit.client.1.smithi152.stdout:2/63: fdatasync da/df/f14 0 2024-03-04T16:04:28.056 INFO:tasks.workunit.client.1.smithi152.stdout:6/127: mkdir d8/d2a 0 2024-03-04T16:04:28.058 INFO:tasks.workunit.client.1.smithi152.stdout:6/128: write d8/d10/f1e [2326178,95905] 0 2024-03-04T16:04:28.058 INFO:tasks.workunit.client.1.smithi152.stdout:2/64: write da/f11 [1338506,73084] 0 2024-03-04T16:04:28.058 INFO:tasks.workunit.client.1.smithi152.stdout:8/93: dread - d2/d8/f1a zero size 2024-03-04T16:04:28.059 INFO:tasks.workunit.client.1.smithi152.stdout:0/55: creat d9/f13 x:0 0 0 2024-03-04T16:04:28.060 INFO:tasks.workunit.client.1.smithi152.stdout:0/56: dread - f6 zero size 2024-03-04T16:04:28.062 INFO:tasks.workunit.client.1.smithi152.stdout:4/68: rename c2 to d8/d9/c19 0 2024-03-04T16:04:28.064 INFO:tasks.workunit.client.1.smithi152.stdout:4/69: write f7 [398365,127273] 0 2024-03-04T16:04:28.065 INFO:tasks.workunit.client.1.smithi152.stdout:6/129: dread d8/f14 [0,4194304] 0 2024-03-04T16:04:28.068 INFO:tasks.workunit.client.1.smithi152.stdout:2/65: rmdir da 39 2024-03-04T16:04:28.071 INFO:tasks.workunit.client.1.smithi152.stdout:8/94: fsync d2/d3/d9/fc 0 2024-03-04T16:04:28.077 INFO:tasks.workunit.client.1.smithi152.stdout:0/57: rmdir d9 39 2024-03-04T16:04:28.083 INFO:tasks.workunit.client.1.smithi152.stdout:4/70: mknod d8/c1a 0 2024-03-04T16:04:28.084 INFO:tasks.workunit.client.1.smithi152.stdout:6/130: mknod d8/d23/c2b 0 2024-03-04T16:04:28.085 INFO:tasks.workunit.client.1.smithi152.stdout:6/131: write d8/f28 [927944,124659] 0 2024-03-04T16:04:28.085 INFO:tasks.workunit.client.1.smithi152.stdout:3/34: getdents d6 0 2024-03-04T16:04:28.089 INFO:tasks.workunit.client.1.smithi152.stdout:6/132: dread d8/f12 [0,4194304] 0 2024-03-04T16:04:28.091 INFO:tasks.workunit.client.1.smithi152.stdout:4/71: rename d8/c12 to d8/c1b 0 2024-03-04T16:04:28.092 INFO:tasks.workunit.client.1.smithi152.stdout:2/66: write da/df/f19 [4120940,44957] 0 2024-03-04T16:04:28.093 INFO:tasks.workunit.client.1.smithi152.stdout:2/67: stat da/f11 0 2024-03-04T16:04:28.093 INFO:tasks.workunit.client.1.smithi152.stdout:2/68: write da/f1a [457144,124960] 0 2024-03-04T16:04:28.094 INFO:tasks.workunit.client.1.smithi152.stdout:6/133: creat d8/d23/f2c x:0 0 0 2024-03-04T16:04:28.096 INFO:tasks.workunit.client.1.smithi152.stdout:0/58: rename d9/fb to d9/f14 0 2024-03-04T16:04:28.096 INFO:tasks.workunit.client.1.smithi152.stdout:3/35: getdents d6 0 2024-03-04T16:04:28.097 INFO:tasks.workunit.client.1.smithi152.stdout:4/72: truncate d8/d9/ff 423969 0 2024-03-04T16:04:28.098 INFO:tasks.workunit.client.1.smithi152.stdout:2/69: symlink da/df/l1b 0 2024-03-04T16:04:28.099 INFO:tasks.workunit.client.1.smithi152.stdout:6/134: unlink f6 0 2024-03-04T16:04:28.101 INFO:tasks.workunit.client.1.smithi152.stdout:0/59: rename d9/f10 to d9/f15 0 2024-03-04T16:04:28.102 INFO:tasks.workunit.client.1.smithi152.stdout:3/36: rename c4 to d6/cb 0 2024-03-04T16:04:28.104 INFO:tasks.workunit.client.1.smithi152.stdout:2/70: creat da/f1c x:0 0 0 2024-03-04T16:04:28.104 INFO:tasks.workunit.client.1.smithi152.stdout:6/135: chown c2 33655 1 2024-03-04T16:04:28.105 INFO:tasks.workunit.client.1.smithi152.stdout:6/136: write d8/f1f [1779284,105153] 0 2024-03-04T16:04:28.111 INFO:tasks.workunit.client.1.smithi152.stdout:4/73: symlink d8/d9/l1c 0 2024-03-04T16:04:28.113 INFO:tasks.workunit.client.1.smithi152.stdout:0/60: dwrite f6 [0,4194304] 0 2024-03-04T16:04:28.122 INFO:tasks.workunit.client.1.smithi152.stdout:4/74: mknod d8/d9/c1d 0 2024-03-04T16:04:28.124 INFO:tasks.workunit.client.1.smithi152.stdout:4/75: mknod d8/c1e 0 2024-03-04T16:04:28.128 INFO:tasks.workunit.client.1.smithi152.stdout:4/76: dread f5 [0,4194304] 0 2024-03-04T16:04:28.131 INFO:tasks.workunit.client.1.smithi152.stdout:3/37: write d6/f8 [1344960,13785] 0 2024-03-04T16:04:28.132 INFO:tasks.workunit.client.1.smithi152.stdout:4/77: symlink d8/l1f 0 2024-03-04T16:04:28.135 INFO:tasks.workunit.client.1.smithi152.stdout:3/38: dread f5 [0,4194304] 0 2024-03-04T16:04:28.136 INFO:tasks.workunit.client.1.smithi152.stdout:4/78: dread d8/f10 [0,4194304] 0 2024-03-04T16:04:28.139 INFO:tasks.workunit.client.1.smithi152.stdout:3/39: creat d6/fc x:0 0 0 2024-03-04T16:04:28.140 INFO:tasks.workunit.client.1.smithi152.stdout:3/40: chown d6/cb 11032623 1 2024-03-04T16:04:28.144 INFO:tasks.workunit.client.1.smithi152.stdout:4/79: dwrite f5 [0,4194304] 0 2024-03-04T16:04:28.158 INFO:tasks.workunit.client.1.smithi152.stdout:4/80: fsync f5 0 2024-03-04T16:04:28.160 INFO:tasks.workunit.client.1.smithi152.stdout:4/81: write d8/f10 [3787130,62443] 0 2024-03-04T16:04:28.161 INFO:tasks.workunit.client.1.smithi152.stdout:4/82: dread - d8/d9/f15 zero size 2024-03-04T16:04:28.230 INFO:tasks.workunit.client.1.smithi152.stdout:5/149: sync 2024-03-04T16:04:28.246 INFO:tasks.workunit.client.1.smithi152.stdout:5/150: dwrite d6/d8/d10/f26 [0,4194304] 0 2024-03-04T16:04:28.252 INFO:tasks.workunit.client.1.smithi152.stdout:5/151: getdents d6 0 2024-03-04T16:04:28.393 INFO:tasks.workunit.client.1.smithi152.stdout:1/74: sync 2024-03-04T16:04:28.448 INFO:tasks.workunit.client.1.smithi152.stdout:1/75: truncate fa 37031 0 2024-03-04T16:04:28.448 INFO:tasks.workunit.client.1.smithi152.stdout:1/76: fsync f17 0 2024-03-04T16:04:28.451 INFO:tasks.workunit.client.1.smithi152.stdout:1/77: rmdir d19 39 2024-03-04T16:04:28.452 INFO:tasks.workunit.client.1.smithi152.stdout:1/78: chown lc 189227759 1 2024-03-04T16:04:28.459 INFO:tasks.workunit.client.1.smithi152.stdout:1/79: dwrite f2 [0,4194304] 0 2024-03-04T16:04:28.461 INFO:tasks.workunit.client.1.smithi152.stdout:1/80: write f5 [2260424,47836] 0 2024-03-04T16:04:28.465 INFO:tasks.workunit.client.1.smithi152.stdout:1/81: dread fe [0,4194304] 0 2024-03-04T16:04:28.475 INFO:tasks.workunit.client.1.smithi152.stdout:1/82: dread f8 [0,4194304] 0 2024-03-04T16:04:28.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:28 smithi116 ceph-mon[29120]: pgmap v30: 65 pgs: 65 active+clean; 2.1 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 59 MiB/s rd, 152 MiB/s wr, 412 op/s 2024-03-04T16:04:28.476 INFO:tasks.workunit.client.1.smithi152.stdout:1/83: rename d19 to d19/d1b 22 2024-03-04T16:04:28.477 INFO:tasks.workunit.client.1.smithi152.stdout:1/84: creat d19/f1c x:0 0 0 2024-03-04T16:04:28.483 INFO:tasks.workunit.client.1.smithi152.stdout:1/85: dwrite f17 [0,4194304] 0 2024-03-04T16:04:28.484 INFO:tasks.workunit.client.1.smithi152.stdout:1/86: stat f1 0 2024-03-04T16:04:28.488 INFO:tasks.workunit.client.1.smithi152.stdout:1/87: unlink f8 0 2024-03-04T16:04:28.489 INFO:tasks.workunit.client.1.smithi152.stdout:1/88: creat d19/f1d x:0 0 0 2024-03-04T16:04:28.492 INFO:tasks.workunit.client.1.smithi152.stdout:1/89: chown l13 131193 1 2024-03-04T16:04:28.492 INFO:tasks.workunit.client.1.smithi152.stdout:1/90: readlink lc 0 2024-03-04T16:04:28.506 INFO:tasks.workunit.client.1.smithi152.stdout:1/91: dwrite fe [0,4194304] 0 2024-03-04T16:04:28.513 INFO:tasks.workunit.client.1.smithi152.stdout:1/92: mknod d19/c1e 0 2024-03-04T16:04:28.517 INFO:tasks.workunit.client.1.smithi152.stdout:1/93: creat d19/f1f x:0 0 0 2024-03-04T16:04:28.524 INFO:tasks.workunit.client.1.smithi152.stdout:1/94: dwrite f16 [0,4194304] 0 2024-03-04T16:04:28.535 INFO:tasks.workunit.client.1.smithi152.stdout:1/95: getdents d19 0 2024-03-04T16:04:28.540 INFO:tasks.workunit.client.1.smithi152.stdout:1/96: unlink f5 0 2024-03-04T16:04:28.544 INFO:tasks.workunit.client.1.smithi152.stdout:1/97: write f10 [187303,95974] 0 2024-03-04T16:04:28.600 INFO:tasks.workunit.client.1.smithi152.stdout:7/34: sync 2024-03-04T16:04:28.601 INFO:tasks.workunit.client.1.smithi152.stdout:7/35: creat f8 x:0 0 0 2024-03-04T16:04:28.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:28 smithi152 ceph-mon[36876]: pgmap v30: 65 pgs: 65 active+clean; 2.1 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 59 MiB/s rd, 152 MiB/s wr, 412 op/s 2024-03-04T16:04:28.609 INFO:tasks.workunit.client.1.smithi152.stdout:7/36: dwrite f7 [0,4194304] 0 2024-03-04T16:04:28.756 INFO:tasks.workunit.client.1.smithi152.stdout:2/71: sync 2024-03-04T16:04:28.756 INFO:tasks.workunit.client.1.smithi152.stdout:9/50: sync 2024-03-04T16:04:28.757 INFO:tasks.workunit.client.1.smithi152.stdout:9/51: write d2/f7 [4060050,94867] 0 2024-03-04T16:04:28.761 INFO:tasks.workunit.client.1.smithi152.stdout:9/52: dread d2/f7 [0,4194304] 0 2024-03-04T16:04:28.788 INFO:tasks.workunit.client.1.smithi152.stdout:9/53: symlink d2/lc 0 2024-03-04T16:04:28.790 INFO:tasks.workunit.client.1.smithi152.stdout:9/54: rename d2/f9 to d2/fd 0 2024-03-04T16:04:28.793 INFO:tasks.workunit.client.1.smithi152.stdout:2/72: getdents da/df 0 2024-03-04T16:04:28.794 INFO:tasks.workunit.client.1.smithi152.stdout:9/55: dread d2/f3 [0,4194304] 0 2024-03-04T16:04:28.796 INFO:tasks.workunit.client.1.smithi152.stdout:2/73: unlink da/df/f19 0 2024-03-04T16:04:28.798 INFO:tasks.workunit.client.1.smithi152.stdout:9/56: rename d2/f7 to d2/fe 0 2024-03-04T16:04:28.798 INFO:tasks.workunit.client.1.smithi152.stdout:9/57: read d2/fd [324444,66386] 0 2024-03-04T16:04:28.799 INFO:tasks.workunit.client.1.smithi152.stdout:9/58: stat c0 0 2024-03-04T16:04:28.799 INFO:tasks.workunit.client.1.smithi152.stdout:9/59: rename d2 to d2/df 22 2024-03-04T16:04:28.801 INFO:tasks.workunit.client.1.smithi152.stdout:9/60: creat d2/f10 x:0 0 0 2024-03-04T16:04:28.806 INFO:tasks.workunit.client.1.smithi152.stdout:9/61: rename d2/la to d2/l11 0 2024-03-04T16:04:28.806 INFO:tasks.workunit.client.1.smithi152.stdout:9/62: dread - d2/f10 zero size 2024-03-04T16:04:28.809 INFO:tasks.workunit.client.1.smithi152.stdout:2/74: getdents da 0 2024-03-04T16:04:28.811 INFO:tasks.workunit.client.1.smithi152.stdout:9/63: creat d2/f12 x:0 0 0 2024-03-04T16:04:28.812 INFO:tasks.workunit.client.1.smithi152.stdout:9/64: chown d2/lc 4725 1 2024-03-04T16:04:28.813 INFO:tasks.workunit.client.1.smithi152.stdout:9/65: write d2/f12 [894428,1838] 0 2024-03-04T16:04:28.813 INFO:tasks.workunit.client.1.smithi152.stdout:2/75: unlink l9 0 2024-03-04T16:04:28.813 INFO:tasks.workunit.client.1.smithi152.stdout:2/76: chown da/fe 1 1 2024-03-04T16:04:28.815 INFO:tasks.workunit.client.1.smithi152.stdout:9/66: mkdir d2/d13 0 2024-03-04T16:04:28.817 INFO:tasks.workunit.client.1.smithi152.stdout:2/77: dread f4 [0,4194304] 0 2024-03-04T16:04:28.819 INFO:tasks.workunit.client.1.smithi152.stdout:9/67: chown d2/l11 62091801 1 2024-03-04T16:04:28.821 INFO:tasks.workunit.client.1.smithi152.stdout:2/78: rename da/c10 to da/df/c1d 0 2024-03-04T16:04:28.822 INFO:tasks.workunit.client.1.smithi152.stdout:2/79: chown c8 1038215 1 2024-03-04T16:04:28.824 INFO:tasks.workunit.client.1.smithi152.stdout:9/68: mknod d2/c14 0 2024-03-04T16:04:28.825 INFO:tasks.workunit.client.1.smithi152.stdout:2/80: rmdir da 39 2024-03-04T16:04:28.829 INFO:tasks.workunit.client.1.smithi152.stdout:2/81: read - da/fb zero size 2024-03-04T16:04:28.831 INFO:tasks.workunit.client.1.smithi152.stdout:2/82: rmdir da 39 2024-03-04T16:04:28.831 INFO:tasks.workunit.client.1.smithi152.stdout:9/69: dwrite d2/f12 [0,4194304] 0 2024-03-04T16:04:28.831 INFO:tasks.workunit.client.1.smithi152.stdout:9/70: readlink d2/l11 0 2024-03-04T16:04:28.834 INFO:tasks.workunit.client.1.smithi152.stdout:9/71: rename d2/f10 to d2/d13/f15 0 2024-03-04T16:04:28.837 INFO:tasks.workunit.client.1.smithi152.stdout:2/83: symlink da/df/l1e 0 2024-03-04T16:04:28.839 INFO:tasks.workunit.client.1.smithi152.stdout:9/72: mknod d2/c16 0 2024-03-04T16:04:28.842 INFO:tasks.workunit.client.1.smithi152.stdout:9/73: mknod d2/d13/c17 0 2024-03-04T16:04:28.845 INFO:tasks.workunit.client.1.smithi152.stdout:2/84: dwrite da/f15 [0,4194304] 0 2024-03-04T16:04:28.846 INFO:tasks.workunit.client.1.smithi152.stdout:9/74: creat d2/d13/f18 x:0 0 0 2024-03-04T16:04:28.849 INFO:tasks.workunit.client.1.smithi152.stdout:9/75: creat d2/f19 x:0 0 0 2024-03-04T16:04:28.851 INFO:tasks.workunit.client.1.smithi152.stdout:9/76: write d2/d13/f18 [717407,24268] 0 2024-03-04T16:04:28.852 INFO:tasks.workunit.client.1.smithi152.stdout:2/85: creat da/df/f1f x:0 0 0 2024-03-04T16:04:28.853 INFO:tasks.workunit.client.1.smithi152.stdout:9/77: creat d2/d13/f1a x:0 0 0 2024-03-04T16:04:28.855 INFO:tasks.workunit.client.1.smithi152.stdout:2/86: mknod da/df/c20 0 2024-03-04T16:04:28.857 INFO:tasks.workunit.client.1.smithi152.stdout:9/78: rename d2/d13/c17 to d2/c1b 0 2024-03-04T16:04:28.858 INFO:tasks.workunit.client.1.smithi152.stdout:2/87: symlink da/l21 0 2024-03-04T16:04:28.859 INFO:tasks.workunit.client.1.smithi152.stdout:2/88: chown c8 186210685 1 2024-03-04T16:04:28.860 INFO:tasks.workunit.client.1.smithi152.stdout:2/89: read f4 [1757547,3796] 0 2024-03-04T16:04:28.861 INFO:tasks.workunit.client.1.smithi152.stdout:2/90: fsync da/f1a 0 2024-03-04T16:04:28.865 INFO:tasks.workunit.client.1.smithi152.stdout:6/137: sync 2024-03-04T16:04:28.865 INFO:tasks.workunit.client.1.smithi152.stdout:8/95: sync 2024-03-04T16:04:28.865 INFO:tasks.workunit.client.1.smithi152.stdout:4/83: sync 2024-03-04T16:04:28.865 INFO:tasks.workunit.client.1.smithi152.stdout:3/41: sync 2024-03-04T16:04:28.865 INFO:tasks.workunit.client.1.smithi152.stdout:0/61: sync 2024-03-04T16:04:28.866 INFO:tasks.workunit.client.1.smithi152.stdout:2/91: rename f2 to da/f22 0 2024-03-04T16:04:28.867 INFO:tasks.workunit.client.1.smithi152.stdout:9/79: dwrite d2/fd [0,4194304] 0 2024-03-04T16:04:28.871 INFO:tasks.workunit.client.1.smithi152.stdout:8/96: dread d2/d3/d9/f1e [0,4194304] 0 2024-03-04T16:04:28.871 INFO:tasks.workunit.client.1.smithi152.stdout:8/97: fsync d2/d3/d17/f19 0 2024-03-04T16:04:28.872 INFO:tasks.workunit.client.1.smithi152.stdout:0/62: creat d9/f16 x:0 0 0 2024-03-04T16:04:28.872 INFO:tasks.workunit.client.1.smithi152.stdout:0/63: fdatasync f7 0 2024-03-04T16:04:28.872 INFO:tasks.workunit.client.1.smithi152.stdout:0/64: read - d9/f13 zero size 2024-03-04T16:04:28.874 INFO:tasks.workunit.client.1.smithi152.stdout:3/42: creat d6/fd x:0 0 0 2024-03-04T16:04:28.875 INFO:tasks.workunit.client.1.smithi152.stdout:3/43: fdatasync f0 0 2024-03-04T16:04:28.875 INFO:tasks.workunit.client.1.smithi152.stdout:3/44: read d6/f8 [355217,127051] 0 2024-03-04T16:04:28.877 INFO:tasks.workunit.client.1.smithi152.stdout:3/45: chown d6/fc 1731 1 2024-03-04T16:04:28.877 INFO:tasks.workunit.client.1.smithi152.stdout:6/138: rmdir d8/d10 39 2024-03-04T16:04:28.877 INFO:tasks.workunit.client.1.smithi152.stdout:3/46: dread - f3 zero size 2024-03-04T16:04:28.878 INFO:tasks.workunit.client.1.smithi152.stdout:6/139: dread - d8/d23/f2c zero size 2024-03-04T16:04:28.882 INFO:tasks.workunit.client.1.smithi152.stdout:3/47: dread d6/f8 [0,4194304] 0 2024-03-04T16:04:28.882 INFO:tasks.workunit.client.1.smithi152.stdout:2/92: creat da/f23 x:0 0 0 2024-03-04T16:04:28.888 INFO:tasks.workunit.client.1.smithi152.stdout:8/98: mknod d2/c20 0 2024-03-04T16:04:28.889 INFO:tasks.workunit.client.1.smithi152.stdout:6/140: dwrite d8/f22 [0,4194304] 0 2024-03-04T16:04:28.895 INFO:tasks.workunit.client.1.smithi152.stdout:0/65: rename d9/f13 to d9/f17 0 2024-03-04T16:04:28.899 INFO:tasks.workunit.client.1.smithi152.stdout:2/93: dwrite da/f1c [0,4194304] 0 2024-03-04T16:04:28.899 INFO:tasks.workunit.client.1.smithi152.stdout:5/152: sync 2024-03-04T16:04:28.899 INFO:tasks.workunit.client.1.smithi152.stdout:7/37: sync 2024-03-04T16:04:28.899 INFO:tasks.workunit.client.1.smithi152.stdout:1/98: sync 2024-03-04T16:04:28.900 INFO:tasks.workunit.client.1.smithi152.stdout:4/84: unlink d8/d9/c13 0 2024-03-04T16:04:28.904 INFO:tasks.workunit.client.1.smithi152.stdout:3/48: symlink d6/le 0 2024-03-04T16:04:28.905 INFO:tasks.workunit.client.1.smithi152.stdout:6/141: dwrite d8/f9 [0,4194304] 0 2024-03-04T16:04:28.905 INFO:tasks.workunit.client.1.smithi152.stdout:8/99: dwrite d2/d8/f1a [0,4194304] 0 2024-03-04T16:04:28.905 INFO:tasks.workunit.client.1.smithi152.stdout:3/49: read - f3 zero size 2024-03-04T16:04:28.915 INFO:tasks.workunit.client.1.smithi152.stdout:0/66: unlink d9/f15 0 2024-03-04T16:04:28.916 INFO:tasks.workunit.client.1.smithi152.stdout:8/100: dwrite d2/d8/f14 [0,4194304] 0 2024-03-04T16:04:28.917 INFO:tasks.workunit.client.1.smithi152.stdout:8/101: chown d2/d3/la 81 1 2024-03-04T16:04:28.918 INFO:tasks.workunit.client.1.smithi152.stdout:1/99: mkdir d19/d20 0 2024-03-04T16:04:28.919 INFO:tasks.workunit.client.1.smithi152.stdout:8/102: write d2/d3/d17/f19 [687932,91492] 0 2024-03-04T16:04:28.920 INFO:tasks.workunit.client.1.smithi152.stdout:2/94: mkdir da/d24 0 2024-03-04T16:04:28.922 INFO:tasks.workunit.client.1.smithi152.stdout:7/38: symlink l9 0 2024-03-04T16:04:28.924 INFO:tasks.workunit.client.1.smithi152.stdout:4/85: creat d8/f20 x:0 0 0 2024-03-04T16:04:28.925 INFO:tasks.workunit.client.1.smithi152.stdout:9/80: link d2/fe d2/f1c 0 2024-03-04T16:04:28.929 INFO:tasks.workunit.client.1.smithi152.stdout:5/153: rename d6/d8/d10/f14 to d6/d8/d1f/f3b 0 2024-03-04T16:04:28.934 INFO:tasks.workunit.client.1.smithi152.stdout:3/50: creat d6/ff x:0 0 0 2024-03-04T16:04:28.941 INFO:tasks.workunit.client.1.smithi152.stdout:8/103: rename d2/d3/d17 to d2/d3/d21 0 2024-03-04T16:04:28.942 INFO:tasks.workunit.client.1.smithi152.stdout:2/95: creat da/df/f25 x:0 0 0 2024-03-04T16:04:28.944 INFO:tasks.workunit.client.1.smithi152.stdout:1/100: dwrite fe [0,4194304] 0 2024-03-04T16:04:28.950 INFO:tasks.workunit.client.1.smithi152.stdout:5/154: rename d6/d8/c29 to d6/c3c 0 2024-03-04T16:04:28.951 INFO:tasks.workunit.client.1.smithi152.stdout:3/51: mkdir d6/d10 0 2024-03-04T16:04:28.952 INFO:tasks.workunit.client.1.smithi152.stdout:3/52: write f0 [1540417,116917] 0 2024-03-04T16:04:28.952 INFO:tasks.workunit.client.1.smithi152.stdout:3/53: chown c2 1 1 2024-03-04T16:04:28.958 INFO:tasks.workunit.client.1.smithi152.stdout:7/39: link f7 fa 0 2024-03-04T16:04:28.959 INFO:tasks.workunit.client.1.smithi152.stdout:2/96: mknod da/df/c26 0 2024-03-04T16:04:28.959 INFO:tasks.workunit.client.1.smithi152.stdout:0/67: write d9/f14 [662802,125780] 0 2024-03-04T16:04:28.961 INFO:tasks.workunit.client.1.smithi152.stdout:1/101: chown f17 95022 1 2024-03-04T16:04:28.963 INFO:tasks.workunit.client.1.smithi152.stdout:2/97: dread da/f22 [0,4194304] 0 2024-03-04T16:04:28.968 INFO:tasks.workunit.client.1.smithi152.stdout:9/81: truncate d2/f12 708379 0 2024-03-04T16:04:28.970 INFO:tasks.workunit.client.1.smithi152.stdout:7/40: creat fb x:0 0 0 2024-03-04T16:04:28.970 INFO:tasks.workunit.client.1.smithi152.stdout:7/41: read f4 [184150,11107] 0 2024-03-04T16:04:28.971 INFO:tasks.workunit.client.1.smithi152.stdout:7/42: write fb [1045242,122199] 0 2024-03-04T16:04:28.981 INFO:tasks.workunit.client.1.smithi152.stdout:5/155: readlink d6/d8/d1f/l36 0 2024-03-04T16:04:28.982 INFO:tasks.workunit.client.1.smithi152.stdout:3/54: mknod d6/d10/c11 0 2024-03-04T16:04:28.982 INFO:tasks.workunit.client.1.smithi152.stdout:0/68: dwrite d9/f17 [0,4194304] 0 2024-03-04T16:04:28.984 INFO:tasks.workunit.client.1.smithi152.stdout:3/55: fdatasync d6/fc 0 2024-03-04T16:04:28.985 INFO:tasks.workunit.client.1.smithi152.stdout:7/43: dwrite f8 [0,4194304] 0 2024-03-04T16:04:28.988 INFO:tasks.workunit.client.1.smithi152.stdout:8/104: truncate d2/d8/f1a 3416834 0 2024-03-04T16:04:28.990 INFO:tasks.workunit.client.1.smithi152.stdout:4/86: getdents d8 0 2024-03-04T16:04:28.991 INFO:tasks.workunit.client.1.smithi152.stdout:1/102: mkdir d19/d20/d21 0 2024-03-04T16:04:28.994 INFO:tasks.workunit.client.1.smithi152.stdout:4/87: write d8/d9/ff [345880,68726] 0 2024-03-04T16:04:28.994 INFO:tasks.workunit.client.1.smithi152.stdout:2/98: dwrite f6 [0,4194304] 0 2024-03-04T16:04:28.996 INFO:tasks.workunit.client.1.smithi152.stdout:4/88: readlink d8/d9/l1c 0 2024-03-04T16:04:28.996 INFO:tasks.workunit.client.1.smithi152.stdout:4/89: chown d8/la 59478 1 2024-03-04T16:04:28.996 INFO:tasks.workunit.client.1.smithi152.stdout:5/156: rmdir d6/d8/d1f 39 2024-03-04T16:04:28.997 INFO:tasks.workunit.client.1.smithi152.stdout:4/90: dread - d8/d9/f15 zero size 2024-03-04T16:04:28.997 INFO:tasks.workunit.client.1.smithi152.stdout:4/91: readlink l1 0 2024-03-04T16:04:28.998 INFO:tasks.workunit.client.1.smithi152.stdout:4/92: truncate f7 4398494 0 2024-03-04T16:04:28.999 INFO:tasks.workunit.client.1.smithi152.stdout:3/56: creat d6/f12 x:0 0 0 2024-03-04T16:04:29.001 INFO:tasks.workunit.client.1.smithi152.stdout:4/93: write d8/f10 [2183382,61487] 0 2024-03-04T16:04:29.006 INFO:tasks.workunit.client.1.smithi152.stdout:9/82: truncate d2/f1c 504396 0 2024-03-04T16:04:29.009 INFO:tasks.workunit.client.1.smithi152.stdout:3/57: dwrite d6/f12 [0,4194304] 0 2024-03-04T16:04:29.010 INFO:tasks.workunit.client.1.smithi152.stdout:3/58: chown d6/f8 3656944 1 2024-03-04T16:04:29.011 INFO:tasks.workunit.client.1.smithi152.stdout:8/105: mkdir d2/d8/dd/d22 0 2024-03-04T16:04:29.015 INFO:tasks.workunit.client.1.smithi152.stdout:7/44: dwrite f8 [4194304,4194304] 0 2024-03-04T16:04:29.024 INFO:tasks.workunit.client.1.smithi152.stdout:3/59: unlink c2 0 2024-03-04T16:04:29.024 INFO:tasks.workunit.client.1.smithi152.stdout:7/45: dwrite fb [0,4194304] 0 2024-03-04T16:04:29.036 INFO:tasks.workunit.client.1.smithi152.stdout:0/69: truncate d9/fa 2413276 0 2024-03-04T16:04:29.038 INFO:tasks.workunit.client.1.smithi152.stdout:5/157: mkdir d6/d8/d1f/d3d 0 2024-03-04T16:04:29.039 INFO:tasks.workunit.client.1.smithi152.stdout:1/103: write f18 [2085306,3075] 0 2024-03-04T16:04:29.044 INFO:tasks.workunit.client.1.smithi152.stdout:3/60: dread d6/f8 [0,4194304] 0 2024-03-04T16:04:29.047 INFO:tasks.workunit.client.1.smithi152.stdout:2/99: truncate da/df/f13 4483717 0 2024-03-04T16:04:29.051 INFO:tasks.workunit.client.1.smithi152.stdout:1/104: chown cf 396550 1 2024-03-04T16:04:29.052 INFO:tasks.workunit.client.1.smithi152.stdout:4/94: rename d8/c1a to d8/d9/c21 0 2024-03-04T16:04:29.052 INFO:tasks.workunit.client.1.smithi152.stdout:5/158: rename d6/d8 to d6/d8/d1f/d3d/d3e 22 2024-03-04T16:04:29.057 INFO:tasks.workunit.client.1.smithi152.stdout:3/61: symlink d6/l13 0 2024-03-04T16:04:29.061 INFO:tasks.workunit.client.1.smithi152.stdout:7/46: dwrite f7 [0,4194304] 0 2024-03-04T16:04:29.061 INFO:tasks.workunit.client.1.smithi152.stdout:8/106: dwrite d2/d8/f1a [0,4194304] 0 2024-03-04T16:04:29.067 INFO:tasks.workunit.client.1.smithi152.stdout:1/105: symlink d19/l22 0 2024-03-04T16:04:29.069 INFO:tasks.workunit.client.1.smithi152.stdout:6/142: sync 2024-03-04T16:04:29.069 INFO:tasks.workunit.client.1.smithi152.stdout:6/143: chown d8/f9 1101768959 1 2024-03-04T16:04:29.071 INFO:tasks.workunit.client.1.smithi152.stdout:9/83: getdents d2 0 2024-03-04T16:04:29.072 INFO:tasks.workunit.client.1.smithi152.stdout:9/84: fsync d2/d13/f1a 0 2024-03-04T16:04:29.074 INFO:tasks.workunit.client.1.smithi152.stdout:7/47: symlink lc 0 2024-03-04T16:04:29.074 INFO:tasks.workunit.client.1.smithi152.stdout:0/70: link d9/l12 d9/l18 0 2024-03-04T16:04:29.075 INFO:tasks.workunit.client.1.smithi152.stdout:0/71: write d9/f17 [2875912,34227] 0 2024-03-04T16:04:29.075 INFO:tasks.workunit.client.1.smithi152.stdout:3/62: read f1 [912247,72443] 0 2024-03-04T16:04:29.079 INFO:tasks.workunit.client.1.smithi152.stdout:7/48: dread f8 [0,4194304] 0 2024-03-04T16:04:29.079 INFO:tasks.workunit.client.1.smithi152.stdout:8/107: fsync d2/d3/d9/fc 0 2024-03-04T16:04:29.083 INFO:tasks.workunit.client.1.smithi152.stdout:1/106: creat d19/f23 x:0 0 0 2024-03-04T16:04:29.086 INFO:tasks.workunit.client.1.smithi152.stdout:4/95: symlink d8/d9/l22 0 2024-03-04T16:04:29.087 INFO:tasks.workunit.client.1.smithi152.stdout:3/63: dread d6/f7 [0,4194304] 0 2024-03-04T16:04:29.087 INFO:tasks.workunit.client.1.smithi152.stdout:5/159: mknod d6/c3f 0 2024-03-04T16:04:29.089 INFO:tasks.workunit.client.1.smithi152.stdout:9/85: rename d2/l11 to d2/l1d 0 2024-03-04T16:04:29.093 INFO:tasks.workunit.client.1.smithi152.stdout:0/72: creat d9/f19 x:0 0 0 2024-03-04T16:04:29.095 INFO:tasks.workunit.client.1.smithi152.stdout:2/100: truncate da/f1c 2503919 0 2024-03-04T16:04:29.096 INFO:tasks.workunit.client.1.smithi152.stdout:9/86: dwrite d2/f3 [4194304,4194304] 0 2024-03-04T16:04:29.097 INFO:tasks.workunit.client.1.smithi152.stdout:9/87: truncate d2/d13/f1a 653651 0 2024-03-04T16:04:29.097 INFO:tasks.workunit.client.1.smithi152.stdout:8/108: mknod d2/d3/d15/c23 0 2024-03-04T16:04:29.097 INFO:tasks.workunit.client.1.smithi152.stdout:9/88: fdatasync d2/fb 0 2024-03-04T16:04:29.098 INFO:tasks.workunit.client.1.smithi152.stdout:7/49: creat fd x:0 0 0 2024-03-04T16:04:29.099 INFO:tasks.workunit.client.1.smithi152.stdout:8/109: chown d2/d3/d21/f1b 4462 1 2024-03-04T16:04:29.101 INFO:tasks.workunit.client.1.smithi152.stdout:1/107: mknod d19/c24 0 2024-03-04T16:04:29.102 INFO:tasks.workunit.client.1.smithi152.stdout:4/96: creat d8/f23 x:0 0 0 2024-03-04T16:04:29.103 INFO:tasks.workunit.client.1.smithi152.stdout:4/97: write d8/f10 [5085919,93036] 0 2024-03-04T16:04:29.106 INFO:tasks.workunit.client.1.smithi152.stdout:3/64: fdatasync f1 0 2024-03-04T16:04:29.107 INFO:tasks.workunit.client.1.smithi152.stdout:6/144: getdents d8/d2a 0 2024-03-04T16:04:29.110 INFO:tasks.workunit.client.1.smithi152.stdout:4/98: dwrite f0 [0,4194304] 0 2024-03-04T16:04:29.110 INFO:tasks.workunit.client.1.smithi152.stdout:5/160: unlink d6/d8/d10/f26 0 2024-03-04T16:04:29.120 INFO:tasks.workunit.client.1.smithi152.stdout:5/161: dread d6/d8/f24 [0,4194304] 0 2024-03-04T16:04:29.121 INFO:tasks.workunit.client.1.smithi152.stdout:9/89: mknod d2/d13/c1e 0 2024-03-04T16:04:29.125 INFO:tasks.workunit.client.1.smithi152.stdout:1/108: readlink l13 0 2024-03-04T16:04:29.269 INFO:tasks.workunit.client.1.smithi152.stdout:3/65: mkdir d6/d14 0 2024-03-04T16:04:29.269 INFO:tasks.workunit.client.1.smithi152.stdout:2/101: getdents da/d24 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:9/90: creat d2/f1f x:0 0 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:2/102: dwrite da/fe [0,4194304] 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:7/50: link f4 fe 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:7/51: write fd [146425,44246] 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:8/110: creat d2/d8/dd/d22/f24 x:0 0 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:1/109: chown f2 911 1 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:1/110: dwrite d19/f1a [0,4194304] 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:6/145: symlink d8/d10/l2d 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:5/162: dwrite d6/d8/f15 [0,4194304] 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:7/52: creat ff x:0 0 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:9/91: dwrite d2/d13/f15 [0,4194304] 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:8/111: mknod d2/d8/c25 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:3/66: write d6/f8 [1127308,16579] 0 2024-03-04T16:04:29.270 INFO:tasks.workunit.client.1.smithi152.stdout:3/67: stat d6/f8 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:3/68: fsync d6/f12 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:1/111: rmdir d19 39 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:6/146: rename d8/d10/f15 to d8/d2a/f2e 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:0/73: getdents d9 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:1/112: dwrite f18 [4194304,4194304] 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:7/53: mknod c10 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:9/92: symlink d2/d13/l20 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:8/112: mkdir d2/d8/dd/d22/d26 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:0/74: dwrite f6 [0,4194304] 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:2/103: truncate da/f1c 650690 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:5/163: stat d6/d8/f24 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:1/113: readlink l12 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:7/54: unlink fb 0 2024-03-04T16:04:29.271 INFO:tasks.workunit.client.1.smithi152.stdout:7/55: chown lc 73258 1 2024-03-04T16:04:29.272 INFO:tasks.workunit.client.1.smithi152.stdout:6/147: dwrite d8/d2a/f2e [4194304,4194304] 0 2024-03-04T16:04:29.272 INFO:tasks.workunit.client.1.smithi152.stdout:9/93: unlink d2/f12 0 2024-03-04T16:04:29.272 INFO:tasks.workunit.client.1.smithi152.stdout:7/56: write f7 [639698,117540] 0 2024-03-04T16:04:29.272 INFO:tasks.workunit.client.1.smithi152.stdout:0/75: mkdir d9/d1a 0 2024-03-04T16:04:29.272 INFO:tasks.workunit.client.1.smithi152.stdout:2/104: mknod da/df/c27 0 2024-03-04T16:04:29.272 INFO:tasks.workunit.client.1.smithi152.stdout:2/105: fsync da/f11 0 2024-03-04T16:04:29.272 INFO:tasks.workunit.client.1.smithi152.stdout:7/57: mknod c11 0 2024-03-04T16:04:29.272 INFO:tasks.workunit.client.1.smithi152.stdout:7/58: write ff [157825,22607] 0 2024-03-04T16:04:29.272 INFO:tasks.workunit.client.1.smithi152.stdout:7/59: chown l3 16072 1 2024-03-04T16:04:29.272 INFO:tasks.workunit.client.1.smithi152.stdout:0/76: dwrite f7 [4194304,4194304] 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:2/106: symlink da/l28 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:6/148: truncate f1 911038 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:7/60: dwrite fa [0,4194304] 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:7/61: stat fd 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:7/62: write f7 [3440458,122204] 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:1/114: symlink d19/d20/d21/l25 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:1/115: truncate f2 8875443 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:3/69: dread f1 [0,4194304] 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:0/77: mkdir d9/d1b 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:5/164: write d6/fa [2112162,110613] 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:1/116: symlink d19/l26 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:5/165: dread d6/d8/f15 [0,4194304] 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:7/63: dread f8 [0,4194304] 0 2024-03-04T16:04:29.273 INFO:tasks.workunit.client.1.smithi152.stdout:2/107: symlink da/d24/l29 0 2024-03-04T16:04:29.274 INFO:tasks.workunit.client.1.smithi152.stdout:1/117: dwrite d19/f23 [0,4194304] 0 2024-03-04T16:04:29.274 INFO:tasks.workunit.client.1.smithi152.stdout:9/94: truncate d2/d13/f15 159320 0 2024-03-04T16:04:29.276 INFO:tasks.workunit.client.1.smithi152.stdout:3/70: creat d6/d14/f15 x:0 0 0 2024-03-04T16:04:29.276 INFO:tasks.workunit.client.1.smithi152.stdout:0/78: creat d9/d1b/f1c x:0 0 0 2024-03-04T16:04:29.281 INFO:tasks.workunit.client.1.smithi152.stdout:0/79: fdatasync f6 0 2024-03-04T16:04:29.281 INFO:tasks.workunit.client.1.smithi152.stdout:7/64: dwrite ff [0,4194304] 0 2024-03-04T16:04:29.281 INFO:tasks.workunit.client.1.smithi152.stdout:7/65: rmdir - no directory 2024-03-04T16:04:29.282 INFO:tasks.workunit.client.1.smithi152.stdout:6/149: creat d8/f2f x:0 0 0 2024-03-04T16:04:29.282 INFO:tasks.workunit.client.1.smithi152.stdout:6/150: stat c2 0 2024-03-04T16:04:29.282 INFO:tasks.workunit.client.1.smithi152.stdout:7/66: stat f7 0 2024-03-04T16:04:29.285 INFO:tasks.workunit.client.1.smithi152.stdout:6/151: truncate d8/f2f 900700 0 2024-03-04T16:04:29.285 INFO:tasks.workunit.client.1.smithi152.stdout:1/118: dwrite f18 [0,4194304] 0 2024-03-04T16:04:29.285 INFO:tasks.workunit.client.1.smithi152.stdout:0/80: dread d9/f17 [0,4194304] 0 2024-03-04T16:04:29.287 INFO:tasks.workunit.client.1.smithi152.stdout:0/81: truncate d9/f16 649331 0 2024-03-04T16:04:29.287 INFO:tasks.workunit.client.1.smithi152.stdout:2/108: creat da/d24/f2a x:0 0 0 2024-03-04T16:04:29.292 INFO:tasks.workunit.client.1.smithi152.stdout:3/71: write f5 [2073727,78315] 0 2024-03-04T16:04:29.294 INFO:tasks.workunit.client.1.smithi152.stdout:3/72: chown f0 1276460 1 2024-03-04T16:04:29.296 INFO:tasks.workunit.client.1.smithi152.stdout:5/166: write d6/d8/f24 [831474,17062] 0 2024-03-04T16:04:29.307 INFO:tasks.workunit.client.1.smithi152.stdout:7/67: fdatasync f4 0 2024-03-04T16:04:29.308 INFO:tasks.workunit.client.1.smithi152.stdout:7/68: chown fe 19319073 1 2024-03-04T16:04:29.309 INFO:tasks.workunit.client.1.smithi152.stdout:1/119: dread fa [0,4194304] 0 2024-03-04T16:04:29.309 INFO:tasks.workunit.client.1.smithi152.stdout:9/95: truncate d2/f1c 987654 0 2024-03-04T16:04:29.309 INFO:tasks.workunit.client.1.smithi152.stdout:3/73: symlink d6/l16 0 2024-03-04T16:04:29.310 INFO:tasks.workunit.client.1.smithi152.stdout:3/74: truncate d6/ff 671348 0 2024-03-04T16:04:29.310 INFO:tasks.workunit.client.1.smithi152.stdout:1/120: dwrite f18 [0,4194304] 0 2024-03-04T16:04:29.310 INFO:tasks.workunit.client.1.smithi152.stdout:5/167: mknod d6/d8/d10/c40 0 2024-03-04T16:04:29.310 INFO:tasks.workunit.client.1.smithi152.stdout:3/75: write d6/d14/f15 [64403,119426] 0 2024-03-04T16:04:29.310 INFO:tasks.workunit.client.1.smithi152.stdout:3/76: dread - d6/fd zero size 2024-03-04T16:04:29.311 INFO:tasks.workunit.client.1.smithi152.stdout:3/77: chown f5 225397 1 2024-03-04T16:04:29.312 INFO:tasks.workunit.client.1.smithi152.stdout:3/78: truncate d6/f8 1398420 0 2024-03-04T16:04:29.316 INFO:tasks.workunit.client.1.smithi152.stdout:3/79: dread - d6/fc zero size 2024-03-04T16:04:29.316 INFO:tasks.workunit.client.1.smithi152.stdout:7/69: creat f12 x:0 0 0 2024-03-04T16:04:29.316 INFO:tasks.workunit.client.1.smithi152.stdout:0/82: getdents d9/d1a 0 2024-03-04T16:04:29.316 INFO:tasks.workunit.client.1.smithi152.stdout:0/83: truncate d9/f14 1674868 0 2024-03-04T16:04:29.321 INFO:tasks.workunit.client.1.smithi152.stdout:0/84: write d9/d1b/f1c [20526,34627] 0 2024-03-04T16:04:29.321 INFO:tasks.workunit.client.1.smithi152.stdout:0/85: fsync f7 0 2024-03-04T16:04:29.321 INFO:tasks.workunit.client.1.smithi152.stdout:4/99: sync 2024-03-04T16:04:29.324 INFO:tasks.workunit.client.1.smithi152.stdout:9/96: rename d2/f3 to d2/d13/f21 0 2024-03-04T16:04:29.324 INFO:tasks.workunit.client.1.smithi152.stdout:6/152: dwrite d8/fe [0,4194304] 0 2024-03-04T16:04:29.324 INFO:tasks.workunit.client.1.smithi152.stdout:9/97: dread - d2/f1f zero size 2024-03-04T16:04:29.325 INFO:tasks.workunit.client.1.smithi152.stdout:4/100: write d8/f10 [4057573,4672] 0 2024-03-04T16:04:29.327 INFO:tasks.workunit.client.1.smithi152.stdout:0/86: dread f7 [4194304,4194304] 0 2024-03-04T16:04:29.331 INFO:tasks.workunit.client.1.smithi152.stdout:2/109: dwrite da/f16 [0,4194304] 0 2024-03-04T16:04:29.343 INFO:tasks.workunit.client.1.smithi152.stdout:9/98: chown d2/fe 39 1 2024-03-04T16:04:29.343 INFO:tasks.workunit.client.1.smithi152.stdout:7/70: dwrite f4 [0,4194304] 0 2024-03-04T16:04:29.353 INFO:tasks.workunit.client.1.smithi152.stdout:8/113: sync 2024-03-04T16:04:29.353 INFO:tasks.workunit.client.1.smithi152.stdout:0/87: unlink d9/d1b/f1c 0 2024-03-04T16:04:29.354 INFO:tasks.workunit.client.1.smithi152.stdout:1/121: truncate f17 2183791 0 2024-03-04T16:04:29.354 INFO:tasks.workunit.client.1.smithi152.stdout:8/114: read - d2/d3/d21/f1f zero size 2024-03-04T16:04:29.355 INFO:tasks.workunit.client.1.smithi152.stdout:2/110: symlink da/d24/l2b 0 2024-03-04T16:04:29.359 INFO:tasks.workunit.client.1.smithi152.stdout:0/88: dread d9/fa [0,4194304] 0 2024-03-04T16:04:29.359 INFO:tasks.workunit.client.1.smithi152.stdout:0/89: fdatasync d9/f16 0 2024-03-04T16:04:29.359 INFO:tasks.workunit.client.1.smithi152.stdout:2/111: write da/fc [934194,76706] 0 2024-03-04T16:04:29.361 INFO:tasks.workunit.client.1.smithi152.stdout:4/101: mkdir d8/d9/d24 0 2024-03-04T16:04:29.364 INFO:tasks.workunit.client.1.smithi152.stdout:5/168: creat d6/d8/f41 x:0 0 0 2024-03-04T16:04:29.367 INFO:tasks.workunit.client.1.smithi152.stdout:7/71: rename l9 to l13 0 2024-03-04T16:04:29.371 INFO:tasks.workunit.client.1.smithi152.stdout:1/122: symlink d19/l27 0 2024-03-04T16:04:29.372 INFO:tasks.workunit.client.1.smithi152.stdout:7/72: dread f4 [0,4194304] 0 2024-03-04T16:04:29.373 INFO:tasks.workunit.client.1.smithi152.stdout:8/115: write d2/d3/d9/f1e [1915885,34600] 0 2024-03-04T16:04:29.375 INFO:tasks.workunit.client.1.smithi152.stdout:6/153: dwrite f1 [0,4194304] 0 2024-03-04T16:04:29.378 INFO:tasks.workunit.client.1.smithi152.stdout:0/90: readlink d9/l12 0 2024-03-04T16:04:29.379 INFO:tasks.workunit.client.1.smithi152.stdout:2/112: mkdir da/d2c 0 2024-03-04T16:04:29.380 INFO:tasks.workunit.client.1.smithi152.stdout:0/91: fsync d9/f16 0 2024-03-04T16:04:29.380 INFO:tasks.workunit.client.1.smithi152.stdout:7/73: dread fe [0,4194304] 0 2024-03-04T16:04:29.381 INFO:tasks.workunit.client.1.smithi152.stdout:1/123: dwrite d19/f1d [0,4194304] 0 2024-03-04T16:04:29.383 INFO:tasks.workunit.client.1.smithi152.stdout:9/99: fsync d2/d13/f15 0 2024-03-04T16:04:29.383 INFO:tasks.workunit.client.1.smithi152.stdout:1/124: stat fa 0 2024-03-04T16:04:29.384 INFO:tasks.workunit.client.1.smithi152.stdout:4/102: unlink l1 0 2024-03-04T16:04:29.388 INFO:tasks.workunit.client.1.smithi152.stdout:5/169: symlink d6/d8/d1f/l42 0 2024-03-04T16:04:29.390 INFO:tasks.workunit.client.1.smithi152.stdout:5/170: write d6/d8/f2f [143908,52019] 0 2024-03-04T16:04:29.390 INFO:tasks.workunit.client.1.smithi152.stdout:2/113: dwrite da/fb [0,4194304] 0 2024-03-04T16:04:29.397 INFO:tasks.workunit.client.1.smithi152.stdout:3/80: getdents d6 0 2024-03-04T16:04:29.397 INFO:tasks.workunit.client.1.smithi152.stdout:2/114: dread da/fc [0,4194304] 0 2024-03-04T16:04:29.402 INFO:tasks.workunit.client.1.smithi152.stdout:2/115: dread da/df/f13 [0,4194304] 0 2024-03-04T16:04:29.402 INFO:tasks.workunit.client.1.smithi152.stdout:6/154: mknod d8/d2a/c30 0 2024-03-04T16:04:29.403 INFO:tasks.workunit.client.1.smithi152.stdout:2/116: write da/fc [855181,104539] 0 2024-03-04T16:04:29.404 INFO:tasks.workunit.client.1.smithi152.stdout:6/155: write d8/f28 [2020558,73870] 0 2024-03-04T16:04:29.408 INFO:tasks.workunit.client.1.smithi152.stdout:6/156: dread d8/f12 [0,4194304] 0 2024-03-04T16:04:29.411 INFO:tasks.workunit.client.1.smithi152.stdout:7/74: mkdir d14 0 2024-03-04T16:04:29.414 INFO:tasks.workunit.client.1.smithi152.stdout:0/92: creat d9/d1b/f1d x:0 0 0 2024-03-04T16:04:29.415 INFO:tasks.workunit.client.1.smithi152.stdout:0/93: read d9/f16 [316658,22645] 0 2024-03-04T16:04:29.416 INFO:tasks.workunit.client.1.smithi152.stdout:1/125: rmdir d19 39 2024-03-04T16:04:29.416 INFO:tasks.workunit.client.1.smithi152.stdout:0/94: dread - d9/d1b/f1d zero size 2024-03-04T16:04:29.417 INFO:tasks.workunit.client.1.smithi152.stdout:9/100: mknod d2/d13/c22 0 2024-03-04T16:04:29.417 INFO:tasks.workunit.client.1.smithi152.stdout:9/101: readlink d2/lc 0 2024-03-04T16:04:29.419 INFO:tasks.workunit.client.1.smithi152.stdout:8/116: dread d2/d3/d9/fc [0,4194304] 0 2024-03-04T16:04:29.422 INFO:tasks.workunit.client.1.smithi152.stdout:5/171: truncate d6/d8/d10/f21 6028634 0 2024-03-04T16:04:29.423 INFO:tasks.workunit.client.1.smithi152.stdout:5/172: truncate d6/d8/d1f/f3a 267262 0 2024-03-04T16:04:29.428 INFO:tasks.workunit.client.1.smithi152.stdout:2/117: rename da/df/c27 to da/df/c2d 0 2024-03-04T16:04:29.430 INFO:tasks.workunit.client.1.smithi152.stdout:6/157: mknod d8/d2a/c31 0 2024-03-04T16:04:29.432 INFO:tasks.workunit.client.1.smithi152.stdout:3/81: dwrite f1 [0,4194304] 0 2024-03-04T16:04:29.436 INFO:tasks.workunit.client.1.smithi152.stdout:6/158: dread d8/f2f [0,4194304] 0 2024-03-04T16:04:29.440 INFO:tasks.workunit.client.1.smithi152.stdout:2/118: dwrite da/f11 [4194304,4194304] 0 2024-03-04T16:04:29.443 INFO:tasks.workunit.client.1.smithi152.stdout:9/102: unlink d2/d13/f21 0 2024-03-04T16:04:29.445 INFO:tasks.workunit.client.1.smithi152.stdout:8/117: stat d2/d8/dd/l11 0 2024-03-04T16:04:29.446 INFO:tasks.workunit.client.1.smithi152.stdout:7/75: dwrite fe [4194304,4194304] 0 2024-03-04T16:04:29.448 INFO:tasks.workunit.client.1.smithi152.stdout:7/76: chown c6 1545978 1 2024-03-04T16:04:29.448 INFO:tasks.workunit.client.1.smithi152.stdout:7/77: readlink lc 0 2024-03-04T16:04:29.449 INFO:tasks.workunit.client.1.smithi152.stdout:1/126: dread d19/f1a [0,4194304] 0 2024-03-04T16:04:29.450 INFO:tasks.workunit.client.1.smithi152.stdout:5/173: creat d6/d8/d10/f43 x:0 0 0 2024-03-04T16:04:29.455 INFO:tasks.workunit.client.1.smithi152.stdout:4/103: write f3 [1218261,56856] 0 2024-03-04T16:04:29.456 INFO:tasks.workunit.client.1.smithi152.stdout:3/82: rmdir d6 39 2024-03-04T16:04:29.457 INFO:tasks.workunit.client.1.smithi152.stdout:9/103: read d2/fb [132744,58807] 0 2024-03-04T16:04:29.459 INFO:tasks.workunit.client.1.smithi152.stdout:0/95: dwrite d9/fa [0,4194304] 0 2024-03-04T16:04:29.459 INFO:tasks.workunit.client.1.smithi152.stdout:6/159: mknod d8/d23/c32 0 2024-03-04T16:04:29.462 INFO:tasks.workunit.client.1.smithi152.stdout:2/119: mkdir da/df/d2e 0 2024-03-04T16:04:29.469 INFO:tasks.workunit.client.1.smithi152.stdout:1/127: creat d19/f28 x:0 0 0 2024-03-04T16:04:29.472 INFO:tasks.workunit.client.1.smithi152.stdout:5/174: creat d6/d8/d1f/f44 x:0 0 0 2024-03-04T16:04:29.475 INFO:tasks.workunit.client.1.smithi152.stdout:4/104: creat d8/f25 x:0 0 0 2024-03-04T16:04:29.475 INFO:tasks.workunit.client.1.smithi152.stdout:4/105: chown d8/c1b 648947506 1 2024-03-04T16:04:29.479 INFO:tasks.workunit.client.1.smithi152.stdout:0/96: creat d9/d1b/f1e x:0 0 0 2024-03-04T16:04:29.482 INFO:tasks.workunit.client.1.smithi152.stdout:6/160: creat d8/d23/f33 x:0 0 0 2024-03-04T16:04:29.484 INFO:tasks.workunit.client.1.smithi152.stdout:8/118: mknod d2/d8/dd/d22/d26/c27 0 2024-03-04T16:04:29.485 INFO:tasks.workunit.client.1.smithi152.stdout:7/78: creat d14/f15 x:0 0 0 2024-03-04T16:04:29.486 INFO:tasks.workunit.client.1.smithi152.stdout:7/79: chown fe 774 1 2024-03-04T16:04:29.488 INFO:tasks.workunit.client.1.smithi152.stdout:7/80: write fd [11644,85958] 0 2024-03-04T16:04:29.488 INFO:tasks.workunit.client.1.smithi152.stdout:3/83: dwrite f0 [0,4194304] 0 2024-03-04T16:04:29.489 INFO:tasks.workunit.client.1.smithi152.stdout:5/175: read d6/d8/fe [2366154,19521] 0 2024-03-04T16:04:29.491 INFO:tasks.workunit.client.1.smithi152.stdout:0/97: mknod d9/c1f 0 2024-03-04T16:04:29.494 INFO:tasks.workunit.client.1.smithi152.stdout:6/161: fsync d8/f2f 0 2024-03-04T16:04:29.497 INFO:tasks.workunit.client.1.smithi152.stdout:2/120: rename da/df/c1d to da/c2f 0 2024-03-04T16:04:29.498 INFO:tasks.workunit.client.1.smithi152.stdout:0/98: dwrite d9/fa [0,4194304] 0 2024-03-04T16:04:29.499 INFO:tasks.workunit.client.1.smithi152.stdout:8/119: creat d2/d8/f28 x:0 0 0 2024-03-04T16:04:29.499 INFO:tasks.workunit.client.1.smithi152.stdout:2/121: chown da/df/c18 6 1 2024-03-04T16:04:29.500 INFO:tasks.workunit.client.1.smithi152.stdout:8/120: fdatasync d2/d3/d21/f1f 0 2024-03-04T16:04:29.507 INFO:tasks.workunit.client.1.smithi152.stdout:4/106: mkdir d8/d9/d24/d26 0 2024-03-04T16:04:29.513 INFO:tasks.workunit.client.1.smithi152.stdout:7/81: creat d14/f16 x:0 0 0 2024-03-04T16:04:29.514 INFO:tasks.workunit.client.1.smithi152.stdout:5/176: symlink d6/d8/d10/l45 0 2024-03-04T16:04:29.517 INFO:tasks.workunit.client.1.smithi152.stdout:6/162: rmdir d8/d23 39 2024-03-04T16:04:29.520 INFO:tasks.workunit.client.1.smithi152.stdout:6/163: dread f7 [0,4194304] 0 2024-03-04T16:04:29.524 INFO:tasks.workunit.client.1.smithi152.stdout:8/121: creat d2/d3/d15/d1c/f29 x:0 0 0 2024-03-04T16:04:29.527 INFO:tasks.workunit.client.1.smithi152.stdout:4/107: symlink d8/l27 0 2024-03-04T16:04:29.529 INFO:tasks.workunit.client.1.smithi152.stdout:3/84: mknod d6/d14/c17 0 2024-03-04T16:04:29.532 INFO:tasks.workunit.client.1.smithi152.stdout:7/82: rmdir d14 39 2024-03-04T16:04:29.533 INFO:tasks.workunit.client.1.smithi152.stdout:4/108: dwrite f7 [0,4194304] 0 2024-03-04T16:04:29.539 INFO:tasks.workunit.client.1.smithi152.stdout:0/99: mknod d9/d1a/c20 0 2024-03-04T16:04:29.540 INFO:tasks.workunit.client.1.smithi152.stdout:0/100: fdatasync d9/fa 0 2024-03-04T16:04:29.541 INFO:tasks.workunit.client.1.smithi152.stdout:5/177: dwrite d6/d8/d1f/f3b [4194304,4194304] 0 2024-03-04T16:04:29.544 INFO:tasks.workunit.client.1.smithi152.stdout:6/164: unlink d8/f21 0 2024-03-04T16:04:29.546 INFO:tasks.workunit.client.1.smithi152.stdout:0/101: dread d9/fa [0,4194304] 0 2024-03-04T16:04:29.547 INFO:tasks.workunit.client.1.smithi152.stdout:2/122: creat da/df/d2e/f30 x:0 0 0 2024-03-04T16:04:29.548 INFO:tasks.workunit.client.1.smithi152.stdout:2/123: readlink da/df/l1e 0 2024-03-04T16:04:29.549 INFO:tasks.workunit.client.1.smithi152.stdout:8/122: mknod d2/d3/d15/d1c/c2a 0 2024-03-04T16:04:29.550 INFO:tasks.workunit.client.1.smithi152.stdout:6/165: read d8/d10/f16 [616680,124900] 0 2024-03-04T16:04:29.552 INFO:tasks.workunit.client.1.smithi152.stdout:6/166: write d8/f9 [1538237,49661] 0 2024-03-04T16:04:29.552 INFO:tasks.workunit.client.1.smithi152.stdout:8/123: dread - d2/d3/d15/d1c/f1d zero size 2024-03-04T16:04:29.554 INFO:tasks.workunit.client.1.smithi152.stdout:1/128: link cd d19/d20/d21/c29 0 2024-03-04T16:04:29.559 INFO:tasks.workunit.client.1.smithi152.stdout:3/85: rename d6/le to d6/d10/l18 0 2024-03-04T16:04:29.567 INFO:tasks.workunit.client.1.smithi152.stdout:0/102: unlink d9/f14 0 2024-03-04T16:04:29.569 INFO:tasks.workunit.client.1.smithi152.stdout:2/124: unlink da/f15 0 2024-03-04T16:04:29.571 INFO:tasks.workunit.client.1.smithi152.stdout:6/167: creat d8/d2a/f34 x:0 0 0 2024-03-04T16:04:29.573 INFO:tasks.workunit.client.1.smithi152.stdout:8/124: creat d2/d3/f2b x:0 0 0 2024-03-04T16:04:29.574 INFO:tasks.workunit.client.1.smithi152.stdout:6/168: write d8/d2a/f34 [880585,10290] 0 2024-03-04T16:04:29.574 INFO:tasks.workunit.client.1.smithi152.stdout:1/129: mknod d19/d20/d21/c2a 0 2024-03-04T16:04:29.575 INFO:tasks.workunit.client.1.smithi152.stdout:8/125: dread - d2/d8/f28 zero size 2024-03-04T16:04:29.575 INFO:tasks.workunit.client.1.smithi152.stdout:3/86: mknod d6/d10/c19 0 2024-03-04T16:04:29.577 INFO:tasks.workunit.client.1.smithi152.stdout:2/125: dwrite da/df/f1f [0,4194304] 0 2024-03-04T16:04:29.578 INFO:tasks.workunit.client.1.smithi152.stdout:7/83: mkdir d14/d17 0 2024-03-04T16:04:29.587 INFO:tasks.workunit.client.1.smithi152.stdout:7/84: dwrite d14/f16 [0,4194304] 0 2024-03-04T16:04:29.593 INFO:tasks.workunit.client.1.smithi152.stdout:6/169: write f1 [3696138,33965] 0 2024-03-04T16:04:29.595 INFO:tasks.workunit.client.1.smithi152.stdout:8/126: creat d2/d3/d9/f2c x:0 0 0 2024-03-04T16:04:29.596 INFO:tasks.workunit.client.1.smithi152.stdout:8/127: chown d2/d8/dd 1698 1 2024-03-04T16:04:29.600 INFO:tasks.workunit.client.1.smithi152.stdout:3/87: rename d6/f7 to d6/f1a 0 2024-03-04T16:04:29.601 INFO:tasks.workunit.client.1.smithi152.stdout:3/88: chown f0 274324 1 2024-03-04T16:04:29.603 INFO:tasks.workunit.client.1.smithi152.stdout:2/126: creat da/d24/f31 x:0 0 0 2024-03-04T16:04:29.603 INFO:tasks.workunit.client.1.smithi152.stdout:8/128: dwrite d2/d3/f13 [4194304,4194304] 0 2024-03-04T16:04:29.604 INFO:tasks.workunit.client.1.smithi152.stdout:8/129: dread - d2/d8/f28 zero size 2024-03-04T16:04:29.606 INFO:tasks.workunit.client.1.smithi152.stdout:4/109: creat d8/d9/f28 x:0 0 0 2024-03-04T16:04:29.615 INFO:tasks.workunit.client.1.smithi152.stdout:1/130: truncate f1 2641298 0 2024-03-04T16:04:29.616 INFO:tasks.workunit.client.1.smithi152.stdout:6/170: dread d8/f1f [0,4194304] 0 2024-03-04T16:04:29.616 INFO:tasks.workunit.client.1.smithi152.stdout:1/131: read f10 [240397,30099] 0 2024-03-04T16:04:29.616 INFO:tasks.workunit.client.1.smithi152.stdout:6/171: readlink d8/d10/l2d 0 2024-03-04T16:04:29.630 INFO:tasks.workunit.client.1.smithi152.stdout:5/178: truncate d6/d8/d1f/f3a 5258 0 2024-03-04T16:04:29.632 INFO:tasks.workunit.client.1.smithi152.stdout:7/85: creat d14/d17/f18 x:0 0 0 2024-03-04T16:04:29.636 INFO:tasks.workunit.client.1.smithi152.stdout:6/172: symlink d8/d10/l35 0 2024-03-04T16:04:29.640 INFO:tasks.workunit.client.1.smithi152.stdout:3/89: rename d6/d10/l18 to d6/d10/l1b 0 2024-03-04T16:04:29.640 INFO:tasks.workunit.client.1.smithi152.stdout:8/130: fsync d2/d3/d9/f2c 0 2024-03-04T16:04:29.641 INFO:tasks.workunit.client.1.smithi152.stdout:3/90: truncate d6/fc 905539 0 2024-03-04T16:04:29.642 INFO:tasks.workunit.client.1.smithi152.stdout:2/127: link da/df/f1f da/d24/f32 0 2024-03-04T16:04:29.646 INFO:tasks.workunit.client.1.smithi152.stdout:4/110: mknod d8/d9/d24/d26/c29 0 2024-03-04T16:04:29.646 INFO:tasks.workunit.client.1.smithi152.stdout:4/111: read - d8/d9/f15 zero size 2024-03-04T16:04:29.647 INFO:tasks.workunit.client.1.smithi152.stdout:8/131: dwrite d2/d3/d21/f19 [0,4194304] 0 2024-03-04T16:04:29.648 INFO:tasks.workunit.client.1.smithi152.stdout:0/103: getdents d9 0 2024-03-04T16:04:29.649 INFO:tasks.workunit.client.1.smithi152.stdout:8/132: write d2/d3/d21/f19 [3420001,68941] 0 2024-03-04T16:04:29.650 INFO:tasks.workunit.client.1.smithi152.stdout:0/104: truncate d9/f19 602958 0 2024-03-04T16:04:29.661 INFO:tasks.workunit.client.1.smithi152.stdout:3/91: creat d6/f1c x:0 0 0 2024-03-04T16:04:29.661 INFO:tasks.workunit.client.1.smithi152.stdout:6/173: dread d8/f14 [0,4194304] 0 2024-03-04T16:04:29.662 INFO:tasks.workunit.client.1.smithi152.stdout:6/174: fsync d8/d10/f24 0 2024-03-04T16:04:29.662 INFO:tasks.workunit.client.1.smithi152.stdout:3/92: write d6/fd [353668,71046] 0 2024-03-04T16:04:29.663 INFO:tasks.workunit.client.1.smithi152.stdout:3/93: chown f0 650254 1 2024-03-04T16:04:29.665 INFO:tasks.workunit.client.1.smithi152.stdout:2/128: mknod da/df/d2e/c33 0 2024-03-04T16:04:29.668 INFO:tasks.workunit.client.1.smithi152.stdout:4/112: creat d8/d9/d24/d26/f2a x:0 0 0 2024-03-04T16:04:29.669 INFO:tasks.workunit.client.1.smithi152.stdout:4/113: write d8/d9/ff [416885,104719] 0 2024-03-04T16:04:29.670 INFO:tasks.workunit.client.1.smithi152.stdout:4/114: chown d8/d9/d24 8242057 1 2024-03-04T16:04:29.671 INFO:tasks.workunit.client.1.smithi152.stdout:4/115: write d8/f10 [680023,88267] 0 2024-03-04T16:04:29.672 INFO:tasks.workunit.client.1.smithi152.stdout:5/179: getdents d6/d8/d1f/d3d 0 2024-03-04T16:04:29.674 INFO:tasks.workunit.client.1.smithi152.stdout:5/180: write d6/d8/f24 [521185,38712] 0 2024-03-04T16:04:29.675 INFO:tasks.workunit.client.1.smithi152.stdout:4/116: dread d8/d9/ff [0,4194304] 0 2024-03-04T16:04:29.675 INFO:tasks.workunit.client.1.smithi152.stdout:0/105: rmdir d9 39 2024-03-04T16:04:29.677 INFO:tasks.workunit.client.1.smithi152.stdout:4/117: write f5 [2503465,113842] 0 2024-03-04T16:04:29.677 INFO:tasks.workunit.client.1.smithi152.stdout:4/118: fsync d8/d9/f28 0 2024-03-04T16:04:29.677 INFO:tasks.workunit.client.1.smithi152.stdout:9/104: sync 2024-03-04T16:04:29.678 INFO:tasks.workunit.client.1.smithi152.stdout:9/105: chown d2/f1f 13774 1 2024-03-04T16:04:29.680 INFO:tasks.workunit.client.1.smithi152.stdout:8/133: rename d2/d3/d9 to d2/d3/d21/d2d 0 2024-03-04T16:04:29.681 INFO:tasks.workunit.client.1.smithi152.stdout:8/134: truncate d2/d8/f28 79202 0 2024-03-04T16:04:29.682 INFO:tasks.workunit.client.1.smithi152.stdout:1/132: link l12 d19/l2b 0 2024-03-04T16:04:29.686 INFO:tasks.workunit.client.1.smithi152.stdout:6/175: chown d8/d23/c32 90197147 1 2024-03-04T16:04:29.687 INFO:tasks.workunit.client.1.smithi152.stdout:3/94: creat d6/f1d x:0 0 0 2024-03-04T16:04:29.688 INFO:tasks.workunit.client.1.smithi152.stdout:3/95: chown d6/ca 75067 1 2024-03-04T16:04:29.691 INFO:tasks.workunit.client.1.smithi152.stdout:4/119: dread f0 [4194304,4194304] 0 2024-03-04T16:04:29.703 INFO:tasks.workunit.client.1.smithi152.stdout:9/106: truncate d2/d13/f15 452645 0 2024-03-04T16:04:29.706 INFO:tasks.workunit.client.1.smithi152.stdout:8/135: creat d2/d8/f2e x:0 0 0 2024-03-04T16:04:29.706 INFO:tasks.workunit.client.1.smithi152.stdout:8/136: chown d2/d3/d21 10 1 2024-03-04T16:04:29.708 INFO:tasks.workunit.client.1.smithi152.stdout:1/133: creat d19/f2c x:0 0 0 2024-03-04T16:04:29.712 INFO:tasks.workunit.client.1.smithi152.stdout:3/96: creat d6/d10/f1e x:0 0 0 2024-03-04T16:04:29.713 INFO:tasks.workunit.client.1.smithi152.stdout:3/97: truncate d6/f1d 501156 0 2024-03-04T16:04:29.713 INFO:tasks.workunit.client.1.smithi152.stdout:3/98: readlink d6/l13 0 2024-03-04T16:04:29.713 INFO:tasks.workunit.client.1.smithi152.stdout:2/129: creat da/d2c/f34 x:0 0 0 2024-03-04T16:04:29.714 INFO:tasks.workunit.client.1.smithi152.stdout:2/130: write da/df/d2e/f30 [958029,17868] 0 2024-03-04T16:04:29.715 INFO:tasks.workunit.client.1.smithi152.stdout:2/131: readlink da/df/l1e 0 2024-03-04T16:04:29.716 INFO:tasks.workunit.client.1.smithi152.stdout:4/120: mknod d8/c2b 0 2024-03-04T16:04:29.723 INFO:tasks.workunit.client.1.smithi152.stdout:9/107: readlink d2/l1d 0 2024-03-04T16:04:29.725 INFO:tasks.workunit.client.1.smithi152.stdout:0/106: write f4 [1391495,7086] 0 2024-03-04T16:04:29.726 INFO:tasks.workunit.client.1.smithi152.stdout:7/86: getdents d14 0 2024-03-04T16:04:29.727 INFO:tasks.workunit.client.1.smithi152.stdout:7/87: write fe [2186072,110163] 0 2024-03-04T16:04:29.728 INFO:tasks.workunit.client.1.smithi152.stdout:5/181: dwrite d6/d8/fe [0,4194304] 0 2024-03-04T16:04:29.728 INFO:tasks.workunit.client.1.smithi152.stdout:8/137: creat d2/d8/dd/f2f x:0 0 0 2024-03-04T16:04:29.729 INFO:tasks.workunit.client.1.smithi152.stdout:7/88: read f8 [186699,129292] 0 2024-03-04T16:04:29.731 INFO:tasks.workunit.client.1.smithi152.stdout:1/134: creat d19/d20/d21/f2d x:0 0 0 2024-03-04T16:04:29.733 INFO:tasks.workunit.client.1.smithi152.stdout:3/99: rmdir d6/d14 39 2024-03-04T16:04:29.734 INFO:tasks.workunit.client.1.smithi152.stdout:5/182: dread d6/d8/d10/f2d [0,4194304] 0 2024-03-04T16:04:29.737 INFO:tasks.workunit.client.1.smithi152.stdout:2/132: fsync da/fc 0 2024-03-04T16:04:29.739 INFO:tasks.workunit.client.1.smithi152.stdout:8/138: dwrite d2/d3/f2b [0,4194304] 0 2024-03-04T16:04:29.739 INFO:tasks.workunit.client.1.smithi152.stdout:4/121: symlink d8/d9/d24/l2c 0 2024-03-04T16:04:29.741 INFO:tasks.workunit.client.1.smithi152.stdout:9/108: mkdir d2/d13/d23 0 2024-03-04T16:04:29.745 INFO:tasks.workunit.client.1.smithi152.stdout:7/89: creat d14/f19 x:0 0 0 2024-03-04T16:04:29.745 INFO:tasks.workunit.client.1.smithi152.stdout:6/176: write d8/f1f [3041843,83131] 0 2024-03-04T16:04:29.745 INFO:tasks.workunit.client.1.smithi152.stdout:1/135: creat d19/d20/d21/f2e x:0 0 0 2024-03-04T16:04:29.750 INFO:tasks.workunit.client.1.smithi152.stdout:5/183: write d6/fd [4168020,77813] 0 2024-03-04T16:04:29.753 INFO:tasks.workunit.client.1.smithi152.stdout:7/90: dwrite fe [0,4194304] 0 2024-03-04T16:04:29.755 INFO:tasks.workunit.client.1.smithi152.stdout:2/133: rmdir da/df/d2e 39 2024-03-04T16:04:29.756 INFO:tasks.workunit.client.1.smithi152.stdout:1/136: dread d19/f1d [0,4194304] 0 2024-03-04T16:04:29.809 INFO:tasks.workunit.client.1.smithi152.stdout:4/122: mknod d8/c2d 0 2024-03-04T16:04:29.809 INFO:tasks.workunit.client.1.smithi152.stdout:8/139: symlink d2/d8/dd/l30 0 2024-03-04T16:04:29.809 INFO:tasks.workunit.client.1.smithi152.stdout:9/109: symlink d2/l24 0 2024-03-04T16:04:29.810 INFO:tasks.workunit.client.1.smithi152.stdout:0/107: creat d9/d1b/f21 x:0 0 0 2024-03-04T16:04:29.810 INFO:tasks.workunit.client.1.smithi152.stdout:3/100: creat d6/d14/f1f x:0 0 0 2024-03-04T16:04:29.811 INFO:tasks.workunit.client.1.smithi152.stdout:5/184: creat d6/d8/d10/f46 x:0 0 0 2024-03-04T16:04:29.812 INFO:tasks.workunit.client.1.smithi152.stdout:1/137: mkdir d19/d20/d2f 0 2024-03-04T16:04:29.813 INFO:tasks.workunit.client.1.smithi152.stdout:3/101: truncate d6/f1c 386612 0 2024-03-04T16:04:29.813 INFO:tasks.workunit.client.1.smithi152.stdout:5/185: read - d6/d8/f41 zero size 2024-03-04T16:04:29.814 INFO:tasks.workunit.client.1.smithi152.stdout:3/102: chown d6/d10/c19 62180253 1 2024-03-04T16:04:29.814 INFO:tasks.workunit.client.1.smithi152.stdout:0/108: unlink d9/d1b/f1e 0 2024-03-04T16:04:29.815 INFO:tasks.workunit.client.1.smithi152.stdout:4/123: creat d8/d9/d24/d26/f2e x:0 0 0 2024-03-04T16:04:29.816 INFO:tasks.workunit.client.1.smithi152.stdout:7/91: unlink l13 0 2024-03-04T16:04:29.816 INFO:tasks.workunit.client.1.smithi152.stdout:9/110: dread d2/d13/f15 [0,4194304] 0 2024-03-04T16:04:29.817 INFO:tasks.workunit.client.1.smithi152.stdout:3/103: write d6/f8 [964289,4460] 0 2024-03-04T16:04:29.817 INFO:tasks.workunit.client.1.smithi152.stdout:2/134: mkdir da/df/d35 0 2024-03-04T16:04:29.818 INFO:tasks.workunit.client.1.smithi152.stdout:3/104: stat d6/f9 0 2024-03-04T16:04:29.819 INFO:tasks.workunit.client.1.smithi152.stdout:8/140: mkdir d2/d3/d21/d2d/d31 0 2024-03-04T16:04:29.821 INFO:tasks.workunit.client.1.smithi152.stdout:7/92: dread f8 [4194304,4194304] 0 2024-03-04T16:04:29.824 INFO:tasks.workunit.client.1.smithi152.stdout:1/138: read f16 [3985942,51251] 0 2024-03-04T16:04:29.826 INFO:tasks.workunit.client.1.smithi152.stdout:0/109: mkdir d9/d22 0 2024-03-04T16:04:29.826 INFO:tasks.workunit.client.1.smithi152.stdout:0/110: write f4 [2164649,65353] 0 2024-03-04T16:04:29.828 INFO:tasks.workunit.client.1.smithi152.stdout:4/124: chown d8/c18 8331 1 2024-03-04T16:04:29.829 INFO:tasks.workunit.client.1.smithi152.stdout:4/125: chown d8/f10 3 1 2024-03-04T16:04:29.832 INFO:tasks.workunit.client.1.smithi152.stdout:3/105: creat d6/d10/f20 x:0 0 0 2024-03-04T16:04:29.834 INFO:tasks.workunit.client.1.smithi152.stdout:8/141: creat d2/d3/d21/f32 x:0 0 0 2024-03-04T16:04:29.837 INFO:tasks.workunit.client.1.smithi152.stdout:4/126: dwrite d8/d9/f28 [0,4194304] 0 2024-03-04T16:04:29.838 INFO:tasks.workunit.client.1.smithi152.stdout:4/127: chown d8/d9 88886 1 2024-03-04T16:04:29.839 INFO:tasks.workunit.client.1.smithi152.stdout:5/186: symlink d6/l47 0 2024-03-04T16:04:29.841 INFO:tasks.workunit.client.1.smithi152.stdout:1/139: unlink d19/f1f 0 2024-03-04T16:04:29.842 INFO:tasks.workunit.client.1.smithi152.stdout:5/187: truncate d6/d8/d1f/f44 936530 0 2024-03-04T16:04:29.842 INFO:tasks.workunit.client.1.smithi152.stdout:1/140: write f2 [8685885,51806] 0 2024-03-04T16:04:29.843 INFO:tasks.workunit.client.1.smithi152.stdout:5/188: chown l5 39516868 1 2024-03-04T16:04:29.844 INFO:tasks.workunit.client.1.smithi152.stdout:0/111: creat d9/d1b/f23 x:0 0 0 2024-03-04T16:04:29.845 INFO:tasks.workunit.client.1.smithi152.stdout:2/135: rmdir da/df/d2e 39 2024-03-04T16:04:29.845 INFO:tasks.workunit.client.1.smithi152.stdout:8/142: dwrite d2/d3/f2b [0,4194304] 0 2024-03-04T16:04:29.845 INFO:tasks.workunit.client.1.smithi152.stdout:5/189: stat d6/d8/d1f/f2e 0 2024-03-04T16:04:29.848 INFO:tasks.workunit.client.1.smithi152.stdout:2/136: write da/d2c/f34 [438692,58724] 0 2024-03-04T16:04:29.849 INFO:tasks.workunit.client.1.smithi152.stdout:5/190: chown d6/d8/f41 50288 1 2024-03-04T16:04:29.851 INFO:tasks.workunit.client.1.smithi152.stdout:3/106: rename d6/f1d to d6/d10/f21 0 2024-03-04T16:04:29.853 INFO:tasks.workunit.client.1.smithi152.stdout:1/141: dwrite f2 [0,4194304] 0 2024-03-04T16:04:29.863 INFO:tasks.workunit.client.1.smithi152.stdout:9/111: link d2/c14 d2/d13/c25 0 2024-03-04T16:04:29.869 INFO:tasks.workunit.client.1.smithi152.stdout:5/191: rename d6/d8/d1f/f25 to d6/d8/d1f/f48 0 2024-03-04T16:04:29.870 INFO:tasks.workunit.client.1.smithi152.stdout:3/107: write d6/f9 [264522,86554] 0 2024-03-04T16:04:29.871 INFO:tasks.workunit.client.1.smithi152.stdout:3/108: stat d6/f12 0 2024-03-04T16:04:29.873 INFO:tasks.workunit.client.1.smithi152.stdout:8/143: rmdir d2/d3 39 2024-03-04T16:04:29.879 INFO:tasks.workunit.client.1.smithi152.stdout:1/142: dread f10 [0,4194304] 0 2024-03-04T16:04:29.881 INFO:tasks.workunit.client.1.smithi152.stdout:2/137: mknod da/df/d35/c36 0 2024-03-04T16:04:29.881 INFO:tasks.workunit.client.1.smithi152.stdout:7/93: dwrite f8 [0,4194304] 0 2024-03-04T16:04:29.885 INFO:tasks.workunit.client.1.smithi152.stdout:5/192: chown d6/d8/d1f/f3a 0 1 2024-03-04T16:04:29.893 INFO:tasks.workunit.client.1.smithi152.stdout:0/112: rmdir d9/d22 0 2024-03-04T16:04:29.896 INFO:tasks.workunit.client.1.smithi152.stdout:1/143: creat d19/d20/f30 x:0 0 0 2024-03-04T16:04:29.898 INFO:tasks.workunit.client.1.smithi152.stdout:9/112: mknod d2/d13/d23/c26 0 2024-03-04T16:04:29.899 INFO:tasks.workunit.client.1.smithi152.stdout:9/113: write d2/d13/f18 [1527574,65360] 0 2024-03-04T16:04:29.900 INFO:tasks.workunit.client.1.smithi152.stdout:9/114: chown d2/f19 1599 1 2024-03-04T16:04:29.901 INFO:tasks.workunit.client.1.smithi152.stdout:7/94: mknod d14/c1a 0 2024-03-04T16:04:29.905 INFO:tasks.workunit.client.1.smithi152.stdout:3/109: symlink d6/l22 0 2024-03-04T16:04:29.906 INFO:tasks.workunit.client.1.smithi152.stdout:3/110: truncate f3 207686 0 2024-03-04T16:04:29.907 INFO:tasks.workunit.client.1.smithi152.stdout:7/95: dwrite fa [0,4194304] 0 2024-03-04T16:04:29.907 INFO:tasks.workunit.client.1.smithi152.stdout:3/111: dread - d6/d10/f1e zero size 2024-03-04T16:04:29.909 INFO:tasks.workunit.client.1.smithi152.stdout:8/144: mkdir d2/d3/d21/d33 0 2024-03-04T16:04:29.910 INFO:tasks.workunit.client.1.smithi152.stdout:4/128: getdents d8/d9/d24/d26 0 2024-03-04T16:04:29.913 INFO:tasks.workunit.client.1.smithi152.stdout:3/112: dread d6/f12 [0,4194304] 0 2024-03-04T16:04:29.923 INFO:tasks.workunit.client.1.smithi152.stdout:8/145: chown d2/d8/l12 1 1 2024-03-04T16:04:29.925 INFO:tasks.workunit.client.1.smithi152.stdout:5/193: dread d6/d8/d10/f21 [0,4194304] 0 2024-03-04T16:04:29.925 INFO:tasks.workunit.client.1.smithi152.stdout:5/194: stat d6/fd 0 2024-03-04T16:04:29.928 INFO:tasks.workunit.client.1.smithi152.stdout:5/195: write d6/d8/d1f/f3b [4433299,83095] 0 2024-03-04T16:04:29.938 INFO:tasks.workunit.client.1.smithi152.stdout:1/144: unlink c9 0 2024-03-04T16:04:29.938 INFO:tasks.workunit.client.1.smithi152.stdout:1/145: fdatasync d19/d20/f30 0 2024-03-04T16:04:29.939 INFO:tasks.workunit.client.1.smithi152.stdout:2/138: creat da/df/f37 x:0 0 0 2024-03-04T16:04:29.951 INFO:tasks.workunit.client.1.smithi152.stdout:5/196: creat d6/d8/d10/f49 x:0 0 0 2024-03-04T16:04:29.953 INFO:tasks.workunit.client.1.smithi152.stdout:5/197: write d6/fd [4757699,85532] 0 2024-03-04T16:04:29.954 INFO:tasks.workunit.client.1.smithi152.stdout:5/198: dread - d6/d8/d1f/f34 zero size 2024-03-04T16:04:29.955 INFO:tasks.workunit.client.1.smithi152.stdout:8/146: dread d2/d3/f13 [0,4194304] 0 2024-03-04T16:04:29.955 INFO:tasks.workunit.client.1.smithi152.stdout:1/146: symlink d19/d20/l31 0 2024-03-04T16:04:29.955 INFO:tasks.workunit.client.1.smithi152.stdout:5/199: readlink d6/d8/d10/l45 0 2024-03-04T16:04:29.956 INFO:tasks.workunit.client.1.smithi152.stdout:8/147: stat d2/d3/d21/f1b 0 2024-03-04T16:04:29.956 INFO:tasks.workunit.client.1.smithi152.stdout:8/148: chown d2/d3/d21/d2d/d31 2510540 1 2024-03-04T16:04:29.962 INFO:tasks.workunit.client.1.smithi152.stdout:5/200: dwrite d6/d8/d1f/f44 [0,4194304] 0 2024-03-04T16:04:29.963 INFO:tasks.workunit.client.1.smithi152.stdout:1/147: creat d19/f32 x:0 0 0 2024-03-04T16:04:29.968 INFO:tasks.workunit.client.1.smithi152.stdout:2/139: creat da/df/d2e/f38 x:0 0 0 2024-03-04T16:04:29.969 INFO:tasks.workunit.client.1.smithi152.stdout:5/201: dwrite d6/d8/d10/f46 [0,4194304] 0 2024-03-04T16:04:29.981 INFO:tasks.workunit.client.1.smithi152.stdout:3/113: getdents d6 0 2024-03-04T16:04:29.984 INFO:tasks.workunit.client.1.smithi152.stdout:2/140: dread da/d24/f32 [0,4194304] 0 2024-03-04T16:04:29.984 INFO:tasks.workunit.client.1.smithi152.stdout:5/202: unlink d6/d8/l22 0 2024-03-04T16:04:29.988 INFO:tasks.workunit.client.1.smithi152.stdout:1/148: rename lb to d19/d20/d2f/l33 0 2024-03-04T16:04:29.993 INFO:tasks.workunit.client.1.smithi152.stdout:3/114: rename d6/fd to d6/f23 0 2024-03-04T16:04:29.993 INFO:tasks.workunit.client.1.smithi152.stdout:2/141: creat da/df/d2e/f39 x:0 0 0 2024-03-04T16:04:29.996 INFO:tasks.workunit.client.1.smithi152.stdout:4/129: write f0 [4020408,67855] 0 2024-03-04T16:04:29.998 INFO:tasks.workunit.client.1.smithi152.stdout:0/113: truncate f4 1185805 0 2024-03-04T16:04:29.999 INFO:tasks.workunit.client.1.smithi152.stdout:1/149: mknod d19/d20/c34 0 2024-03-04T16:04:30.000 INFO:tasks.workunit.client.1.smithi152.stdout:1/150: truncate f2 9863715 0 2024-03-04T16:04:30.000 INFO:tasks.workunit.client.1.smithi152.stdout:1/151: dread - d19/f2c zero size 2024-03-04T16:04:30.003 INFO:tasks.workunit.client.1.smithi152.stdout:7/96: truncate fe 1368288 0 2024-03-04T16:04:30.004 INFO:tasks.workunit.client.1.smithi152.stdout:7/97: fdatasync fa 0 2024-03-04T16:04:30.010 INFO:tasks.workunit.client.1.smithi152.stdout:5/203: unlink d6/c28 0 2024-03-04T16:04:30.013 INFO:tasks.workunit.client.1.smithi152.stdout:4/130: read d8/d9/ff [340630,60191] 0 2024-03-04T16:04:30.014 INFO:tasks.workunit.client.1.smithi152.stdout:4/131: dread - d8/d9/d24/d26/f2e zero size 2024-03-04T16:04:30.015 INFO:tasks.workunit.client.1.smithi152.stdout:0/114: symlink d9/d1a/l24 0 2024-03-04T16:04:30.016 INFO:tasks.workunit.client.1.smithi152.stdout:1/152: rmdir d19/d20/d2f 39 2024-03-04T16:04:30.018 INFO:tasks.workunit.client.1.smithi152.stdout:8/149: truncate d2/d3/d21/f19 322505 0 2024-03-04T16:04:30.019 INFO:tasks.workunit.client.1.smithi152.stdout:8/150: fsync d2/d3/d21/f1b 0 2024-03-04T16:04:30.024 INFO:tasks.workunit.client.1.smithi152.stdout:4/132: mkdir d8/d2f 0 2024-03-04T16:04:30.028 INFO:tasks.workunit.client.1.smithi152.stdout:0/115: creat d9/f25 x:0 0 0 2024-03-04T16:04:30.028 INFO:tasks.workunit.client.1.smithi152.stdout:0/116: stat d9/l18 0 2024-03-04T16:04:30.031 INFO:tasks.workunit.client.1.smithi152.stdout:1/153: rmdir d19/d20/d21 39 2024-03-04T16:04:30.032 INFO:tasks.workunit.client.1.smithi152.stdout:1/154: readlink l13 0 2024-03-04T16:04:30.037 INFO:tasks.workunit.client.1.smithi152.stdout:8/151: symlink d2/d3/l34 0 2024-03-04T16:04:30.040 INFO:tasks.workunit.client.1.smithi152.stdout:7/98: dread f4 [0,4194304] 0 2024-03-04T16:04:30.043 INFO:tasks.workunit.client.1.smithi152.stdout:6/177: sync 2024-03-04T16:04:30.050 INFO:tasks.workunit.client.1.smithi152.stdout:6/178: dwrite d8/d23/f2c [0,4194304] 0 2024-03-04T16:04:30.051 INFO:tasks.workunit.client.1.smithi152.stdout:6/179: stat d8/d2a/f34 0 2024-03-04T16:04:30.054 INFO:tasks.workunit.client.1.smithi152.stdout:4/133: getdents d8/d9/d24/d26 0 2024-03-04T16:04:30.055 INFO:tasks.workunit.client.1.smithi152.stdout:4/134: read - d8/d9/f15 zero size 2024-03-04T16:04:30.058 INFO:tasks.workunit.client.1.smithi152.stdout:6/180: dwrite d8/f1f [0,4194304] 0 2024-03-04T16:04:30.061 INFO:tasks.workunit.client.1.smithi152.stdout:3/115: symlink d6/d14/l24 0 2024-03-04T16:04:30.069 INFO:tasks.workunit.client.1.smithi152.stdout:7/99: mknod d14/d17/c1b 0 2024-03-04T16:04:30.069 INFO:tasks.workunit.client.1.smithi152.stdout:4/135: dwrite f7 [0,4194304] 0 2024-03-04T16:04:30.071 INFO:tasks.workunit.client.1.smithi152.stdout:4/136: write d8/f16 [543885,20205] 0 2024-03-04T16:04:30.071 INFO:tasks.workunit.client.1.smithi152.stdout:2/142: dwrite da/f1c [0,4194304] 0 2024-03-04T16:04:30.073 INFO:tasks.workunit.client.1.smithi152.stdout:2/143: write da/fb [492398,82170] 0 2024-03-04T16:04:30.082 INFO:tasks.workunit.client.1.smithi152.stdout:8/152: creat d2/d3/d21/d2d/d31/f35 x:0 0 0 2024-03-04T16:04:30.085 INFO:tasks.workunit.client.1.smithi152.stdout:1/155: dwrite f16 [4194304,4194304] 0 2024-03-04T16:04:30.086 INFO:tasks.workunit.client.1.smithi152.stdout:1/156: readlink l13 0 2024-03-04T16:04:30.087 INFO:tasks.workunit.client.1.smithi152.stdout:6/181: creat d8/d2a/f36 x:0 0 0 2024-03-04T16:04:30.089 INFO:tasks.workunit.client.1.smithi152.stdout:8/153: dread d2/d3/d21/d2d/fc [0,4194304] 0 2024-03-04T16:04:30.089 INFO:tasks.workunit.client.1.smithi152.stdout:6/182: read d8/d10/f16 [407246,109122] 0 2024-03-04T16:04:30.090 INFO:tasks.workunit.client.1.smithi152.stdout:8/154: dread - d2/d3/d21/f1f zero size 2024-03-04T16:04:30.091 INFO:tasks.workunit.client.1.smithi152.stdout:6/183: fdatasync d8/d2a/f34 0 2024-03-04T16:04:30.091 INFO:tasks.workunit.client.1.smithi152.stdout:7/100: creat d14/f1c x:0 0 0 2024-03-04T16:04:30.092 INFO:tasks.workunit.client.1.smithi152.stdout:4/137: creat d8/f30 x:0 0 0 2024-03-04T16:04:30.095 INFO:tasks.workunit.client.1.smithi152.stdout:4/138: write d8/f25 [227818,4639] 0 2024-03-04T16:04:30.098 INFO:tasks.workunit.client.1.smithi152.stdout:3/116: rmdir d6/d14 39 2024-03-04T16:04:30.102 INFO:tasks.workunit.client.1.smithi152.stdout:4/139: truncate d8/f25 750410 0 2024-03-04T16:04:30.104 INFO:tasks.workunit.client.1.smithi152.stdout:6/184: dwrite d8/d2a/f2e [0,4194304] 0 2024-03-04T16:04:30.105 INFO:tasks.workunit.client.1.smithi152.stdout:4/140: write f5 [2106819,53649] 0 2024-03-04T16:04:30.105 INFO:tasks.workunit.client.1.smithi152.stdout:8/155: dwrite d2/d8/f1a [0,4194304] 0 2024-03-04T16:04:30.107 INFO:tasks.workunit.client.1.smithi152.stdout:8/156: write d2/d8/f1a [1261871,85871] 0 2024-03-04T16:04:30.108 INFO:tasks.workunit.client.1.smithi152.stdout:7/101: truncate f4 1478933 0 2024-03-04T16:04:30.109 INFO:tasks.workunit.client.1.smithi152.stdout:8/157: readlink d2/d3/la 0 2024-03-04T16:04:30.112 INFO:tasks.workunit.client.1.smithi152.stdout:8/158: write d2/d3/d21/d2d/f1e [3265668,125799] 0 2024-03-04T16:04:30.119 INFO:tasks.workunit.client.1.smithi152.stdout:4/141: dwrite d8/f16 [0,4194304] 0 2024-03-04T16:04:30.121 INFO:tasks.workunit.client.1.smithi152.stdout:7/102: dwrite d14/d17/f18 [0,4194304] 0 2024-03-04T16:04:30.121 INFO:tasks.workunit.client.1.smithi152.stdout:6/185: mknod d8/d2a/c37 0 2024-03-04T16:04:30.122 INFO:tasks.workunit.client.1.smithi152.stdout:6/186: fsync d8/f2f 0 2024-03-04T16:04:30.133 INFO:tasks.workunit.client.1.smithi152.stdout:8/159: symlink d2/d8/dd/d22/l36 0 2024-03-04T16:04:30.140 INFO:tasks.workunit.client.1.smithi152.stdout:2/144: truncate da/f11 3708806 0 2024-03-04T16:04:30.140 INFO:tasks.workunit.client.1.smithi152.stdout:1/157: dwrite fa [0,4194304] 0 2024-03-04T16:04:30.140 INFO:tasks.workunit.client.1.smithi152.stdout:7/103: mknod d14/c1d 0 2024-03-04T16:04:30.142 INFO:tasks.workunit.client.1.smithi152.stdout:7/104: readlink l3 0 2024-03-04T16:04:30.142 INFO:tasks.workunit.client.1.smithi152.stdout:2/145: dread - da/df/f14 zero size 2024-03-04T16:04:30.143 INFO:tasks.workunit.client.1.smithi152.stdout:3/117: dwrite d6/d10/f1e [0,4194304] 0 2024-03-04T16:04:30.145 INFO:tasks.workunit.client.1.smithi152.stdout:7/105: read - d14/f15 zero size 2024-03-04T16:04:30.145 INFO:tasks.workunit.client.1.smithi152.stdout:2/146: stat c8 0 2024-03-04T16:04:30.146 INFO:tasks.workunit.client.1.smithi152.stdout:8/160: dread d2/d3/f13 [0,4194304] 0 2024-03-04T16:04:30.148 INFO:tasks.workunit.client.1.smithi152.stdout:0/117: write f4 [1455261,33032] 0 2024-03-04T16:04:30.152 INFO:tasks.workunit.client.1.smithi152.stdout:8/161: chown d2/d8/dd/d22/l36 15516 1 2024-03-04T16:04:30.152 INFO:tasks.workunit.client.1.smithi152.stdout:0/118: write d9/d1b/f23 [224628,81412] 0 2024-03-04T16:04:30.154 INFO:tasks.workunit.client.1.smithi152.stdout:6/187: mkdir d8/d38 0 2024-03-04T16:04:30.154 INFO:tasks.workunit.client.1.smithi152.stdout:6/188: readlink d8/d23/l29 0 2024-03-04T16:04:30.157 INFO:tasks.workunit.client.1.smithi152.stdout:4/142: write d8/d9/ff [503241,125335] 0 2024-03-04T16:04:30.158 INFO:tasks.workunit.client.1.smithi152.stdout:3/118: read d6/f8 [766721,107243] 0 2024-03-04T16:04:30.158 INFO:tasks.workunit.client.1.smithi152.stdout:4/143: dread - d8/f23 zero size 2024-03-04T16:04:30.161 INFO:tasks.workunit.client.1.smithi152.stdout:9/115: sync 2024-03-04T16:04:30.163 INFO:tasks.workunit.client.1.smithi152.stdout:8/162: unlink d2/d3/l16 0 2024-03-04T16:04:30.166 INFO:tasks.workunit.client.1.smithi152.stdout:7/106: fsync fe 0 2024-03-04T16:04:30.170 INFO:tasks.workunit.client.1.smithi152.stdout:2/147: link da/d24/f2a da/df/d2e/f3a 0 2024-03-04T16:04:30.172 INFO:tasks.workunit.client.1.smithi152.stdout:1/158: dread f17 [0,4194304] 0 2024-03-04T16:04:30.175 INFO:tasks.workunit.client.1.smithi152.stdout:4/144: symlink d8/d9/d24/d26/l31 0 2024-03-04T16:04:30.177 INFO:tasks.workunit.client.1.smithi152.stdout:9/116: mknod d2/d13/d23/c27 0 2024-03-04T16:04:30.183 INFO:tasks.workunit.client.1.smithi152.stdout:1/159: chown d19/l2b 10 1 2024-03-04T16:04:30.184 INFO:tasks.workunit.client.1.smithi152.stdout:5/204: sync 2024-03-04T16:04:30.185 INFO:tasks.workunit.client.1.smithi152.stdout:1/160: write d19/f23 [2333127,87649] 0 2024-03-04T16:04:30.185 INFO:tasks.workunit.client.1.smithi152.stdout:3/119: mknod d6/d14/c25 0 2024-03-04T16:04:30.185 INFO:tasks.workunit.client.1.smithi152.stdout:5/205: dread - d6/d8/d10/f49 zero size 2024-03-04T16:04:30.188 INFO:tasks.workunit.client.1.smithi152.stdout:4/145: unlink d8/d9/d24/d26/c29 0 2024-03-04T16:04:30.188 INFO:tasks.workunit.client.1.smithi152.stdout:4/146: read - d8/f20 zero size 2024-03-04T16:04:30.191 INFO:tasks.workunit.client.1.smithi152.stdout:9/117: creat d2/f28 x:0 0 0 2024-03-04T16:04:30.191 INFO:tasks.workunit.client.1.smithi152.stdout:9/118: stat d2/f1c 0 2024-03-04T16:04:30.193 INFO:tasks.workunit.client.1.smithi152.stdout:9/119: dread d2/d13/f1a [0,4194304] 0 2024-03-04T16:04:30.193 INFO:tasks.workunit.client.1.smithi152.stdout:6/189: rename d8/f14 to d8/f39 0 2024-03-04T16:04:30.202 INFO:tasks.workunit.client.1.smithi152.stdout:4/147: chown d8/d9/c21 17 1 2024-03-04T16:04:30.203 INFO:tasks.workunit.client.1.smithi152.stdout:0/119: getdents d9/d1b 0 2024-03-04T16:04:30.203 INFO:tasks.workunit.client.1.smithi152.stdout:2/148: write da/f11 [3164484,62664] 0 2024-03-04T16:04:30.205 INFO:tasks.workunit.client.1.smithi152.stdout:3/120: dwrite d6/f1c [0,4194304] 0 2024-03-04T16:04:30.205 INFO:tasks.workunit.client.1.smithi152.stdout:0/120: fsync d9/d1b/f23 0 2024-03-04T16:04:30.206 INFO:tasks.workunit.client.1.smithi152.stdout:8/163: rename d2/d3/f13 to d2/f37 0 2024-03-04T16:04:30.207 INFO:tasks.workunit.client.1.smithi152.stdout:4/148: dread d8/d9/ff [0,4194304] 0 2024-03-04T16:04:30.208 INFO:tasks.workunit.client.1.smithi152.stdout:0/121: readlink d9/d1a/l24 0 2024-03-04T16:04:30.208 INFO:tasks.workunit.client.1.smithi152.stdout:3/121: write f0 [2906645,31989] 0 2024-03-04T16:04:30.210 INFO:tasks.workunit.client.1.smithi152.stdout:9/120: rmdir d2 39 2024-03-04T16:04:30.210 INFO:tasks.workunit.client.1.smithi152.stdout:6/190: mkdir d8/d23/d3a 0 2024-03-04T16:04:30.211 INFO:tasks.workunit.client.1.smithi152.stdout:8/164: truncate d2/d3/d21/f32 331401 0 2024-03-04T16:04:30.212 INFO:tasks.workunit.client.1.smithi152.stdout:4/149: write d8/d9/d24/d26/f2e [334941,8397] 0 2024-03-04T16:04:30.218 INFO:tasks.workunit.client.1.smithi152.stdout:2/149: dread da/f1c [0,4194304] 0 2024-03-04T16:04:30.221 INFO:tasks.workunit.client.1.smithi152.stdout:6/191: dwrite d8/f28 [0,4194304] 0 2024-03-04T16:04:30.223 INFO:tasks.workunit.client.1.smithi152.stdout:7/107: rename fe to d14/f1e 0 2024-03-04T16:04:30.225 INFO:tasks.workunit.client.1.smithi152.stdout:3/122: fdatasync d6/ff 0 2024-03-04T16:04:30.225 INFO:tasks.workunit.client.1.smithi152.stdout:7/108: chown d14/c1d 222557687 1 2024-03-04T16:04:30.226 INFO:tasks.workunit.client.1.smithi152.stdout:0/122: creat d9/f26 x:0 0 0 2024-03-04T16:04:30.229 INFO:tasks.workunit.client.1.smithi152.stdout:2/150: dwrite da/f1a [0,4194304] 0 2024-03-04T16:04:30.230 INFO:tasks.workunit.client.1.smithi152.stdout:0/123: dread - d9/d1b/f21 zero size 2024-03-04T16:04:30.230 INFO:tasks.workunit.client.1.smithi152.stdout:3/123: write d6/d10/f1e [313155,18459] 0 2024-03-04T16:04:30.232 INFO:tasks.workunit.client.1.smithi152.stdout:2/151: chown da/f16 831 1 2024-03-04T16:04:30.233 INFO:tasks.workunit.client.1.smithi152.stdout:5/206: creat d6/f4a x:0 0 0 2024-03-04T16:04:30.234 INFO:tasks.workunit.client.1.smithi152.stdout:0/124: chown d9/f25 12 1 2024-03-04T16:04:30.234 INFO:tasks.workunit.client.1.smithi152.stdout:4/150: creat d8/d9/d24/d26/f32 x:0 0 0 2024-03-04T16:04:30.234 INFO:tasks.workunit.client.1.smithi152.stdout:5/207: readlink d6/d8/l33 0 2024-03-04T16:04:30.236 INFO:tasks.workunit.client.1.smithi152.stdout:6/192: dwrite d8/d2a/f36 [0,4194304] 0 2024-03-04T16:04:30.236 INFO:tasks.workunit.client.1.smithi152.stdout:0/125: write f7 [2013215,55287] 0 2024-03-04T16:04:30.241 INFO:tasks.workunit.client.1.smithi152.stdout:7/109: rename fa to d14/f1f 0 2024-03-04T16:04:30.245 INFO:tasks.workunit.client.1.smithi152.stdout:3/124: symlink d6/d14/l26 0 2024-03-04T16:04:30.249 INFO:tasks.workunit.client.1.smithi152.stdout:7/110: dread fd [0,4194304] 0 2024-03-04T16:04:30.249 INFO:tasks.workunit.client.1.smithi152.stdout:8/165: link d2/d3/d21/d2d/f1e d2/d8/dd/d22/d26/f38 0 2024-03-04T16:04:30.250 INFO:tasks.workunit.client.1.smithi152.stdout:7/111: chown d14/c1a 12 1 2024-03-04T16:04:30.251 INFO:tasks.workunit.client.1.smithi152.stdout:9/121: truncate d2/d13/f1a 1109115 0 2024-03-04T16:04:30.254 INFO:tasks.workunit.client.1.smithi152.stdout:6/193: mknod d8/d10/c3b 0 2024-03-04T16:04:30.255 INFO:tasks.workunit.client.1.smithi152.stdout:0/126: symlink d9/d1b/l27 0 2024-03-04T16:04:30.261 INFO:tasks.workunit.client.1.smithi152.stdout:2/152: rename da/d24/l29 to da/df/d2e/l3b 0 2024-03-04T16:04:30.262 INFO:tasks.workunit.client.1.smithi152.stdout:1/161: link l11 d19/d20/d21/l35 0 2024-03-04T16:04:30.262 INFO:tasks.workunit.client.1.smithi152.stdout:0/127: dwrite d9/f16 [0,4194304] 0 2024-03-04T16:04:30.264 INFO:tasks.workunit.client.1.smithi152.stdout:8/166: creat d2/d8/f39 x:0 0 0 2024-03-04T16:04:30.265 INFO:tasks.workunit.client.1.smithi152.stdout:8/167: chown d2/d8/l12 0 1 2024-03-04T16:04:30.266 INFO:tasks.workunit.client.1.smithi152.stdout:7/112: creat d14/f20 x:0 0 0 2024-03-04T16:04:30.271 INFO:tasks.workunit.client.1.smithi152.stdout:6/194: creat d8/d2a/f3c x:0 0 0 2024-03-04T16:04:30.273 INFO:tasks.workunit.client.1.smithi152.stdout:4/151: dwrite d8/d9/ff [0,4194304] 0 2024-03-04T16:04:30.274 INFO:tasks.workunit.client.1.smithi152.stdout:2/153: creat da/f3c x:0 0 0 2024-03-04T16:04:30.275 INFO:tasks.workunit.client.1.smithi152.stdout:4/152: write f0 [3434874,7841] 0 2024-03-04T16:04:30.277 INFO:tasks.workunit.client.1.smithi152.stdout:0/128: mknod d9/d1a/c28 0 2024-03-04T16:04:30.277 INFO:tasks.workunit.client.1.smithi152.stdout:7/113: dwrite d14/f1c [0,4194304] 0 2024-03-04T16:04:30.278 INFO:tasks.workunit.client.1.smithi152.stdout:7/114: read d14/f16 [1656935,51534] 0 2024-03-04T16:04:30.279 INFO:tasks.workunit.client.1.smithi152.stdout:8/168: unlink d2/d3/d15/d1c/f29 0 2024-03-04T16:04:30.279 INFO:tasks.workunit.client.1.smithi152.stdout:7/115: read - d14/f19 zero size 2024-03-04T16:04:30.280 INFO:tasks.workunit.client.1.smithi152.stdout:0/129: write d9/f26 [737110,50845] 0 2024-03-04T16:04:30.281 INFO:tasks.workunit.client.1.smithi152.stdout:8/169: write d2/d3/d21/f1b [510361,4930] 0 2024-03-04T16:04:30.283 INFO:tasks.workunit.client.1.smithi152.stdout:5/208: creat d6/d8/f4b x:0 0 0 2024-03-04T16:04:30.283 INFO:tasks.workunit.client.1.smithi152.stdout:8/170: chown d2/d3/d21/f32 0 1 2024-03-04T16:04:30.286 INFO:tasks.workunit.client.1.smithi152.stdout:6/195: symlink d8/d23/l3d 0 2024-03-04T16:04:30.287 INFO:tasks.workunit.client.1.smithi152.stdout:6/196: dread - d8/d2a/f3c zero size 2024-03-04T16:04:30.290 INFO:tasks.workunit.client.1.smithi152.stdout:3/125: rename d6/f12 to d6/d10/f27 0 2024-03-04T16:04:30.292 INFO:tasks.workunit.client.1.smithi152.stdout:3/126: write f0 [1658343,116238] 0 2024-03-04T16:04:30.294 INFO:tasks.workunit.client.1.smithi152.stdout:3/127: write d6/d10/f20 [807033,57924] 0 2024-03-04T16:04:30.296 INFO:tasks.workunit.client.1.smithi152.stdout:3/128: write f0 [4307468,124318] 0 2024-03-04T16:04:30.297 INFO:tasks.workunit.client.1.smithi152.stdout:6/197: dwrite d8/d10/f24 [0,4194304] 0 2024-03-04T16:04:30.297 INFO:tasks.workunit.client.1.smithi152.stdout:7/116: mkdir d14/d21 0 2024-03-04T16:04:30.297 INFO:tasks.workunit.client.1.smithi152.stdout:7/117: dread - d14/f19 zero size 2024-03-04T16:04:30.298 INFO:tasks.workunit.client.1.smithi152.stdout:6/198: fsync d8/d23/f25 0 2024-03-04T16:04:30.298 INFO:tasks.workunit.client.1.smithi152.stdout:0/130: unlink d9/f17 0 2024-03-04T16:04:30.302 INFO:tasks.workunit.client.1.smithi152.stdout:4/153: write d8/f16 [5159488,12155] 0 2024-03-04T16:04:30.304 INFO:tasks.workunit.client.1.smithi152.stdout:9/122: rename d2/f1f to d2/d13/d23/f29 0 2024-03-04T16:04:30.310 INFO:tasks.workunit.client.1.smithi152.stdout:7/118: unlink c10 0 2024-03-04T16:04:30.314 INFO:tasks.workunit.client.1.smithi152.stdout:6/199: creat d8/d2a/f3e x:0 0 0 2024-03-04T16:04:30.314 INFO:tasks.workunit.client.1.smithi152.stdout:9/123: dwrite d2/fb [0,4194304] 0 2024-03-04T16:04:30.314 INFO:tasks.workunit.client.1.smithi152.stdout:6/200: chown d8/f2f 128486 1 2024-03-04T16:04:30.316 INFO:tasks.workunit.client.1.smithi152.stdout:4/154: fdatasync d8/f16 0 2024-03-04T16:04:30.317 INFO:tasks.workunit.client.1.smithi152.stdout:9/124: chown d2/d13/c22 1367311884 1 2024-03-04T16:04:30.319 INFO:tasks.workunit.client.1.smithi152.stdout:8/171: fsync d2/d3/d21/f19 0 2024-03-04T16:04:30.321 INFO:tasks.workunit.client.1.smithi152.stdout:5/209: link d6/d8/f32 d6/d8/d1f/d3d/f4c 0 2024-03-04T16:04:30.322 INFO:tasks.workunit.client.1.smithi152.stdout:5/210: stat d6/d8/f1d 0 2024-03-04T16:04:30.324 INFO:tasks.workunit.client.1.smithi152.stdout:7/119: dwrite f12 [0,4194304] 0 2024-03-04T16:04:30.328 INFO:tasks.workunit.client.1.smithi152.stdout:2/154: rename da/l21 to da/d24/l3d 0 2024-03-04T16:04:30.331 INFO:tasks.workunit.client.1.smithi152.stdout:3/129: unlink d6/cb 0 2024-03-04T16:04:30.333 INFO:tasks.workunit.client.1.smithi152.stdout:0/131: unlink c8 0 2024-03-04T16:04:30.336 INFO:tasks.workunit.client.1.smithi152.stdout:9/125: symlink d2/d13/d23/l2a 0 2024-03-04T16:04:30.338 INFO:tasks.workunit.client.1.smithi152.stdout:8/172: mknod d2/d8/dd/c3a 0 2024-03-04T16:04:30.338 INFO:tasks.workunit.client.1.smithi152.stdout:8/173: readlink l0 0 2024-03-04T16:04:30.339 INFO:tasks.workunit.client.1.smithi152.stdout:8/174: chown d2/c20 0 1 2024-03-04T16:04:30.340 INFO:tasks.workunit.client.1.smithi152.stdout:5/211: mkdir d6/d8/d10/d4d 0 2024-03-04T16:04:30.343 INFO:tasks.workunit.client.1.smithi152.stdout:3/130: mknod d6/d14/c28 0 2024-03-04T16:04:30.344 INFO:tasks.workunit.client.1.smithi152.stdout:3/131: readlink d6/d14/l26 0 2024-03-04T16:04:30.345 INFO:tasks.workunit.client.1.smithi152.stdout:4/155: creat d8/d2f/f33 x:0 0 0 2024-03-04T16:04:30.346 INFO:tasks.workunit.client.1.smithi152.stdout:0/132: creat d9/d1b/f29 x:0 0 0 2024-03-04T16:04:30.348 INFO:tasks.workunit.client.1.smithi152.stdout:0/133: read d9/f26 [32628,99011] 0 2024-03-04T16:04:30.350 INFO:tasks.workunit.client.1.smithi152.stdout:9/126: mknod d2/d13/c2b 0 2024-03-04T16:04:30.355 INFO:tasks.workunit.client.1.smithi152.stdout:4/156: dwrite d8/d9/d24/d26/f2a [0,4194304] 0 2024-03-04T16:04:30.362 INFO:tasks.workunit.client.1.smithi152.stdout:3/132: fsync d6/f1a 0 2024-03-04T16:04:30.363 INFO:tasks.workunit.client.1.smithi152.stdout:0/134: dwrite f6 [4194304,4194304] 0 2024-03-04T16:04:30.365 INFO:tasks.workunit.client.1.smithi152.stdout:6/201: dwrite d8/d10/f18 [4194304,4194304] 0 2024-03-04T16:04:30.368 INFO:tasks.workunit.client.1.smithi152.stdout:5/212: getdents d6/d8/d10/d4d 0 2024-03-04T16:04:30.371 INFO:tasks.workunit.client.1.smithi152.stdout:8/175: link d2/d3/d21/d2d/f2c d2/d8/dd/d22/f3b 0 2024-03-04T16:04:30.375 INFO:tasks.workunit.client.1.smithi152.stdout:7/120: dwrite fd [0,4194304] 0 2024-03-04T16:04:30.378 INFO:tasks.workunit.client.1.smithi152.stdout:3/133: dwrite f5 [4194304,4194304] 0 2024-03-04T16:04:30.379 INFO:tasks.workunit.client.1.smithi152.stdout:4/157: creat d8/d2f/f34 x:0 0 0 2024-03-04T16:04:30.382 INFO:tasks.workunit.client.1.smithi152.stdout:7/121: dread d14/f1f [0,4194304] 0 2024-03-04T16:04:30.390 INFO:tasks.workunit.client.1.smithi152.stdout:7/122: dwrite fd [0,4194304] 0 2024-03-04T16:04:30.390 INFO:tasks.workunit.client.1.smithi152.stdout:7/123: readlink lc 0 2024-03-04T16:04:30.397 INFO:tasks.workunit.client.1.smithi152.stdout:3/134: creat d6/d14/f29 x:0 0 0 2024-03-04T16:04:30.397 INFO:tasks.workunit.client.1.smithi152.stdout:7/124: dread f7 [0,4194304] 0 2024-03-04T16:04:30.404 INFO:tasks.workunit.client.1.smithi152.stdout:0/135: dread d9/fa [0,4194304] 0 2024-03-04T16:04:30.406 INFO:tasks.workunit.client.1.smithi152.stdout:6/202: creat d8/d23/d3a/f3f x:0 0 0 2024-03-04T16:04:30.407 INFO:tasks.workunit.client.1.smithi152.stdout:8/176: mknod d2/c3c 0 2024-03-04T16:04:30.408 INFO:tasks.workunit.client.1.smithi152.stdout:8/177: chown d2/d3/d21/d2d/d31/f35 1065 1 2024-03-04T16:04:30.408 INFO:tasks.workunit.client.1.smithi152.stdout:7/125: symlink d14/l22 0 2024-03-04T16:04:30.409 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:30 smithi116 ceph-mon[29120]: pgmap v31: 65 pgs: 65 active+clean; 2.2 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 62 MiB/s rd, 168 MiB/s wr, 428 op/s 2024-03-04T16:04:30.410 INFO:tasks.workunit.client.1.smithi152.stdout:9/127: getdents d2/d13 0 2024-03-04T16:04:30.411 INFO:tasks.workunit.client.1.smithi152.stdout:5/213: creat d6/f4e x:0 0 0 2024-03-04T16:04:30.412 INFO:tasks.workunit.client.1.smithi152.stdout:0/136: readlink d9/l12 0 2024-03-04T16:04:30.413 INFO:tasks.workunit.client.1.smithi152.stdout:6/203: symlink d8/d2a/l40 0 2024-03-04T16:04:30.414 INFO:tasks.workunit.client.1.smithi152.stdout:6/204: fdatasync d8/d23/f33 0 2024-03-04T16:04:30.416 INFO:tasks.workunit.client.1.smithi152.stdout:7/126: dwrite f8 [0,4194304] 0 2024-03-04T16:04:30.416 INFO:tasks.workunit.client.1.smithi152.stdout:6/205: write d8/d23/f33 [511335,33846] 0 2024-03-04T16:04:30.436 INFO:tasks.workunit.client.1.smithi152.stdout:3/135: truncate d6/f1a 2541284 0 2024-03-04T16:04:30.440 INFO:tasks.workunit.client.1.smithi152.stdout:4/158: getdents d8/d9 0 2024-03-04T16:04:30.444 INFO:tasks.workunit.client.1.smithi152.stdout:8/178: mknod d2/c3d 0 2024-03-04T16:04:30.445 INFO:tasks.workunit.client.1.smithi152.stdout:7/127: write f7 [4845325,46489] 0 2024-03-04T16:04:30.447 INFO:tasks.workunit.client.1.smithi152.stdout:3/136: creat d6/d14/f2a x:0 0 0 2024-03-04T16:04:30.449 INFO:tasks.workunit.client.1.smithi152.stdout:3/137: dread d6/ff [0,4194304] 0 2024-03-04T16:04:30.449 INFO:tasks.workunit.client.1.smithi152.stdout:0/137: creat d9/d1b/f2a x:0 0 0 2024-03-04T16:04:30.450 INFO:tasks.workunit.client.1.smithi152.stdout:0/138: stat d9 0 2024-03-04T16:04:30.452 INFO:tasks.workunit.client.1.smithi152.stdout:8/179: unlink d2/d3/d15/c23 0 2024-03-04T16:04:30.453 INFO:tasks.workunit.client.1.smithi152.stdout:7/128: mkdir d14/d23 0 2024-03-04T16:04:30.459 INFO:tasks.workunit.client.1.smithi152.stdout:5/214: truncate d6/d8/f1d 594671 0 2024-03-04T16:04:30.460 INFO:tasks.workunit.client.1.smithi152.stdout:3/138: fsync d6/f1a 0 2024-03-04T16:04:30.465 INFO:tasks.workunit.client.1.smithi152.stdout:6/206: truncate d8/d10/f18 1996000 0 2024-03-04T16:04:30.465 INFO:tasks.workunit.client.1.smithi152.stdout:6/207: fdatasync d8/f1f 0 2024-03-04T16:04:30.466 INFO:tasks.workunit.client.1.smithi152.stdout:6/208: chown d8/d23/d3a 2692 1 2024-03-04T16:04:30.469 INFO:tasks.workunit.client.1.smithi152.stdout:9/128: getdents d2 0 2024-03-04T16:04:30.471 INFO:tasks.workunit.client.1.smithi152.stdout:8/180: write d2/d3/d21/d2d/fc [4512188,23888] 0 2024-03-04T16:04:30.473 INFO:tasks.workunit.client.1.smithi152.stdout:3/139: creat d6/d14/f2b x:0 0 0 2024-03-04T16:04:30.479 INFO:tasks.workunit.client.1.smithi152.stdout:3/140: dwrite f3 [0,4194304] 0 2024-03-04T16:04:30.479 INFO:tasks.workunit.client.1.smithi152.stdout:7/129: symlink d14/d21/l24 0 2024-03-04T16:04:30.480 INFO:tasks.workunit.client.1.smithi152.stdout:3/141: fdatasync d6/f1a 0 2024-03-04T16:04:30.481 INFO:tasks.workunit.client.1.smithi152.stdout:9/129: symlink d2/l2c 0 2024-03-04T16:04:30.483 INFO:tasks.workunit.client.1.smithi152.stdout:5/215: link d6/d8/f41 d6/d8/d1f/d3d/f4f 0 2024-03-04T16:04:30.484 INFO:tasks.workunit.client.1.smithi152.stdout:5/216: chown d6/fd 97 1 2024-03-04T16:04:30.485 INFO:tasks.workunit.client.1.smithi152.stdout:5/217: truncate d6/d8/d10/f49 740816 0 2024-03-04T16:04:30.494 INFO:tasks.workunit.client.1.smithi152.stdout:9/130: creat d2/f2d x:0 0 0 2024-03-04T16:04:30.496 INFO:tasks.workunit.client.1.smithi152.stdout:8/181: truncate d2/d8/dd/d22/d26/f38 2455326 0 2024-03-04T16:04:30.498 INFO:tasks.workunit.client.1.smithi152.stdout:5/218: unlink d6/d8/d1f/f34 0 2024-03-04T16:04:30.500 INFO:tasks.workunit.client.1.smithi152.stdout:6/209: truncate d8/f12 1410660 0 2024-03-04T16:04:30.501 INFO:tasks.workunit.client.1.smithi152.stdout:9/131: mkdir d2/d13/d2e 0 2024-03-04T16:04:30.504 INFO:tasks.workunit.client.1.smithi152.stdout:8/182: rename d2/d3/la to d2/d8/dd/d22/d26/l3e 0 2024-03-04T16:04:30.505 INFO:tasks.workunit.client.1.smithi152.stdout:8/183: chown d2/d8/f2e 0 1 2024-03-04T16:04:30.505 INFO:tasks.workunit.client.1.smithi152.stdout:3/142: creat d6/f2c x:0 0 0 2024-03-04T16:04:30.506 INFO:tasks.workunit.client.1.smithi152.stdout:9/132: dread d2/fe [0,4194304] 0 2024-03-04T16:04:30.507 INFO:tasks.workunit.client.1.smithi152.stdout:9/133: stat d2/lc 0 2024-03-04T16:04:30.507 INFO:tasks.workunit.client.1.smithi152.stdout:9/134: readlink d2/l2c 0 2024-03-04T16:04:30.508 INFO:tasks.workunit.client.1.smithi152.stdout:6/210: symlink d8/d23/l41 0 2024-03-04T16:04:30.508 INFO:tasks.workunit.client.1.smithi152.stdout:9/135: write d2/d13/f18 [1490217,49071] 0 2024-03-04T16:04:30.509 INFO:tasks.workunit.client.1.smithi152.stdout:5/219: mkdir d6/d50 0 2024-03-04T16:04:30.511 INFO:tasks.workunit.client.1.smithi152.stdout:8/184: fdatasync d2/f37 0 2024-03-04T16:04:30.512 INFO:tasks.workunit.client.1.smithi152.stdout:6/211: dread f1 [0,4194304] 0 2024-03-04T16:04:30.513 INFO:tasks.workunit.client.1.smithi152.stdout:8/185: rename d2/d3/d21/d2d to d2/d3/d21/d2d/d3f 22 2024-03-04T16:04:30.515 INFO:tasks.workunit.client.1.smithi152.stdout:8/186: dread d2/d3/d21/f32 [0,4194304] 0 2024-03-04T16:04:30.517 INFO:tasks.workunit.client.1.smithi152.stdout:7/130: getdents d14/d17 0 2024-03-04T16:04:30.519 INFO:tasks.workunit.client.1.smithi152.stdout:7/131: chown d14/f20 51333 1 2024-03-04T16:04:30.521 INFO:tasks.workunit.client.1.smithi152.stdout:6/212: dwrite d8/d23/d3a/f3f [0,4194304] 0 2024-03-04T16:04:30.532 INFO:tasks.workunit.client.1.smithi152.stdout:9/136: mkdir d2/d13/d23/d2f 0 2024-03-04T16:04:30.532 INFO:tasks.workunit.client.1.smithi152.stdout:9/137: dread - d2/f28 zero size 2024-03-04T16:04:30.532 INFO:tasks.workunit.client.1.smithi152.stdout:9/138: write d2/fb [3192604,57234] 0 2024-03-04T16:04:30.533 INFO:tasks.workunit.client.1.smithi152.stdout:9/139: dread - d2/d13/d23/f29 zero size 2024-03-04T16:04:30.537 INFO:tasks.workunit.client.1.smithi152.stdout:8/187: creat d2/d3/f40 x:0 0 0 2024-03-04T16:04:30.542 INFO:tasks.workunit.client.1.smithi152.stdout:7/132: dread ff [0,4194304] 0 2024-03-04T16:04:30.542 INFO:tasks.workunit.client.1.smithi152.stdout:7/133: read d14/f1c [2354793,81654] 0 2024-03-04T16:04:30.543 INFO:tasks.workunit.client.1.smithi152.stdout:7/134: stat d14/f19 0 2024-03-04T16:04:30.563 INFO:tasks.workunit.client.1.smithi152.stdout:6/213: mkdir d8/d42 0 2024-03-04T16:04:30.564 INFO:tasks.workunit.client.1.smithi152.stdout:7/135: creat d14/d21/f25 x:0 0 0 2024-03-04T16:04:30.565 INFO:tasks.workunit.client.1.smithi152.stdout:7/136: fsync f12 0 2024-03-04T16:04:30.566 INFO:tasks.workunit.client.1.smithi152.stdout:5/220: rmdir d6/d8/d10/d4d 0 2024-03-04T16:04:30.573 INFO:tasks.workunit.client.1.smithi152.stdout:3/143: getdents d6/d14 0 2024-03-04T16:04:30.575 INFO:tasks.workunit.client.1.smithi152.stdout:8/188: fsync d2/d8/dd/d22/d26/f38 0 2024-03-04T16:04:30.576 INFO:tasks.workunit.client.1.smithi152.stdout:8/189: chown d2/d3/f2b 18 1 2024-03-04T16:04:30.577 INFO:tasks.workunit.client.1.smithi152.stdout:8/190: dread - d2/d8/dd/d22/f24 zero size 2024-03-04T16:04:30.579 INFO:tasks.workunit.client.1.smithi152.stdout:6/214: creat d8/d10/f43 x:0 0 0 2024-03-04T16:04:30.582 INFO:tasks.workunit.client.1.smithi152.stdout:9/140: truncate d2/fd 1822951 0 2024-03-04T16:04:30.582 INFO:tasks.workunit.client.1.smithi152.stdout:9/141: readlink d2/d13/d23/l2a 0 2024-03-04T16:04:30.587 INFO:tasks.workunit.client.1.smithi152.stdout:3/144: mkdir d6/d10/d2d 0 2024-03-04T16:04:30.588 INFO:tasks.workunit.client.1.smithi152.stdout:3/145: truncate d6/d10/f21 1268569 0 2024-03-04T16:04:30.591 INFO:tasks.workunit.client.1.smithi152.stdout:8/191: symlink d2/d3/d21/d2d/d31/l41 0 2024-03-04T16:04:30.591 INFO:tasks.workunit.client.1.smithi152.stdout:8/192: chown d2/d3/d15/d1c/f1d 1522490 1 2024-03-04T16:04:30.595 INFO:tasks.workunit.client.1.smithi152.stdout:3/146: creat d6/d14/f2e x:0 0 0 2024-03-04T16:04:30.596 INFO:tasks.workunit.client.1.smithi152.stdout:3/147: write f5 [7990809,49621] 0 2024-03-04T16:04:30.598 INFO:tasks.workunit.client.1.smithi152.stdout:7/137: rmdir d14/d23 0 2024-03-04T16:04:30.601 INFO:tasks.workunit.client.1.smithi152.stdout:1/162: sync 2024-03-04T16:04:30.601 INFO:tasks.workunit.client.1.smithi152.stdout:1/163: readlink d19/d20/d21/l25 0 2024-03-04T16:04:30.602 INFO:tasks.workunit.client.1.smithi152.stdout:3/148: chown d6/d10/l1b 16436945 1 2024-03-04T16:04:30.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:30 smithi152 ceph-mon[36876]: pgmap v31: 65 pgs: 65 active+clean; 2.2 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 62 MiB/s rd, 168 MiB/s wr, 428 op/s 2024-03-04T16:04:30.608 INFO:tasks.workunit.client.1.smithi152.stdout:7/138: mknod d14/d21/c26 0 2024-03-04T16:04:30.610 INFO:tasks.workunit.client.1.smithi152.stdout:1/164: dwrite f2 [8388608,4194304] 0 2024-03-04T16:04:30.612 INFO:tasks.workunit.client.1.smithi152.stdout:5/221: getdents d6/d8/d10 0 2024-03-04T16:04:30.613 INFO:tasks.workunit.client.1.smithi152.stdout:3/149: dwrite d6/d14/f15 [0,4194304] 0 2024-03-04T16:04:30.619 INFO:tasks.workunit.client.1.smithi152.stdout:7/139: dwrite d14/d21/f25 [0,4194304] 0 2024-03-04T16:04:30.619 INFO:tasks.workunit.client.1.smithi152.stdout:6/215: link d8/d10/f18 d8/d10/f44 0 2024-03-04T16:04:30.624 INFO:tasks.workunit.client.1.smithi152.stdout:3/150: write d6/fc [1707648,18786] 0 2024-03-04T16:04:30.627 INFO:tasks.workunit.client.1.smithi152.stdout:5/222: creat d6/d8/d10/f51 x:0 0 0 2024-03-04T16:04:30.628 INFO:tasks.workunit.client.1.smithi152.stdout:5/223: truncate d6/f4e 185895 0 2024-03-04T16:04:30.633 INFO:tasks.workunit.client.1.smithi152.stdout:6/216: creat d8/d2a/f45 x:0 0 0 2024-03-04T16:04:30.634 INFO:tasks.workunit.client.1.smithi152.stdout:6/217: readlink d8/d23/l29 0 2024-03-04T16:04:30.650 INFO:tasks.workunit.client.1.smithi152.stdout:3/151: mknod d6/d10/d2d/c2f 0 2024-03-04T16:04:30.655 INFO:tasks.workunit.client.1.smithi152.stdout:6/218: mknod d8/d38/c46 0 2024-03-04T16:04:30.657 INFO:tasks.workunit.client.1.smithi152.stdout:1/165: getdents d19 0 2024-03-04T16:04:30.658 INFO:tasks.workunit.client.1.smithi152.stdout:3/152: write d6/d10/f27 [2941487,2198] 0 2024-03-04T16:04:30.660 INFO:tasks.workunit.client.1.smithi152.stdout:5/224: link d6/d8/d1f/d3d/f4c d6/d50/f52 0 2024-03-04T16:04:30.662 INFO:tasks.workunit.client.1.smithi152.stdout:3/153: fsync d6/f8 0 2024-03-04T16:04:30.665 INFO:tasks.workunit.client.1.smithi152.stdout:1/166: dwrite d19/d20/d21/f2d [0,4194304] 0 2024-03-04T16:04:30.665 INFO:tasks.workunit.client.1.smithi152.stdout:7/140: truncate d14/f1c 167984 0 2024-03-04T16:04:30.666 INFO:tasks.workunit.client.1.smithi152.stdout:7/141: truncate d14/f19 41347 0 2024-03-04T16:04:30.667 INFO:tasks.workunit.client.1.smithi152.stdout:7/142: fdatasync d14/d17/f18 0 2024-03-04T16:04:30.670 INFO:tasks.workunit.client.1.smithi152.stdout:5/225: rename d6/f4a to d6/d8/d10/f53 0 2024-03-04T16:04:30.674 INFO:tasks.workunit.client.1.smithi152.stdout:1/167: dwrite d19/f32 [0,4194304] 0 2024-03-04T16:04:30.676 INFO:tasks.workunit.client.1.smithi152.stdout:3/154: unlink d6/d14/l24 0 2024-03-04T16:04:30.684 INFO:tasks.workunit.client.1.smithi152.stdout:3/155: dwrite d6/d14/f15 [0,4194304] 0 2024-03-04T16:04:30.684 INFO:tasks.workunit.client.1.smithi152.stdout:7/143: symlink d14/d21/l27 0 2024-03-04T16:04:30.685 INFO:tasks.workunit.client.1.smithi152.stdout:3/156: chown d6/d14/f2e 15536996 1 2024-03-04T16:04:30.688 INFO:tasks.workunit.client.1.smithi152.stdout:5/226: creat d6/d8/d10/f54 x:0 0 0 2024-03-04T16:04:30.688 INFO:tasks.workunit.client.1.smithi152.stdout:5/227: stat d6/d8/d1f/f2e 0 2024-03-04T16:04:30.698 INFO:tasks.workunit.client.1.smithi152.stdout:5/228: write d6/f39 [983222,117265] 0 2024-03-04T16:04:30.699 INFO:tasks.workunit.client.1.smithi152.stdout:1/168: symlink d19/d20/d2f/l36 0 2024-03-04T16:04:30.704 INFO:tasks.workunit.client.1.smithi152.stdout:5/229: dread d6/d8/d1f/f48 [0,4194304] 0 2024-03-04T16:04:30.704 INFO:tasks.workunit.client.1.smithi152.stdout:1/169: dread d19/f32 [0,4194304] 0 2024-03-04T16:04:30.708 INFO:tasks.workunit.client.1.smithi152.stdout:1/170: write d19/f1a [40695,67175] 0 2024-03-04T16:04:30.710 INFO:tasks.workunit.client.1.smithi152.stdout:5/230: mknod d6/d8/c55 0 2024-03-04T16:04:30.718 INFO:tasks.workunit.client.1.smithi152.stdout:1/171: dwrite d19/f32 [0,4194304] 0 2024-03-04T16:04:30.719 INFO:tasks.workunit.client.1.smithi152.stdout:5/231: creat d6/d8/f56 x:0 0 0 2024-03-04T16:04:30.722 INFO:tasks.workunit.client.1.smithi152.stdout:1/172: unlink d19/l26 0 2024-03-04T16:04:30.727 INFO:tasks.workunit.client.1.smithi152.stdout:5/232: dwrite d6/d8/ff [0,4194304] 0 2024-03-04T16:04:30.728 INFO:tasks.workunit.client.1.smithi152.stdout:5/233: readlink d6/d8/l33 0 2024-03-04T16:04:30.731 INFO:tasks.workunit.client.1.smithi152.stdout:5/234: creat d6/d8/d1f/f57 x:0 0 0 2024-03-04T16:04:30.746 INFO:tasks.workunit.client.1.smithi152.stdout:5/235: getdents d6/d8/d1f 0 2024-03-04T16:04:30.746 INFO:tasks.workunit.client.1.smithi152.stdout:1/173: dread f1 [0,4194304] 0 2024-03-04T16:04:30.747 INFO:tasks.workunit.client.1.smithi152.stdout:5/236: chown d6/d8/d10/f54 439755 1 2024-03-04T16:04:30.749 INFO:tasks.workunit.client.1.smithi152.stdout:5/237: fsync d6/d8/f2c 0 2024-03-04T16:04:30.750 INFO:tasks.workunit.client.1.smithi152.stdout:1/174: link d19/d20/d21/f2e d19/d20/d21/f37 0 2024-03-04T16:04:30.754 INFO:tasks.workunit.client.1.smithi152.stdout:1/175: write d19/d20/d21/f2e [91076,124905] 0 2024-03-04T16:04:30.755 INFO:tasks.workunit.client.1.smithi152.stdout:1/176: chown f2 48255 1 2024-03-04T16:04:30.757 INFO:tasks.workunit.client.1.smithi152.stdout:1/177: write d19/f32 [1862712,67074] 0 2024-03-04T16:04:30.757 INFO:tasks.workunit.client.1.smithi152.stdout:5/238: dwrite d6/d8/d10/f54 [0,4194304] 0 2024-03-04T16:04:30.758 INFO:tasks.workunit.client.1.smithi152.stdout:5/239: fdatasync d6/d8/d10/f51 0 2024-03-04T16:04:30.762 INFO:tasks.workunit.client.1.smithi152.stdout:1/178: mkdir d19/d20/d21/d38 0 2024-03-04T16:04:30.770 INFO:tasks.workunit.client.1.smithi152.stdout:1/179: getdents d19/d20/d21 0 2024-03-04T16:04:30.773 INFO:tasks.workunit.client.1.smithi152.stdout:5/240: dread d6/d8/d10/f12 [0,4194304] 0 2024-03-04T16:04:30.774 INFO:tasks.workunit.client.1.smithi152.stdout:1/180: creat d19/d20/d2f/f39 x:0 0 0 2024-03-04T16:04:30.777 INFO:tasks.workunit.client.1.smithi152.stdout:5/241: dread d6/d8/ff [0,4194304] 0 2024-03-04T16:04:30.778 INFO:tasks.workunit.client.1.smithi152.stdout:5/242: chown d6/d8/d10/f27 6488311 1 2024-03-04T16:04:30.789 INFO:tasks.workunit.client.1.smithi152.stdout:5/243: write d6/d8/d1f/f23 [1140996,66105] 0 2024-03-04T16:04:30.790 INFO:tasks.workunit.client.1.smithi152.stdout:5/244: fdatasync d6/d8/d10/f51 0 2024-03-04T16:04:30.801 INFO:tasks.workunit.client.1.smithi152.stdout:5/245: truncate d6/d8/d1f/f3b 1126021 0 2024-03-04T16:04:30.805 INFO:tasks.workunit.client.1.smithi152.stdout:5/246: dread d6/d8/fe [0,4194304] 0 2024-03-04T16:04:30.805 INFO:tasks.workunit.client.1.smithi152.stdout:5/247: dread - d6/d8/d1f/f57 zero size 2024-03-04T16:04:30.807 INFO:tasks.workunit.client.1.smithi152.stdout:5/248: mknod d6/d8/c58 0 2024-03-04T16:04:30.809 INFO:tasks.workunit.client.1.smithi152.stdout:2/155: sync 2024-03-04T16:04:30.809 INFO:tasks.workunit.client.1.smithi152.stdout:4/159: sync 2024-03-04T16:04:30.810 INFO:tasks.workunit.client.1.smithi152.stdout:0/139: sync 2024-03-04T16:04:30.810 INFO:tasks.workunit.client.1.smithi152.stdout:5/249: creat d6/d50/f59 x:0 0 0 2024-03-04T16:04:30.812 INFO:tasks.workunit.client.1.smithi152.stdout:0/140: chown d9/d1a/c28 282810646 1 2024-03-04T16:04:30.812 INFO:tasks.workunit.client.1.smithi152.stdout:4/160: symlink d8/d2f/l35 0 2024-03-04T16:04:30.814 INFO:tasks.workunit.client.1.smithi152.stdout:5/250: mkdir d6/d8/d5a 0 2024-03-04T16:04:30.814 INFO:tasks.workunit.client.1.smithi152.stdout:2/156: dread da/df/f1f [0,4194304] 0 2024-03-04T16:04:30.814 INFO:tasks.workunit.client.1.smithi152.stdout:5/251: read - d6/d8/f4b zero size 2024-03-04T16:04:30.815 INFO:tasks.workunit.client.1.smithi152.stdout:5/252: dread - d6/d50/f59 zero size 2024-03-04T16:04:30.816 INFO:tasks.workunit.client.1.smithi152.stdout:4/161: rename d8/d9/l1c to d8/d9/l36 0 2024-03-04T16:04:30.821 INFO:tasks.workunit.client.1.smithi152.stdout:2/157: mknod da/df/c3e 0 2024-03-04T16:04:30.822 INFO:tasks.workunit.client.1.smithi152.stdout:5/253: link d6/d8/f56 d6/d8/d10/f5b 0 2024-03-04T16:04:30.822 INFO:tasks.workunit.client.1.smithi152.stdout:4/162: dwrite d8/f25 [0,4194304] 0 2024-03-04T16:04:30.825 INFO:tasks.workunit.client.1.smithi152.stdout:4/163: chown f5 813279792 1 2024-03-04T16:04:30.830 INFO:tasks.workunit.client.1.smithi152.stdout:0/141: getdents d9/d1b 0 2024-03-04T16:04:30.831 INFO:tasks.workunit.client.1.smithi152.stdout:2/158: truncate da/fc 429465 0 2024-03-04T16:04:30.832 INFO:tasks.workunit.client.1.smithi152.stdout:2/159: write da/df/f37 [23584,116743] 0 2024-03-04T16:04:30.833 INFO:tasks.workunit.client.1.smithi152.stdout:2/160: write da/df/d2e/f39 [130155,13385] 0 2024-03-04T16:04:30.833 INFO:tasks.workunit.client.1.smithi152.stdout:5/254: creat d6/d50/f5c x:0 0 0 2024-03-04T16:04:30.835 INFO:tasks.workunit.client.1.smithi152.stdout:2/161: dread - da/d24/f2a zero size 2024-03-04T16:04:30.835 INFO:tasks.workunit.client.1.smithi152.stdout:5/255: write d6/d8/d10/f46 [1125816,71507] 0 2024-03-04T16:04:30.837 INFO:tasks.workunit.client.1.smithi152.stdout:0/142: chown d9/f16 366729 1 2024-03-04T16:04:30.837 INFO:tasks.workunit.client.1.smithi152.stdout:5/256: truncate d6/d8/d1f/f2e 278098 0 2024-03-04T16:04:30.844 INFO:tasks.workunit.client.1.smithi152.stdout:0/143: dwrite d9/f16 [4194304,4194304] 0 2024-03-04T16:04:30.847 INFO:tasks.workunit.client.1.smithi152.stdout:2/162: mkdir da/d2c/d3f 0 2024-03-04T16:04:30.847 INFO:tasks.workunit.client.1.smithi152.stdout:2/163: dread - da/d24/f2a zero size 2024-03-04T16:04:30.849 INFO:tasks.workunit.client.1.smithi152.stdout:0/144: dread d9/fa [0,4194304] 0 2024-03-04T16:04:30.850 INFO:tasks.workunit.client.1.smithi152.stdout:0/145: stat d9/d1b/f21 0 2024-03-04T16:04:30.851 INFO:tasks.workunit.client.1.smithi152.stdout:0/146: write f7 [9329222,27830] 0 2024-03-04T16:04:30.852 INFO:tasks.workunit.client.1.smithi152.stdout:0/147: fdatasync d9/d1b/f23 0 2024-03-04T16:04:30.857 INFO:tasks.workunit.client.1.smithi152.stdout:4/164: getdents d8/d9 0 2024-03-04T16:04:30.859 INFO:tasks.workunit.client.1.smithi152.stdout:0/148: symlink d9/l2b 0 2024-03-04T16:04:30.864 INFO:tasks.workunit.client.1.smithi152.stdout:2/164: dwrite da/d24/f2a [0,4194304] 0 2024-03-04T16:04:30.866 INFO:tasks.workunit.client.1.smithi152.stdout:2/165: chown da/df/c26 262002 1 2024-03-04T16:04:30.867 INFO:tasks.workunit.client.1.smithi152.stdout:4/165: link d8/d9/f15 d8/f37 0 2024-03-04T16:04:30.869 INFO:tasks.workunit.client.1.smithi152.stdout:0/149: dwrite d9/d1b/f1d [0,4194304] 0 2024-03-04T16:04:30.875 INFO:tasks.workunit.client.1.smithi152.stdout:2/166: mknod da/d24/c40 0 2024-03-04T16:04:30.876 INFO:tasks.workunit.client.1.smithi152.stdout:4/166: mknod d8/d9/d24/c38 0 2024-03-04T16:04:30.879 INFO:tasks.workunit.client.1.smithi152.stdout:0/150: dwrite d9/f16 [4194304,4194304] 0 2024-03-04T16:04:30.881 INFO:tasks.workunit.client.1.smithi152.stdout:4/167: mknod d8/d9/d24/d26/c39 0 2024-03-04T16:04:30.882 INFO:tasks.workunit.client.1.smithi152.stdout:4/168: write d8/f16 [3277619,57439] 0 2024-03-04T16:04:30.887 INFO:tasks.workunit.client.1.smithi152.stdout:2/167: link da/df/d2e/f39 da/d2c/f41 0 2024-03-04T16:04:30.888 INFO:tasks.workunit.client.1.smithi152.stdout:0/151: mkdir d9/d2c 0 2024-03-04T16:04:30.891 INFO:tasks.workunit.client.1.smithi152.stdout:2/168: creat da/d24/f42 x:0 0 0 2024-03-04T16:04:30.913 INFO:tasks.workunit.client.1.smithi152.stdout:0/152: getdents d9 0 2024-03-04T16:04:30.915 INFO:tasks.workunit.client.1.smithi152.stdout:0/153: read d9/d1b/f23 [47391,112605] 0 2024-03-04T16:04:30.923 INFO:tasks.workunit.client.1.smithi152.stdout:0/154: dwrite d9/fa [4194304,4194304] 0 2024-03-04T16:04:30.928 INFO:tasks.workunit.client.1.smithi152.stdout:0/155: mknod d9/c2d 0 2024-03-04T16:04:30.978 INFO:tasks.workunit.client.1.smithi152.stdout:0/156: dread d9/f26 [0,4194304] 0 2024-03-04T16:04:30.980 INFO:tasks.workunit.client.1.smithi152.stdout:0/157: write d9/f19 [1458519,73003] 0 2024-03-04T16:04:30.982 INFO:tasks.workunit.client.1.smithi152.stdout:0/158: dread d9/f26 [0,4194304] 0 2024-03-04T16:04:30.986 INFO:tasks.workunit.client.1.smithi152.stdout:0/159: mkdir d9/d2c/d2e 0 2024-03-04T16:04:30.987 INFO:tasks.workunit.client.1.smithi152.stdout:0/160: fsync d9/d1b/f2a 0 2024-03-04T16:04:30.992 INFO:tasks.workunit.client.1.smithi152.stdout:0/161: getdents d9/d2c 0 2024-03-04T16:04:30.993 INFO:tasks.workunit.client.1.smithi152.stdout:0/162: write d9/f19 [1150021,40816] 0 2024-03-04T16:04:30.998 INFO:tasks.workunit.client.1.smithi152.stdout:0/163: write d9/f26 [207550,10348] 0 2024-03-04T16:04:30.998 INFO:tasks.workunit.client.1.smithi152.stdout:0/164: chown d9/d1b/f2a 27083421 1 2024-03-04T16:04:31.002 INFO:tasks.workunit.client.1.smithi152.stdout:0/165: rename d9/d1b/f21 to d9/d2c/f2f 0 2024-03-04T16:04:31.005 INFO:tasks.workunit.client.1.smithi152.stdout:0/166: link l0 d9/d1b/l30 0 2024-03-04T16:04:31.006 INFO:tasks.workunit.client.1.smithi152.stdout:0/167: fdatasync d9/f25 0 2024-03-04T16:04:31.013 INFO:tasks.workunit.client.1.smithi152.stdout:0/168: dread f4 [0,4194304] 0 2024-03-04T16:04:31.014 INFO:tasks.workunit.client.1.smithi152.stdout:0/169: write f7 [526631,12805] 0 2024-03-04T16:04:31.016 INFO:tasks.workunit.client.1.smithi152.stdout:9/142: sync 2024-03-04T16:04:31.017 INFO:tasks.workunit.client.1.smithi152.stdout:9/143: fdatasync d2/f19 0 2024-03-04T16:04:31.019 INFO:tasks.workunit.client.1.smithi152.stdout:0/170: dread d9/f19 [0,4194304] 0 2024-03-04T16:04:31.019 INFO:tasks.workunit.client.1.smithi152.stdout:9/144: symlink d2/l30 0 2024-03-04T16:04:31.022 INFO:tasks.workunit.client.1.smithi152.stdout:9/145: fdatasync d2/f1c 0 2024-03-04T16:04:31.024 INFO:tasks.workunit.client.1.smithi152.stdout:0/171: dread d9/f16 [4194304,4194304] 0 2024-03-04T16:04:31.028 INFO:tasks.workunit.client.1.smithi152.stdout:9/146: rmdir d2/d13 39 2024-03-04T16:04:31.030 INFO:tasks.workunit.client.1.smithi152.stdout:0/172: dwrite d9/d1b/f29 [0,4194304] 0 2024-03-04T16:04:31.032 INFO:tasks.workunit.client.1.smithi152.stdout:0/173: chown d9/d1b/f23 58 1 2024-03-04T16:04:31.037 INFO:tasks.workunit.client.1.smithi152.stdout:0/174: dread f6 [4194304,4194304] 0 2024-03-04T16:04:31.037 INFO:tasks.workunit.client.1.smithi152.stdout:9/147: link d2/c16 d2/d13/d23/d2f/c31 0 2024-03-04T16:04:31.046 INFO:tasks.workunit.client.1.smithi152.stdout:9/148: rmdir d2 39 2024-03-04T16:04:31.048 INFO:tasks.workunit.client.1.smithi152.stdout:0/175: truncate d9/fa 6155068 0 2024-03-04T16:04:31.052 INFO:tasks.workunit.client.1.smithi152.stdout:9/149: link d2/d13/c25 d2/d13/d23/c32 0 2024-03-04T16:04:31.062 INFO:tasks.workunit.client.1.smithi152.stdout:9/150: dread d2/fd [0,4194304] 0 2024-03-04T16:04:31.064 INFO:tasks.workunit.client.1.smithi152.stdout:9/151: creat d2/d13/d23/f33 x:0 0 0 2024-03-04T16:04:31.066 INFO:tasks.workunit.client.1.smithi152.stdout:9/152: getdents d2/d13 0 2024-03-04T16:04:31.067 INFO:tasks.workunit.client.1.smithi152.stdout:9/153: chown d2/d13/d23/c27 1 1 2024-03-04T16:04:31.072 INFO:tasks.workunit.client.1.smithi152.stdout:9/154: creat d2/d13/d23/d2f/f34 x:0 0 0 2024-03-04T16:04:31.077 INFO:tasks.workunit.client.1.smithi152.stdout:9/155: rename d2/d13/d23/d2f to d2/d13/d35 0 2024-03-04T16:04:31.078 INFO:tasks.workunit.client.1.smithi152.stdout:9/156: fdatasync d2/d13/d23/f33 0 2024-03-04T16:04:31.079 INFO:tasks.workunit.client.1.smithi152.stdout:9/157: stat d2/c16 0 2024-03-04T16:04:31.082 INFO:tasks.workunit.client.1.smithi152.stdout:9/158: creat d2/f36 x:0 0 0 2024-03-04T16:04:31.083 INFO:tasks.workunit.client.1.smithi152.stdout:9/159: dread - d2/f19 zero size 2024-03-04T16:04:31.084 INFO:tasks.workunit.client.1.smithi152.stdout:9/160: read d2/d13/f18 [565090,35083] 0 2024-03-04T16:04:31.089 INFO:tasks.workunit.client.1.smithi152.stdout:9/161: getdents d2/d13/d35 0 2024-03-04T16:04:31.092 INFO:tasks.workunit.client.1.smithi152.stdout:9/162: dread d2/fb [0,4194304] 0 2024-03-04T16:04:31.095 INFO:tasks.workunit.client.1.smithi152.stdout:9/163: rmdir d2/d13/d23 39 2024-03-04T16:04:31.096 INFO:tasks.workunit.client.1.smithi152.stdout:9/164: dread - d2/f19 zero size 2024-03-04T16:04:31.097 INFO:tasks.workunit.client.1.smithi152.stdout:9/165: write d2/f2d [114029,12880] 0 2024-03-04T16:04:31.097 INFO:tasks.workunit.client.1.smithi152.stdout:9/166: dread - d2/f28 zero size 2024-03-04T16:04:31.098 INFO:tasks.workunit.client.1.smithi152.stdout:9/167: read - d2/f36 zero size 2024-03-04T16:04:31.214 INFO:tasks.workunit.client.1.smithi152.stdout:8/193: sync 2024-03-04T16:04:31.218 INFO:tasks.workunit.client.1.smithi152.stdout:8/194: creat d2/f42 x:0 0 0 2024-03-04T16:04:31.237 INFO:tasks.workunit.client.1.smithi152.stdout:8/195: dread d2/d8/f14 [0,4194304] 0 2024-03-04T16:04:31.242 INFO:tasks.workunit.client.1.smithi152.stdout:8/196: dwrite d2/d8/f1a [0,4194304] 0 2024-03-04T16:04:31.248 INFO:tasks.workunit.client.1.smithi152.stdout:8/197: dwrite d2/d3/f2b [4194304,4194304] 0 2024-03-04T16:04:31.253 INFO:tasks.workunit.client.1.smithi152.stdout:8/198: rename d2/d8/dd/l30 to d2/d8/dd/d22/d26/l43 0 2024-03-04T16:04:31.254 INFO:tasks.workunit.client.1.smithi152.stdout:8/199: read d2/d8/f1a [2178367,63630] 0 2024-03-04T16:04:31.276 INFO:tasks.workunit.client.1.smithi152.stdout:8/200: dwrite d2/d8/dd/d22/d26/f38 [0,4194304] 0 2024-03-04T16:04:31.280 INFO:tasks.workunit.client.1.smithi152.stdout:8/201: creat d2/d3/d21/d33/f44 x:0 0 0 2024-03-04T16:04:31.284 INFO:tasks.workunit.client.1.smithi152.stdout:8/202: dread d2/d8/f1a [0,4194304] 0 2024-03-04T16:04:31.285 INFO:tasks.workunit.client.1.smithi152.stdout:8/203: dread - d2/d3/d15/d1c/f1d zero size 2024-03-04T16:04:31.285 INFO:tasks.workunit.client.1.smithi152.stdout:8/204: chown d2/d8/dd/d22/d26/f38 0 1 2024-03-04T16:04:31.288 INFO:tasks.workunit.client.1.smithi152.stdout:8/205: rename d2/d3/f2b to d2/d8/f45 0 2024-03-04T16:04:31.295 INFO:tasks.workunit.client.1.smithi152.stdout:8/206: dwrite d2/d8/dd/d22/f24 [0,4194304] 0 2024-03-04T16:04:31.301 INFO:tasks.workunit.client.1.smithi152.stdout:8/207: symlink d2/l46 0 2024-03-04T16:04:31.307 INFO:tasks.workunit.client.1.smithi152.stdout:8/208: getdents d2/d3/d21/d2d/d31 0 2024-03-04T16:04:31.307 INFO:tasks.workunit.client.1.smithi152.stdout:8/209: fdatasync d2/d8/f28 0 2024-03-04T16:04:31.308 INFO:tasks.workunit.client.1.smithi152.stdout:8/210: chown d2/d8/dd/c3a 15 1 2024-03-04T16:04:31.308 INFO:tasks.workunit.client.1.smithi152.stdout:8/211: chown d2/d8 217325 1 2024-03-04T16:04:31.311 INFO:tasks.workunit.client.1.smithi152.stdout:8/212: getdents d2/d3/d21/d2d/d31 0 2024-03-04T16:04:31.312 INFO:tasks.workunit.client.1.smithi152.stdout:8/213: dread - d2/d3/d21/d2d/f2c zero size 2024-03-04T16:04:31.318 INFO:tasks.workunit.client.1.smithi152.stdout:8/214: dwrite d2/f42 [0,4194304] 0 2024-03-04T16:04:31.326 INFO:tasks.workunit.client.1.smithi152.stdout:8/215: dwrite d2/d8/f28 [0,4194304] 0 2024-03-04T16:04:31.335 INFO:tasks.workunit.client.1.smithi152.stdout:8/216: link d2/c3c d2/d3/c47 0 2024-03-04T16:04:31.335 INFO:tasks.workunit.client.1.smithi152.stdout:8/217: chown d2/l46 7 1 2024-03-04T16:04:31.337 INFO:tasks.workunit.client.1.smithi152.stdout:8/218: rmdir d2/d3/d21/d33 39 2024-03-04T16:04:31.339 INFO:tasks.workunit.client.1.smithi152.stdout:8/219: mkdir d2/d3/d21/d2d/d48 0 2024-03-04T16:04:31.343 INFO:tasks.workunit.client.1.smithi152.stdout:7/144: sync 2024-03-04T16:04:31.343 INFO:tasks.workunit.client.1.smithi152.stdout:6/219: sync 2024-03-04T16:04:31.343 INFO:tasks.workunit.client.1.smithi152.stdout:7/145: chown c6 1961484 1 2024-03-04T16:04:31.345 INFO:tasks.workunit.client.1.smithi152.stdout:8/220: dwrite d2/d8/f1a [0,4194304] 0 2024-03-04T16:04:31.348 INFO:tasks.workunit.client.1.smithi152.stdout:7/146: rename d14/c1d to d14/d21/c28 0 2024-03-04T16:04:31.349 INFO:tasks.workunit.client.1.smithi152.stdout:7/147: chown d14/d21 76594 1 2024-03-04T16:04:31.351 INFO:tasks.workunit.client.1.smithi152.stdout:6/220: dwrite d8/d10/f24 [0,4194304] 0 2024-03-04T16:04:31.351 INFO:tasks.workunit.client.1.smithi152.stdout:7/148: mknod d14/c29 0 2024-03-04T16:04:31.354 INFO:tasks.workunit.client.1.smithi152.stdout:6/221: unlink d8/fe 0 2024-03-04T16:04:31.358 INFO:tasks.workunit.client.1.smithi152.stdout:6/222: mknod d8/d2a/c47 0 2024-03-04T16:04:31.360 INFO:tasks.workunit.client.1.smithi152.stdout:7/149: rename d14/f1e to d14/f2a 0 2024-03-04T16:04:31.362 INFO:tasks.workunit.client.1.smithi152.stdout:7/150: mkdir d14/d2b 0 2024-03-04T16:04:31.363 INFO:tasks.workunit.client.1.smithi152.stdout:7/151: readlink lc 0 2024-03-04T16:04:31.364 INFO:tasks.workunit.client.1.smithi152.stdout:7/152: creat d14/f2c x:0 0 0 2024-03-04T16:04:31.365 INFO:tasks.workunit.client.1.smithi152.stdout:7/153: mknod d14/c2d 0 2024-03-04T16:04:31.367 INFO:tasks.workunit.client.1.smithi152.stdout:7/154: mknod d14/c2e 0 2024-03-04T16:04:31.368 INFO:tasks.workunit.client.1.smithi152.stdout:7/155: rmdir d14/d17 39 2024-03-04T16:04:31.368 INFO:tasks.workunit.client.1.smithi152.stdout:7/156: stat l3 0 2024-03-04T16:04:31.369 INFO:tasks.workunit.client.1.smithi152.stdout:7/157: write f12 [1076855,90878] 0 2024-03-04T16:04:31.374 INFO:tasks.workunit.client.1.smithi152.stdout:6/223: dread d8/f12 [0,4194304] 0 2024-03-04T16:04:31.375 INFO:tasks.workunit.client.1.smithi152.stdout:7/158: creat d14/f2f x:0 0 0 2024-03-04T16:04:31.375 INFO:tasks.workunit.client.1.smithi152.stdout:6/224: fsync f1 0 2024-03-04T16:04:31.383 INFO:tasks.workunit.client.1.smithi152.stdout:7/159: rmdir d14/d17 39 2024-03-04T16:04:31.387 INFO:tasks.workunit.client.1.smithi152.stdout:6/225: dread d8/d2a/f2e [0,4194304] 0 2024-03-04T16:04:31.388 INFO:tasks.workunit.client.1.smithi152.stdout:7/160: link d14/f1f d14/d21/f30 0 2024-03-04T16:04:31.390 INFO:tasks.workunit.client.1.smithi152.stdout:6/226: rmdir d8/d2a 39 2024-03-04T16:04:31.392 INFO:tasks.workunit.client.1.smithi152.stdout:7/161: chown d14/f1c 77465 1 2024-03-04T16:04:31.396 INFO:tasks.workunit.client.1.smithi152.stdout:7/162: dread f12 [0,4194304] 0 2024-03-04T16:04:31.403 INFO:tasks.workunit.client.1.smithi152.stdout:7/163: mkdir d14/d21/d31 0 2024-03-04T16:04:31.404 INFO:tasks.workunit.client.1.smithi152.stdout:7/164: dread d14/f1c [0,4194304] 0 2024-03-04T16:04:31.407 INFO:tasks.workunit.client.1.smithi152.stdout:6/227: dwrite d8/d2a/f2e [4194304,4194304] 0 2024-03-04T16:04:31.409 INFO:tasks.workunit.client.1.smithi152.stdout:7/165: mkdir d14/d17/d32 0 2024-03-04T16:04:31.416 INFO:tasks.workunit.client.1.smithi152.stdout:7/166: creat d14/d21/d31/f33 x:0 0 0 2024-03-04T16:04:31.418 INFO:tasks.workunit.client.1.smithi152.stdout:7/167: fdatasync d14/f1c 0 2024-03-04T16:04:31.420 INFO:tasks.workunit.client.1.smithi152.stdout:7/168: creat d14/d17/f34 x:0 0 0 2024-03-04T16:04:31.423 INFO:tasks.workunit.client.1.smithi152.stdout:7/169: rename ff to d14/d17/f35 0 2024-03-04T16:04:31.425 INFO:tasks.workunit.client.1.smithi152.stdout:7/170: mkdir d14/d17/d36 0 2024-03-04T16:04:31.431 INFO:tasks.workunit.client.1.smithi152.stdout:6/228: truncate d8/d23/d3a/f3f 1967433 0 2024-03-04T16:04:31.431 INFO:tasks.workunit.client.1.smithi152.stdout:6/229: fsync d8/d23/f25 0 2024-03-04T16:04:31.432 INFO:tasks.workunit.client.1.smithi152.stdout:6/230: chown d8/d2a/c37 18 1 2024-03-04T16:04:31.433 INFO:tasks.workunit.client.1.smithi152.stdout:6/231: write d8/d2a/f2e [9206177,72759] 0 2024-03-04T16:04:31.434 INFO:tasks.workunit.client.1.smithi152.stdout:6/232: stat d8/d23/f2c 0 2024-03-04T16:04:31.436 INFO:tasks.workunit.client.1.smithi152.stdout:6/233: creat d8/d38/f48 x:0 0 0 2024-03-04T16:04:31.437 INFO:tasks.workunit.client.1.smithi152.stdout:7/171: link d14/d21/c28 d14/c37 0 2024-03-04T16:04:31.437 INFO:tasks.workunit.client.1.smithi152.stdout:7/172: stat d14/d21/c26 0 2024-03-04T16:04:31.447 INFO:tasks.workunit.client.1.smithi152.stdout:7/173: rmdir d14 39 2024-03-04T16:04:31.450 INFO:tasks.workunit.client.1.smithi152.stdout:6/234: write d8/f12 [1621615,82761] 0 2024-03-04T16:04:31.453 INFO:tasks.workunit.client.1.smithi152.stdout:7/174: mknod d14/d17/d36/c38 0 2024-03-04T16:04:31.455 INFO:tasks.workunit.client.1.smithi152.stdout:6/235: dread d8/d2a/f34 [0,4194304] 0 2024-03-04T16:04:31.459 INFO:tasks.workunit.client.1.smithi152.stdout:6/236: rename l5 to d8/d10/l49 0 2024-03-04T16:04:31.460 INFO:tasks.workunit.client.1.smithi152.stdout:7/175: dwrite d14/d17/f35 [0,4194304] 0 2024-03-04T16:04:31.461 INFO:tasks.workunit.client.1.smithi152.stdout:6/237: fsync d8/d23/f33 0 2024-03-04T16:04:31.463 INFO:tasks.workunit.client.1.smithi152.stdout:7/176: mknod d14/d21/c39 0 2024-03-04T16:04:31.465 INFO:tasks.workunit.client.1.smithi152.stdout:6/238: dread d8/d2a/f36 [0,4194304] 0 2024-03-04T16:04:31.465 INFO:tasks.workunit.client.1.smithi152.stdout:6/239: readlink d8/lf 0 2024-03-04T16:04:31.469 INFO:tasks.workunit.client.1.smithi152.stdout:6/240: creat d8/d42/f4a x:0 0 0 2024-03-04T16:04:31.470 INFO:tasks.workunit.client.1.smithi152.stdout:6/241: stat d8/d38/f48 0 2024-03-04T16:04:31.472 INFO:tasks.workunit.client.1.smithi152.stdout:6/242: fsync d8/d10/f16 0 2024-03-04T16:04:31.473 INFO:tasks.workunit.client.1.smithi152.stdout:7/177: dwrite d14/f16 [0,4194304] 0 2024-03-04T16:04:31.478 INFO:tasks.workunit.client.1.smithi152.stdout:6/243: mkdir d8/d4b 0 2024-03-04T16:04:31.481 INFO:tasks.workunit.client.1.smithi152.stdout:6/244: truncate d8/d10/f18 2572529 0 2024-03-04T16:04:31.487 INFO:tasks.workunit.client.1.smithi152.stdout:7/178: fsync d14/d17/f35 0 2024-03-04T16:04:31.491 INFO:tasks.workunit.client.1.smithi152.stdout:7/179: symlink d14/d21/d31/l3a 0 2024-03-04T16:04:31.492 INFO:tasks.workunit.client.1.smithi152.stdout:6/245: rmdir d8 39 2024-03-04T16:04:31.495 INFO:tasks.workunit.client.1.smithi152.stdout:7/180: creat d14/d2b/f3b x:0 0 0 2024-03-04T16:04:31.496 INFO:tasks.workunit.client.1.smithi152.stdout:7/181: write d14/f2c [727514,104520] 0 2024-03-04T16:04:31.496 INFO:tasks.workunit.client.1.smithi152.stdout:7/182: readlink lc 0 2024-03-04T16:04:31.498 INFO:tasks.workunit.client.1.smithi152.stdout:6/246: mkdir d8/d4c 0 2024-03-04T16:04:31.500 INFO:tasks.workunit.client.1.smithi152.stdout:7/183: unlink d14/f1f 0 2024-03-04T16:04:31.500 INFO:tasks.workunit.client.1.smithi152.stdout:6/247: dread d8/f12 [0,4194304] 0 2024-03-04T16:04:31.501 INFO:tasks.workunit.client.1.smithi152.stdout:6/248: dread - d8/d10/f43 zero size 2024-03-04T16:04:31.505 INFO:tasks.workunit.client.1.smithi152.stdout:6/249: rmdir d8/d2a 39 2024-03-04T16:04:31.507 INFO:tasks.workunit.client.1.smithi152.stdout:7/184: dwrite d14/d2b/f3b [0,4194304] 0 2024-03-04T16:04:31.508 INFO:tasks.workunit.client.1.smithi152.stdout:7/185: chown d14/d17 7693664 1 2024-03-04T16:04:31.521 INFO:tasks.workunit.client.1.smithi152.stdout:6/250: write d8/d10/f44 [1503318,46257] 0 2024-03-04T16:04:31.526 INFO:tasks.workunit.client.1.smithi152.stdout:7/186: dwrite d14/f2a [0,4194304] 0 2024-03-04T16:04:31.528 INFO:tasks.workunit.client.1.smithi152.stdout:7/187: write d14/f2c [1640943,120649] 0 2024-03-04T16:04:31.529 INFO:tasks.workunit.client.1.smithi152.stdout:7/188: write d14/f20 [149910,12512] 0 2024-03-04T16:04:31.535 INFO:tasks.workunit.client.1.smithi152.stdout:7/189: mknod d14/d17/d32/c3c 0 2024-03-04T16:04:31.542 INFO:tasks.workunit.client.1.smithi152.stdout:7/190: getdents d14/d17 0 2024-03-04T16:04:31.543 INFO:tasks.workunit.client.1.smithi152.stdout:7/191: read d14/f16 [2821644,86214] 0 2024-03-04T16:04:31.548 INFO:tasks.workunit.client.1.smithi152.stdout:6/251: dread d8/d10/f18 [0,4194304] 0 2024-03-04T16:04:31.551 INFO:tasks.workunit.client.1.smithi152.stdout:7/192: dwrite d14/d17/f35 [0,4194304] 0 2024-03-04T16:04:31.553 INFO:tasks.workunit.client.1.smithi152.stdout:6/252: fsync d8/f39 0 2024-03-04T16:04:31.555 INFO:tasks.workunit.client.1.smithi152.stdout:7/193: mknod d14/c3d 0 2024-03-04T16:04:31.555 INFO:tasks.workunit.client.1.smithi152.stdout:3/157: sync 2024-03-04T16:04:31.559 INFO:tasks.workunit.client.1.smithi152.stdout:7/194: dread f4 [0,4194304] 0 2024-03-04T16:04:31.560 INFO:tasks.workunit.client.1.smithi152.stdout:7/195: write d14/f2a [3900308,39043] 0 2024-03-04T16:04:31.567 INFO:tasks.workunit.client.1.smithi152.stdout:6/253: dwrite d8/d2a/f3c [0,4194304] 0 2024-03-04T16:04:31.567 INFO:tasks.workunit.client.1.smithi152.stdout:3/158: mknod d6/d10/d2d/c30 0 2024-03-04T16:04:31.575 INFO:tasks.workunit.client.1.smithi152.stdout:3/159: dread d6/fc [0,4194304] 0 2024-03-04T16:04:31.576 INFO:tasks.workunit.client.1.smithi152.stdout:6/254: rmdir d8/d2a 39 2024-03-04T16:04:31.583 INFO:tasks.workunit.client.1.smithi152.stdout:3/160: dwrite d6/d10/f20 [0,4194304] 0 2024-03-04T16:04:31.591 INFO:tasks.workunit.client.1.smithi152.stdout:1/181: sync 2024-03-04T16:04:31.591 INFO:tasks.workunit.client.1.smithi152.stdout:8/221: sync 2024-03-04T16:04:31.591 INFO:tasks.workunit.client.1.smithi152.stdout:5/257: sync 2024-03-04T16:04:31.591 INFO:tasks.workunit.client.1.smithi152.stdout:9/168: sync 2024-03-04T16:04:31.591 INFO:tasks.workunit.client.1.smithi152.stdout:0/176: sync 2024-03-04T16:04:31.591 INFO:tasks.workunit.client.1.smithi152.stdout:4/169: sync 2024-03-04T16:04:31.591 INFO:tasks.workunit.client.1.smithi152.stdout:2/169: sync 2024-03-04T16:04:31.592 INFO:tasks.workunit.client.1.smithi152.stdout:4/170: fdatasync f0 0 2024-03-04T16:04:31.592 INFO:tasks.workunit.client.1.smithi152.stdout:7/196: dread d14/d17/f18 [0,4194304] 0 2024-03-04T16:04:31.593 INFO:tasks.workunit.client.1.smithi152.stdout:1/182: stat d19/f32 0 2024-03-04T16:04:31.594 INFO:tasks.workunit.client.1.smithi152.stdout:3/161: dwrite f3 [0,4194304] 0 2024-03-04T16:04:31.594 INFO:tasks.workunit.client.1.smithi152.stdout:7/197: chown d14 35 1 2024-03-04T16:04:31.595 INFO:tasks.workunit.client.1.smithi152.stdout:3/162: chown d6/f1a 570 1 2024-03-04T16:04:31.599 INFO:tasks.workunit.client.1.smithi152.stdout:3/163: write d6/d10/f27 [3355096,43452] 0 2024-03-04T16:04:31.601 INFO:tasks.workunit.client.1.smithi152.stdout:0/177: write f6 [4766788,20654] 0 2024-03-04T16:04:31.601 INFO:tasks.workunit.client.1.smithi152.stdout:1/183: dread f10 [0,4194304] 0 2024-03-04T16:04:31.607 INFO:tasks.workunit.client.1.smithi152.stdout:6/255: truncate d8/f39 1349146 0 2024-03-04T16:04:31.608 INFO:tasks.workunit.client.1.smithi152.stdout:1/184: truncate d19/f28 381364 0 2024-03-04T16:04:31.609 INFO:tasks.workunit.client.1.smithi152.stdout:4/171: symlink d8/d9/l3a 0 2024-03-04T16:04:31.611 INFO:tasks.workunit.client.1.smithi152.stdout:2/170: dwrite da/df/f14 [0,4194304] 0 2024-03-04T16:04:31.611 INFO:tasks.workunit.client.1.smithi152.stdout:5/258: dwrite d6/d8/f2f [0,4194304] 0 2024-03-04T16:04:31.612 INFO:tasks.workunit.client.1.smithi152.stdout:6/256: chown d8/d42/f4a 401 1 2024-03-04T16:04:31.614 INFO:tasks.workunit.client.1.smithi152.stdout:7/198: creat d14/d21/d31/f3e x:0 0 0 2024-03-04T16:04:31.614 INFO:tasks.workunit.client.1.smithi152.stdout:2/171: chown da/f3c 455649683 1 2024-03-04T16:04:31.615 INFO:tasks.workunit.client.1.smithi152.stdout:9/169: dread d2/d13/f18 [0,4194304] 0 2024-03-04T16:04:31.630 INFO:tasks.workunit.client.1.smithi152.stdout:9/170: rmdir d2 39 2024-03-04T16:04:31.633 INFO:tasks.workunit.client.1.smithi152.stdout:2/172: readlink da/df/d2e/l3b 0 2024-03-04T16:04:31.634 INFO:tasks.workunit.client.1.smithi152.stdout:2/173: truncate da/d24/f42 630068 0 2024-03-04T16:04:31.635 INFO:tasks.workunit.client.1.smithi152.stdout:2/174: fsync da/f1a 0 2024-03-04T16:04:31.635 INFO:tasks.workunit.client.1.smithi152.stdout:3/164: mknod d6/c31 0 2024-03-04T16:04:31.636 INFO:tasks.workunit.client.1.smithi152.stdout:0/178: creat d9/d2c/d2e/f31 x:0 0 0 2024-03-04T16:04:31.640 INFO:tasks.workunit.client.1.smithi152.stdout:1/185: creat d19/d20/d21/d38/f3a x:0 0 0 2024-03-04T16:04:31.641 INFO:tasks.workunit.client.1.smithi152.stdout:2/175: dwrite da/d24/f42 [0,4194304] 0 2024-03-04T16:04:31.641 INFO:tasks.workunit.client.1.smithi152.stdout:1/186: fsync f16 0 2024-03-04T16:04:31.642 INFO:tasks.workunit.client.1.smithi152.stdout:2/176: chown da/d24/f2a 1326543384 1 2024-03-04T16:04:31.643 INFO:tasks.workunit.client.1.smithi152.stdout:6/257: symlink d8/d2a/l4d 0 2024-03-04T16:04:31.645 INFO:tasks.workunit.client.1.smithi152.stdout:4/172: rmdir d8 39 2024-03-04T16:04:31.645 INFO:tasks.workunit.client.1.smithi152.stdout:6/258: write d8/d2a/f3e [143324,30127] 0 2024-03-04T16:04:31.648 INFO:tasks.workunit.client.1.smithi152.stdout:7/199: rename d14/c37 to d14/d21/c3f 0 2024-03-04T16:04:31.651 INFO:tasks.workunit.client.1.smithi152.stdout:5/259: link d6/d8/f24 d6/d8/d1f/d3d/f5d 0 2024-03-04T16:04:31.653 INFO:tasks.workunit.client.1.smithi152.stdout:5/260: dread d6/d8/d1f/f48 [0,4194304] 0 2024-03-04T16:04:31.659 INFO:tasks.workunit.client.1.smithi152.stdout:0/179: creat d9/d2c/d2e/f32 x:0 0 0 2024-03-04T16:04:31.662 INFO:tasks.workunit.client.1.smithi152.stdout:3/165: rmdir d6 39 2024-03-04T16:04:31.662 INFO:tasks.workunit.client.1.smithi152.stdout:0/180: dread d9/d1b/f23 [0,4194304] 0 2024-03-04T16:04:31.663 INFO:tasks.workunit.client.1.smithi152.stdout:0/181: write d9/d1b/f2a [16615,74608] 0 2024-03-04T16:04:31.667 INFO:tasks.workunit.client.1.smithi152.stdout:4/173: readlink d8/la 0 2024-03-04T16:04:31.671 INFO:tasks.workunit.client.1.smithi152.stdout:6/259: symlink d8/d23/d3a/l4e 0 2024-03-04T16:04:31.674 INFO:tasks.workunit.client.1.smithi152.stdout:7/200: rmdir d14/d17 39 2024-03-04T16:04:31.679 INFO:tasks.workunit.client.1.smithi152.stdout:0/182: mknod d9/d1b/c33 0 2024-03-04T16:04:31.683 INFO:tasks.workunit.client.1.smithi152.stdout:6/260: mkdir d8/d38/d4f 0 2024-03-04T16:04:31.684 INFO:tasks.workunit.client.1.smithi152.stdout:9/171: creat d2/d13/d2e/f37 x:0 0 0 2024-03-04T16:04:31.685 INFO:tasks.workunit.client.1.smithi152.stdout:9/172: chown d2/d13/f15 0 1 2024-03-04T16:04:31.693 INFO:tasks.workunit.client.1.smithi152.stdout:1/187: truncate d19/f32 1144741 0 2024-03-04T16:04:31.693 INFO:tasks.workunit.client.1.smithi152.stdout:1/188: chown cf 25565 1 2024-03-04T16:04:31.693 INFO:tasks.workunit.client.1.smithi152.stdout:1/189: stat d19/d20/d2f 0 2024-03-04T16:04:31.699 INFO:tasks.workunit.client.1.smithi152.stdout:6/261: creat d8/d23/d3a/f50 x:0 0 0 2024-03-04T16:04:31.700 INFO:tasks.workunit.client.1.smithi152.stdout:4/174: truncate f3 2162797 0 2024-03-04T16:04:31.701 INFO:tasks.workunit.client.1.smithi152.stdout:9/173: read - d2/d13/d23/f33 zero size 2024-03-04T16:04:31.704 INFO:tasks.workunit.client.1.smithi152.stdout:7/201: mkdir d14/d17/d40 0 2024-03-04T16:04:31.707 INFO:tasks.workunit.client.1.smithi152.stdout:3/166: dread d6/f9 [0,4194304] 0 2024-03-04T16:04:31.707 INFO:tasks.workunit.client.1.smithi152.stdout:9/174: dwrite d2/d13/d2e/f37 [0,4194304] 0 2024-03-04T16:04:31.709 INFO:tasks.workunit.client.1.smithi152.stdout:1/190: mkdir d19/d3b 0 2024-03-04T16:04:31.710 INFO:tasks.workunit.client.1.smithi152.stdout:6/262: creat d8/d42/f51 x:0 0 0 2024-03-04T16:04:31.710 INFO:tasks.workunit.client.1.smithi152.stdout:6/263: chown f1 1013955 1 2024-03-04T16:04:31.717 INFO:tasks.workunit.client.1.smithi152.stdout:5/261: getdents d6/d50 0 2024-03-04T16:04:31.718 INFO:tasks.workunit.client.1.smithi152.stdout:5/262: chown d6/fa 9 1 2024-03-04T16:04:31.721 INFO:tasks.workunit.client.1.smithi152.stdout:3/167: rename d6/d14/f2a to d6/d10/d2d/f32 0 2024-03-04T16:04:31.727 INFO:tasks.workunit.client.1.smithi152.stdout:9/175: mkdir d2/d13/d38 0 2024-03-04T16:04:31.733 INFO:tasks.workunit.client.1.smithi152.stdout:3/168: symlink d6/d10/d2d/l33 0 2024-03-04T16:04:31.733 INFO:tasks.workunit.client.1.smithi152.stdout:7/202: link d14/d21/l27 d14/d17/d32/l41 0 2024-03-04T16:04:31.734 INFO:tasks.workunit.client.1.smithi152.stdout:6/264: dread d8/f1f [0,4194304] 0 2024-03-04T16:04:31.736 INFO:tasks.workunit.client.1.smithi152.stdout:5/263: symlink d6/d8/d1f/d3d/l5e 0 2024-03-04T16:04:31.737 INFO:tasks.workunit.client.1.smithi152.stdout:9/176: symlink d2/d13/d23/l39 0 2024-03-04T16:04:31.738 INFO:tasks.workunit.client.1.smithi152.stdout:9/177: write d2/d13/d23/f33 [763639,100066] 0 2024-03-04T16:04:31.738 INFO:tasks.workunit.client.1.smithi152.stdout:1/191: link d19/d20/l31 d19/d20/d21/d38/l3c 0 2024-03-04T16:04:31.740 INFO:tasks.workunit.client.1.smithi152.stdout:7/203: fdatasync f4 0 2024-03-04T16:04:31.741 INFO:tasks.workunit.client.1.smithi152.stdout:7/204: readlink d14/d21/l24 0 2024-03-04T16:04:31.743 INFO:tasks.workunit.client.1.smithi152.stdout:7/205: write d14/d21/f25 [1228325,9624] 0 2024-03-04T16:04:31.744 INFO:tasks.workunit.client.1.smithi152.stdout:7/206: write d14/d2b/f3b [2571289,50414] 0 2024-03-04T16:04:31.745 INFO:tasks.workunit.client.1.smithi152.stdout:3/169: rename d6/d14/f2b to d6/d10/d2d/f34 0 2024-03-04T16:04:31.746 INFO:tasks.workunit.client.1.smithi152.stdout:5/264: symlink d6/d50/l5f 0 2024-03-04T16:04:31.748 INFO:tasks.workunit.client.1.smithi152.stdout:4/175: dwrite f3 [0,4194304] 0 2024-03-04T16:04:31.748 INFO:tasks.workunit.client.1.smithi152.stdout:4/176: chown d8 57984 1 2024-03-04T16:04:31.754 INFO:tasks.workunit.client.1.smithi152.stdout:6/265: mkdir d8/d52 0 2024-03-04T16:04:31.755 INFO:tasks.workunit.client.1.smithi152.stdout:6/266: write d8/f28 [3360601,57120] 0 2024-03-04T16:04:31.759 INFO:tasks.workunit.client.1.smithi152.stdout:6/267: dread d8/d23/f33 [0,4194304] 0 2024-03-04T16:04:31.760 INFO:tasks.workunit.client.1.smithi152.stdout:6/268: write d8/d2a/f45 [571706,8477] 0 2024-03-04T16:04:31.761 INFO:tasks.workunit.client.1.smithi152.stdout:3/170: readlink d6/l16 0 2024-03-04T16:04:31.765 INFO:tasks.workunit.client.1.smithi152.stdout:4/177: creat d8/f3b x:0 0 0 2024-03-04T16:04:31.765 INFO:tasks.workunit.client.1.smithi152.stdout:9/178: write d2/d13/f18 [1231291,85165] 0 2024-03-04T16:04:31.771 INFO:tasks.workunit.client.1.smithi152.stdout:5/265: dwrite d6/d8/d10/f12 [4194304,4194304] 0 2024-03-04T16:04:31.776 INFO:tasks.workunit.client.1.smithi152.stdout:9/179: mknod d2/d13/d2e/c3a 0 2024-03-04T16:04:31.776 INFO:tasks.workunit.client.1.smithi152.stdout:9/180: chown d2/f1c 3 1 2024-03-04T16:04:31.778 INFO:tasks.workunit.client.1.smithi152.stdout:6/269: truncate d8/f39 286584 0 2024-03-04T16:04:31.780 INFO:tasks.workunit.client.1.smithi152.stdout:3/171: mknod d6/c35 0 2024-03-04T16:04:31.781 INFO:tasks.workunit.client.1.smithi152.stdout:7/207: creat d14/f42 x:0 0 0 2024-03-04T16:04:31.782 INFO:tasks.workunit.client.1.smithi152.stdout:5/266: symlink d6/d50/l60 0 2024-03-04T16:04:31.782 INFO:tasks.workunit.client.1.smithi152.stdout:5/267: write d6/f4e [461155,19379] 0 2024-03-04T16:04:31.784 INFO:tasks.workunit.client.1.smithi152.stdout:5/268: truncate d6/d8/f41 573739 0 2024-03-04T16:04:31.791 INFO:tasks.workunit.client.1.smithi152.stdout:7/208: readlink d14/d21/l27 0 2024-03-04T16:04:31.793 INFO:tasks.workunit.client.1.smithi152.stdout:1/192: link c7 d19/d20/c3d 0 2024-03-04T16:04:31.796 INFO:tasks.workunit.client.1.smithi152.stdout:3/172: dwrite d6/d10/d2d/f32 [0,4194304] 0 2024-03-04T16:04:31.799 INFO:tasks.workunit.client.1.smithi152.stdout:5/269: unlink d6/d8/d10/f21 0 2024-03-04T16:04:31.799 INFO:tasks.workunit.client.1.smithi152.stdout:4/178: link d8/cc d8/d2f/c3c 0 2024-03-04T16:04:31.799 INFO:tasks.workunit.client.1.smithi152.stdout:4/179: dread - d8/f23 zero size 2024-03-04T16:04:31.802 INFO:tasks.workunit.client.1.smithi152.stdout:1/193: fdatasync f1 0 2024-03-04T16:04:31.804 INFO:tasks.workunit.client.1.smithi152.stdout:9/181: link d2/c16 d2/d13/d23/c3b 0 2024-03-04T16:04:31.804 INFO:tasks.workunit.client.1.smithi152.stdout:1/194: read d19/d20/d21/f2e [115317,59674] 0 2024-03-04T16:04:31.806 INFO:tasks.workunit.client.1.smithi152.stdout:9/182: chown d2/d13/l20 7106170 1 2024-03-04T16:04:31.808 INFO:tasks.workunit.client.1.smithi152.stdout:6/270: rmdir d8/d4c 0 2024-03-04T16:04:31.809 INFO:tasks.workunit.client.1.smithi152.stdout:4/180: dwrite d8/f23 [0,4194304] 0 2024-03-04T16:04:31.809 INFO:tasks.workunit.client.1.smithi152.stdout:5/270: mkdir d6/d8/d1f/d3d/d61 0 2024-03-04T16:04:31.810 INFO:tasks.workunit.client.1.smithi152.stdout:6/271: dread - d8/d42/f4a zero size 2024-03-04T16:04:31.810 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:31 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:31.810 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:31 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:31.810 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:31 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:31.811 INFO:tasks.workunit.client.1.smithi152.stdout:7/209: creat d14/d17/d40/f43 x:0 0 0 2024-03-04T16:04:31.812 INFO:tasks.workunit.client.1.smithi152.stdout:5/271: chown d6/d8/l33 1787307 1 2024-03-04T16:04:31.812 INFO:tasks.workunit.client.1.smithi152.stdout:6/272: dread - d8/d23/d3a/f50 zero size 2024-03-04T16:04:31.816 INFO:tasks.workunit.client.1.smithi152.stdout:5/272: write d6/d8/d10/f43 [866925,114942] 0 2024-03-04T16:04:31.820 INFO:tasks.workunit.client.1.smithi152.stdout:4/181: dwrite d8/f23 [0,4194304] 0 2024-03-04T16:04:31.820 INFO:tasks.workunit.client.1.smithi152.stdout:9/183: creat d2/d13/d2e/f3c x:0 0 0 2024-03-04T16:04:31.821 INFO:tasks.workunit.client.1.smithi152.stdout:4/182: dread - d8/d2f/f34 zero size 2024-03-04T16:04:31.827 INFO:tasks.workunit.client.1.smithi152.stdout:6/273: write d8/d10/f18 [1945493,110531] 0 2024-03-04T16:04:31.829 INFO:tasks.workunit.client.1.smithi152.stdout:4/183: dwrite f3 [0,4194304] 0 2024-03-04T16:04:31.830 INFO:tasks.workunit.client.1.smithi152.stdout:4/184: chown d8/f25 6410760 1 2024-03-04T16:04:31.838 INFO:tasks.workunit.client.1.smithi152.stdout:5/273: mknod d6/d8/d1f/c62 0 2024-03-04T16:04:31.839 INFO:tasks.workunit.client.1.smithi152.stdout:5/274: write d6/f39 [1995158,25447] 0 2024-03-04T16:04:31.841 INFO:tasks.workunit.client.1.smithi152.stdout:9/184: mknod d2/d13/d35/c3d 0 2024-03-04T16:04:31.845 INFO:tasks.workunit.client.1.smithi152.stdout:6/274: mkdir d8/d10/d53 0 2024-03-04T16:04:31.852 INFO:tasks.workunit.client.1.smithi152.stdout:6/275: dwrite d8/f22 [0,4194304] 0 2024-03-04T16:04:31.855 INFO:tasks.workunit.client.1.smithi152.stdout:3/173: dwrite d6/fc [0,4194304] 0 2024-03-04T16:04:31.855 INFO:tasks.workunit.client.1.smithi152.stdout:3/174: read - d6/f2c zero size 2024-03-04T16:04:31.857 INFO:tasks.workunit.client.1.smithi152.stdout:6/276: write d8/d10/f44 [1416258,25378] 0 2024-03-04T16:04:31.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:31 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:31.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:31 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:31.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:31 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:31.859 INFO:tasks.workunit.client.1.smithi152.stdout:3/175: chown d6/d10/d2d/f34 261960 1 2024-03-04T16:04:31.860 INFO:tasks.workunit.client.1.smithi152.stdout:3/176: write d6/d10/f27 [3964193,116546] 0 2024-03-04T16:04:31.860 INFO:tasks.workunit.client.1.smithi152.stdout:3/177: dread - d6/d14/f2e zero size 2024-03-04T16:04:31.861 INFO:tasks.workunit.client.1.smithi152.stdout:4/185: write d8/d9/d24/d26/f2e [234799,128063] 0 2024-03-04T16:04:31.862 INFO:tasks.workunit.client.1.smithi152.stdout:3/178: readlink d6/d10/d2d/l33 0 2024-03-04T16:04:31.863 INFO:tasks.workunit.client.1.smithi152.stdout:3/179: write d6/f2c [263979,32141] 0 2024-03-04T16:04:31.874 INFO:tasks.workunit.client.1.smithi152.stdout:3/180: rename d6/d14/f29 to d6/d10/d2d/f36 0 2024-03-04T16:04:31.876 INFO:tasks.workunit.client.1.smithi152.stdout:5/275: link c4 d6/d8/d1f/d3d/d61/c63 0 2024-03-04T16:04:31.879 INFO:tasks.workunit.client.1.smithi152.stdout:9/185: link d2/d13/c25 d2/c3e 0 2024-03-04T16:04:31.879 INFO:tasks.workunit.client.1.smithi152.stdout:9/186: stat d2/fe 0 2024-03-04T16:04:31.880 INFO:tasks.workunit.client.1.smithi152.stdout:9/187: chown d2/d13/c22 6 1 2024-03-04T16:04:31.881 INFO:tasks.workunit.client.1.smithi152.stdout:5/276: mkdir d6/d8/d5a/d64 0 2024-03-04T16:04:31.884 INFO:tasks.workunit.client.1.smithi152.stdout:5/277: dread d6/d8/d10/f2d [0,4194304] 0 2024-03-04T16:04:31.884 INFO:tasks.workunit.client.1.smithi152.stdout:3/181: getdents d6/d10 0 2024-03-04T16:04:31.889 INFO:tasks.workunit.client.1.smithi152.stdout:5/278: mknod d6/d8/d1f/d3d/d61/c65 0 2024-03-04T16:04:31.890 INFO:tasks.workunit.client.1.smithi152.stdout:9/188: dwrite d2/d13/d23/f29 [0,4194304] 0 2024-03-04T16:04:31.895 INFO:tasks.workunit.client.1.smithi152.stdout:5/279: unlink d6/d8/d1f/f23 0 2024-03-04T16:04:31.897 INFO:tasks.workunit.client.1.smithi152.stdout:3/182: link d6/f23 d6/d14/f37 0 2024-03-04T16:04:31.897 INFO:tasks.workunit.client.1.smithi152.stdout:9/189: dwrite d2/d13/d23/f33 [0,4194304] 0 2024-03-04T16:04:31.904 INFO:tasks.workunit.client.1.smithi152.stdout:5/280: unlink d6/d8/d1f/d3d/d61/c65 0 2024-03-04T16:04:31.905 INFO:tasks.workunit.client.1.smithi152.stdout:3/183: dwrite d6/d14/f2e [0,4194304] 0 2024-03-04T16:04:31.906 INFO:tasks.workunit.client.1.smithi152.stdout:5/281: write d6/fd [431266,47378] 0 2024-03-04T16:04:31.907 INFO:tasks.workunit.client.1.smithi152.stdout:5/282: chown d6/d50 1341966 1 2024-03-04T16:04:31.919 INFO:tasks.workunit.client.1.smithi152.stdout:9/190: getdents d2 0 2024-03-04T16:04:31.925 INFO:tasks.workunit.client.1.smithi152.stdout:9/191: mknod d2/d13/d38/c3f 0 2024-03-04T16:04:31.926 INFO:tasks.workunit.client.1.smithi152.stdout:9/192: write d2/f19 [547006,124863] 0 2024-03-04T16:04:31.943 INFO:tasks.workunit.client.1.smithi152.stdout:3/184: dread d6/d10/f1e [0,4194304] 0 2024-03-04T16:04:31.947 INFO:tasks.workunit.client.1.smithi152.stdout:3/185: creat d6/d10/d2d/f38 x:0 0 0 2024-03-04T16:04:31.952 INFO:tasks.workunit.client.1.smithi152.stdout:3/186: creat d6/d10/f39 x:0 0 0 2024-03-04T16:04:31.961 INFO:tasks.workunit.client.1.smithi152.stdout:3/187: getdents d6/d10/d2d 0 2024-03-04T16:04:31.967 INFO:tasks.workunit.client.1.smithi152.stdout:3/188: creat d6/f3a x:0 0 0 2024-03-04T16:04:31.968 INFO:tasks.workunit.client.1.smithi152.stdout:3/189: creat d6/d10/d2d/f3b x:0 0 0 2024-03-04T16:04:31.972 INFO:tasks.workunit.client.1.smithi152.stdout:3/190: dwrite f1 [8388608,4194304] 0 2024-03-04T16:04:31.984 INFO:tasks.workunit.client.1.smithi152.stdout:3/191: rmdir d6/d10 39 2024-03-04T16:04:31.987 INFO:tasks.workunit.client.1.smithi152.stdout:3/192: write d6/d10/f27 [3875604,58668] 0 2024-03-04T16:04:31.989 INFO:tasks.workunit.client.1.smithi152.stdout:3/193: mknod d6/d10/c3c 0 2024-03-04T16:04:31.990 INFO:tasks.workunit.client.1.smithi152.stdout:3/194: dread - d6/d14/f1f zero size 2024-03-04T16:04:31.993 INFO:tasks.workunit.client.1.smithi152.stdout:3/195: symlink d6/d10/l3d 0 2024-03-04T16:04:31.999 INFO:tasks.workunit.client.1.smithi152.stdout:3/196: symlink d6/d10/d2d/l3e 0 2024-03-04T16:04:32.000 INFO:tasks.workunit.client.1.smithi152.stdout:3/197: write d6/d10/d2d/f38 [316434,2655] 0 2024-03-04T16:04:32.001 INFO:tasks.workunit.client.1.smithi152.stdout:3/198: write f5 [1908643,23996] 0 2024-03-04T16:04:32.010 INFO:tasks.workunit.client.1.smithi152.stdout:3/199: getdents d6/d14 0 2024-03-04T16:04:32.014 INFO:tasks.workunit.client.1.smithi152.stdout:3/200: creat d6/f3f x:0 0 0 2024-03-04T16:04:32.016 INFO:tasks.workunit.client.1.smithi152.stdout:3/201: write d6/d10/d2d/f32 [1604794,70616] 0 2024-03-04T16:04:32.019 INFO:tasks.workunit.client.1.smithi152.stdout:3/202: rename f0 to d6/d10/d2d/f40 0 2024-03-04T16:04:32.023 INFO:tasks.workunit.client.1.smithi152.stdout:3/203: dread d6/f9 [0,4194304] 0 2024-03-04T16:04:32.023 INFO:tasks.workunit.client.1.smithi152.stdout:3/204: dread - d6/d10/d2d/f36 zero size 2024-03-04T16:04:32.025 INFO:tasks.workunit.client.1.smithi152.stdout:3/205: mkdir d6/d14/d41 0 2024-03-04T16:04:32.028 INFO:tasks.workunit.client.1.smithi152.stdout:3/206: dread d6/f23 [0,4194304] 0 2024-03-04T16:04:32.036 INFO:tasks.workunit.client.1.smithi152.stdout:3/207: rename f1 to d6/f42 0 2024-03-04T16:04:32.037 INFO:tasks.workunit.client.1.smithi152.stdout:8/222: sync 2024-03-04T16:04:32.039 INFO:tasks.workunit.client.1.smithi152.stdout:3/208: symlink d6/d14/l43 0 2024-03-04T16:04:32.047 INFO:tasks.workunit.client.1.smithi152.stdout:3/209: rename d6/ca to d6/d10/c44 0 2024-03-04T16:04:32.055 INFO:tasks.workunit.client.1.smithi152.stdout:8/223: getdents d2/d3/d15/d1c 0 2024-03-04T16:04:32.056 INFO:tasks.workunit.client.1.smithi152.stdout:8/224: fdatasync d2/d3/d21/d2d/fc 0 2024-03-04T16:04:32.057 INFO:tasks.workunit.client.1.smithi152.stdout:8/225: fdatasync d2/d3/f40 0 2024-03-04T16:04:32.058 INFO:tasks.workunit.client.1.smithi152.stdout:3/210: mknod d6/c45 0 2024-03-04T16:04:32.058 INFO:tasks.workunit.client.1.smithi152.stdout:8/226: write d2/d8/dd/d22/f24 [1019997,31650] 0 2024-03-04T16:04:32.059 INFO:tasks.workunit.client.1.smithi152.stdout:3/211: read d6/f8 [744639,70987] 0 2024-03-04T16:04:32.060 INFO:tasks.workunit.client.1.smithi152.stdout:3/212: chown d6/d10/c3c 0 1 2024-03-04T16:04:32.060 INFO:tasks.workunit.client.1.smithi152.stdout:8/227: write d2/d3/d21/d2d/f1e [2028051,71455] 0 2024-03-04T16:04:32.063 INFO:tasks.workunit.client.1.smithi152.stdout:3/213: rmdir d6/d14 39 2024-03-04T16:04:32.064 INFO:tasks.workunit.client.1.smithi152.stdout:8/228: mkdir d2/d8/dd/d22/d49 0 2024-03-04T16:04:32.070 INFO:tasks.workunit.client.1.smithi152.stdout:3/214: rename d6/d10/d2d/f38 to d6/f46 0 2024-03-04T16:04:32.073 INFO:tasks.workunit.client.1.smithi152.stdout:8/229: dwrite d2/d3/d21/f1b [0,4194304] 0 2024-03-04T16:04:32.074 INFO:tasks.workunit.client.1.smithi152.stdout:8/230: truncate d2/d8/dd/f2f 570587 0 2024-03-04T16:04:32.075 INFO:tasks.workunit.client.1.smithi152.stdout:8/231: chown d2/d3/f40 823 1 2024-03-04T16:04:32.075 INFO:tasks.workunit.client.1.smithi152.stdout:8/232: stat l0 0 2024-03-04T16:04:32.076 INFO:tasks.workunit.client.1.smithi152.stdout:3/215: dread - d6/d14/f1f zero size 2024-03-04T16:04:32.076 INFO:tasks.workunit.client.1.smithi152.stdout:8/233: write d2/d3/d21/f1b [1536575,62269] 0 2024-03-04T16:04:32.078 INFO:tasks.workunit.client.1.smithi152.stdout:3/216: dread d6/d14/f37 [0,4194304] 0 2024-03-04T16:04:32.081 INFO:tasks.workunit.client.1.smithi152.stdout:3/217: fdatasync d6/f8 0 2024-03-04T16:04:32.087 INFO:tasks.workunit.client.1.smithi152.stdout:3/218: mkdir d6/d10/d47 0 2024-03-04T16:04:32.088 INFO:tasks.workunit.client.1.smithi152.stdout:8/234: link d2/d8/dd/d22/l36 d2/d8/dd/d22/d49/l4a 0 2024-03-04T16:04:32.095 INFO:tasks.workunit.client.1.smithi152.stdout:8/235: fdatasync d2/d3/d21/f32 0 2024-03-04T16:04:32.096 INFO:tasks.workunit.client.1.smithi152.stdout:3/219: symlink d6/d14/d41/l48 0 2024-03-04T16:04:32.097 INFO:tasks.workunit.client.1.smithi152.stdout:3/220: write d6/fc [5202818,85257] 0 2024-03-04T16:04:32.098 INFO:tasks.workunit.client.1.smithi152.stdout:8/236: creat d2/d3/f4b x:0 0 0 2024-03-04T16:04:32.102 INFO:tasks.workunit.client.1.smithi152.stdout:3/221: rename d6/d10/d47 to d6/d14/d41/d49 0 2024-03-04T16:04:32.103 INFO:tasks.workunit.client.1.smithi152.stdout:8/237: write d2/d3/d21/d33/f44 [745377,92026] 0 2024-03-04T16:04:32.108 INFO:tasks.workunit.client.1.smithi152.stdout:3/222: rename d6/f23 to d6/f4a 0 2024-03-04T16:04:32.109 INFO:tasks.workunit.client.1.smithi152.stdout:8/238: dread d2/d3/d21/f19 [0,4194304] 0 2024-03-04T16:04:32.109 INFO:tasks.workunit.client.1.smithi152.stdout:3/223: creat d6/d10/d2d/f4b x:0 0 0 2024-03-04T16:04:32.111 INFO:tasks.workunit.client.1.smithi152.stdout:8/239: rmdir d2/d3/d15 39 2024-03-04T16:04:32.113 INFO:tasks.workunit.client.1.smithi152.stdout:3/224: creat d6/d10/d2d/f4c x:0 0 0 2024-03-04T16:04:32.117 INFO:tasks.workunit.client.1.smithi152.stdout:3/225: mkdir d6/d10/d2d/d4d 0 2024-03-04T16:04:32.118 INFO:tasks.workunit.client.1.smithi152.stdout:8/240: dwrite d2/d8/f28 [0,4194304] 0 2024-03-04T16:04:32.125 INFO:tasks.workunit.client.1.smithi152.stdout:3/226: getdents d6/d14/d41/d49 0 2024-03-04T16:04:32.126 INFO:tasks.workunit.client.1.smithi152.stdout:8/241: symlink d2/d3/d21/d2d/d48/l4c 0 2024-03-04T16:04:32.129 INFO:tasks.workunit.client.1.smithi152.stdout:8/242: truncate d2/d8/dd/d22/f3b 178716 0 2024-03-04T16:04:32.136 INFO:tasks.workunit.client.1.smithi152.stdout:3/227: getdents d6/d10/d2d 0 2024-03-04T16:04:32.137 INFO:tasks.workunit.client.1.smithi152.stdout:8/243: link d2/d8/dd/l11 d2/d3/d21/d2d/l4d 0 2024-03-04T16:04:32.234 INFO:tasks.workunit.client.1.smithi152.stdout:8/244: truncate d2/f42 2891637 0 2024-03-04T16:04:32.234 INFO:tasks.workunit.client.1.smithi152.stdout:8/245: readlink d2/d3/l34 0 2024-03-04T16:04:32.236 INFO:tasks.workunit.client.1.smithi152.stdout:3/228: dwrite d6/d14/f37 [0,4194304] 0 2024-03-04T16:04:32.237 INFO:tasks.workunit.client.1.smithi152.stdout:3/229: fsync f5 0 2024-03-04T16:04:32.242 INFO:tasks.workunit.client.1.smithi152.stdout:3/230: dread d6/d14/f2e [0,4194304] 0 2024-03-04T16:04:32.243 INFO:tasks.workunit.client.1.smithi152.stdout:3/231: fsync d6/d10/d2d/f4c 0 2024-03-04T16:04:32.243 INFO:tasks.workunit.client.1.smithi152.stdout:3/232: stat d6/d10 0 2024-03-04T16:04:32.255 INFO:tasks.workunit.client.1.smithi152.stdout:3/233: mknod d6/d10/d2d/d4d/c4e 0 2024-03-04T16:04:32.264 INFO:tasks.workunit.client.1.smithi152.stdout:3/234: getdents d6/d10 0 2024-03-04T16:04:32.270 INFO:tasks.workunit.client.1.smithi152.stdout:3/235: getdents d6/d10/d2d 0 2024-03-04T16:04:32.270 INFO:tasks.workunit.client.1.smithi152.stdout:3/236: dread - d6/d10/f39 zero size 2024-03-04T16:04:32.274 INFO:tasks.workunit.client.1.smithi152.stdout:3/237: symlink d6/l4f 0 2024-03-04T16:04:32.276 INFO:tasks.workunit.client.1.smithi152.stdout:3/238: write d6/d10/d2d/f32 [2989094,28842] 0 2024-03-04T16:04:32.361 INFO:tasks.workunit.client.1.smithi152.stdout:3/239: dwrite d6/f8 [0,4194304] 0 2024-03-04T16:04:32.370 INFO:tasks.workunit.client.1.smithi152.stdout:3/240: link d6/d10/l3d d6/d14/l50 0 2024-03-04T16:04:32.372 INFO:tasks.workunit.client.1.smithi152.stdout:3/241: write d6/f9 [1095892,93746] 0 2024-03-04T16:04:32.379 INFO:tasks.workunit.client.1.smithi152.stdout:3/242: dwrite f5 [4194304,4194304] 0 2024-03-04T16:04:32.379 INFO:tasks.workunit.client.1.smithi152.stdout:3/243: write d6/ff [1680183,82874] 0 2024-03-04T16:04:32.381 INFO:tasks.workunit.client.1.smithi152.stdout:3/244: rmdir d6/d10/d2d/d4d 39 2024-03-04T16:04:32.382 INFO:tasks.workunit.client.1.smithi152.stdout:3/245: write d6/f1c [44296,111861] 0 2024-03-04T16:04:32.385 INFO:tasks.workunit.client.1.smithi152.stdout:3/246: creat d6/d14/f51 x:0 0 0 2024-03-04T16:04:32.389 INFO:tasks.workunit.client.1.smithi152.stdout:3/247: symlink d6/d10/d2d/d4d/l52 0 2024-03-04T16:04:32.389 INFO:tasks.workunit.client.1.smithi152.stdout:3/248: read f5 [6104726,86771] 0 2024-03-04T16:04:32.393 INFO:tasks.workunit.client.1.smithi152.stdout:3/249: mkdir d6/d53 0 2024-03-04T16:04:32.395 INFO:tasks.workunit.client.1.smithi152.stdout:3/250: mknod d6/d14/c54 0 2024-03-04T16:04:32.400 INFO:tasks.workunit.client.1.smithi152.stdout:3/251: link d6/d10/d2d/f3b d6/d10/d2d/d4d/f55 0 2024-03-04T16:04:32.401 INFO:tasks.workunit.client.1.smithi152.stdout:3/252: stat d6/d10/d2d/l3e 0 2024-03-04T16:04:32.403 INFO:tasks.workunit.client.1.smithi152.stdout:3/253: mknod d6/d10/d2d/c56 0 2024-03-04T16:04:32.405 INFO:tasks.workunit.client.1.smithi152.stdout:3/254: mknod d6/c57 0 2024-03-04T16:04:32.405 INFO:tasks.workunit.client.1.smithi152.stdout:3/255: fdatasync d6/f4a 0 2024-03-04T16:04:32.411 INFO:tasks.workunit.client.1.smithi152.stdout:3/256: dwrite d6/d14/f15 [0,4194304] 0 2024-03-04T16:04:32.415 INFO:tasks.workunit.client.1.smithi152.stdout:3/257: chown d6/d14/l50 193205 1 2024-03-04T16:04:32.424 INFO:tasks.workunit.client.1.smithi152.stdout:3/258: truncate d6/f42 1111353 0 2024-03-04T16:04:32.430 INFO:tasks.workunit.client.1.smithi152.stdout:3/259: dwrite d6/d10/f27 [4194304,4194304] 0 2024-03-04T16:04:32.431 INFO:tasks.workunit.client.1.smithi152.stdout:3/260: write d6/f8 [962251,98124] 0 2024-03-04T16:04:32.487 INFO:tasks.workunit.client.1.smithi152.stdout:2/177: sync 2024-03-04T16:04:32.490 INFO:tasks.workunit.client.1.smithi152.stdout:3/261: unlink f3 0 2024-03-04T16:04:32.490 INFO:tasks.workunit.client.1.smithi152.stdout:3/262: stat d6/d10 0 2024-03-04T16:04:32.492 INFO:tasks.workunit.client.1.smithi152.stdout:3/263: write d6/d14/f37 [4079593,46407] 0 2024-03-04T16:04:32.494 INFO:tasks.workunit.client.1.smithi152.stdout:3/264: mkdir d6/d14/d58 0 2024-03-04T16:04:32.496 INFO:tasks.workunit.client.1.smithi152.stdout:2/178: dwrite da/df/d2e/f3a [0,4194304] 0 2024-03-04T16:04:32.496 INFO:tasks.workunit.client.1.smithi152.stdout:3/265: truncate d6/d14/f1f 574377 0 2024-03-04T16:04:32.497 INFO:tasks.workunit.client.1.smithi152.stdout:2/179: chown da/f1c 76604050 1 2024-03-04T16:04:32.498 INFO:tasks.workunit.client.1.smithi152.stdout:3/266: write d6/f4a [958363,90654] 0 2024-03-04T16:04:32.498 INFO:tasks.workunit.client.1.smithi152.stdout:2/180: rename da/df to da/df/d35/d43 22 2024-03-04T16:04:32.603 INFO:tasks.workunit.client.1.smithi152.stdout:2/181: chown da/df/d35/c36 1805316912 1 2024-03-04T16:04:32.604 INFO:tasks.workunit.client.1.smithi152.stdout:3/267: mkdir d6/d10/d2d/d59 0 2024-03-04T16:04:32.604 INFO:tasks.workunit.client.1.smithi152.stdout:3/268: dread - d6/f3a zero size 2024-03-04T16:04:32.606 INFO:tasks.workunit.client.1.smithi152.stdout:2/182: truncate da/d24/f32 4874717 0 2024-03-04T16:04:32.607 INFO:tasks.workunit.client.1.smithi152.stdout:2/183: read da/f22 [1240237,87709] 0 2024-03-04T16:04:32.609 INFO:tasks.workunit.client.1.smithi152.stdout:2/184: readlink da/df/d2e/l3b 0 2024-03-04T16:04:32.611 INFO:tasks.workunit.client.1.smithi152.stdout:3/269: dwrite d6/d14/f1f [0,4194304] 0 2024-03-04T16:04:32.613 INFO:tasks.workunit.client.1.smithi152.stdout:2/185: creat da/df/d35/f44 x:0 0 0 2024-03-04T16:04:32.616 INFO:tasks.workunit.client.1.smithi152.stdout:3/270: creat d6/d14/d41/d49/f5a x:0 0 0 2024-03-04T16:04:32.617 INFO:tasks.workunit.client.1.smithi152.stdout:3/271: write d6/f1a [1008372,124538] 0 2024-03-04T16:04:32.618 INFO:tasks.workunit.client.1.smithi152.stdout:2/186: creat da/f45 x:0 0 0 2024-03-04T16:04:32.622 INFO:tasks.workunit.client.1.smithi152.stdout:3/272: creat d6/d53/f5b x:0 0 0 2024-03-04T16:04:32.622 INFO:tasks.workunit.client.1.smithi152.stdout:3/273: readlink d6/d10/l1b 0 2024-03-04T16:04:32.627 INFO:tasks.workunit.client.1.smithi152.stdout:2/187: rmdir da/df/d35 39 2024-03-04T16:04:32.630 INFO:tasks.workunit.client.1.smithi152.stdout:3/274: dwrite d6/d10/d2d/f36 [0,4194304] 0 2024-03-04T16:04:32.632 INFO:tasks.workunit.client.1.smithi152.stdout:2/188: creat da/df/d35/f46 x:0 0 0 2024-03-04T16:04:32.637 INFO:tasks.workunit.client.1.smithi152.stdout:2/189: dread da/d24/f2a [0,4194304] 0 2024-03-04T16:04:32.638 INFO:tasks.workunit.client.1.smithi152.stdout:2/190: chown da/d24/l2b 13606 1 2024-03-04T16:04:32.639 INFO:tasks.workunit.client.1.smithi152.stdout:3/275: mkdir d6/d5c 0 2024-03-04T16:04:32.639 INFO:tasks.workunit.client.1.smithi152.stdout:2/191: fdatasync da/df/d2e/f38 0 2024-03-04T16:04:32.643 INFO:tasks.workunit.client.1.smithi152.stdout:2/192: mkdir da/d24/d47 0 2024-03-04T16:04:32.645 INFO:tasks.workunit.client.1.smithi152.stdout:3/276: dwrite d6/d14/f1f [0,4194304] 0 2024-03-04T16:04:32.647 INFO:tasks.workunit.client.1.smithi152.stdout:2/193: rename da/d2c/f34 to da/f48 0 2024-03-04T16:04:32.649 INFO:tasks.workunit.client.1.smithi152.stdout:2/194: write da/fb [3892280,101934] 0 2024-03-04T16:04:32.649 INFO:tasks.workunit.client.1.smithi152.stdout:3/277: symlink d6/d10/d2d/d4d/l5d 0 2024-03-04T16:04:32.651 INFO:tasks.workunit.client.1.smithi152.stdout:3/278: symlink d6/d14/d41/d49/l5e 0 2024-03-04T16:04:32.655 INFO:tasks.workunit.client.1.smithi152.stdout:3/279: mknod d6/d10/c5f 0 2024-03-04T16:04:32.660 INFO:tasks.workunit.client.1.smithi152.stdout:2/195: link da/df/c2d da/d24/c49 0 2024-03-04T16:04:32.661 INFO:tasks.workunit.client.1.smithi152.stdout:3/280: dwrite d6/f4a [0,4194304] 0 2024-03-04T16:04:32.664 INFO:tasks.workunit.client.1.smithi152.stdout:3/281: write d6/d10/d2d/f34 [533034,83001] 0 2024-03-04T16:04:32.666 INFO:tasks.workunit.client.1.smithi152.stdout:3/282: mkdir d6/d60 0 2024-03-04T16:04:32.667 INFO:tasks.workunit.client.1.smithi152.stdout:3/283: fdatasync d6/f9 0 2024-03-04T16:04:32.668 INFO:tasks.workunit.client.1.smithi152.stdout:3/284: symlink d6/d60/l61 0 2024-03-04T16:04:32.671 INFO:tasks.workunit.client.1.smithi152.stdout:3/285: creat d6/d14/d41/d49/f62 x:0 0 0 2024-03-04T16:04:32.674 INFO:tasks.workunit.client.1.smithi152.stdout:3/286: mknod d6/d10/c63 0 2024-03-04T16:04:32.679 INFO:tasks.workunit.client.1.smithi152.stdout:0/183: sync 2024-03-04T16:04:32.683 INFO:tasks.workunit.client.1.smithi152.stdout:0/184: mknod d9/c34 0 2024-03-04T16:04:32.688 INFO:tasks.workunit.client.1.smithi152.stdout:0/185: dread f4 [0,4194304] 0 2024-03-04T16:04:32.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:32 smithi116 ceph-mon[29120]: pgmap v32: 65 pgs: 65 active+clean; 2.5 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 58 MiB/s rd, 181 MiB/s wr, 457 op/s 2024-03-04T16:04:32.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:32 smithi152 ceph-mon[36876]: pgmap v32: 65 pgs: 65 active+clean; 2.5 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 58 MiB/s rd, 181 MiB/s wr, 457 op/s 2024-03-04T16:04:32.911 INFO:tasks.workunit.client.1.smithi152.stdout:1/195: sync 2024-03-04T16:04:32.913 INFO:tasks.workunit.client.1.smithi152.stdout:1/196: creat d19/d20/d21/d38/f3e x:0 0 0 2024-03-04T16:04:32.915 INFO:tasks.workunit.client.1.smithi152.stdout:1/197: symlink d19/d20/d21/l3f 0 2024-03-04T16:04:32.925 INFO:tasks.workunit.client.1.smithi152.stdout:1/198: dread f1 [0,4194304] 0 2024-03-04T16:04:32.927 INFO:tasks.workunit.client.1.smithi152.stdout:1/199: creat d19/d20/d21/f40 x:0 0 0 2024-03-04T16:04:32.931 INFO:tasks.workunit.client.1.smithi152.stdout:1/200: creat d19/d20/f41 x:0 0 0 2024-03-04T16:04:32.949 INFO:tasks.workunit.client.1.smithi152.stdout:1/201: dwrite d19/d20/d21/f2e [0,4194304] 0 2024-03-04T16:04:32.951 INFO:tasks.workunit.client.1.smithi152.stdout:1/202: creat d19/d20/d21/f42 x:0 0 0 2024-03-04T16:04:32.953 INFO:tasks.workunit.client.1.smithi152.stdout:1/203: mknod d19/d20/d21/d38/c43 0 2024-03-04T16:04:32.956 INFO:tasks.workunit.client.1.smithi152.stdout:1/204: symlink d19/d20/d21/l44 0 2024-03-04T16:04:32.970 INFO:tasks.workunit.client.1.smithi152.stdout:1/205: write f10 [721020,114189] 0 2024-03-04T16:04:32.975 INFO:tasks.workunit.client.1.smithi152.stdout:1/206: rename d19/f32 to d19/d20/f45 0 2024-03-04T16:04:32.979 INFO:tasks.workunit.client.1.smithi152.stdout:1/207: rename f18 to d19/d20/d21/f46 0 2024-03-04T16:04:32.983 INFO:tasks.workunit.client.1.smithi152.stdout:1/208: mknod d19/d20/d2f/c47 0 2024-03-04T16:04:32.984 INFO:tasks.workunit.client.1.smithi152.stdout:1/209: readlink d19/d20/d21/l44 0 2024-03-04T16:04:32.989 INFO:tasks.workunit.client.1.smithi152.stdout:1/210: dwrite fa [0,4194304] 0 2024-03-04T16:04:32.992 INFO:tasks.workunit.client.1.smithi152.stdout:1/211: creat d19/f48 x:0 0 0 2024-03-04T16:04:32.994 INFO:tasks.workunit.client.1.smithi152.stdout:1/212: truncate d19/d20/f41 852542 0 2024-03-04T16:04:32.997 INFO:tasks.workunit.client.1.smithi152.stdout:1/213: creat d19/d3b/f49 x:0 0 0 2024-03-04T16:04:33.143 INFO:tasks.workunit.client.1.smithi152.stdout:6/277: sync 2024-03-04T16:04:33.143 INFO:tasks.workunit.client.1.smithi152.stdout:7/210: sync 2024-03-04T16:04:33.144 INFO:tasks.workunit.client.1.smithi152.stdout:6/278: write d8/d38/f48 [410622,105337] 0 2024-03-04T16:04:33.146 INFO:tasks.workunit.client.1.smithi152.stdout:7/211: creat d14/d21/d31/f44 x:0 0 0 2024-03-04T16:04:33.150 INFO:tasks.workunit.client.1.smithi152.stdout:7/212: dread d14/d17/f18 [0,4194304] 0 2024-03-04T16:04:33.150 INFO:tasks.workunit.client.1.smithi152.stdout:7/213: chown f4 1276588081 1 2024-03-04T16:04:33.151 INFO:tasks.workunit.client.1.smithi152.stdout:7/214: write d14/f2f [731315,125690] 0 2024-03-04T16:04:33.158 INFO:tasks.workunit.client.1.smithi152.stdout:7/215: dwrite d14/d17/f18 [4194304,4194304] 0 2024-03-04T16:04:33.158 INFO:tasks.workunit.client.1.smithi152.stdout:6/279: dwrite f7 [4194304,4194304] 0 2024-03-04T16:04:33.161 INFO:tasks.workunit.client.1.smithi152.stdout:6/280: fsync d8/d10/f18 0 2024-03-04T16:04:33.166 INFO:tasks.workunit.client.1.smithi152.stdout:7/216: rename d14/f1c to d14/d21/f45 0 2024-03-04T16:04:33.173 INFO:tasks.workunit.client.1.smithi152.stdout:0/186: sync 2024-03-04T16:04:33.173 INFO:tasks.workunit.client.1.smithi152.stdout:9/193: sync 2024-03-04T16:04:33.173 INFO:tasks.workunit.client.1.smithi152.stdout:5/283: sync 2024-03-04T16:04:33.173 INFO:tasks.workunit.client.1.smithi152.stdout:4/186: sync 2024-03-04T16:04:33.174 INFO:tasks.workunit.client.1.smithi152.stdout:8/246: sync 2024-03-04T16:04:33.174 INFO:tasks.workunit.client.1.smithi152.stdout:2/196: sync 2024-03-04T16:04:33.174 INFO:tasks.workunit.client.1.smithi152.stdout:3/287: sync 2024-03-04T16:04:33.174 INFO:tasks.workunit.client.1.smithi152.stdout:1/214: sync 2024-03-04T16:04:33.174 INFO:tasks.workunit.client.1.smithi152.stdout:0/187: dread - d9/f25 zero size 2024-03-04T16:04:33.177 INFO:tasks.workunit.client.1.smithi152.stdout:2/197: fsync da/f1c 0 2024-03-04T16:04:33.178 INFO:tasks.workunit.client.1.smithi152.stdout:9/194: creat d2/d13/d23/f40 x:0 0 0 2024-03-04T16:04:33.179 INFO:tasks.workunit.client.1.smithi152.stdout:1/215: write f2 [6389135,84246] 0 2024-03-04T16:04:33.183 INFO:tasks.workunit.client.1.smithi152.stdout:6/281: fdatasync f1 0 2024-03-04T16:04:33.183 INFO:tasks.workunit.client.1.smithi152.stdout:0/188: rename d9/f19 to d9/d1b/f35 0 2024-03-04T16:04:33.184 INFO:tasks.workunit.client.1.smithi152.stdout:0/189: chown d9/d1b/f23 13 1 2024-03-04T16:04:33.187 INFO:tasks.workunit.client.1.smithi152.stdout:8/247: unlink d2/d8/dd/d22/d26/c27 0 2024-03-04T16:04:33.188 INFO:tasks.workunit.client.1.smithi152.stdout:8/248: write d2/d8/dd/d22/f24 [2693815,89520] 0 2024-03-04T16:04:33.188 INFO:tasks.workunit.client.1.smithi152.stdout:7/217: truncate d14/f2c 219364 0 2024-03-04T16:04:33.190 INFO:tasks.workunit.client.1.smithi152.stdout:8/249: write d2/d8/f39 [891956,30311] 0 2024-03-04T16:04:33.194 INFO:tasks.workunit.client.1.smithi152.stdout:1/216: creat d19/d20/d21/f4a x:0 0 0 2024-03-04T16:04:33.197 INFO:tasks.workunit.client.1.smithi152.stdout:5/284: dwrite d6/d8/d1f/d3d/f5d [0,4194304] 0 2024-03-04T16:04:33.203 INFO:tasks.workunit.client.1.smithi152.stdout:9/195: dwrite d2/fd [0,4194304] 0 2024-03-04T16:04:33.206 INFO:tasks.workunit.client.1.smithi152.stdout:7/218: chown d14/d21/c3f 288020 1 2024-03-04T16:04:33.206 INFO:tasks.workunit.client.1.smithi152.stdout:2/198: link da/df/d2e/f30 da/df/d35/f4a 0 2024-03-04T16:04:33.207 INFO:tasks.workunit.client.1.smithi152.stdout:2/199: stat da/df/d2e/f39 0 2024-03-04T16:04:33.213 INFO:tasks.workunit.client.1.smithi152.stdout:1/217: rmdir d19/d20/d21 39 2024-03-04T16:04:33.217 INFO:tasks.workunit.client.1.smithi152.stdout:5/285: mknod d6/d8/d10/c66 0 2024-03-04T16:04:33.219 INFO:tasks.workunit.client.1.smithi152.stdout:6/282: creat d8/d4b/f54 x:0 0 0 2024-03-04T16:04:33.226 INFO:tasks.workunit.client.1.smithi152.stdout:9/196: stat d2/c14 0 2024-03-04T16:04:33.227 INFO:tasks.workunit.client.1.smithi152.stdout:9/197: write d2/f36 [509018,28806] 0 2024-03-04T16:04:33.230 INFO:tasks.workunit.client.1.smithi152.stdout:1/218: mkdir d19/d4b 0 2024-03-04T16:04:33.233 INFO:tasks.workunit.client.1.smithi152.stdout:3/288: rename d6/d14/f1f to d6/d10/d2d/f64 0 2024-03-04T16:04:33.236 INFO:tasks.workunit.client.1.smithi152.stdout:0/190: creat d9/f36 x:0 0 0 2024-03-04T16:04:33.240 INFO:tasks.workunit.client.1.smithi152.stdout:7/219: truncate d14/f2a 2142983 0 2024-03-04T16:04:33.241 INFO:tasks.workunit.client.1.smithi152.stdout:9/198: mkdir d2/d13/d35/d41 0 2024-03-04T16:04:33.241 INFO:tasks.workunit.client.1.smithi152.stdout:5/286: write d6/d8/d10/f27 [180842,53735] 0 2024-03-04T16:04:33.244 INFO:tasks.workunit.client.1.smithi152.stdout:2/200: fsync da/df/f1f 0 2024-03-04T16:04:33.249 INFO:tasks.workunit.client.1.smithi152.stdout:9/199: dwrite d2/d13/d23/f33 [0,4194304] 0 2024-03-04T16:04:33.258 INFO:tasks.workunit.client.1.smithi152.stdout:9/200: dwrite d2/f28 [0,4194304] 0 2024-03-04T16:04:33.260 INFO:tasks.workunit.client.1.smithi152.stdout:3/289: fsync d6/d14/f2e 0 2024-03-04T16:04:33.260 INFO:tasks.workunit.client.1.smithi152.stdout:6/283: truncate d8/f2f 94389 0 2024-03-04T16:04:33.263 INFO:tasks.workunit.client.1.smithi152.stdout:5/287: dread d6/d8/d1f/f44 [0,4194304] 0 2024-03-04T16:04:33.265 INFO:tasks.workunit.client.1.smithi152.stdout:2/201: creat da/d2c/f4b x:0 0 0 2024-03-04T16:04:33.265 INFO:tasks.workunit.client.1.smithi152.stdout:8/250: link d2/f42 d2/d8/f4e 0 2024-03-04T16:04:33.266 INFO:tasks.workunit.client.1.smithi152.stdout:5/288: truncate d6/d50/f59 677739 0 2024-03-04T16:04:33.266 INFO:tasks.workunit.client.1.smithi152.stdout:2/202: readlink da/df/d2e/l3b 0 2024-03-04T16:04:33.267 INFO:tasks.workunit.client.1.smithi152.stdout:9/201: dread d2/f28 [0,4194304] 0 2024-03-04T16:04:33.267 INFO:tasks.workunit.client.1.smithi152.stdout:1/219: creat d19/d20/d21/d38/f4c x:0 0 0 2024-03-04T16:04:33.268 INFO:tasks.workunit.client.1.smithi152.stdout:2/203: dread - da/d2c/f4b zero size 2024-03-04T16:04:33.270 INFO:tasks.workunit.client.1.smithi152.stdout:2/204: fdatasync da/df/d35/f44 0 2024-03-04T16:04:33.272 INFO:tasks.workunit.client.1.smithi152.stdout:1/220: chown d19/d20/f41 75898 1 2024-03-04T16:04:33.274 INFO:tasks.workunit.client.1.smithi152.stdout:6/284: creat d8/d23/f55 x:0 0 0 2024-03-04T16:04:33.276 INFO:tasks.workunit.client.1.smithi152.stdout:7/220: fsync d14/d21/f45 0 2024-03-04T16:04:33.276 INFO:tasks.workunit.client.1.smithi152.stdout:8/251: dwrite d2/d3/d21/f1f [0,4194304] 0 2024-03-04T16:04:33.278 INFO:tasks.workunit.client.1.smithi152.stdout:7/221: write d14/f20 [11766,32694] 0 2024-03-04T16:04:33.279 INFO:tasks.workunit.client.1.smithi152.stdout:3/290: creat d6/d10/f65 x:0 0 0 2024-03-04T16:04:33.283 INFO:tasks.workunit.client.1.smithi152.stdout:5/289: chown d6/d50/f52 136153427 1 2024-03-04T16:04:33.286 INFO:tasks.workunit.client.1.smithi152.stdout:9/202: mkdir d2/d13/d23/d42 0 2024-03-04T16:04:33.287 INFO:tasks.workunit.client.1.smithi152.stdout:9/203: write d2/fd [847268,29342] 0 2024-03-04T16:04:33.288 INFO:tasks.workunit.client.1.smithi152.stdout:2/205: rmdir da/d24 39 2024-03-04T16:04:33.293 INFO:tasks.workunit.client.1.smithi152.stdout:0/191: creat d9/f37 x:0 0 0 2024-03-04T16:04:33.300 INFO:tasks.workunit.client.1.smithi152.stdout:2/206: mknod da/d2c/c4c 0 2024-03-04T16:04:33.301 INFO:tasks.workunit.client.1.smithi152.stdout:2/207: chown da/fe 31 1 2024-03-04T16:04:33.301 INFO:tasks.workunit.client.1.smithi152.stdout:9/204: truncate d2/d13/f15 203789 0 2024-03-04T16:04:33.302 INFO:tasks.workunit.client.1.smithi152.stdout:2/208: dread - da/f23 zero size 2024-03-04T16:04:33.311 INFO:tasks.workunit.client.1.smithi152.stdout:0/192: dread d9/d1b/f1d [0,4194304] 0 2024-03-04T16:04:33.312 INFO:tasks.workunit.client.1.smithi152.stdout:3/291: getdents d6/d10/d2d/d59 0 2024-03-04T16:04:33.313 INFO:tasks.workunit.client.1.smithi152.stdout:2/209: dwrite da/d2c/f4b [0,4194304] 0 2024-03-04T16:04:33.313 INFO:tasks.workunit.client.1.smithi152.stdout:0/193: write d9/f26 [606413,48894] 0 2024-03-04T16:04:33.314 INFO:tasks.workunit.client.1.smithi152.stdout:2/210: write da/f1a [1199130,14469] 0 2024-03-04T16:04:33.315 INFO:tasks.workunit.client.1.smithi152.stdout:5/290: truncate d6/d8/f1d 572453 0 2024-03-04T16:04:33.316 INFO:tasks.workunit.client.1.smithi152.stdout:2/211: chown da/d2c/f41 1 1 2024-03-04T16:04:33.322 INFO:tasks.workunit.client.1.smithi152.stdout:6/285: creat d8/f56 x:0 0 0 2024-03-04T16:04:33.322 INFO:tasks.workunit.client.1.smithi152.stdout:4/187: sync 2024-03-04T16:04:33.325 INFO:tasks.workunit.client.1.smithi152.stdout:3/292: dread d6/d10/d2d/f32 [0,4194304] 0 2024-03-04T16:04:33.325 INFO:tasks.workunit.client.1.smithi152.stdout:8/252: dwrite d2/d8/dd/d22/f3b [0,4194304] 0 2024-03-04T16:04:33.326 INFO:tasks.workunit.client.1.smithi152.stdout:4/188: write d8/f10 [2494706,115850] 0 2024-03-04T16:04:33.330 INFO:tasks.workunit.client.1.smithi152.stdout:7/222: dwrite d14/f2a [0,4194304] 0 2024-03-04T16:04:33.330 INFO:tasks.workunit.client.1.smithi152.stdout:4/189: read - d8/f37 zero size 2024-03-04T16:04:33.331 INFO:tasks.workunit.client.1.smithi152.stdout:7/223: chown d14/d21/f45 101105 1 2024-03-04T16:04:33.331 INFO:tasks.workunit.client.1.smithi152.stdout:4/190: dread - d8/f3b zero size 2024-03-04T16:04:33.333 INFO:tasks.workunit.client.1.smithi152.stdout:0/194: mkdir d9/d2c/d2e/d38 0 2024-03-04T16:04:33.334 INFO:tasks.workunit.client.1.smithi152.stdout:7/224: read d14/d21/f25 [2509856,75299] 0 2024-03-04T16:04:33.339 INFO:tasks.workunit.client.1.smithi152.stdout:2/212: dread - da/d24/f31 zero size 2024-03-04T16:04:33.339 INFO:tasks.workunit.client.1.smithi152.stdout:2/213: dread - da/f3c zero size 2024-03-04T16:04:33.340 INFO:tasks.workunit.client.1.smithi152.stdout:5/291: symlink d6/d8/d1f/l67 0 2024-03-04T16:04:33.341 INFO:tasks.workunit.client.1.smithi152.stdout:5/292: chown d6/d8/d1f/d3d/d61 0 1 2024-03-04T16:04:33.345 INFO:tasks.workunit.client.1.smithi152.stdout:1/221: getdents d19/d20/d21 0 2024-03-04T16:04:33.351 INFO:tasks.workunit.client.1.smithi152.stdout:3/293: symlink d6/d10/l66 0 2024-03-04T16:04:33.352 INFO:tasks.workunit.client.1.smithi152.stdout:3/294: write d6/d10/f65 [423320,83997] 0 2024-03-04T16:04:33.353 INFO:tasks.workunit.client.1.smithi152.stdout:8/253: rmdir d2/d8/dd/d22/d49 39 2024-03-04T16:04:33.358 INFO:tasks.workunit.client.1.smithi152.stdout:0/195: write d9/d1b/f1d [486509,126425] 0 2024-03-04T16:04:33.361 INFO:tasks.workunit.client.1.smithi152.stdout:2/214: read da/df/d2e/f3a [1933707,68314] 0 2024-03-04T16:04:33.366 INFO:tasks.workunit.client.1.smithi152.stdout:9/205: creat d2/d13/f43 x:0 0 0 2024-03-04T16:04:33.368 INFO:tasks.workunit.client.1.smithi152.stdout:9/206: write d2/d13/f18 [280838,40327] 0 2024-03-04T16:04:33.369 INFO:tasks.workunit.client.1.smithi152.stdout:1/222: readlink d19/d20/d2f/l33 0 2024-03-04T16:04:33.371 INFO:tasks.workunit.client.1.smithi152.stdout:6/286: mkdir d8/d38/d4f/d57 0 2024-03-04T16:04:33.375 INFO:tasks.workunit.client.1.smithi152.stdout:3/295: creat d6/d60/f67 x:0 0 0 2024-03-04T16:04:33.376 INFO:tasks.workunit.client.1.smithi152.stdout:9/207: dwrite d2/f2d [0,4194304] 0 2024-03-04T16:04:33.383 INFO:tasks.workunit.client.1.smithi152.stdout:2/215: unlink da/df/f25 0 2024-03-04T16:04:33.384 INFO:tasks.workunit.client.1.smithi152.stdout:3/296: dwrite d6/ff [0,4194304] 0 2024-03-04T16:04:33.385 INFO:tasks.workunit.client.1.smithi152.stdout:3/297: chown d6/d10/d2d/f40 0 1 2024-03-04T16:04:33.394 INFO:tasks.workunit.client.1.smithi152.stdout:1/223: rename d19/d20/d21/f2e to d19/d20/d2f/f4d 0 2024-03-04T16:04:33.395 INFO:tasks.workunit.client.1.smithi152.stdout:1/224: dread - d19/f1c zero size 2024-03-04T16:04:33.397 INFO:tasks.workunit.client.1.smithi152.stdout:4/191: dwrite d8/d9/ff [4194304,4194304] 0 2024-03-04T16:04:33.397 INFO:tasks.workunit.client.1.smithi152.stdout:7/225: link d14/d21/f45 d14/d17/f46 0 2024-03-04T16:04:33.399 INFO:tasks.workunit.client.1.smithi152.stdout:0/196: rename d9/d1a/l24 to d9/d1a/l39 0 2024-03-04T16:04:33.402 INFO:tasks.workunit.client.1.smithi152.stdout:2/216: fdatasync da/df/f13 0 2024-03-04T16:04:33.408 INFO:tasks.workunit.client.1.smithi152.stdout:5/293: creat d6/f68 x:0 0 0 2024-03-04T16:04:33.408 INFO:tasks.workunit.client.1.smithi152.stdout:6/287: mkdir d8/d52/d58 0 2024-03-04T16:04:33.408 INFO:tasks.workunit.client.1.smithi152.stdout:0/197: dwrite f7 [8388608,4194304] 0 2024-03-04T16:04:33.409 INFO:tasks.workunit.client.1.smithi152.stdout:6/288: fsync d8/d10/f18 0 2024-03-04T16:04:33.415 INFO:tasks.workunit.client.1.smithi152.stdout:1/225: symlink d19/d20/d21/l4e 0 2024-03-04T16:04:33.419 INFO:tasks.workunit.client.1.smithi152.stdout:4/192: mkdir d8/d3d 0 2024-03-04T16:04:33.424 INFO:tasks.workunit.client.1.smithi152.stdout:1/226: dwrite d19/d20/d21/d38/f3a [0,4194304] 0 2024-03-04T16:04:33.424 INFO:tasks.workunit.client.1.smithi152.stdout:7/226: read d14/d21/f45 [80280,72456] 0 2024-03-04T16:04:33.428 INFO:tasks.workunit.client.1.smithi152.stdout:4/193: dwrite f5 [0,4194304] 0 2024-03-04T16:04:33.430 INFO:tasks.workunit.client.1.smithi152.stdout:1/227: truncate d19/d20/d21/f42 293365 0 2024-03-04T16:04:33.431 INFO:tasks.workunit.client.1.smithi152.stdout:2/217: rmdir da/d2c 39 2024-03-04T16:04:33.431 INFO:tasks.workunit.client.1.smithi152.stdout:4/194: read - d8/d2f/f34 zero size 2024-03-04T16:04:33.433 INFO:tasks.workunit.client.1.smithi152.stdout:1/228: readlink l13 0 2024-03-04T16:04:33.434 INFO:tasks.workunit.client.1.smithi152.stdout:2/218: chown da/d24/f2a 121752040 1 2024-03-04T16:04:33.435 INFO:tasks.workunit.client.1.smithi152.stdout:1/229: dread - d19/f2c zero size 2024-03-04T16:04:33.437 INFO:tasks.workunit.client.1.smithi152.stdout:6/289: rmdir d8/d23 39 2024-03-04T16:04:33.438 INFO:tasks.workunit.client.1.smithi152.stdout:1/230: write d19/d3b/f49 [206472,69665] 0 2024-03-04T16:04:33.442 INFO:tasks.workunit.client.1.smithi152.stdout:7/227: dwrite d14/d17/f18 [0,4194304] 0 2024-03-04T16:04:33.443 INFO:tasks.workunit.client.1.smithi152.stdout:3/298: creat d6/f68 x:0 0 0 2024-03-04T16:04:33.444 INFO:tasks.workunit.client.1.smithi152.stdout:3/299: chown d6/d14/l43 119474157 1 2024-03-04T16:04:33.451 INFO:tasks.workunit.client.1.smithi152.stdout:4/195: rename d8/f20 to d8/f3e 0 2024-03-04T16:04:33.452 INFO:tasks.workunit.client.1.smithi152.stdout:7/228: dwrite d14/d2b/f3b [4194304,4194304] 0 2024-03-04T16:04:33.457 INFO:tasks.workunit.client.1.smithi152.stdout:0/198: fsync d9/fa 0 2024-03-04T16:04:33.463 INFO:tasks.workunit.client.1.smithi152.stdout:1/231: dread d19/f1a [0,4194304] 0 2024-03-04T16:04:33.469 INFO:tasks.workunit.client.1.smithi152.stdout:4/196: rmdir d8/d9/d24/d26 39 2024-03-04T16:04:33.470 INFO:tasks.workunit.client.1.smithi152.stdout:6/290: dwrite d8/d2a/f36 [4194304,4194304] 0 2024-03-04T16:04:33.471 INFO:tasks.workunit.client.1.smithi152.stdout:4/197: chown f5 735535759 1 2024-03-04T16:04:33.473 INFO:tasks.workunit.client.1.smithi152.stdout:3/300: dwrite d6/d10/d2d/f64 [0,4194304] 0 2024-03-04T16:04:33.475 INFO:tasks.workunit.client.1.smithi152.stdout:7/229: rename d14/d17 to d14/d21/d47 0 2024-03-04T16:04:33.476 INFO:tasks.workunit.client.1.smithi152.stdout:2/219: creat da/d24/d47/f4d x:0 0 0 2024-03-04T16:04:33.477 INFO:tasks.workunit.client.1.smithi152.stdout:0/199: readlink d9/d1b/l30 0 2024-03-04T16:04:33.482 INFO:tasks.workunit.client.1.smithi152.stdout:7/230: read f8 [8133004,53266] 0 2024-03-04T16:04:33.483 INFO:tasks.workunit.client.1.smithi152.stdout:7/231: fsync d14/f15 0 2024-03-04T16:04:33.488 INFO:tasks.workunit.client.1.smithi152.stdout:0/200: creat d9/d2c/f3a x:0 0 0 2024-03-04T16:04:33.490 INFO:tasks.workunit.client.1.smithi152.stdout:5/294: getdents d6/d8/d10 0 2024-03-04T16:04:33.491 INFO:tasks.workunit.client.1.smithi152.stdout:1/232: mknod d19/d4b/c4f 0 2024-03-04T16:04:33.492 INFO:tasks.workunit.client.1.smithi152.stdout:1/233: write f2 [2372452,14730] 0 2024-03-04T16:04:33.494 INFO:tasks.workunit.client.1.smithi152.stdout:1/234: write d19/d20/f41 [1605926,106805] 0 2024-03-04T16:04:33.496 INFO:tasks.workunit.client.1.smithi152.stdout:4/198: dwrite d8/d9/f15 [0,4194304] 0 2024-03-04T16:04:33.496 INFO:tasks.workunit.client.1.smithi152.stdout:7/232: mkdir d14/d21/d48 0 2024-03-04T16:04:33.500 INFO:tasks.workunit.client.1.smithi152.stdout:2/220: link da/f23 da/df/d2e/f4e 0 2024-03-04T16:04:33.501 INFO:tasks.workunit.client.1.smithi152.stdout:0/201: rmdir d9/d1a 39 2024-03-04T16:04:33.504 INFO:tasks.workunit.client.1.smithi152.stdout:3/301: truncate d6/fc 701082 0 2024-03-04T16:04:33.506 INFO:tasks.workunit.client.1.smithi152.stdout:3/302: chown d6/d14/d41/d49 3535741 1 2024-03-04T16:04:33.507 INFO:tasks.workunit.client.1.smithi152.stdout:5/295: mknod d6/d8/d10/c69 0 2024-03-04T16:04:33.508 INFO:tasks.workunit.client.1.smithi152.stdout:3/303: write d6/f3a [952034,3889] 0 2024-03-04T16:04:33.508 INFO:tasks.workunit.client.1.smithi152.stdout:2/221: dwrite da/f45 [0,4194304] 0 2024-03-04T16:04:33.510 INFO:tasks.workunit.client.1.smithi152.stdout:1/235: symlink d19/d3b/l50 0 2024-03-04T16:04:33.512 INFO:tasks.workunit.client.1.smithi152.stdout:1/236: dread - d19/d20/d21/d38/f3e zero size 2024-03-04T16:04:33.513 INFO:tasks.workunit.client.1.smithi152.stdout:7/233: creat d14/d21/f49 x:0 0 0 2024-03-04T16:04:33.515 INFO:tasks.workunit.client.1.smithi152.stdout:1/237: truncate d19/d20/d21/f4a 1021338 0 2024-03-04T16:04:33.516 INFO:tasks.workunit.client.1.smithi152.stdout:1/238: write d19/d20/f41 [1691067,103844] 0 2024-03-04T16:04:33.518 INFO:tasks.workunit.client.1.smithi152.stdout:0/202: fdatasync d9/fa 0 2024-03-04T16:04:33.522 INFO:tasks.workunit.client.1.smithi152.stdout:0/203: dread - d9/d2c/d2e/f31 zero size 2024-03-04T16:04:33.522 INFO:tasks.workunit.client.1.smithi152.stdout:5/296: symlink d6/d50/l6a 0 2024-03-04T16:04:33.525 INFO:tasks.workunit.client.1.smithi152.stdout:2/222: mkdir da/d24/d47/d4f 0 2024-03-04T16:04:33.525 INFO:tasks.workunit.client.1.smithi152.stdout:4/199: dwrite d8/d9/d24/d26/f2e [0,4194304] 0 2024-03-04T16:04:33.528 INFO:tasks.workunit.client.1.smithi152.stdout:1/239: dwrite d19/f23 [0,4194304] 0 2024-03-04T16:04:33.545 INFO:tasks.workunit.client.1.smithi152.stdout:0/204: symlink d9/d2c/l3b 0 2024-03-04T16:04:33.546 INFO:tasks.workunit.client.1.smithi152.stdout:4/200: rmdir d8 39 2024-03-04T16:04:33.551 INFO:tasks.workunit.client.1.smithi152.stdout:5/297: symlink d6/d8/d1f/d3d/l6b 0 2024-03-04T16:04:33.551 INFO:tasks.workunit.client.1.smithi152.stdout:2/223: mkdir da/df/d35/d50 0 2024-03-04T16:04:33.553 INFO:tasks.workunit.client.1.smithi152.stdout:7/234: dwrite f12 [0,4194304] 0 2024-03-04T16:04:33.558 INFO:tasks.workunit.client.1.smithi152.stdout:5/298: dwrite d6/d8/fe [0,4194304] 0 2024-03-04T16:04:33.558 INFO:tasks.workunit.client.1.smithi152.stdout:2/224: dread da/d24/f2a [0,4194304] 0 2024-03-04T16:04:33.560 INFO:tasks.workunit.client.1.smithi152.stdout:2/225: write da/d24/f31 [501886,52177] 0 2024-03-04T16:04:33.561 INFO:tasks.workunit.client.1.smithi152.stdout:0/205: rmdir d9/d1b 39 2024-03-04T16:04:33.563 INFO:tasks.workunit.client.1.smithi152.stdout:2/226: readlink da/l28 0 2024-03-04T16:04:33.566 INFO:tasks.workunit.client.1.smithi152.stdout:5/299: dwrite d6/d8/f2c [0,4194304] 0 2024-03-04T16:04:33.572 INFO:tasks.workunit.client.1.smithi152.stdout:4/201: write d8/f23 [2207249,105321] 0 2024-03-04T16:04:33.575 INFO:tasks.workunit.client.1.smithi152.stdout:0/206: readlink d9/d1b/l27 0 2024-03-04T16:04:33.576 INFO:tasks.workunit.client.1.smithi152.stdout:0/207: fsync d9/d1b/f23 0 2024-03-04T16:04:33.580 INFO:tasks.workunit.client.1.smithi152.stdout:2/227: mkdir da/df/d2e/d51 0 2024-03-04T16:04:33.580 INFO:tasks.workunit.client.1.smithi152.stdout:1/240: write f1 [2485521,44040] 0 2024-03-04T16:04:33.583 INFO:tasks.workunit.client.1.smithi152.stdout:5/300: creat d6/d8/d1f/d3d/d61/f6c x:0 0 0 2024-03-04T16:04:33.586 INFO:tasks.workunit.client.1.smithi152.stdout:4/202: fdatasync d8/d9/f28 0 2024-03-04T16:04:33.587 INFO:tasks.workunit.client.1.smithi152.stdout:0/208: mkdir d9/d1b/d3c 0 2024-03-04T16:04:33.588 INFO:tasks.workunit.client.1.smithi152.stdout:0/209: fsync d9/d1b/f29 0 2024-03-04T16:04:33.592 INFO:tasks.workunit.client.1.smithi152.stdout:1/241: rename d19/f48 to d19/d4b/f51 0 2024-03-04T16:04:33.595 INFO:tasks.workunit.client.1.smithi152.stdout:7/235: dread d14/f16 [0,4194304] 0 2024-03-04T16:04:33.595 INFO:tasks.workunit.client.1.smithi152.stdout:7/236: dread - d14/f15 zero size 2024-03-04T16:04:33.598 INFO:tasks.workunit.client.1.smithi152.stdout:7/237: dread d14/f19 [0,4194304] 0 2024-03-04T16:04:33.600 INFO:tasks.workunit.client.1.smithi152.stdout:1/242: creat d19/d3b/f52 x:0 0 0 2024-03-04T16:04:33.600 INFO:tasks.workunit.client.1.smithi152.stdout:9/208: sync 2024-03-04T16:04:33.601 INFO:tasks.workunit.client.1.smithi152.stdout:4/203: mknod d8/d3d/c3f 0 2024-03-04T16:04:33.604 INFO:tasks.workunit.client.1.smithi152.stdout:9/209: write d2/d13/f18 [1854025,115070] 0 2024-03-04T16:04:33.605 INFO:tasks.workunit.client.1.smithi152.stdout:0/210: creat d9/d1a/f3d x:0 0 0 2024-03-04T16:04:33.607 INFO:tasks.workunit.client.1.smithi152.stdout:7/238: dwrite d14/d21/d47/f34 [0,4194304] 0 2024-03-04T16:04:33.610 INFO:tasks.workunit.client.1.smithi152.stdout:9/210: write d2/d13/d23/f33 [3483878,113138] 0 2024-03-04T16:04:33.611 INFO:tasks.workunit.client.1.smithi152.stdout:2/228: write da/df/f13 [3472261,73516] 0 2024-03-04T16:04:33.614 INFO:tasks.workunit.client.1.smithi152.stdout:9/211: write d2/d13/d2e/f3c [318779,65742] 0 2024-03-04T16:04:33.616 INFO:tasks.workunit.client.1.smithi152.stdout:5/301: getdents d6/d8/d10 0 2024-03-04T16:04:33.618 INFO:tasks.workunit.client.1.smithi152.stdout:4/204: dwrite f0 [4194304,4194304] 0 2024-03-04T16:04:33.622 INFO:tasks.workunit.client.1.smithi152.stdout:9/212: truncate d2/d13/f43 441766 0 2024-03-04T16:04:33.622 INFO:tasks.workunit.client.1.smithi152.stdout:1/243: dwrite d19/d20/d21/d38/f3e [0,4194304] 0 2024-03-04T16:04:33.623 INFO:tasks.workunit.client.1.smithi152.stdout:2/229: fdatasync da/f23 0 2024-03-04T16:04:33.633 INFO:tasks.workunit.client.1.smithi152.stdout:0/211: dwrite d9/d2c/f2f [0,4194304] 0 2024-03-04T16:04:33.636 INFO:tasks.workunit.client.1.smithi152.stdout:5/302: dwrite d6/d8/d10/f27 [0,4194304] 0 2024-03-04T16:04:33.636 INFO:tasks.workunit.client.1.smithi152.stdout:1/244: dread d19/d20/d21/d38/f3e [0,4194304] 0 2024-03-04T16:04:33.638 INFO:tasks.workunit.client.1.smithi152.stdout:2/230: write da/df/d2e/f4e [726040,78308] 0 2024-03-04T16:04:33.642 INFO:tasks.workunit.client.1.smithi152.stdout:9/213: rename d2/d13/f43 to d2/d13/d2e/f44 0 2024-03-04T16:04:33.645 INFO:tasks.workunit.client.1.smithi152.stdout:1/245: mkdir d19/d4b/d53 0 2024-03-04T16:04:33.647 INFO:tasks.workunit.client.1.smithi152.stdout:2/231: mknod da/d24/d47/c52 0 2024-03-04T16:04:33.647 INFO:tasks.workunit.client.1.smithi152.stdout:5/303: fsync d6/d8/f30 0 2024-03-04T16:04:33.648 INFO:tasks.workunit.client.1.smithi152.stdout:1/246: chown d19/d20/f30 0 1 2024-03-04T16:04:33.649 INFO:tasks.workunit.client.1.smithi152.stdout:4/205: creat d8/d9/d24/f40 x:0 0 0 2024-03-04T16:04:33.650 INFO:tasks.workunit.client.1.smithi152.stdout:1/247: read d19/d20/d21/d38/f3e [2019032,81898] 0 2024-03-04T16:04:33.651 INFO:tasks.workunit.client.1.smithi152.stdout:5/304: truncate d6/d8/d1f/f57 616807 0 2024-03-04T16:04:33.654 INFO:tasks.workunit.client.1.smithi152.stdout:9/214: rename d2/d13/d23/c3b to d2/d13/d23/c45 0 2024-03-04T16:04:33.654 INFO:tasks.workunit.client.1.smithi152.stdout:4/206: mknod d8/c41 0 2024-03-04T16:04:33.656 INFO:tasks.workunit.client.1.smithi152.stdout:2/232: mknod da/df/d2e/d51/c53 0 2024-03-04T16:04:33.658 INFO:tasks.workunit.client.1.smithi152.stdout:0/212: link d9/le d9/d1a/l3e 0 2024-03-04T16:04:33.661 INFO:tasks.workunit.client.1.smithi152.stdout:1/248: mknod d19/d4b/d53/c54 0 2024-03-04T16:04:33.663 INFO:tasks.workunit.client.1.smithi152.stdout:9/215: dwrite d2/f2d [0,4194304] 0 2024-03-04T16:04:33.664 INFO:tasks.workunit.client.1.smithi152.stdout:9/216: fsync d2/d13/d23/f29 0 2024-03-04T16:04:33.665 INFO:tasks.workunit.client.1.smithi152.stdout:0/213: rename d9/d1a/c20 to d9/d2c/d2e/c3f 0 2024-03-04T16:04:33.666 INFO:tasks.workunit.client.1.smithi152.stdout:2/233: symlink da/d24/d47/d4f/l54 0 2024-03-04T16:04:33.667 INFO:tasks.workunit.client.1.smithi152.stdout:7/239: truncate d14/d2b/f3b 5699690 0 2024-03-04T16:04:33.667 INFO:tasks.workunit.client.1.smithi152.stdout:1/249: dread d19/d20/d21/f37 [0,4194304] 0 2024-03-04T16:04:33.674 INFO:tasks.workunit.client.1.smithi152.stdout:0/214: dwrite d9/d1b/f2a [0,4194304] 0 2024-03-04T16:04:33.677 INFO:tasks.workunit.client.1.smithi152.stdout:1/250: dread d19/d20/d21/f37 [0,4194304] 0 2024-03-04T16:04:33.681 INFO:tasks.workunit.client.1.smithi152.stdout:7/240: stat d14/d21/f30 0 2024-03-04T16:04:33.681 INFO:tasks.workunit.client.1.smithi152.stdout:7/241: dread - d14/f42 zero size 2024-03-04T16:04:33.682 INFO:tasks.workunit.client.1.smithi152.stdout:9/217: creat d2/d13/d23/d42/f46 x:0 0 0 2024-03-04T16:04:33.685 INFO:tasks.workunit.client.1.smithi152.stdout:0/215: readlink d9/l18 0 2024-03-04T16:04:33.690 INFO:tasks.workunit.client.1.smithi152.stdout:7/242: rename d14/f19 to d14/d21/d47/d40/f4a 0 2024-03-04T16:04:33.690 INFO:tasks.workunit.client.1.smithi152.stdout:1/251: chown d19/d20/d21/d38/l3c 0 1 2024-03-04T16:04:33.691 INFO:tasks.workunit.client.1.smithi152.stdout:9/218: dwrite d2/f2d [0,4194304] 0 2024-03-04T16:04:33.698 INFO:tasks.workunit.client.1.smithi152.stdout:9/219: write d2/d13/f18 [2251445,121994] 0 2024-03-04T16:04:33.699 INFO:tasks.workunit.client.1.smithi152.stdout:5/305: dwrite d6/d8/d10/f5b [0,4194304] 0 2024-03-04T16:04:33.701 INFO:tasks.workunit.client.1.smithi152.stdout:0/216: dwrite d9/d1a/f3d [0,4194304] 0 2024-03-04T16:04:33.705 INFO:tasks.workunit.client.1.smithi152.stdout:2/234: getdents da/df 0 2024-03-04T16:04:33.712 INFO:tasks.workunit.client.1.smithi152.stdout:1/252: dwrite d19/f1d [4194304,4194304] 0 2024-03-04T16:04:33.716 INFO:tasks.workunit.client.1.smithi152.stdout:0/217: readlink d9/d1a/l3e 0 2024-03-04T16:04:33.718 INFO:tasks.workunit.client.1.smithi152.stdout:0/218: write f6 [4604884,61094] 0 2024-03-04T16:04:33.720 INFO:tasks.workunit.client.1.smithi152.stdout:0/219: chown d9/c34 8288 1 2024-03-04T16:04:33.720 INFO:tasks.workunit.client.1.smithi152.stdout:7/243: rename d14/d21/d47/d40/f4a to d14/d21/d48/f4b 0 2024-03-04T16:04:33.721 INFO:tasks.workunit.client.1.smithi152.stdout:2/235: dwrite da/fe [0,4194304] 0 2024-03-04T16:04:33.721 INFO:tasks.workunit.client.1.smithi152.stdout:5/306: symlink d6/d8/l6d 0 2024-03-04T16:04:33.724 INFO:tasks.workunit.client.1.smithi152.stdout:1/253: rename d19/d20/d2f/f39 to d19/d4b/d53/f55 0 2024-03-04T16:04:33.731 INFO:tasks.workunit.client.1.smithi152.stdout:2/236: rmdir da/df/d35 39 2024-03-04T16:04:33.733 INFO:tasks.workunit.client.1.smithi152.stdout:0/220: rename l0 to d9/d2c/l40 0 2024-03-04T16:04:33.734 INFO:tasks.workunit.client.1.smithi152.stdout:1/254: dwrite d19/d20/d21/f4a [0,4194304] 0 2024-03-04T16:04:33.743 INFO:tasks.workunit.client.1.smithi152.stdout:0/221: creat d9/d2c/d2e/f41 x:0 0 0 2024-03-04T16:04:33.746 INFO:tasks.workunit.client.1.smithi152.stdout:2/237: fdatasync da/df/d35/f44 0 2024-03-04T16:04:33.750 INFO:tasks.workunit.client.1.smithi152.stdout:0/222: dwrite f6 [4194304,4194304] 0 2024-03-04T16:04:33.750 INFO:tasks.workunit.client.1.smithi152.stdout:0/223: chown d9/f36 3218869 1 2024-03-04T16:04:33.751 INFO:tasks.workunit.client.1.smithi152.stdout:0/224: chown d9/f25 1519156 1 2024-03-04T16:04:33.752 INFO:tasks.workunit.client.1.smithi152.stdout:0/225: write f7 [13462547,8740] 0 2024-03-04T16:04:33.755 INFO:tasks.workunit.client.1.smithi152.stdout:0/226: symlink d9/d2c/d2e/d38/l42 0 2024-03-04T16:04:33.760 INFO:tasks.workunit.client.1.smithi152.stdout:0/227: link d9/d1a/l39 d9/d2c/d2e/l43 0 2024-03-04T16:04:33.764 INFO:tasks.workunit.client.1.smithi152.stdout:5/307: write d6/d8/f32 [61473,5953] 0 2024-03-04T16:04:33.765 INFO:tasks.workunit.client.1.smithi152.stdout:0/228: read d9/d1b/f29 [1921965,69037] 0 2024-03-04T16:04:33.768 INFO:tasks.workunit.client.1.smithi152.stdout:7/244: truncate f4 3880029 0 2024-03-04T16:04:33.770 INFO:tasks.workunit.client.1.smithi152.stdout:7/245: write d14/d21/f49 [272342,126095] 0 2024-03-04T16:04:33.770 INFO:tasks.workunit.client.1.smithi152.stdout:5/308: creat d6/d8/d5a/d64/f6e x:0 0 0 2024-03-04T16:04:33.771 INFO:tasks.workunit.client.1.smithi152.stdout:7/246: chown d14/d21/d47/d36/c38 604917 1 2024-03-04T16:04:33.772 INFO:tasks.workunit.client.1.smithi152.stdout:0/229: rename c2 to d9/d2c/d2e/c44 0 2024-03-04T16:04:33.775 INFO:tasks.workunit.client.1.smithi152.stdout:5/309: mkdir d6/d8/d1f/d3d/d6f 0 2024-03-04T16:04:33.775 INFO:tasks.workunit.client.1.smithi152.stdout:1/255: truncate f2 1319965 0 2024-03-04T16:04:33.776 INFO:tasks.workunit.client.1.smithi152.stdout:1/256: chown f17 60 1 2024-03-04T16:04:33.777 INFO:tasks.workunit.client.1.smithi152.stdout:7/247: creat d14/d21/d47/d40/f4c x:0 0 0 2024-03-04T16:04:33.777 INFO:tasks.workunit.client.1.smithi152.stdout:2/238: dread da/f16 [0,4194304] 0 2024-03-04T16:04:33.783 INFO:tasks.workunit.client.1.smithi152.stdout:2/239: chown da/df/d2e/f30 5927387 1 2024-03-04T16:04:33.783 INFO:tasks.workunit.client.1.smithi152.stdout:2/240: dread - da/df/d35/f46 zero size 2024-03-04T16:04:33.784 INFO:tasks.workunit.client.1.smithi152.stdout:5/310: mkdir d6/d8/d1f/d3d/d6f/d70 0 2024-03-04T16:04:33.787 INFO:tasks.workunit.client.1.smithi152.stdout:5/311: readlink d6/d8/d1f/d3d/l6b 0 2024-03-04T16:04:33.788 INFO:tasks.workunit.client.1.smithi152.stdout:0/230: rename d9/f25 to d9/f45 0 2024-03-04T16:04:33.789 INFO:tasks.workunit.client.1.smithi152.stdout:0/231: dread - d9/d2c/d2e/f32 zero size 2024-03-04T16:04:33.794 INFO:tasks.workunit.client.1.smithi152.stdout:8/254: sync 2024-03-04T16:04:33.798 INFO:tasks.workunit.client.1.smithi152.stdout:0/232: rename d9/d1a/f3d to d9/d1a/f46 0 2024-03-04T16:04:33.800 INFO:tasks.workunit.client.1.smithi152.stdout:2/241: rename da/df/l1e to da/d24/d47/d4f/l55 0 2024-03-04T16:04:33.801 INFO:tasks.workunit.client.1.smithi152.stdout:7/248: write d14/d2b/f3b [1643833,5883] 0 2024-03-04T16:04:33.802 INFO:tasks.workunit.client.1.smithi152.stdout:5/312: creat d6/d8/d1f/d3d/d6f/d70/f71 x:0 0 0 2024-03-04T16:04:33.804 INFO:tasks.workunit.client.1.smithi152.stdout:1/257: write d19/d20/d21/f46 [1552142,14378] 0 2024-03-04T16:04:33.807 INFO:tasks.workunit.client.1.smithi152.stdout:2/242: chown da/df/f1f 29 1 2024-03-04T16:04:33.810 INFO:tasks.workunit.client.1.smithi152.stdout:0/233: rename d9/d2c/f3a to d9/d2c/d2e/f47 0 2024-03-04T16:04:33.812 INFO:tasks.workunit.client.1.smithi152.stdout:0/234: read d9/d1b/f29 [1584317,122195] 0 2024-03-04T16:04:33.812 INFO:tasks.workunit.client.1.smithi152.stdout:0/235: fdatasync d9/d1b/f29 0 2024-03-04T16:04:33.813 INFO:tasks.workunit.client.1.smithi152.stdout:0/236: read d9/d1b/f35 [898187,99915] 0 2024-03-04T16:04:33.815 INFO:tasks.workunit.client.1.smithi152.stdout:8/255: creat d2/d3/d15/f4f x:0 0 0 2024-03-04T16:04:33.820 INFO:tasks.workunit.client.1.smithi152.stdout:0/237: rmdir d9/d1a 39 2024-03-04T16:04:33.821 INFO:tasks.workunit.client.1.smithi152.stdout:8/256: rmdir d2/d3/d21 39 2024-03-04T16:04:33.822 INFO:tasks.workunit.client.1.smithi152.stdout:0/238: truncate d9/d1b/f23 342106 0 2024-03-04T16:04:33.826 INFO:tasks.workunit.client.1.smithi152.stdout:2/243: write da/df/d2e/f3a [704791,74593] 0 2024-03-04T16:04:33.828 INFO:tasks.workunit.client.1.smithi152.stdout:1/258: truncate f2 1353837 0 2024-03-04T16:04:33.830 INFO:tasks.workunit.client.1.smithi152.stdout:1/259: chown d19/d20/d21/f4a 389342 1 2024-03-04T16:04:33.830 INFO:tasks.workunit.client.1.smithi152.stdout:0/239: mknod d9/d1b/c48 0 2024-03-04T16:04:33.831 INFO:tasks.workunit.client.1.smithi152.stdout:7/249: dwrite d14/d21/d48/f4b [0,4194304] 0 2024-03-04T16:04:33.833 INFO:tasks.workunit.client.1.smithi152.stdout:0/240: write d9/d2c/d2e/f31 [377984,109713] 0 2024-03-04T16:04:33.834 INFO:tasks.workunit.client.1.smithi152.stdout:0/241: fsync f7 0 2024-03-04T16:04:33.835 INFO:tasks.workunit.client.1.smithi152.stdout:1/260: read d19/d20/d21/f2d [3063123,109547] 0 2024-03-04T16:04:33.836 INFO:tasks.workunit.client.1.smithi152.stdout:7/250: truncate d14/d21/d47/d40/f4c 385471 0 2024-03-04T16:04:33.838 INFO:tasks.workunit.client.1.smithi152.stdout:2/244: creat da/f56 x:0 0 0 2024-03-04T16:04:33.848 INFO:tasks.workunit.client.1.smithi152.stdout:1/261: symlink d19/d20/d21/l56 0 2024-03-04T16:04:33.848 INFO:tasks.workunit.client.1.smithi152.stdout:8/257: write d2/d3/d21/f32 [1040477,17118] 0 2024-03-04T16:04:33.848 INFO:tasks.workunit.client.1.smithi152.stdout:1/262: dread - d19/f2c zero size 2024-03-04T16:04:33.851 INFO:tasks.workunit.client.1.smithi152.stdout:2/245: write da/d2c/f4b [1859313,67230] 0 2024-03-04T16:04:33.856 INFO:tasks.workunit.client.1.smithi152.stdout:1/263: dwrite d19/d20/d21/d38/f4c [0,4194304] 0 2024-03-04T16:04:33.858 INFO:tasks.workunit.client.1.smithi152.stdout:0/242: rename d9/d1b/f29 to d9/f49 0 2024-03-04T16:04:33.867 INFO:tasks.workunit.client.1.smithi152.stdout:1/264: mkdir d19/d4b/d53/d57 0 2024-03-04T16:04:33.869 INFO:tasks.workunit.client.1.smithi152.stdout:8/258: rename d2/d8/f39 to d2/d3/d21/d2d/f50 0 2024-03-04T16:04:33.870 INFO:tasks.workunit.client.1.smithi152.stdout:8/259: fdatasync d2/d3/d15/f4f 0 2024-03-04T16:04:33.873 INFO:tasks.workunit.client.1.smithi152.stdout:8/260: read d2/d3/d21/f1f [591190,74220] 0 2024-03-04T16:04:33.875 INFO:tasks.workunit.client.1.smithi152.stdout:0/243: creat d9/d2c/d2e/f4a x:0 0 0 2024-03-04T16:04:33.876 INFO:tasks.workunit.client.1.smithi152.stdout:2/246: dwrite da/df/d35/f4a [0,4194304] 0 2024-03-04T16:04:33.884 INFO:tasks.workunit.client.1.smithi152.stdout:2/247: mkdir da/d24/d57 0 2024-03-04T16:04:33.885 INFO:tasks.workunit.client.1.smithi152.stdout:8/261: dwrite d2/d3/d15/d1c/f1d [0,4194304] 0 2024-03-04T16:04:33.886 INFO:tasks.workunit.client.1.smithi152.stdout:1/265: symlink d19/d20/l58 0 2024-03-04T16:04:33.886 INFO:tasks.workunit.client.1.smithi152.stdout:7/251: dread d14/d21/f25 [0,4194304] 0 2024-03-04T16:04:33.887 INFO:tasks.workunit.client.1.smithi152.stdout:1/266: stat d19/d20/d21/f2d 0 2024-03-04T16:04:33.889 INFO:tasks.workunit.client.1.smithi152.stdout:8/262: write d2/d8/f45 [4129080,48254] 0 2024-03-04T16:04:33.890 INFO:tasks.workunit.client.1.smithi152.stdout:7/252: truncate f8 4201115 0 2024-03-04T16:04:33.891 INFO:tasks.workunit.client.1.smithi152.stdout:1/267: chown d19/d20/c3d 4415 1 2024-03-04T16:04:33.894 INFO:tasks.workunit.client.1.smithi152.stdout:7/253: rmdir d14/d21/d47 39 2024-03-04T16:04:33.895 INFO:tasks.workunit.client.1.smithi152.stdout:1/268: creat d19/d20/d2f/f59 x:0 0 0 2024-03-04T16:04:33.896 INFO:tasks.workunit.client.1.smithi152.stdout:1/269: readlink d19/d20/d2f/l33 0 2024-03-04T16:04:33.897 INFO:tasks.workunit.client.1.smithi152.stdout:0/244: link d9/d1b/l30 d9/d2c/d2e/d38/l4b 0 2024-03-04T16:04:33.898 INFO:tasks.workunit.client.1.smithi152.stdout:1/270: write d19/d20/d21/f42 [543457,125679] 0 2024-03-04T16:04:33.898 INFO:tasks.workunit.client.1.smithi152.stdout:1/271: readlink d19/d20/d2f/l36 0 2024-03-04T16:04:33.899 INFO:tasks.workunit.client.1.smithi152.stdout:8/263: symlink d2/d3/d21/l51 0 2024-03-04T16:04:33.903 INFO:tasks.workunit.client.1.smithi152.stdout:7/254: unlink f12 0 2024-03-04T16:04:33.906 INFO:tasks.workunit.client.1.smithi152.stdout:1/272: creat d19/d20/d21/f5a x:0 0 0 2024-03-04T16:04:33.911 INFO:tasks.workunit.client.1.smithi152.stdout:1/273: creat d19/f5b x:0 0 0 2024-03-04T16:04:33.911 INFO:tasks.workunit.client.1.smithi152.stdout:7/255: mknod d14/d21/d47/d36/c4d 0 2024-03-04T16:04:33.914 INFO:tasks.workunit.client.1.smithi152.stdout:0/245: dwrite d9/d1a/f46 [0,4194304] 0 2024-03-04T16:04:33.916 INFO:tasks.workunit.client.1.smithi152.stdout:1/274: mkdir d19/d4b/d53/d5c 0 2024-03-04T16:04:33.921 INFO:tasks.workunit.client.1.smithi152.stdout:0/246: mkdir d9/d2c/d2e/d38/d4c 0 2024-03-04T16:04:33.922 INFO:tasks.workunit.client.1.smithi152.stdout:1/275: dread d19/f1a [0,4194304] 0 2024-03-04T16:04:33.923 INFO:tasks.workunit.client.1.smithi152.stdout:0/247: truncate d9/fa 2617222 0 2024-03-04T16:04:33.923 INFO:tasks.workunit.client.1.smithi152.stdout:0/248: chown f7 0 1 2024-03-04T16:04:33.928 INFO:tasks.workunit.client.1.smithi152.stdout:0/249: dwrite d9/d2c/f2f [0,4194304] 0 2024-03-04T16:04:33.993 INFO:tasks.workunit.client.1.smithi152.stdout:6/291: sync 2024-03-04T16:04:33.993 INFO:tasks.workunit.client.1.smithi152.stdout:3/304: sync 2024-03-04T16:04:34.035 INFO:tasks.workunit.client.1.smithi152.stdout:1/276: mkdir d19/d20/d2f/d5d 0 2024-03-04T16:04:34.038 INFO:tasks.workunit.client.1.smithi152.stdout:3/305: rmdir d6/d10/d2d/d4d 39 2024-03-04T16:04:34.042 INFO:tasks.workunit.client.1.smithi152.stdout:0/250: symlink d9/l4d 0 2024-03-04T16:04:34.042 INFO:tasks.workunit.client.1.smithi152.stdout:0/251: fdatasync f6 0 2024-03-04T16:04:34.047 INFO:tasks.workunit.client.1.smithi152.stdout:6/292: unlink d8/d23/c32 0 2024-03-04T16:04:34.052 INFO:tasks.workunit.client.1.smithi152.stdout:1/277: creat d19/d4b/d53/d57/f5e x:0 0 0 2024-03-04T16:04:34.053 INFO:tasks.workunit.client.1.smithi152.stdout:3/306: symlink d6/d53/l69 0 2024-03-04T16:04:34.055 INFO:tasks.workunit.client.1.smithi152.stdout:1/278: write d19/d20/d21/d38/f4c [1387327,87530] 0 2024-03-04T16:04:34.058 INFO:tasks.workunit.client.1.smithi152.stdout:0/252: symlink d9/d2c/d2e/l4e 0 2024-03-04T16:04:34.058 INFO:tasks.workunit.client.1.smithi152.stdout:0/253: dread - d9/d2c/d2e/f41 zero size 2024-03-04T16:04:34.059 INFO:tasks.workunit.client.1.smithi152.stdout:0/254: dread - d9/d2c/d2e/f32 zero size 2024-03-04T16:04:34.068 INFO:tasks.workunit.client.1.smithi152.stdout:1/279: creat d19/d4b/f5f x:0 0 0 2024-03-04T16:04:34.077 INFO:tasks.workunit.client.1.smithi152.stdout:0/255: creat d9/d1b/f4f x:0 0 0 2024-03-04T16:04:34.079 INFO:tasks.workunit.client.1.smithi152.stdout:0/256: write d9/d1b/f2a [2409385,130353] 0 2024-03-04T16:04:34.079 INFO:tasks.workunit.client.1.smithi152.stdout:1/280: creat d19/d4b/d53/d57/f60 x:0 0 0 2024-03-04T16:04:34.083 INFO:tasks.workunit.client.1.smithi152.stdout:0/257: dread d9/d2c/d2e/f31 [0,4194304] 0 2024-03-04T16:04:34.084 INFO:tasks.workunit.client.1.smithi152.stdout:7/256: dread f8 [0,4194304] 0 2024-03-04T16:04:34.085 INFO:tasks.workunit.client.1.smithi152.stdout:7/257: write d14/d21/d31/f33 [801059,82005] 0 2024-03-04T16:04:34.087 INFO:tasks.workunit.client.1.smithi152.stdout:1/281: mkdir d19/d4b/d61 0 2024-03-04T16:04:34.088 INFO:tasks.workunit.client.1.smithi152.stdout:7/258: read d14/d21/d47/f35 [4024868,91389] 0 2024-03-04T16:04:34.093 INFO:tasks.workunit.client.1.smithi152.stdout:1/282: mkdir d19/d4b/d53/d57/d62 0 2024-03-04T16:04:34.096 INFO:tasks.workunit.client.1.smithi152.stdout:7/259: rename f8 to d14/d2b/f4e 0 2024-03-04T16:04:34.097 INFO:tasks.workunit.client.1.smithi152.stdout:7/260: stat d14/d21/d31/f33 0 2024-03-04T16:04:34.099 INFO:tasks.workunit.client.1.smithi152.stdout:0/258: mkdir d9/d1b/d3c/d50 0 2024-03-04T16:04:34.101 INFO:tasks.workunit.client.1.smithi152.stdout:1/283: mkdir d19/d20/d21/d38/d63 0 2024-03-04T16:04:34.102 INFO:tasks.workunit.client.1.smithi152.stdout:7/261: truncate d14/f16 3614019 0 2024-03-04T16:04:34.106 INFO:tasks.workunit.client.1.smithi152.stdout:0/259: write d9/d2c/d2e/f31 [567139,67977] 0 2024-03-04T16:04:34.108 INFO:tasks.workunit.client.1.smithi152.stdout:4/207: sync 2024-03-04T16:04:34.109 INFO:tasks.workunit.client.1.smithi152.stdout:7/262: creat d14/d21/d31/f4f x:0 0 0 2024-03-04T16:04:34.111 INFO:tasks.workunit.client.1.smithi152.stdout:0/260: mknod d9/d1a/c51 0 2024-03-04T16:04:34.112 INFO:tasks.workunit.client.1.smithi152.stdout:1/284: dread f1 [0,4194304] 0 2024-03-04T16:04:34.112 INFO:tasks.workunit.client.1.smithi152.stdout:0/261: write d9/d1a/f46 [2374307,57360] 0 2024-03-04T16:04:34.113 INFO:tasks.workunit.client.1.smithi152.stdout:1/285: stat d19/d4b/d53/d5c 0 2024-03-04T16:04:34.114 INFO:tasks.workunit.client.1.smithi152.stdout:0/262: stat d9/d1b/d3c/d50 0 2024-03-04T16:04:34.116 INFO:tasks.workunit.client.1.smithi152.stdout:4/208: mkdir d8/d9/d24/d26/d42 0 2024-03-04T16:04:34.116 INFO:tasks.workunit.client.1.smithi152.stdout:4/209: fsync d8/f16 0 2024-03-04T16:04:34.118 INFO:tasks.workunit.client.1.smithi152.stdout:4/210: write d8/d9/f15 [4873920,25914] 0 2024-03-04T16:04:34.118 INFO:tasks.workunit.client.1.smithi152.stdout:7/263: symlink d14/d21/d47/d32/l50 0 2024-03-04T16:04:34.118 INFO:tasks.workunit.client.1.smithi152.stdout:7/264: chown d14/d21/c39 14564 1 2024-03-04T16:04:34.124 INFO:tasks.workunit.client.1.smithi152.stdout:7/265: dwrite d14/d21/d47/f18 [4194304,4194304] 0 2024-03-04T16:04:34.127 INFO:tasks.workunit.client.1.smithi152.stdout:3/307: write d6/d10/d2d/f3b [322865,82165] 0 2024-03-04T16:04:34.130 INFO:tasks.workunit.client.1.smithi152.stdout:6/293: write d8/f39 [532463,101192] 0 2024-03-04T16:04:34.131 INFO:tasks.workunit.client.1.smithi152.stdout:1/286: mkdir d19/d20/d21/d38/d63/d64 0 2024-03-04T16:04:34.138 INFO:tasks.workunit.client.1.smithi152.stdout:1/287: dwrite d19/d4b/d53/d57/f5e [0,4194304] 0 2024-03-04T16:04:34.140 INFO:tasks.workunit.client.1.smithi152.stdout:4/211: mknod d8/d9/c43 0 2024-03-04T16:04:34.148 INFO:tasks.workunit.client.1.smithi152.stdout:6/294: dread d8/d10/f44 [0,4194304] 0 2024-03-04T16:04:34.154 INFO:tasks.workunit.client.1.smithi152.stdout:6/295: dwrite f1 [0,4194304] 0 2024-03-04T16:04:34.154 INFO:tasks.workunit.client.1.smithi152.stdout:6/296: chown d8/d42 73372 1 2024-03-04T16:04:34.156 INFO:tasks.workunit.client.1.smithi152.stdout:5/313: sync 2024-03-04T16:04:34.156 INFO:tasks.workunit.client.1.smithi152.stdout:9/220: sync 2024-03-04T16:04:34.156 INFO:tasks.workunit.client.1.smithi152.stdout:8/264: sync 2024-03-04T16:04:34.157 INFO:tasks.workunit.client.1.smithi152.stdout:2/248: sync 2024-03-04T16:04:34.161 INFO:tasks.workunit.client.1.smithi152.stdout:6/297: mkdir d8/d38/d59 0 2024-03-04T16:04:34.165 INFO:tasks.workunit.client.1.smithi152.stdout:9/221: rename d2/d13/c25 to d2/d13/d35/c47 0 2024-03-04T16:04:34.166 INFO:tasks.workunit.client.1.smithi152.stdout:5/314: symlink d6/d8/d1f/d3d/d6f/l72 0 2024-03-04T16:04:34.170 INFO:tasks.workunit.client.1.smithi152.stdout:5/315: dread d6/d8/f2c [0,4194304] 0 2024-03-04T16:04:34.170 INFO:tasks.workunit.client.1.smithi152.stdout:8/265: mkdir d2/d3/d21/d2d/d52 0 2024-03-04T16:04:34.171 INFO:tasks.workunit.client.1.smithi152.stdout:8/266: write d2/d3/d21/d2d/f1e [3251909,120365] 0 2024-03-04T16:04:34.172 INFO:tasks.workunit.client.1.smithi152.stdout:6/298: mknod d8/d23/c5a 0 2024-03-04T16:04:34.177 INFO:tasks.workunit.client.1.smithi152.stdout:5/316: stat d6/f38 0 2024-03-04T16:04:34.178 INFO:tasks.workunit.client.1.smithi152.stdout:4/212: getdents d8/d3d 0 2024-03-04T16:04:34.179 INFO:tasks.workunit.client.1.smithi152.stdout:4/213: write d8/d9/d24/d26/f32 [427425,55310] 0 2024-03-04T16:04:34.184 INFO:tasks.workunit.client.1.smithi152.stdout:5/317: creat d6/d8/d1f/d3d/d6f/f73 x:0 0 0 2024-03-04T16:04:34.187 INFO:tasks.workunit.client.1.smithi152.stdout:6/299: creat d8/d10/d53/f5b x:0 0 0 2024-03-04T16:04:34.188 INFO:tasks.workunit.client.1.smithi152.stdout:0/263: truncate d9/d1a/f46 1639950 0 2024-03-04T16:04:34.189 INFO:tasks.workunit.client.1.smithi152.stdout:0/264: truncate d9/d1b/f23 428374 0 2024-03-04T16:04:34.189 INFO:tasks.workunit.client.1.smithi152.stdout:9/222: rmdir d2/d13/d35/d41 0 2024-03-04T16:04:34.189 INFO:tasks.workunit.client.1.smithi152.stdout:7/266: write d14/d21/f30 [2332467,64876] 0 2024-03-04T16:04:34.190 INFO:tasks.workunit.client.1.smithi152.stdout:7/267: readlink lc 0 2024-03-04T16:04:34.192 INFO:tasks.workunit.client.1.smithi152.stdout:9/223: dread d2/d13/f1a [0,4194304] 0 2024-03-04T16:04:34.195 INFO:tasks.workunit.client.1.smithi152.stdout:6/300: creat d8/d23/f5c x:0 0 0 2024-03-04T16:04:34.196 INFO:tasks.workunit.client.1.smithi152.stdout:7/268: dread d14/d21/d48/f4b [0,4194304] 0 2024-03-04T16:04:34.197 INFO:tasks.workunit.client.1.smithi152.stdout:0/265: write d9/f49 [817900,44777] 0 2024-03-04T16:04:34.198 INFO:tasks.workunit.client.1.smithi152.stdout:9/224: dread d2/fe [0,4194304] 0 2024-03-04T16:04:34.199 INFO:tasks.workunit.client.1.smithi152.stdout:7/269: chown d14/d21/d31/f33 3143 1 2024-03-04T16:04:34.200 INFO:tasks.workunit.client.1.smithi152.stdout:3/308: dwrite d6/d14/f2e [0,4194304] 0 2024-03-04T16:04:34.201 INFO:tasks.workunit.client.1.smithi152.stdout:3/309: chown d6/d53/f5b 232 1 2024-03-04T16:04:34.201 INFO:tasks.workunit.client.1.smithi152.stdout:1/288: truncate d19/d20/d21/d38/f4c 1970985 0 2024-03-04T16:04:34.205 INFO:tasks.workunit.client.1.smithi152.stdout:9/225: write d2/fb [2564252,40471] 0 2024-03-04T16:04:34.207 INFO:tasks.workunit.client.1.smithi152.stdout:0/266: symlink d9/d2c/d2e/l52 0 2024-03-04T16:04:34.207 INFO:tasks.workunit.client.1.smithi152.stdout:0/267: fdatasync d9/f36 0 2024-03-04T16:04:34.209 INFO:tasks.workunit.client.1.smithi152.stdout:7/270: stat d14/f2a 0 2024-03-04T16:04:34.216 INFO:tasks.workunit.client.1.smithi152.stdout:9/226: creat d2/f48 x:0 0 0 2024-03-04T16:04:34.217 INFO:tasks.workunit.client.1.smithi152.stdout:0/268: mknod d9/d1a/c53 0 2024-03-04T16:04:34.217 INFO:tasks.workunit.client.1.smithi152.stdout:9/227: read - d2/d13/d23/d42/f46 zero size 2024-03-04T16:04:34.220 INFO:tasks.workunit.client.1.smithi152.stdout:3/310: rename d6/d10/d2d/d4d/c4e to d6/d10/d2d/d4d/c6a 0 2024-03-04T16:04:34.220 INFO:tasks.workunit.client.1.smithi152.stdout:3/311: readlink d6/d53/l69 0 2024-03-04T16:04:34.225 INFO:tasks.workunit.client.1.smithi152.stdout:8/267: dwrite d2/d8/f4e [0,4194304] 0 2024-03-04T16:04:34.227 INFO:tasks.workunit.client.1.smithi152.stdout:4/214: truncate f3 2819001 0 2024-03-04T16:04:34.232 INFO:tasks.workunit.client.1.smithi152.stdout:5/318: truncate d6/d8/d10/f5b 407317 0 2024-03-04T16:04:34.232 INFO:tasks.workunit.client.1.smithi152.stdout:8/268: dread d2/d8/dd/d22/f3b [0,4194304] 0 2024-03-04T16:04:34.233 INFO:tasks.workunit.client.1.smithi152.stdout:1/289: rmdir d19/d4b/d61 0 2024-03-04T16:04:34.235 INFO:tasks.workunit.client.1.smithi152.stdout:5/319: write d6/d8/f4b [237223,86538] 0 2024-03-04T16:04:34.236 INFO:tasks.workunit.client.1.smithi152.stdout:8/269: chown d2/d3/d15/f4f 13805 1 2024-03-04T16:04:34.236 INFO:tasks.workunit.client.1.smithi152.stdout:1/290: write d19/d4b/f5f [261408,112065] 0 2024-03-04T16:04:34.237 INFO:tasks.workunit.client.1.smithi152.stdout:6/301: getdents d8/d23/d3a 0 2024-03-04T16:04:34.237 INFO:tasks.workunit.client.1.smithi152.stdout:5/320: read d6/f4e [193585,53074] 0 2024-03-04T16:04:34.239 INFO:tasks.workunit.client.1.smithi152.stdout:1/291: chown c7 732772 1 2024-03-04T16:04:34.240 INFO:tasks.workunit.client.1.smithi152.stdout:4/215: dwrite d8/d9/d24/d26/f32 [0,4194304] 0 2024-03-04T16:04:34.246 INFO:tasks.workunit.client.1.smithi152.stdout:9/228: rename d2/d13/c2b to d2/d13/c49 0 2024-03-04T16:04:34.247 INFO:tasks.workunit.client.1.smithi152.stdout:9/229: readlink d2/l2c 0 2024-03-04T16:04:34.247 INFO:tasks.workunit.client.1.smithi152.stdout:9/230: readlink d2/l2c 0 2024-03-04T16:04:34.248 INFO:tasks.workunit.client.1.smithi152.stdout:6/302: mknod d8/d38/c5d 0 2024-03-04T16:04:34.248 INFO:tasks.workunit.client.1.smithi152.stdout:5/321: read d6/d8/d1f/f48 [514422,102175] 0 2024-03-04T16:04:34.248 INFO:tasks.workunit.client.1.smithi152.stdout:9/231: dread - d2/d13/d23/f40 zero size 2024-03-04T16:04:34.248 INFO:tasks.workunit.client.1.smithi152.stdout:6/303: stat d8/d10/c3b 0 2024-03-04T16:04:34.249 INFO:tasks.workunit.client.1.smithi152.stdout:6/304: chown d8/f56 20 1 2024-03-04T16:04:34.256 INFO:tasks.workunit.client.1.smithi152.stdout:9/232: dwrite d2/d13/d23/f29 [0,4194304] 0 2024-03-04T16:04:34.261 INFO:tasks.workunit.client.1.smithi152.stdout:3/312: truncate f5 328010 0 2024-03-04T16:04:34.263 INFO:tasks.workunit.client.1.smithi152.stdout:0/269: dwrite d9/fa [0,4194304] 0 2024-03-04T16:04:34.265 INFO:tasks.workunit.client.1.smithi152.stdout:8/270: rename d2/d3/d21/f19 to d2/f53 0 2024-03-04T16:04:34.266 INFO:tasks.workunit.client.1.smithi152.stdout:8/271: chown d2/c3d 260498557 1 2024-03-04T16:04:34.267 INFO:tasks.workunit.client.1.smithi152.stdout:6/305: link d8/d23/f2c d8/d38/f5e 0 2024-03-04T16:04:34.268 INFO:tasks.workunit.client.1.smithi152.stdout:6/306: write d8/d10/d53/f5b [377827,64962] 0 2024-03-04T16:04:34.269 INFO:tasks.workunit.client.1.smithi152.stdout:5/322: link d6/d8/d10/f49 d6/d8/f74 0 2024-03-04T16:04:34.273 INFO:tasks.workunit.client.1.smithi152.stdout:4/216: rename d8/d9/f15 to d8/d3d/f44 0 2024-03-04T16:04:34.274 INFO:tasks.workunit.client.1.smithi152.stdout:9/233: rename d2 to d2/d13/d23/d42/d4a 22 2024-03-04T16:04:34.276 INFO:tasks.workunit.client.1.smithi152.stdout:9/234: readlink d2/l2c 0 2024-03-04T16:04:34.281 INFO:tasks.workunit.client.1.smithi152.stdout:1/292: dwrite fe [4194304,4194304] 0 2024-03-04T16:04:34.283 INFO:tasks.workunit.client.1.smithi152.stdout:6/307: dread d8/f1f [0,4194304] 0 2024-03-04T16:04:34.288 INFO:tasks.workunit.client.1.smithi152.stdout:3/313: rename d6/d10/f20 to d6/d14/d58/f6b 0 2024-03-04T16:04:34.289 INFO:tasks.workunit.client.1.smithi152.stdout:0/270: rmdir d9/d1b/d3c/d50 0 2024-03-04T16:04:34.292 INFO:tasks.workunit.client.1.smithi152.stdout:9/235: creat d2/d13/d23/d42/f4b x:0 0 0 2024-03-04T16:04:34.294 INFO:tasks.workunit.client.1.smithi152.stdout:1/293: mkdir d19/d20/d2f/d65 0 2024-03-04T16:04:34.300 INFO:tasks.workunit.client.1.smithi152.stdout:4/217: dread d8/f23 [0,4194304] 0 2024-03-04T16:04:34.301 INFO:tasks.workunit.client.1.smithi152.stdout:4/218: write d8/f16 [4146012,4902] 0 2024-03-04T16:04:34.303 INFO:tasks.workunit.client.1.smithi152.stdout:0/271: write d9/f16 [3754202,108429] 0 2024-03-04T16:04:34.308 INFO:tasks.workunit.client.1.smithi152.stdout:9/236: write d2/d13/d2e/f44 [351625,82532] 0 2024-03-04T16:04:34.310 INFO:tasks.workunit.client.1.smithi152.stdout:3/314: dwrite d6/f46 [0,4194304] 0 2024-03-04T16:04:34.311 INFO:tasks.workunit.client.1.smithi152.stdout:6/308: mkdir d8/d5f 0 2024-03-04T16:04:34.314 INFO:tasks.workunit.client.1.smithi152.stdout:1/294: dread fa [0,4194304] 0 2024-03-04T16:04:34.316 INFO:tasks.workunit.client.1.smithi152.stdout:8/272: getdents d2/d8/dd/d22/d49 0 2024-03-04T16:04:34.317 INFO:tasks.workunit.client.1.smithi152.stdout:4/219: creat d8/f45 x:0 0 0 2024-03-04T16:04:34.322 INFO:tasks.workunit.client.1.smithi152.stdout:5/323: getdents d6/d8/d1f/d3d/d6f/d70 0 2024-03-04T16:04:34.323 INFO:tasks.workunit.client.1.smithi152.stdout:5/324: read d6/d8/fe [8250706,29992] 0 2024-03-04T16:04:34.325 INFO:tasks.workunit.client.1.smithi152.stdout:9/237: chown d2/d13/d23/c32 8 1 2024-03-04T16:04:34.327 INFO:tasks.workunit.client.1.smithi152.stdout:3/315: unlink d6/d10/c11 0 2024-03-04T16:04:34.329 INFO:tasks.workunit.client.1.smithi152.stdout:2/249: sync 2024-03-04T16:04:34.329 INFO:tasks.workunit.client.1.smithi152.stdout:5/325: dwrite d6/d50/f59 [0,4194304] 0 2024-03-04T16:04:34.332 INFO:tasks.workunit.client.1.smithi152.stdout:5/326: dread d6/d8/f74 [0,4194304] 0 2024-03-04T16:04:34.348 INFO:tasks.workunit.client.1.smithi152.stdout:0/272: truncate d9/f49 480748 0 2024-03-04T16:04:34.350 INFO:tasks.workunit.client.1.smithi152.stdout:0/273: chown d9/d1b/f4f 10404 1 2024-03-04T16:04:34.350 INFO:tasks.workunit.client.1.smithi152.stdout:8/273: dwrite d2/d3/d21/d2d/f2c [4194304,4194304] 0 2024-03-04T16:04:34.353 INFO:tasks.workunit.client.1.smithi152.stdout:6/309: mknod d8/d5f/c60 0 2024-03-04T16:04:34.357 INFO:tasks.workunit.client.1.smithi152.stdout:2/250: chown da/fc 318755 1 2024-03-04T16:04:34.358 INFO:tasks.workunit.client.1.smithi152.stdout:5/327: creat d6/d8/d1f/d3d/d6f/d70/f75 x:0 0 0 2024-03-04T16:04:34.360 INFO:tasks.workunit.client.1.smithi152.stdout:5/328: chown d6/d8/d1f/d3d/d6f/d70/f71 67 1 2024-03-04T16:04:34.362 INFO:tasks.workunit.client.1.smithi152.stdout:2/251: dread da/df/d2e/f3a [0,4194304] 0 2024-03-04T16:04:34.363 INFO:tasks.workunit.client.1.smithi152.stdout:4/220: symlink d8/d9/l46 0 2024-03-04T16:04:34.364 INFO:tasks.workunit.client.1.smithi152.stdout:1/295: write f2 [2119796,51045] 0 2024-03-04T16:04:34.365 INFO:tasks.workunit.client.1.smithi152.stdout:1/296: readlink d19/d20/l31 0 2024-03-04T16:04:34.365 INFO:tasks.workunit.client.1.smithi152.stdout:1/297: stat d19/d20/d21/l44 0 2024-03-04T16:04:34.369 INFO:tasks.workunit.client.1.smithi152.stdout:8/274: stat d2/d3/d21/d2d/l4d 0 2024-03-04T16:04:34.370 INFO:tasks.workunit.client.1.smithi152.stdout:6/310: dread d8/f9 [0,4194304] 0 2024-03-04T16:04:34.372 INFO:tasks.workunit.client.1.smithi152.stdout:3/316: creat d6/d10/d2d/d59/f6c x:0 0 0 2024-03-04T16:04:34.378 INFO:tasks.workunit.client.1.smithi152.stdout:5/329: mkdir d6/d8/d5a/d64/d76 0 2024-03-04T16:04:34.380 INFO:tasks.workunit.client.1.smithi152.stdout:4/221: creat d8/d2f/f47 x:0 0 0 2024-03-04T16:04:34.382 INFO:tasks.workunit.client.1.smithi152.stdout:7/271: sync 2024-03-04T16:04:34.383 INFO:tasks.workunit.client.1.smithi152.stdout:2/252: dread da/fb [0,4194304] 0 2024-03-04T16:04:34.390 INFO:tasks.workunit.client.1.smithi152.stdout:6/311: rename d8/d38/d4f to d8/d38/d59/d61 0 2024-03-04T16:04:34.391 INFO:tasks.workunit.client.1.smithi152.stdout:6/312: fdatasync d8/d4b/f54 0 2024-03-04T16:04:34.393 INFO:tasks.workunit.client.1.smithi152.stdout:5/330: mknod d6/d8/d1f/d3d/d6f/d70/c77 0 2024-03-04T16:04:34.394 INFO:tasks.workunit.client.1.smithi152.stdout:5/331: dread - d6/d8/d10/f51 zero size 2024-03-04T16:04:34.396 INFO:tasks.workunit.client.1.smithi152.stdout:4/222: creat d8/d9/d24/d26/f48 x:0 0 0 2024-03-04T16:04:34.399 INFO:tasks.workunit.client.1.smithi152.stdout:9/238: getdents d2/d13 0 2024-03-04T16:04:34.401 INFO:tasks.workunit.client.1.smithi152.stdout:1/298: getdents d19/d20/d2f/d65 0 2024-03-04T16:04:34.404 INFO:tasks.workunit.client.1.smithi152.stdout:1/299: truncate d19/d20/d21/f5a 493216 0 2024-03-04T16:04:34.405 INFO:tasks.workunit.client.1.smithi152.stdout:1/300: write d19/d20/d2f/f59 [975523,92908] 0 2024-03-04T16:04:34.405 INFO:tasks.workunit.client.1.smithi152.stdout:7/272: unlink c6 0 2024-03-04T16:04:34.406 INFO:tasks.workunit.client.1.smithi152.stdout:9/239: dwrite d2/f19 [0,4194304] 0 2024-03-04T16:04:34.408 INFO:tasks.workunit.client.1.smithi152.stdout:1/301: chown d19/d20/d21/d38/d63/d64 417476 1 2024-03-04T16:04:34.408 INFO:tasks.workunit.client.1.smithi152.stdout:7/273: write d14/d21/d47/f34 [392510,82753] 0 2024-03-04T16:04:34.408 INFO:tasks.workunit.client.1.smithi152.stdout:9/240: fsync d2/d13/d23/f29 0 2024-03-04T16:04:34.409 INFO:tasks.workunit.client.1.smithi152.stdout:0/274: rmdir d9/d2c/d2e/d38/d4c 0 2024-03-04T16:04:34.411 INFO:tasks.workunit.client.1.smithi152.stdout:1/302: dread d19/f28 [0,4194304] 0 2024-03-04T16:04:34.412 INFO:tasks.workunit.client.1.smithi152.stdout:8/275: truncate d2/f37 5861978 0 2024-03-04T16:04:34.416 INFO:tasks.workunit.client.1.smithi152.stdout:6/313: symlink d8/d42/l62 0 2024-03-04T16:04:34.417 INFO:tasks.workunit.client.1.smithi152.stdout:6/314: stat d8/d23/f55 0 2024-03-04T16:04:34.417 INFO:tasks.workunit.client.1.smithi152.stdout:6/315: rename d8 to d8/d52/d58/d63 22 2024-03-04T16:04:34.419 INFO:tasks.workunit.client.1.smithi152.stdout:5/332: creat d6/d8/d1f/d3d/f78 x:0 0 0 2024-03-04T16:04:34.421 INFO:tasks.workunit.client.1.smithi152.stdout:6/316: dread d8/d2a/f45 [0,4194304] 0 2024-03-04T16:04:34.421 INFO:tasks.workunit.client.1.smithi152.stdout:6/317: read - d8/d23/f5c zero size 2024-03-04T16:04:34.425 INFO:tasks.workunit.client.1.smithi152.stdout:2/253: mkdir da/df/d58 0 2024-03-04T16:04:34.432 INFO:tasks.workunit.client.1.smithi152.stdout:0/275: mkdir d9/d1b/d54 0 2024-03-04T16:04:34.433 INFO:tasks.workunit.client.1.smithi152.stdout:0/276: stat d9/d1b/f1d 0 2024-03-04T16:04:34.434 INFO:tasks.workunit.client.1.smithi152.stdout:7/274: truncate d14/d21/d47/f35 2279434 0 2024-03-04T16:04:34.435 INFO:tasks.workunit.client.1.smithi152.stdout:0/277: write d9/f16 [5972231,42273] 0 2024-03-04T16:04:34.435 INFO:tasks.workunit.client.1.smithi152.stdout:3/317: dwrite d6/f42 [0,4194304] 0 2024-03-04T16:04:34.437 INFO:tasks.workunit.client.1.smithi152.stdout:3/318: write d6/f1a [1430625,21934] 0 2024-03-04T16:04:34.438 INFO:tasks.workunit.client.1.smithi152.stdout:7/275: chown d14/d21/d47/d40/f43 12480560 1 2024-03-04T16:04:34.438 INFO:tasks.workunit.client.1.smithi152.stdout:0/278: write d9/d2c/f2f [4342621,99122] 0 2024-03-04T16:04:34.438 INFO:tasks.workunit.client.1.smithi152.stdout:4/223: getdents d8/d9/d24/d26/d42 0 2024-03-04T16:04:34.439 INFO:tasks.workunit.client.1.smithi152.stdout:3/319: readlink d6/d14/d41/d49/l5e 0 2024-03-04T16:04:34.441 INFO:tasks.workunit.client.1.smithi152.stdout:5/333: write d6/d8/d1f/f44 [2609915,32138] 0 2024-03-04T16:04:34.442 INFO:tasks.workunit.client.1.smithi152.stdout:6/318: rmdir d8/d38 39 2024-03-04T16:04:34.448 INFO:tasks.workunit.client.1.smithi152.stdout:7/276: dwrite d14/d21/d47/d40/f43 [0,4194304] 0 2024-03-04T16:04:34.453 INFO:tasks.workunit.client.1.smithi152.stdout:2/254: write f6 [1061854,110751] 0 2024-03-04T16:04:34.454 INFO:tasks.workunit.client.1.smithi152.stdout:2/255: write f6 [2393725,22555] 0 2024-03-04T16:04:34.455 INFO:tasks.workunit.client.1.smithi152.stdout:8/276: getdents d2/d8/dd/d22/d49 0 2024-03-04T16:04:34.455 INFO:tasks.workunit.client.1.smithi152.stdout:2/256: write da/f23 [946925,81283] 0 2024-03-04T16:04:34.455 INFO:tasks.workunit.client.1.smithi152.stdout:8/277: chown d2/d8/dd 819668 1 2024-03-04T16:04:34.460 INFO:tasks.workunit.client.1.smithi152.stdout:4/224: symlink d8/d3d/l49 0 2024-03-04T16:04:34.461 INFO:tasks.workunit.client.1.smithi152.stdout:0/279: symlink d9/d2c/d2e/l55 0 2024-03-04T16:04:34.462 INFO:tasks.workunit.client.1.smithi152.stdout:0/280: fdatasync d9/d2c/d2e/f32 0 2024-03-04T16:04:34.463 INFO:tasks.workunit.client.1.smithi152.stdout:8/278: dread d2/d3/d21/d2d/f50 [0,4194304] 0 2024-03-04T16:04:34.464 INFO:tasks.workunit.client.1.smithi152.stdout:5/334: symlink d6/d8/d1f/l79 0 2024-03-04T16:04:34.465 INFO:tasks.workunit.client.1.smithi152.stdout:5/335: chown d6/d8/d1f/d3d/f5d 18 1 2024-03-04T16:04:34.465 INFO:tasks.workunit.client.1.smithi152.stdout:8/279: write d2/d8/dd/d22/d26/f38 [3451184,83320] 0 2024-03-04T16:04:34.466 INFO:tasks.workunit.client.1.smithi152.stdout:6/319: symlink d8/d2a/l64 0 2024-03-04T16:04:34.466 INFO:tasks.workunit.client.1.smithi152.stdout:8/280: chown d2/d3/d21/d2d/f50 1806343958 1 2024-03-04T16:04:34.468 INFO:tasks.workunit.client.1.smithi152.stdout:7/277: creat d14/d21/d47/f51 x:0 0 0 2024-03-04T16:04:34.469 INFO:tasks.workunit.client.1.smithi152.stdout:7/278: readlink d14/d21/l24 0 2024-03-04T16:04:34.470 INFO:tasks.workunit.client.1.smithi152.stdout:7/279: read d14/f20 [78856,22313] 0 2024-03-04T16:04:34.475 INFO:tasks.workunit.client.1.smithi152.stdout:4/225: creat d8/d2f/f4a x:0 0 0 2024-03-04T16:04:34.476 INFO:tasks.workunit.client.1.smithi152.stdout:7/280: dwrite d14/d21/d47/f18 [4194304,4194304] 0 2024-03-04T16:04:34.484 INFO:tasks.workunit.client.1.smithi152.stdout:3/320: symlink d6/d5c/l6d 0 2024-03-04T16:04:34.484 INFO:tasks.workunit.client.1.smithi152.stdout:0/281: mkdir d9/d2c/d2e/d38/d56 0 2024-03-04T16:04:34.484 INFO:tasks.workunit.client.1.smithi152.stdout:0/282: write d9/d2c/d2e/f41 [707973,12476] 0 2024-03-04T16:04:34.486 INFO:tasks.workunit.client.1.smithi152.stdout:6/320: mknod d8/d10/c65 0 2024-03-04T16:04:34.487 INFO:tasks.workunit.client.1.smithi152.stdout:8/281: creat d2/d8/dd/f54 x:0 0 0 2024-03-04T16:04:34.488 INFO:tasks.workunit.client.1.smithi152.stdout:3/321: dwrite d6/d10/d2d/f34 [0,4194304] 0 2024-03-04T16:04:34.490 INFO:tasks.workunit.client.1.smithi152.stdout:3/322: write d6/d14/d41/d49/f5a [216941,87623] 0 2024-03-04T16:04:34.491 INFO:tasks.workunit.client.1.smithi152.stdout:2/257: symlink da/df/d58/l59 0 2024-03-04T16:04:34.492 INFO:tasks.workunit.client.1.smithi152.stdout:2/258: chown da/f23 11 1 2024-03-04T16:04:34.494 INFO:tasks.workunit.client.1.smithi152.stdout:7/281: mknod d14/d21/d47/d32/c52 0 2024-03-04T16:04:34.495 INFO:tasks.workunit.client.1.smithi152.stdout:7/282: readlink lc 0 2024-03-04T16:04:34.496 INFO:tasks.workunit.client.1.smithi152.stdout:0/283: mknod d9/d2c/c57 0 2024-03-04T16:04:34.496 INFO:tasks.workunit.client.1.smithi152.stdout:7/283: chown d14/c1a 1284 1 2024-03-04T16:04:34.496 INFO:tasks.workunit.client.1.smithi152.stdout:6/321: dwrite d8/d23/d3a/f50 [0,4194304] 0 2024-03-04T16:04:34.497 INFO:tasks.workunit.client.1.smithi152.stdout:5/336: symlink d6/d8/l7a 0 2024-03-04T16:04:34.500 INFO:tasks.workunit.client.1.smithi152.stdout:8/282: creat d2/d8/dd/d22/d49/f55 x:0 0 0 2024-03-04T16:04:34.500 INFO:tasks.workunit.client.1.smithi152.stdout:6/322: chown d8/d23/d3a/f50 3 1 2024-03-04T16:04:34.501 INFO:tasks.workunit.client.1.smithi152.stdout:4/226: creat d8/d9/d24/d26/d42/f4b x:0 0 0 2024-03-04T16:04:34.503 INFO:tasks.workunit.client.1.smithi152.stdout:2/259: dwrite da/df/d35/f4a [0,4194304] 0 2024-03-04T16:04:34.506 INFO:tasks.workunit.client.1.smithi152.stdout:7/284: fsync d14/d21/d47/f46 0 2024-03-04T16:04:34.506 INFO:tasks.workunit.client.1.smithi152.stdout:0/284: read d9/d1b/f1d [3977411,98221] 0 2024-03-04T16:04:34.506 INFO:tasks.workunit.client.1.smithi152.stdout:5/337: truncate d6/d8/f30 3466441 0 2024-03-04T16:04:34.508 INFO:tasks.workunit.client.1.smithi152.stdout:5/338: dread d6/d8/f32 [0,4194304] 0 2024-03-04T16:04:34.510 INFO:tasks.workunit.client.1.smithi152.stdout:3/323: mkdir d6/d10/d2d/d6e 0 2024-03-04T16:04:34.511 INFO:tasks.workunit.client.1.smithi152.stdout:4/227: creat d8/d3d/f4c x:0 0 0 2024-03-04T16:04:34.512 INFO:tasks.workunit.client.1.smithi152.stdout:4/228: chown d8/d3d/c3f 2085975 1 2024-03-04T16:04:34.514 INFO:tasks.workunit.client.1.smithi152.stdout:4/229: fsync d8/f45 0 2024-03-04T16:04:34.514 INFO:tasks.workunit.client.1.smithi152.stdout:4/230: write f0 [5303285,53577] 0 2024-03-04T16:04:34.514 INFO:tasks.workunit.client.1.smithi152.stdout:8/283: dwrite d2/d8/f14 [0,4194304] 0 2024-03-04T16:04:34.515 INFO:tasks.workunit.client.1.smithi152.stdout:2/260: rename da/df/d35/f44 to da/d24/d47/f5a 0 2024-03-04T16:04:34.519 INFO:tasks.workunit.client.1.smithi152.stdout:2/261: dread da/f11 [0,4194304] 0 2024-03-04T16:04:34.519 INFO:tasks.workunit.client.1.smithi152.stdout:7/285: symlink d14/d2b/l53 0 2024-03-04T16:04:34.520 INFO:tasks.workunit.client.1.smithi152.stdout:0/285: symlink d9/d1a/l58 0 2024-03-04T16:04:34.525 INFO:tasks.workunit.client.1.smithi152.stdout:2/262: dwrite da/f1a [0,4194304] 0 2024-03-04T16:04:34.528 INFO:tasks.workunit.client.1.smithi152.stdout:5/339: creat d6/d50/f7b x:0 0 0 2024-03-04T16:04:34.532 INFO:tasks.workunit.client.1.smithi152.stdout:2/263: dwrite da/f56 [0,4194304] 0 2024-03-04T16:04:34.546 INFO:tasks.workunit.client.1.smithi152.stdout:7/286: symlink d14/d21/d47/l54 0 2024-03-04T16:04:34.548 INFO:tasks.workunit.client.1.smithi152.stdout:0/286: mknod d9/d1b/d3c/c59 0 2024-03-04T16:04:34.551 INFO:tasks.workunit.client.1.smithi152.stdout:5/340: creat d6/d8/d10/f7c x:0 0 0 2024-03-04T16:04:34.555 INFO:tasks.workunit.client.1.smithi152.stdout:2/264: fsync f4 0 2024-03-04T16:04:34.566 INFO:tasks.workunit.client.1.smithi152.stdout:3/324: creat d6/d10/d2d/f6f x:0 0 0 2024-03-04T16:04:34.566 INFO:tasks.workunit.client.1.smithi152.stdout:3/325: fdatasync d6/d14/f37 0 2024-03-04T16:04:34.567 INFO:tasks.workunit.client.1.smithi152.stdout:3/326: chown d6/d10/l1b 5049 1 2024-03-04T16:04:34.567 INFO:tasks.workunit.client.1.smithi152.stdout:2/265: creat da/df/d2e/d51/f5b x:0 0 0 2024-03-04T16:04:34.568 INFO:tasks.workunit.client.1.smithi152.stdout:2/266: dread - da/df/d35/f46 zero size 2024-03-04T16:04:34.569 INFO:tasks.workunit.client.1.smithi152.stdout:4/231: creat d8/d9/d24/f4d x:0 0 0 2024-03-04T16:04:34.570 INFO:tasks.workunit.client.1.smithi152.stdout:4/232: chown f7 708169 1 2024-03-04T16:04:34.572 INFO:tasks.workunit.client.1.smithi152.stdout:6/323: getdents d8/d38 0 2024-03-04T16:04:34.573 INFO:tasks.workunit.client.1.smithi152.stdout:5/341: creat d6/d8/d5a/d64/d76/f7d x:0 0 0 2024-03-04T16:04:34.576 INFO:tasks.workunit.client.1.smithi152.stdout:3/327: creat d6/d10/f70 x:0 0 0 2024-03-04T16:04:34.576 INFO:tasks.workunit.client.1.smithi152.stdout:5/342: dread d6/d8/d10/f27 [0,4194304] 0 2024-03-04T16:04:34.577 INFO:tasks.workunit.client.1.smithi152.stdout:8/284: truncate d2/d3/d21/d2d/f1e 1702745 0 2024-03-04T16:04:34.577 INFO:tasks.workunit.client.1.smithi152.stdout:5/343: chown d6/d8/d5a 248219 1 2024-03-04T16:04:34.578 INFO:tasks.workunit.client.1.smithi152.stdout:2/267: fdatasync da/d24/f32 0 2024-03-04T16:04:34.598 INFO:tasks.workunit.client.1.smithi152.stdout:7/287: rename d14/d21/f45 to d14/d21/f55 0 2024-03-04T16:04:34.599 INFO:tasks.workunit.client.1.smithi152.stdout:8/285: write d2/d8/dd/f2f [1544327,112806] 0 2024-03-04T16:04:34.599 INFO:tasks.workunit.client.1.smithi152.stdout:7/288: chown d14 1830974 1 2024-03-04T16:04:34.599 INFO:tasks.workunit.client.1.smithi152.stdout:6/324: symlink d8/d10/l66 0 2024-03-04T16:04:34.601 INFO:tasks.workunit.client.1.smithi152.stdout:5/344: creat d6/d50/f7e x:0 0 0 2024-03-04T16:04:34.601 INFO:tasks.workunit.client.1.smithi152.stdout:8/286: write d2/f42 [1451493,98155] 0 2024-03-04T16:04:34.602 INFO:tasks.workunit.client.1.smithi152.stdout:5/345: chown d6/d50/l5f 24032983 1 2024-03-04T16:04:34.603 INFO:tasks.workunit.client.1.smithi152.stdout:0/287: rename d9/d2c/d2e/d38/l4b to d9/l5a 0 2024-03-04T16:04:34.607 INFO:tasks.workunit.client.1.smithi152.stdout:2/268: rmdir da/d24/d47/d4f 39 2024-03-04T16:04:34.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:34 smithi152 ceph-mon[36876]: pgmap v33: 65 pgs: 65 active+clean; 2.5 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 39 MiB/s rd, 133 MiB/s wr, 305 op/s 2024-03-04T16:04:34.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:34 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:34.608 INFO:tasks.workunit.client.1.smithi152.stdout:3/328: dwrite d6/d10/d2d/f34 [4194304,4194304] 0 2024-03-04T16:04:34.610 INFO:tasks.workunit.client.1.smithi152.stdout:7/289: creat d14/d2b/f56 x:0 0 0 2024-03-04T16:04:34.610 INFO:tasks.workunit.client.1.smithi152.stdout:3/329: write d6/d10/f70 [154746,128088] 0 2024-03-04T16:04:34.613 INFO:tasks.workunit.client.1.smithi152.stdout:6/325: creat d8/d52/f67 x:0 0 0 2024-03-04T16:04:34.614 INFO:tasks.workunit.client.1.smithi152.stdout:6/326: write d8/f22 [4524926,59167] 0 2024-03-04T16:04:34.615 INFO:tasks.workunit.client.1.smithi152.stdout:8/287: creat d2/d3/d15/d1c/f56 x:0 0 0 2024-03-04T16:04:34.622 INFO:tasks.workunit.client.1.smithi152.stdout:2/269: dread da/d2c/f41 [0,4194304] 0 2024-03-04T16:04:34.624 INFO:tasks.workunit.client.1.smithi152.stdout:7/290: creat d14/d21/d31/f57 x:0 0 0 2024-03-04T16:04:34.627 INFO:tasks.workunit.client.1.smithi152.stdout:3/330: dread f5 [0,4194304] 0 2024-03-04T16:04:34.628 INFO:tasks.workunit.client.1.smithi152.stdout:6/327: rmdir d8/d23 39 2024-03-04T16:04:34.632 INFO:tasks.workunit.client.1.smithi152.stdout:0/288: mkdir d9/d2c/d2e/d38/d56/d5b 0 2024-03-04T16:04:34.635 INFO:tasks.workunit.client.1.smithi152.stdout:6/328: dwrite d8/d10/f18 [0,4194304] 0 2024-03-04T16:04:34.635 INFO:tasks.workunit.client.1.smithi152.stdout:7/291: creat d14/d2b/f58 x:0 0 0 2024-03-04T16:04:34.636 INFO:tasks.workunit.client.1.smithi152.stdout:8/288: dwrite d2/d3/d21/d2d/f50 [0,4194304] 0 2024-03-04T16:04:34.638 INFO:tasks.workunit.client.1.smithi152.stdout:3/331: creat d6/d14/d58/f71 x:0 0 0 2024-03-04T16:04:34.638 INFO:tasks.workunit.client.1.smithi152.stdout:5/346: creat d6/f7f x:0 0 0 2024-03-04T16:04:34.639 INFO:tasks.workunit.client.1.smithi152.stdout:3/332: truncate d6/f68 963477 0 2024-03-04T16:04:34.641 INFO:tasks.workunit.client.1.smithi152.stdout:0/289: write d9/f45 [903071,54352] 0 2024-03-04T16:04:34.641 INFO:tasks.workunit.client.1.smithi152.stdout:2/270: creat da/df/d35/d50/f5c x:0 0 0 2024-03-04T16:04:34.643 INFO:tasks.workunit.client.1.smithi152.stdout:7/292: dread d14/d21/d31/f33 [0,4194304] 0 2024-03-04T16:04:34.645 INFO:tasks.workunit.client.1.smithi152.stdout:7/293: truncate d14/d21/f30 5021282 0 2024-03-04T16:04:34.646 INFO:tasks.workunit.client.1.smithi152.stdout:7/294: dread - d14/f15 zero size 2024-03-04T16:04:34.649 INFO:tasks.workunit.client.1.smithi152.stdout:5/347: mknod d6/d8/d5a/c80 0 2024-03-04T16:04:34.650 INFO:tasks.workunit.client.1.smithi152.stdout:3/333: creat d6/d53/f72 x:0 0 0 2024-03-04T16:04:34.650 INFO:tasks.workunit.client.1.smithi152.stdout:2/271: dwrite da/f23 [0,4194304] 0 2024-03-04T16:04:34.651 INFO:tasks.workunit.client.1.smithi152.stdout:5/348: dread - d6/d8/d1f/d3d/f78 zero size 2024-03-04T16:04:34.652 INFO:tasks.workunit.client.1.smithi152.stdout:3/334: write d6/d10/d2d/d4d/f55 [1158802,31406] 0 2024-03-04T16:04:34.654 INFO:tasks.workunit.client.1.smithi152.stdout:0/290: unlink d9/le 0 2024-03-04T16:04:34.657 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:34 smithi116 ceph-mon[29120]: pgmap v33: 65 pgs: 65 active+clean; 2.5 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 39 MiB/s rd, 133 MiB/s wr, 305 op/s 2024-03-04T16:04:34.657 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:34 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:34.659 INFO:tasks.workunit.client.1.smithi152.stdout:6/329: getdents d8/d38/d59/d61/d57 0 2024-03-04T16:04:34.660 INFO:tasks.workunit.client.1.smithi152.stdout:8/289: mknod d2/d3/d21/c57 0 2024-03-04T16:04:34.662 INFO:tasks.workunit.client.1.smithi152.stdout:8/290: chown d2/d8/dd/d22/f24 139437 1 2024-03-04T16:04:34.663 INFO:tasks.workunit.client.1.smithi152.stdout:7/295: symlink d14/d21/d47/d40/l59 0 2024-03-04T16:04:34.665 INFO:tasks.workunit.client.1.smithi152.stdout:5/349: creat d6/d8/d5a/d64/d76/f81 x:0 0 0 2024-03-04T16:04:34.665 INFO:tasks.workunit.client.1.smithi152.stdout:5/350: readlink d6/d8/l33 0 2024-03-04T16:04:34.666 INFO:tasks.workunit.client.1.smithi152.stdout:7/296: dread d14/d2b/f4e [0,4194304] 0 2024-03-04T16:04:34.669 INFO:tasks.workunit.client.1.smithi152.stdout:6/330: symlink d8/d52/l68 0 2024-03-04T16:04:34.671 INFO:tasks.workunit.client.1.smithi152.stdout:5/351: dread d6/d50/f59 [0,4194304] 0 2024-03-04T16:04:34.672 INFO:tasks.workunit.client.1.smithi152.stdout:5/352: stat d6/d8/d1f/d3d/d6f/d70 0 2024-03-04T16:04:34.673 INFO:tasks.workunit.client.1.smithi152.stdout:8/291: creat d2/d8/dd/d22/d26/f58 x:0 0 0 2024-03-04T16:04:34.673 INFO:tasks.workunit.client.1.smithi152.stdout:0/291: dwrite d9/d1b/f1d [0,4194304] 0 2024-03-04T16:04:34.678 INFO:tasks.workunit.client.1.smithi152.stdout:2/272: write da/d24/f32 [5790263,61641] 0 2024-03-04T16:04:34.678 INFO:tasks.workunit.client.1.smithi152.stdout:2/273: fdatasync da/f22 0 2024-03-04T16:04:34.679 INFO:tasks.workunit.client.1.smithi152.stdout:2/274: chown da/df/f1f 1081 1 2024-03-04T16:04:34.681 INFO:tasks.workunit.client.1.smithi152.stdout:2/275: write da/df/d2e/f4e [467806,48346] 0 2024-03-04T16:04:34.682 INFO:tasks.workunit.client.1.smithi152.stdout:2/276: chown da/df/d2e 49443 1 2024-03-04T16:04:34.684 INFO:tasks.workunit.client.1.smithi152.stdout:3/335: truncate d6/d14/f37 2284700 0 2024-03-04T16:04:34.684 INFO:tasks.workunit.client.1.smithi152.stdout:2/277: fsync da/df/d35/f46 0 2024-03-04T16:04:34.684 INFO:tasks.workunit.client.1.smithi152.stdout:7/297: symlink d14/d21/d47/l5a 0 2024-03-04T16:04:34.688 INFO:tasks.workunit.client.1.smithi152.stdout:7/298: chown d14/d21/d47/d32/c52 119 1 2024-03-04T16:04:34.691 INFO:tasks.workunit.client.1.smithi152.stdout:3/336: dwrite d6/d10/f27 [0,4194304] 0 2024-03-04T16:04:34.692 INFO:tasks.workunit.client.1.smithi152.stdout:5/353: dread d6/d8/ff [0,4194304] 0 2024-03-04T16:04:34.692 INFO:tasks.workunit.client.1.smithi152.stdout:3/337: fdatasync d6/d10/f65 0 2024-03-04T16:04:34.693 INFO:tasks.workunit.client.1.smithi152.stdout:3/338: fdatasync d6/f2c 0 2024-03-04T16:04:34.697 INFO:tasks.workunit.client.1.smithi152.stdout:6/331: dread d8/d23/f33 [0,4194304] 0 2024-03-04T16:04:34.701 INFO:tasks.workunit.client.1.smithi152.stdout:2/278: rename c8 to da/d24/c5d 0 2024-03-04T16:04:34.707 INFO:tasks.workunit.client.1.smithi152.stdout:3/339: symlink d6/d10/d2d/d4d/l73 0 2024-03-04T16:04:34.712 INFO:tasks.workunit.client.1.smithi152.stdout:6/332: mknod d8/c69 0 2024-03-04T16:04:34.713 INFO:tasks.workunit.client.1.smithi152.stdout:8/292: write d2/d3/d21/d2d/f1e [2551857,118481] 0 2024-03-04T16:04:34.713 INFO:tasks.workunit.client.1.smithi152.stdout:6/333: write d8/d2a/f2e [3469221,127933] 0 2024-03-04T16:04:34.717 INFO:tasks.workunit.client.1.smithi152.stdout:5/354: write d6/d8/f1d [21544,7838] 0 2024-03-04T16:04:34.718 INFO:tasks.workunit.client.1.smithi152.stdout:2/279: dread da/d24/f42 [0,4194304] 0 2024-03-04T16:04:34.787 INFO:tasks.workunit.client.1.smithi152.stdout:6/334: symlink d8/d5f/l6a 0 2024-03-04T16:04:34.788 INFO:tasks.workunit.client.1.smithi152.stdout:8/293: creat d2/d3/d15/d1c/f59 x:0 0 0 2024-03-04T16:04:34.788 INFO:tasks.workunit.client.1.smithi152.stdout:7/299: link d14/f2c d14/d21/d47/d36/f5b 0 2024-03-04T16:04:34.788 INFO:tasks.workunit.client.1.smithi152.stdout:2/280: mkdir da/d2c/d5e 0 2024-03-04T16:04:34.790 INFO:tasks.workunit.client.1.smithi152.stdout:2/281: truncate da/fb 4609732 0 2024-03-04T16:04:34.791 INFO:tasks.workunit.client.1.smithi152.stdout:6/335: creat d8/d23/d3a/f6b x:0 0 0 2024-03-04T16:04:34.792 INFO:tasks.workunit.client.1.smithi152.stdout:5/355: dwrite d6/d50/f5c [0,4194304] 0 2024-03-04T16:04:34.792 INFO:tasks.workunit.client.1.smithi152.stdout:6/336: stat d8/f56 0 2024-03-04T16:04:34.800 INFO:tasks.workunit.client.1.smithi152.stdout:8/294: rename d2/d8/dd/c3a to d2/d3/d21/d2d/c5a 0 2024-03-04T16:04:34.803 INFO:tasks.workunit.client.1.smithi152.stdout:6/337: mknod d8/d38/d59/c6c 0 2024-03-04T16:04:34.804 INFO:tasks.workunit.client.1.smithi152.stdout:5/356: dwrite d6/fd [4194304,4194304] 0 2024-03-04T16:04:34.804 INFO:tasks.workunit.client.1.smithi152.stdout:2/282: dwrite da/df/d35/d50/f5c [0,4194304] 0 2024-03-04T16:04:34.810 INFO:tasks.workunit.client.1.smithi152.stdout:8/295: rmdir d2/d8/dd/d22 39 2024-03-04T16:04:34.814 INFO:tasks.workunit.client.1.smithi152.stdout:6/338: mkdir d8/d23/d3a/d6d 0 2024-03-04T16:04:34.814 INFO:tasks.workunit.client.1.smithi152.stdout:2/283: dread da/fb [0,4194304] 0 2024-03-04T16:04:34.815 INFO:tasks.workunit.client.1.smithi152.stdout:2/284: stat da/df/d35 0 2024-03-04T16:04:34.815 INFO:tasks.workunit.client.1.smithi152.stdout:7/300: dwrite d14/d21/f25 [0,4194304] 0 2024-03-04T16:04:34.816 INFO:tasks.workunit.client.1.smithi152.stdout:8/296: write d2/d3/d21/d2d/d31/f35 [192346,5820] 0 2024-03-04T16:04:34.816 INFO:tasks.workunit.client.1.smithi152.stdout:2/285: chown f4 218 1 2024-03-04T16:04:34.816 INFO:tasks.workunit.client.1.smithi152.stdout:7/301: fdatasync d14/f15 0 2024-03-04T16:04:34.818 INFO:tasks.workunit.client.1.smithi152.stdout:8/297: write d2/d3/d15/d1c/f56 [742816,73342] 0 2024-03-04T16:04:34.818 INFO:tasks.workunit.client.1.smithi152.stdout:7/302: write d14/f42 [760366,67525] 0 2024-03-04T16:04:34.818 INFO:tasks.workunit.client.1.smithi152.stdout:5/357: mkdir d6/d8/d5a/d82 0 2024-03-04T16:04:34.822 INFO:tasks.workunit.client.1.smithi152.stdout:5/358: write d6/f7f [453958,106242] 0 2024-03-04T16:04:34.822 INFO:tasks.workunit.client.1.smithi152.stdout:8/298: write d2/d8/f2e [462895,113898] 0 2024-03-04T16:04:34.822 INFO:tasks.workunit.client.1.smithi152.stdout:6/339: mknod d8/c6e 0 2024-03-04T16:04:34.825 INFO:tasks.workunit.client.1.smithi152.stdout:2/286: mkdir da/d5f 0 2024-03-04T16:04:34.828 INFO:tasks.workunit.client.1.smithi152.stdout:5/359: truncate d6/d8/d1f/d3d/d61/f6c 551141 0 2024-03-04T16:04:34.832 INFO:tasks.workunit.client.1.smithi152.stdout:5/360: creat d6/d50/f83 x:0 0 0 2024-03-04T16:04:34.835 INFO:tasks.workunit.client.1.smithi152.stdout:6/340: dwrite d8/d23/f25 [0,4194304] 0 2024-03-04T16:04:34.837 INFO:tasks.workunit.client.1.smithi152.stdout:5/361: read d6/fd [790571,13660] 0 2024-03-04T16:04:34.839 INFO:tasks.workunit.client.1.smithi152.stdout:6/341: mkdir d8/d38/d6f 0 2024-03-04T16:04:34.839 INFO:tasks.workunit.client.1.smithi152.stdout:6/342: read - d8/d23/d3a/f6b zero size 2024-03-04T16:04:34.841 INFO:tasks.workunit.client.1.smithi152.stdout:5/362: symlink d6/d8/d10/l84 0 2024-03-04T16:04:34.852 INFO:tasks.workunit.client.1.smithi152.stdout:5/363: creat d6/d8/d1f/d3d/f85 x:0 0 0 2024-03-04T16:04:34.854 INFO:tasks.workunit.client.1.smithi152.stdout:7/303: write d14/f2c [1162809,119248] 0 2024-03-04T16:04:34.856 INFO:tasks.workunit.client.1.smithi152.stdout:7/304: write d14/f15 [189713,23846] 0 2024-03-04T16:04:34.856 INFO:tasks.workunit.client.1.smithi152.stdout:8/299: truncate d2/d3/d21/d2d/f1e 1799568 0 2024-03-04T16:04:34.857 INFO:tasks.workunit.client.1.smithi152.stdout:8/300: stat d2/d3/d15/f4f 0 2024-03-04T16:04:34.858 INFO:tasks.workunit.client.1.smithi152.stdout:5/364: getdents d6 0 2024-03-04T16:04:34.858 INFO:tasks.workunit.client.1.smithi152.stdout:8/301: readlink d2/d8/dd/l11 0 2024-03-04T16:04:34.859 INFO:tasks.workunit.client.1.smithi152.stdout:5/365: readlink d6/d8/d1f/d3d/l6b 0 2024-03-04T16:04:34.859 INFO:tasks.workunit.client.1.smithi152.stdout:7/305: fdatasync d14/d21/d31/f33 0 2024-03-04T16:04:34.861 INFO:tasks.workunit.client.1.smithi152.stdout:8/302: mknod d2/d3/d21/d2d/c5b 0 2024-03-04T16:04:34.862 INFO:tasks.workunit.client.1.smithi152.stdout:2/287: truncate da/df/d35/f4a 2188188 0 2024-03-04T16:04:34.865 INFO:tasks.workunit.client.1.smithi152.stdout:7/306: mknod d14/d21/d47/c5c 0 2024-03-04T16:04:34.865 INFO:tasks.workunit.client.1.smithi152.stdout:7/307: chown d14/d21/c28 50716260 1 2024-03-04T16:04:34.866 INFO:tasks.workunit.client.1.smithi152.stdout:7/308: dread - d14/d21/d31/f4f zero size 2024-03-04T16:04:34.867 INFO:tasks.workunit.client.1.smithi152.stdout:1/303: sync 2024-03-04T16:04:34.867 INFO:tasks.workunit.client.1.smithi152.stdout:9/241: sync 2024-03-04T16:04:34.867 INFO:tasks.workunit.client.1.smithi152.stdout:1/304: chown d19/f1a 60 1 2024-03-04T16:04:34.868 INFO:tasks.workunit.client.1.smithi152.stdout:9/242: chown d2/f28 280 1 2024-03-04T16:04:34.868 INFO:tasks.workunit.client.1.smithi152.stdout:6/343: truncate d8/d2a/f36 4723775 0 2024-03-04T16:04:34.871 INFO:tasks.workunit.client.1.smithi152.stdout:2/288: dread da/fe [0,4194304] 0 2024-03-04T16:04:34.872 INFO:tasks.workunit.client.1.smithi152.stdout:8/303: rmdir d2/d8/dd/d22 39 2024-03-04T16:04:34.873 INFO:tasks.workunit.client.1.smithi152.stdout:9/243: rename d2/f36 to d2/d13/d23/f4c 0 2024-03-04T16:04:34.874 INFO:tasks.workunit.client.1.smithi152.stdout:9/244: chown d2/d13/d38 296 1 2024-03-04T16:04:34.891 INFO:tasks.workunit.client.1.smithi152.stdout:6/344: creat d8/d23/d3a/f70 x:0 0 0 2024-03-04T16:04:34.892 INFO:tasks.workunit.client.1.smithi152.stdout:6/345: chown d8/d10/d53 110608153 1 2024-03-04T16:04:34.893 INFO:tasks.workunit.client.1.smithi152.stdout:5/366: creat d6/d8/f86 x:0 0 0 2024-03-04T16:04:34.895 INFO:tasks.workunit.client.1.smithi152.stdout:2/289: rmdir da/d24/d47 39 2024-03-04T16:04:34.899 INFO:tasks.workunit.client.1.smithi152.stdout:8/304: symlink d2/d3/d21/d2d/d48/l5c 0 2024-03-04T16:04:34.902 INFO:tasks.workunit.client.1.smithi152.stdout:9/245: mkdir d2/d13/d23/d42/d4d 0 2024-03-04T16:04:34.903 INFO:tasks.workunit.client.1.smithi152.stdout:9/246: chown d2/d13/c49 3 1 2024-03-04T16:04:34.904 INFO:tasks.workunit.client.1.smithi152.stdout:6/346: mkdir d8/d23/d3a/d71 0 2024-03-04T16:04:34.907 INFO:tasks.workunit.client.1.smithi152.stdout:5/367: creat d6/d8/d5a/f87 x:0 0 0 2024-03-04T16:04:34.911 INFO:tasks.workunit.client.1.smithi152.stdout:8/305: rmdir d2 39 2024-03-04T16:04:34.911 INFO:tasks.workunit.client.1.smithi152.stdout:6/347: dwrite d8/f1f [4194304,4194304] 0 2024-03-04T16:04:34.912 INFO:tasks.workunit.client.1.smithi152.stdout:6/348: write d8/d23/d3a/f50 [1809224,24018] 0 2024-03-04T16:04:34.913 INFO:tasks.workunit.client.1.smithi152.stdout:6/349: readlink d8/d10/l13 0 2024-03-04T16:04:34.915 INFO:tasks.workunit.client.1.smithi152.stdout:1/305: rename lc to d19/d20/d21/d38/l66 0 2024-03-04T16:04:34.917 INFO:tasks.workunit.client.1.smithi152.stdout:9/247: unlink d2/f48 0 2024-03-04T16:04:34.920 INFO:tasks.workunit.client.1.smithi152.stdout:2/290: symlink da/l60 0 2024-03-04T16:04:34.921 INFO:tasks.workunit.client.1.smithi152.stdout:5/368: mkdir d6/d8/d1f/d88 0 2024-03-04T16:04:34.930 INFO:tasks.workunit.client.1.smithi152.stdout:2/291: chown da/df/c2d 1127141 1 2024-03-04T16:04:34.934 INFO:tasks.workunit.client.1.smithi152.stdout:1/306: creat d19/d20/d2f/d5d/f67 x:0 0 0 2024-03-04T16:04:34.935 INFO:tasks.workunit.client.1.smithi152.stdout:1/307: stat d19/d20/d21/d38/l3c 0 2024-03-04T16:04:34.937 INFO:tasks.workunit.client.1.smithi152.stdout:8/306: rename d2/d3/d21/d33 to d2/d8/dd/d22/d49/d5d 0 2024-03-04T16:04:34.938 INFO:tasks.workunit.client.1.smithi152.stdout:6/350: link d8/d23/l3d d8/d38/d6f/l72 0 2024-03-04T16:04:34.939 INFO:tasks.workunit.client.1.smithi152.stdout:6/351: write d8/d38/f48 [629172,1628] 0 2024-03-04T16:04:34.940 INFO:tasks.workunit.client.1.smithi152.stdout:1/308: fsync d19/d20/d21/f37 0 2024-03-04T16:04:34.942 INFO:tasks.workunit.client.1.smithi152.stdout:8/307: mknod d2/d3/d15/d1c/c5e 0 2024-03-04T16:04:34.944 INFO:tasks.workunit.client.1.smithi152.stdout:6/352: creat d8/d5f/f73 x:0 0 0 2024-03-04T16:04:34.946 INFO:tasks.workunit.client.1.smithi152.stdout:6/353: write d8/d10/f18 [1040236,65228] 0 2024-03-04T16:04:34.946 INFO:tasks.workunit.client.1.smithi152.stdout:1/309: dwrite d19/d20/d2f/f4d [0,4194304] 0 2024-03-04T16:04:34.951 INFO:tasks.workunit.client.1.smithi152.stdout:8/308: rename d2/d8/dd/d22/f24 to d2/d3/d21/f5f 0 2024-03-04T16:04:34.952 INFO:tasks.workunit.client.1.smithi152.stdout:6/354: creat d8/d42/f74 x:0 0 0 2024-03-04T16:04:34.957 INFO:tasks.workunit.client.1.smithi152.stdout:6/355: dread d8/d23/f25 [0,4194304] 0 2024-03-04T16:04:34.962 INFO:tasks.workunit.client.1.smithi152.stdout:6/356: mkdir d8/d38/d75 0 2024-03-04T16:04:34.966 INFO:tasks.workunit.client.1.smithi152.stdout:6/357: link d8/d10/c65 d8/d52/d58/c76 0 2024-03-04T16:04:34.969 INFO:tasks.workunit.client.1.smithi152.stdout:6/358: rename d8/d2a/f2e to d8/d5f/f77 0 2024-03-04T16:04:34.972 INFO:tasks.workunit.client.1.smithi152.stdout:6/359: truncate d8/f2f 240161 0 2024-03-04T16:04:34.979 INFO:tasks.workunit.client.1.smithi152.stdout:6/360: dread d8/f12 [0,4194304] 0 2024-03-04T16:04:34.979 INFO:tasks.workunit.client.1.smithi152.stdout:6/361: chown d8 1491 1 2024-03-04T16:04:34.986 INFO:tasks.workunit.client.1.smithi152.stdout:2/292: dwrite da/d24/f2a [0,4194304] 0 2024-03-04T16:04:34.989 INFO:tasks.workunit.client.1.smithi152.stdout:2/293: truncate da/fe 3812492 0 2024-03-04T16:04:35.002 INFO:tasks.workunit.client.1.smithi152.stdout:1/310: write d19/d20/d21/f2d [4009091,62453] 0 2024-03-04T16:04:35.003 INFO:tasks.workunit.client.1.smithi152.stdout:1/311: write d19/d20/d21/d38/f3a [2216744,60570] 0 2024-03-04T16:04:35.006 INFO:tasks.workunit.client.1.smithi152.stdout:6/362: chown d8/d2a/f36 2772694 1 2024-03-04T16:04:35.006 INFO:tasks.workunit.client.1.smithi152.stdout:6/363: readlink d8/d2a/l4d 0 2024-03-04T16:04:35.007 INFO:tasks.workunit.client.1.smithi152.stdout:6/364: write d8/d23/f5c [665778,2079] 0 2024-03-04T16:04:35.012 INFO:tasks.workunit.client.1.smithi152.stdout:4/233: sync 2024-03-04T16:04:35.013 INFO:tasks.workunit.client.1.smithi152.stdout:4/234: chown d8/la 1 1 2024-03-04T16:04:35.014 INFO:tasks.workunit.client.1.smithi152.stdout:8/309: dwrite d2/f53 [0,4194304] 0 2024-03-04T16:04:35.016 INFO:tasks.workunit.client.1.smithi152.stdout:4/235: truncate d8/d9/d24/f40 600972 0 2024-03-04T16:04:35.017 INFO:tasks.workunit.client.1.smithi152.stdout:4/236: write d8/f16 [876691,2530] 0 2024-03-04T16:04:35.020 INFO:tasks.workunit.client.1.smithi152.stdout:8/310: write d2/d3/d21/f5f [1630116,22874] 0 2024-03-04T16:04:35.027 INFO:tasks.workunit.client.1.smithi152.stdout:8/311: mkdir d2/d8/dd/d22/d60 0 2024-03-04T16:04:35.029 INFO:tasks.workunit.client.1.smithi152.stdout:8/312: write d2/d8/dd/d22/d49/d5d/f44 [234944,32547] 0 2024-03-04T16:04:35.033 INFO:tasks.workunit.client.1.smithi152.stdout:8/313: stat d2/d8/dd/d22/d26/l3e 0 2024-03-04T16:04:35.037 INFO:tasks.workunit.client.1.smithi152.stdout:8/314: creat d2/d8/dd/d22/d26/f61 x:0 0 0 2024-03-04T16:04:35.038 INFO:tasks.workunit.client.1.smithi152.stdout:8/315: readlink d2/d3/d21/d2d/l4d 0 2024-03-04T16:04:35.040 INFO:tasks.workunit.client.1.smithi152.stdout:8/316: chown d2/d8/dd/f2f 1768729780 1 2024-03-04T16:04:35.041 INFO:tasks.workunit.client.1.smithi152.stdout:8/317: chown d2/d3/d15/d1c/f59 36201836 1 2024-03-04T16:04:35.048 INFO:tasks.workunit.client.1.smithi152.stdout:2/294: truncate da/f56 2962906 0 2024-03-04T16:04:35.050 INFO:tasks.workunit.client.1.smithi152.stdout:8/318: dwrite d2/d8/f45 [0,4194304] 0 2024-03-04T16:04:35.051 INFO:tasks.workunit.client.1.smithi152.stdout:8/319: dread - d2/d8/dd/d22/d26/f61 zero size 2024-03-04T16:04:35.055 INFO:tasks.workunit.client.1.smithi152.stdout:1/312: dread d19/d20/d21/d38/f3a [0,4194304] 0 2024-03-04T16:04:35.056 INFO:tasks.workunit.client.1.smithi152.stdout:2/295: rename da/d24/d57 to da/df/d2e/d61 0 2024-03-04T16:04:35.061 INFO:tasks.workunit.client.1.smithi152.stdout:2/296: truncate da/df/f37 924220 0 2024-03-04T16:04:35.062 INFO:tasks.workunit.client.1.smithi152.stdout:4/237: dread d8/d9/d24/d26/f2a [0,4194304] 0 2024-03-04T16:04:35.064 INFO:tasks.workunit.client.1.smithi152.stdout:4/238: dread - d8/d9/d24/f4d zero size 2024-03-04T16:04:35.069 INFO:tasks.workunit.client.1.smithi152.stdout:2/297: symlink da/d2c/l62 0 2024-03-04T16:04:35.082 INFO:tasks.workunit.client.1.smithi152.stdout:6/365: truncate d8/d23/d3a/f50 875155 0 2024-03-04T16:04:35.091 INFO:tasks.workunit.client.1.smithi152.stdout:6/366: dwrite d8/d52/f67 [0,4194304] 0 2024-03-04T16:04:35.092 INFO:tasks.workunit.client.1.smithi152.stdout:1/313: dread d19/d20/d21/d38/f3e [0,4194304] 0 2024-03-04T16:04:35.097 INFO:tasks.workunit.client.1.smithi152.stdout:8/320: dread d2/d3/d21/f5f [0,4194304] 0 2024-03-04T16:04:35.097 INFO:tasks.workunit.client.1.smithi152.stdout:8/321: chown d2/d8/c25 1985 1 2024-03-04T16:04:35.100 INFO:tasks.workunit.client.1.smithi152.stdout:2/298: truncate f6 3415122 0 2024-03-04T16:04:35.102 INFO:tasks.workunit.client.1.smithi152.stdout:1/314: rmdir d19/d20/d21 39 2024-03-04T16:04:35.107 INFO:tasks.workunit.client.1.smithi152.stdout:1/315: rmdir d19/d4b/d53 39 2024-03-04T16:04:35.109 INFO:tasks.workunit.client.1.smithi152.stdout:8/322: mknod d2/d3/c62 0 2024-03-04T16:04:35.110 INFO:tasks.workunit.client.1.smithi152.stdout:8/323: chown d2/d8/dd/d22/d49/f55 0 1 2024-03-04T16:04:35.111 INFO:tasks.workunit.client.1.smithi152.stdout:8/324: truncate d2/d3/f4b 104136 0 2024-03-04T16:04:35.112 INFO:tasks.workunit.client.1.smithi152.stdout:2/299: read da/df/f14 [2124258,52803] 0 2024-03-04T16:04:35.116 INFO:tasks.workunit.client.1.smithi152.stdout:1/316: truncate d19/d4b/d53/d57/f60 84289 0 2024-03-04T16:04:35.116 INFO:tasks.workunit.client.1.smithi152.stdout:1/317: chown d19/d20/d2f/f4d 145711414 1 2024-03-04T16:04:35.118 INFO:tasks.workunit.client.1.smithi152.stdout:6/367: getdents d8/d38/d59 0 2024-03-04T16:04:35.118 INFO:tasks.workunit.client.1.smithi152.stdout:6/368: fdatasync d8/d10/f16 0 2024-03-04T16:04:35.119 INFO:tasks.workunit.client.1.smithi152.stdout:6/369: stat d8/c20 0 2024-03-04T16:04:35.120 INFO:tasks.workunit.client.1.smithi152.stdout:6/370: truncate d8/d10/f16 811785 0 2024-03-04T16:04:35.122 INFO:tasks.workunit.client.1.smithi152.stdout:2/300: fdatasync da/df/f14 0 2024-03-04T16:04:35.124 INFO:tasks.workunit.client.1.smithi152.stdout:8/325: dwrite d2/d3/d21/f5f [0,4194304] 0 2024-03-04T16:04:35.124 INFO:tasks.workunit.client.1.smithi152.stdout:1/318: fsync f1 0 2024-03-04T16:04:35.126 INFO:tasks.workunit.client.1.smithi152.stdout:6/371: creat d8/d38/d59/d61/f78 x:0 0 0 2024-03-04T16:04:35.126 INFO:tasks.workunit.client.1.smithi152.stdout:6/372: fdatasync d8/f22 0 2024-03-04T16:04:35.132 INFO:tasks.workunit.client.1.smithi152.stdout:8/326: mknod d2/d3/d21/d2d/c63 0 2024-03-04T16:04:35.135 INFO:tasks.workunit.client.1.smithi152.stdout:6/373: creat d8/d2a/f79 x:0 0 0 2024-03-04T16:04:35.135 INFO:tasks.workunit.client.1.smithi152.stdout:1/319: dwrite d19/d20/d21/f37 [0,4194304] 0 2024-03-04T16:04:35.138 INFO:tasks.workunit.client.1.smithi152.stdout:8/327: symlink d2/d3/d15/l64 0 2024-03-04T16:04:35.139 INFO:tasks.workunit.client.1.smithi152.stdout:8/328: chown l0 16 1 2024-03-04T16:04:35.140 INFO:tasks.workunit.client.1.smithi152.stdout:8/329: read d2/f53 [2750010,71009] 0 2024-03-04T16:04:35.141 INFO:tasks.workunit.client.1.smithi152.stdout:8/330: stat d2/d3/d15 0 2024-03-04T16:04:35.141 INFO:tasks.workunit.client.1.smithi152.stdout:1/320: mkdir d19/d20/d21/d38/d68 0 2024-03-04T16:04:35.142 INFO:tasks.workunit.client.1.smithi152.stdout:1/321: fsync f16 0 2024-03-04T16:04:35.142 INFO:tasks.workunit.client.1.smithi152.stdout:8/331: chown d2/d8/dd/d22/d49/d5d 14 1 2024-03-04T16:04:35.144 INFO:tasks.workunit.client.1.smithi152.stdout:6/374: creat d8/d23/d3a/d6d/f7a x:0 0 0 2024-03-04T16:04:35.144 INFO:tasks.workunit.client.1.smithi152.stdout:6/375: read - d8/d42/f4a zero size 2024-03-04T16:04:35.145 INFO:tasks.workunit.client.1.smithi152.stdout:1/322: read d19/d20/d21/f42 [446697,94389] 0 2024-03-04T16:04:35.145 INFO:tasks.workunit.client.1.smithi152.stdout:6/376: readlink d8/d2a/l64 0 2024-03-04T16:04:35.149 INFO:tasks.workunit.client.1.smithi152.stdout:2/301: getdents da/d24/d47/d4f 0 2024-03-04T16:04:35.151 INFO:tasks.workunit.client.1.smithi152.stdout:1/323: dwrite d19/d20/d21/f4a [0,4194304] 0 2024-03-04T16:04:35.153 INFO:tasks.workunit.client.1.smithi152.stdout:6/377: symlink d8/d23/l7b 0 2024-03-04T16:04:35.161 INFO:tasks.workunit.client.1.smithi152.stdout:6/378: dwrite d8/d4b/f54 [0,4194304] 0 2024-03-04T16:04:35.166 INFO:tasks.workunit.client.1.smithi152.stdout:8/332: rmdir d2/d8/dd/d22/d60 0 2024-03-04T16:04:35.171 INFO:tasks.workunit.client.1.smithi152.stdout:2/302: truncate f6 1347363 0 2024-03-04T16:04:35.179 INFO:tasks.workunit.client.1.smithi152.stdout:2/303: mknod da/df/d58/c63 0 2024-03-04T16:04:35.180 INFO:tasks.workunit.client.1.smithi152.stdout:8/333: dread d2/f42 [0,4194304] 0 2024-03-04T16:04:35.181 INFO:tasks.workunit.client.1.smithi152.stdout:1/324: rmdir d19/d20/d21/d38/d68 0 2024-03-04T16:04:35.187 INFO:tasks.workunit.client.1.smithi152.stdout:1/325: mknod d19/d3b/c69 0 2024-03-04T16:04:35.188 INFO:tasks.workunit.client.1.smithi152.stdout:6/379: truncate d8/f12 1278323 0 2024-03-04T16:04:35.189 INFO:tasks.workunit.client.1.smithi152.stdout:8/334: truncate d2/d8/dd/d22/d26/f38 761309 0 2024-03-04T16:04:35.190 INFO:tasks.workunit.client.1.smithi152.stdout:8/335: dread - d2/d3/d15/d1c/f59 zero size 2024-03-04T16:04:35.191 INFO:tasks.workunit.client.1.smithi152.stdout:8/336: stat d2/d3/d21/d2d/d52 0 2024-03-04T16:04:35.192 INFO:tasks.workunit.client.1.smithi152.stdout:7/309: sync 2024-03-04T16:04:35.192 INFO:tasks.workunit.client.1.smithi152.stdout:0/292: sync 2024-03-04T16:04:35.192 INFO:tasks.workunit.client.1.smithi152.stdout:3/340: sync 2024-03-04T16:04:35.193 INFO:tasks.workunit.client.1.smithi152.stdout:3/341: stat d6/f3f 0 2024-03-04T16:04:35.194 INFO:tasks.workunit.client.1.smithi152.stdout:6/380: creat d8/d38/d59/f7c x:0 0 0 2024-03-04T16:04:35.195 INFO:tasks.workunit.client.1.smithi152.stdout:2/304: dread da/df/f14 [0,4194304] 0 2024-03-04T16:04:35.195 INFO:tasks.workunit.client.1.smithi152.stdout:1/326: symlink d19/d4b/d53/d57/d62/l6a 0 2024-03-04T16:04:35.197 INFO:tasks.workunit.client.1.smithi152.stdout:8/337: symlink d2/d3/d21/d2d/d48/l65 0 2024-03-04T16:04:35.201 INFO:tasks.workunit.client.1.smithi152.stdout:0/293: dwrite d9/f45 [0,4194304] 0 2024-03-04T16:04:35.203 INFO:tasks.workunit.client.1.smithi152.stdout:7/310: dread d14/d2b/f3b [4194304,4194304] 0 2024-03-04T16:04:35.206 INFO:tasks.workunit.client.1.smithi152.stdout:6/381: mkdir d8/d38/d59/d7d 0 2024-03-04T16:04:35.206 INFO:tasks.workunit.client.1.smithi152.stdout:6/382: chown d8/d42/f4a 314698 1 2024-03-04T16:04:35.207 INFO:tasks.workunit.client.1.smithi152.stdout:1/327: dread f1 [0,4194304] 0 2024-03-04T16:04:35.208 INFO:tasks.workunit.client.1.smithi152.stdout:2/305: mkdir da/d24/d64 0 2024-03-04T16:04:35.211 INFO:tasks.workunit.client.1.smithi152.stdout:0/294: symlink d9/d2c/d2e/d38/l5c 0 2024-03-04T16:04:35.212 INFO:tasks.workunit.client.1.smithi152.stdout:6/383: symlink d8/d38/d59/d61/l7e 0 2024-03-04T16:04:35.213 INFO:tasks.workunit.client.1.smithi152.stdout:6/384: readlink d8/lf 0 2024-03-04T16:04:35.213 INFO:tasks.workunit.client.1.smithi152.stdout:3/342: link d6/c45 d6/d10/d2d/d4d/c74 0 2024-03-04T16:04:35.215 INFO:tasks.workunit.client.1.smithi152.stdout:1/328: rename d19/d20/d21 to d19/d20/d2f/d6b 0 2024-03-04T16:04:35.217 INFO:tasks.workunit.client.1.smithi152.stdout:7/311: read d14/d21/f55 [56554,113243] 0 2024-03-04T16:04:35.220 INFO:tasks.workunit.client.1.smithi152.stdout:3/343: dwrite d6/f1a [0,4194304] 0 2024-03-04T16:04:35.230 INFO:tasks.workunit.client.1.smithi152.stdout:1/329: dwrite d19/d20/d2f/d6b/d38/f3e [0,4194304] 0 2024-03-04T16:04:35.234 INFO:tasks.workunit.client.1.smithi152.stdout:2/306: creat da/d24/f65 x:0 0 0 2024-03-04T16:04:35.235 INFO:tasks.workunit.client.1.smithi152.stdout:3/344: symlink d6/d14/d41/l75 0 2024-03-04T16:04:35.236 INFO:tasks.workunit.client.1.smithi152.stdout:3/345: readlink d6/d10/d2d/l3e 0 2024-03-04T16:04:35.237 INFO:tasks.workunit.client.1.smithi152.stdout:3/346: write d6/f3a [325189,3036] 0 2024-03-04T16:04:35.238 INFO:tasks.workunit.client.1.smithi152.stdout:3/347: chown d6/d10/f39 14 1 2024-03-04T16:04:35.238 INFO:tasks.workunit.client.1.smithi152.stdout:2/307: dread da/fb [0,4194304] 0 2024-03-04T16:04:35.238 INFO:tasks.workunit.client.1.smithi152.stdout:7/312: mknod d14/c5d 0 2024-03-04T16:04:35.240 INFO:tasks.workunit.client.1.smithi152.stdout:2/308: write da/d24/f32 [2058587,31195] 0 2024-03-04T16:04:35.241 INFO:tasks.workunit.client.1.smithi152.stdout:0/295: dread d9/fa [0,4194304] 0 2024-03-04T16:04:35.242 INFO:tasks.workunit.client.1.smithi152.stdout:2/309: write da/f23 [4419289,9719] 0 2024-03-04T16:04:35.242 INFO:tasks.workunit.client.1.smithi152.stdout:2/310: chown da/df/f1f 1676 1 2024-03-04T16:04:35.243 INFO:tasks.workunit.client.1.smithi152.stdout:1/330: rename cf to d19/d20/d2f/d5d/c6c 0 2024-03-04T16:04:35.250 INFO:tasks.workunit.client.1.smithi152.stdout:1/331: dwrite d19/d20/d2f/d6b/f46 [0,4194304] 0 2024-03-04T16:04:35.252 INFO:tasks.workunit.client.1.smithi152.stdout:7/313: creat d14/d21/d47/f5e x:0 0 0 2024-03-04T16:04:35.255 INFO:tasks.workunit.client.1.smithi152.stdout:0/296: write d9/fa [1582471,34531] 0 2024-03-04T16:04:35.257 INFO:tasks.workunit.client.1.smithi152.stdout:1/332: dwrite f2 [0,4194304] 0 2024-03-04T16:04:35.259 INFO:tasks.workunit.client.1.smithi152.stdout:6/385: truncate d8/f1f 517439 0 2024-03-04T16:04:35.262 INFO:tasks.workunit.client.1.smithi152.stdout:7/314: write d14/d2b/f3b [4786596,5954] 0 2024-03-04T16:04:35.267 INFO:tasks.workunit.client.1.smithi152.stdout:2/311: rename da/fe to da/df/d35/d50/f66 0 2024-03-04T16:04:35.267 INFO:tasks.workunit.client.1.smithi152.stdout:3/348: write f5 [323024,11205] 0 2024-03-04T16:04:35.268 INFO:tasks.workunit.client.1.smithi152.stdout:2/312: write f4 [1466889,4532] 0 2024-03-04T16:04:35.271 INFO:tasks.workunit.client.1.smithi152.stdout:0/297: mknod d9/d1b/d3c/c5d 0 2024-03-04T16:04:35.271 INFO:tasks.workunit.client.1.smithi152.stdout:2/313: dread da/fc [0,4194304] 0 2024-03-04T16:04:35.273 INFO:tasks.workunit.client.1.smithi152.stdout:2/314: chown da/df/d2e/d51/f5b 59 1 2024-03-04T16:04:35.275 INFO:tasks.workunit.client.1.smithi152.stdout:6/386: creat d8/d38/d59/f7f x:0 0 0 2024-03-04T16:04:35.280 INFO:tasks.workunit.client.1.smithi152.stdout:0/298: symlink d9/d2c/d2e/d38/l5e 0 2024-03-04T16:04:35.282 INFO:tasks.workunit.client.1.smithi152.stdout:1/333: creat d19/d4b/d53/d5c/f6d x:0 0 0 2024-03-04T16:04:35.285 INFO:tasks.workunit.client.1.smithi152.stdout:3/349: mknod d6/d10/d2d/c76 0 2024-03-04T16:04:35.287 INFO:tasks.workunit.client.1.smithi152.stdout:0/299: fsync f4 0 2024-03-04T16:04:35.288 INFO:tasks.workunit.client.1.smithi152.stdout:2/315: fdatasync da/f11 0 2024-03-04T16:04:35.289 INFO:tasks.workunit.client.1.smithi152.stdout:3/350: dread d6/f2c [0,4194304] 0 2024-03-04T16:04:35.292 INFO:tasks.workunit.client.1.smithi152.stdout:7/315: dread d14/d21/d47/f34 [0,4194304] 0 2024-03-04T16:04:35.292 INFO:tasks.workunit.client.1.smithi152.stdout:1/334: mkdir d19/d3b/d6e 0 2024-03-04T16:04:35.293 INFO:tasks.workunit.client.1.smithi152.stdout:6/387: mknod d8/c80 0 2024-03-04T16:04:35.297 INFO:tasks.workunit.client.1.smithi152.stdout:0/300: symlink d9/d2c/d2e/d38/l5f 0 2024-03-04T16:04:35.297 INFO:tasks.workunit.client.1.smithi152.stdout:7/316: dread d14/f42 [0,4194304] 0 2024-03-04T16:04:35.299 INFO:tasks.workunit.client.1.smithi152.stdout:3/351: dwrite d6/d14/d41/d49/f62 [0,4194304] 0 2024-03-04T16:04:35.300 INFO:tasks.workunit.client.1.smithi152.stdout:1/335: dread fa [0,4194304] 0 2024-03-04T16:04:35.306 INFO:tasks.workunit.client.1.smithi152.stdout:2/316: write da/d2c/f41 [764945,114661] 0 2024-03-04T16:04:35.307 INFO:tasks.workunit.client.1.smithi152.stdout:6/388: dread d8/d23/d3a/f50 [0,4194304] 0 2024-03-04T16:04:35.308 INFO:tasks.workunit.client.1.smithi152.stdout:2/317: write da/df/d35/f46 [747238,14229] 0 2024-03-04T16:04:35.311 INFO:tasks.workunit.client.1.smithi152.stdout:3/352: rename d6/d10/d2d/d59/f6c to d6/d10/d2d/d59/f77 0 2024-03-04T16:04:35.314 INFO:tasks.workunit.client.1.smithi152.stdout:6/389: mknod d8/d10/d53/c81 0 2024-03-04T16:04:35.315 INFO:tasks.workunit.client.1.smithi152.stdout:7/317: mknod d14/c5f 0 2024-03-04T16:04:35.317 INFO:tasks.workunit.client.1.smithi152.stdout:1/336: rename l0 to d19/d20/d2f/d6b/d38/d63/d64/l6f 0 2024-03-04T16:04:35.318 INFO:tasks.workunit.client.1.smithi152.stdout:2/318: symlink da/d24/l67 0 2024-03-04T16:04:35.319 INFO:tasks.workunit.client.1.smithi152.stdout:1/337: stat d19/d3b/c69 0 2024-03-04T16:04:35.319 INFO:tasks.workunit.client.1.smithi152.stdout:3/353: dwrite d6/d14/d58/f71 [0,4194304] 0 2024-03-04T16:04:35.322 INFO:tasks.workunit.client.1.smithi152.stdout:6/390: creat d8/d23/d3a/d6d/f82 x:0 0 0 2024-03-04T16:04:35.323 INFO:tasks.workunit.client.1.smithi152.stdout:0/301: dread f7 [4194304,4194304] 0 2024-03-04T16:04:35.323 INFO:tasks.workunit.client.1.smithi152.stdout:1/338: rename d19/d20/d2f/d6b/d38/d63/d64 to d19/d20/d2f/d6b/d38/d63/d64/d70 22 2024-03-04T16:04:35.326 INFO:tasks.workunit.client.1.smithi152.stdout:7/318: creat d14/d2b/f60 x:0 0 0 2024-03-04T16:04:35.326 INFO:tasks.workunit.client.1.smithi152.stdout:0/302: write d9/f16 [5110701,33347] 0 2024-03-04T16:04:35.330 INFO:tasks.workunit.client.1.smithi152.stdout:6/391: symlink d8/d52/l83 0 2024-03-04T16:04:35.331 INFO:tasks.workunit.client.1.smithi152.stdout:6/392: dread - d8/d38/d59/f7c zero size 2024-03-04T16:04:35.332 INFO:tasks.workunit.client.1.smithi152.stdout:1/339: creat d19/d4b/d53/d5c/f71 x:0 0 0 2024-03-04T16:04:35.332 INFO:tasks.workunit.client.1.smithi152.stdout:1/340: readlink l11 0 2024-03-04T16:04:35.333 INFO:tasks.workunit.client.1.smithi152.stdout:0/303: dwrite f7 [8388608,4194304] 0 2024-03-04T16:04:35.333 INFO:tasks.workunit.client.1.smithi152.stdout:0/304: chown d9/f16 45379 1 2024-03-04T16:04:35.334 INFO:tasks.workunit.client.1.smithi152.stdout:6/393: truncate d8/d23/d3a/f6b 798257 0 2024-03-04T16:04:35.334 INFO:tasks.workunit.client.1.smithi152.stdout:1/341: stat d19/d20/d2f/d6b/l56 0 2024-03-04T16:04:35.335 INFO:tasks.workunit.client.1.smithi152.stdout:1/342: write fe [6303181,47794] 0 2024-03-04T16:04:35.335 INFO:tasks.workunit.client.1.smithi152.stdout:6/394: dread - d8/d42/f4a zero size 2024-03-04T16:04:35.337 INFO:tasks.workunit.client.1.smithi152.stdout:2/319: creat da/d5f/f68 x:0 0 0 2024-03-04T16:04:35.343 INFO:tasks.workunit.client.1.smithi152.stdout:1/343: dread d19/d20/d2f/d6b/f46 [0,4194304] 0 2024-03-04T16:04:35.346 INFO:tasks.workunit.client.1.smithi152.stdout:6/395: mkdir d8/d52/d84 0 2024-03-04T16:04:35.346 INFO:tasks.workunit.client.1.smithi152.stdout:4/239: sync 2024-03-04T16:04:35.346 INFO:tasks.workunit.client.1.smithi152.stdout:5/369: sync 2024-03-04T16:04:35.346 INFO:tasks.workunit.client.1.smithi152.stdout:9/248: sync 2024-03-04T16:04:35.347 INFO:tasks.workunit.client.1.smithi152.stdout:5/370: readlink d6/d50/l60 0 2024-03-04T16:04:35.350 INFO:tasks.workunit.client.1.smithi152.stdout:0/305: creat d9/d2c/d2e/d38/d56/d5b/f60 x:0 0 0 2024-03-04T16:04:35.351 INFO:tasks.workunit.client.1.smithi152.stdout:4/240: mkdir d8/d4e 0 2024-03-04T16:04:35.353 INFO:tasks.workunit.client.1.smithi152.stdout:6/396: mknod d8/d52/c85 0 2024-03-04T16:04:35.356 INFO:tasks.workunit.client.1.smithi152.stdout:9/249: rename d2/d13/d23/d42/f4b to d2/d13/d23/f4e 0 2024-03-04T16:04:35.357 INFO:tasks.workunit.client.1.smithi152.stdout:5/371: mknod d6/d8/d1f/d3d/c89 0 2024-03-04T16:04:35.358 INFO:tasks.workunit.client.1.smithi152.stdout:2/320: creat da/d2c/d5e/f69 x:0 0 0 2024-03-04T16:04:35.359 INFO:tasks.workunit.client.1.smithi152.stdout:0/306: dread d9/d1b/f35 [0,4194304] 0 2024-03-04T16:04:35.362 INFO:tasks.workunit.client.1.smithi152.stdout:7/319: dread d14/f16 [0,4194304] 0 2024-03-04T16:04:35.364 INFO:tasks.workunit.client.1.smithi152.stdout:1/344: link cd d19/d20/d2f/d5d/c72 0 2024-03-04T16:04:35.364 INFO:tasks.workunit.client.1.smithi152.stdout:5/372: dwrite d6/f7f [0,4194304] 0 2024-03-04T16:04:35.365 INFO:tasks.workunit.client.1.smithi152.stdout:2/321: write da/f16 [4732030,3300] 0 2024-03-04T16:04:35.367 INFO:tasks.workunit.client.1.smithi152.stdout:6/397: mknod d8/d23/d3a/d71/c86 0 2024-03-04T16:04:35.377 INFO:tasks.workunit.client.1.smithi152.stdout:5/373: truncate d6/d8/f32 433475 0 2024-03-04T16:04:35.381 INFO:tasks.workunit.client.1.smithi152.stdout:6/398: truncate d8/d2a/f45 1446049 0 2024-03-04T16:04:35.383 INFO:tasks.workunit.client.1.smithi152.stdout:5/374: dwrite d6/d8/f4b [0,4194304] 0 2024-03-04T16:04:35.391 INFO:tasks.workunit.client.1.smithi152.stdout:9/250: write d2/d13/f1a [153069,6846] 0 2024-03-04T16:04:35.395 INFO:tasks.workunit.client.1.smithi152.stdout:0/307: truncate d9/fa 1018566 0 2024-03-04T16:04:35.396 INFO:tasks.workunit.client.1.smithi152.stdout:0/308: read - d9/d2c/d2e/f4a zero size 2024-03-04T16:04:35.397 INFO:tasks.workunit.client.1.smithi152.stdout:2/322: rmdir da/d24/d64 0 2024-03-04T16:04:35.399 INFO:tasks.workunit.client.1.smithi152.stdout:6/399: mkdir d8/d52/d84/d87 0 2024-03-04T16:04:35.399 INFO:tasks.workunit.client.1.smithi152.stdout:1/345: write d19/d20/f45 [2169376,122039] 0 2024-03-04T16:04:35.400 INFO:tasks.workunit.client.1.smithi152.stdout:9/251: creat d2/d13/d2e/f4f x:0 0 0 2024-03-04T16:04:35.402 INFO:tasks.workunit.client.1.smithi152.stdout:5/375: write d6/d8/f41 [570979,102591] 0 2024-03-04T16:04:35.402 INFO:tasks.workunit.client.1.smithi152.stdout:1/346: dread - d19/d3b/f52 zero size 2024-03-04T16:04:35.403 INFO:tasks.workunit.client.1.smithi152.stdout:9/252: dread - d2/d13/d2e/f4f zero size 2024-03-04T16:04:35.405 INFO:tasks.workunit.client.1.smithi152.stdout:7/320: dwrite d14/d21/d47/f35 [0,4194304] 0 2024-03-04T16:04:35.411 INFO:tasks.workunit.client.1.smithi152.stdout:1/347: dwrite d19/f1d [0,4194304] 0 2024-03-04T16:04:35.411 INFO:tasks.workunit.client.1.smithi152.stdout:1/348: chown d19/d20/d2f/d6b/l3f 953721872 1 2024-03-04T16:04:35.413 INFO:tasks.workunit.client.1.smithi152.stdout:9/253: symlink d2/d13/d38/l50 0 2024-03-04T16:04:35.416 INFO:tasks.workunit.client.1.smithi152.stdout:0/309: chown d9/fa 135576832 1 2024-03-04T16:04:35.421 INFO:tasks.workunit.client.1.smithi152.stdout:9/254: rmdir d2/d13/d35 39 2024-03-04T16:04:35.423 INFO:tasks.workunit.client.1.smithi152.stdout:0/310: dread d9/f45 [0,4194304] 0 2024-03-04T16:04:35.434 INFO:tasks.workunit.client.1.smithi152.stdout:7/321: creat d14/f61 x:0 0 0 2024-03-04T16:04:35.435 INFO:tasks.workunit.client.1.smithi152.stdout:6/400: write d8/f1f [752865,75182] 0 2024-03-04T16:04:35.439 INFO:tasks.workunit.client.1.smithi152.stdout:0/311: getdents d9/d1b/d54 0 2024-03-04T16:04:35.439 INFO:tasks.workunit.client.1.smithi152.stdout:5/376: dread d6/d8/f30 [0,4194304] 0 2024-03-04T16:04:35.440 INFO:tasks.workunit.client.1.smithi152.stdout:9/255: dread - d2/d13/d23/d42/f46 zero size 2024-03-04T16:04:35.440 INFO:tasks.workunit.client.1.smithi152.stdout:7/322: symlink d14/d21/d31/l62 0 2024-03-04T16:04:35.442 INFO:tasks.workunit.client.1.smithi152.stdout:2/323: dwrite da/df/d35/f4a [0,4194304] 0 2024-03-04T16:04:35.443 INFO:tasks.workunit.client.1.smithi152.stdout:5/377: readlink d6/d8/d1f/l79 0 2024-03-04T16:04:35.445 INFO:tasks.workunit.client.1.smithi152.stdout:0/312: symlink d9/d1b/d3c/l61 0 2024-03-04T16:04:35.447 INFO:tasks.workunit.client.1.smithi152.stdout:1/349: dwrite d19/d20/d2f/d6b/f42 [0,4194304] 0 2024-03-04T16:04:35.450 INFO:tasks.workunit.client.1.smithi152.stdout:9/256: dwrite d2/d13/d23/f33 [4194304,4194304] 0 2024-03-04T16:04:35.451 INFO:tasks.workunit.client.1.smithi152.stdout:5/378: mkdir d6/d8/d5a/d64/d8a 0 2024-03-04T16:04:35.452 INFO:tasks.workunit.client.1.smithi152.stdout:0/313: mkdir d9/d1b/d62 0 2024-03-04T16:04:35.455 INFO:tasks.workunit.client.1.smithi152.stdout:2/324: mknod da/c6a 0 2024-03-04T16:04:35.457 INFO:tasks.workunit.client.1.smithi152.stdout:6/401: link d8/d23/f25 d8/d38/d75/f88 0 2024-03-04T16:04:35.457 INFO:tasks.workunit.client.1.smithi152.stdout:1/350: write d19/d4b/f51 [285509,1772] 0 2024-03-04T16:04:35.458 INFO:tasks.workunit.client.1.smithi152.stdout:6/402: write d8/f39 [996493,120185] 0 2024-03-04T16:04:35.461 INFO:tasks.workunit.client.1.smithi152.stdout:0/314: read d9/f45 [323233,86982] 0 2024-03-04T16:04:35.463 INFO:tasks.workunit.client.1.smithi152.stdout:2/325: dread da/d24/f2a [0,4194304] 0 2024-03-04T16:04:35.464 INFO:tasks.workunit.client.1.smithi152.stdout:0/315: truncate d9/d2c/d2e/f41 1460821 0 2024-03-04T16:04:35.466 INFO:tasks.workunit.client.1.smithi152.stdout:2/326: truncate da/d24/d47/f5a 437702 0 2024-03-04T16:04:35.472 INFO:tasks.workunit.client.1.smithi152.stdout:9/257: rmdir d2/d13/d23/d42/d4d 0 2024-03-04T16:04:35.472 INFO:tasks.workunit.client.1.smithi152.stdout:2/327: dwrite f4 [0,4194304] 0 2024-03-04T16:04:35.476 INFO:tasks.workunit.client.1.smithi152.stdout:5/379: getdents d6/d8/d1f/d3d/d61 0 2024-03-04T16:04:35.479 INFO:tasks.workunit.client.1.smithi152.stdout:2/328: creat da/d24/d47/f6b x:0 0 0 2024-03-04T16:04:35.482 INFO:tasks.workunit.client.1.smithi152.stdout:5/380: getdents d6/d8 0 2024-03-04T16:04:35.483 INFO:tasks.workunit.client.1.smithi152.stdout:2/329: symlink da/d24/d47/d4f/l6c 0 2024-03-04T16:04:35.486 INFO:tasks.workunit.client.1.smithi152.stdout:7/323: dread d14/f2a [0,4194304] 0 2024-03-04T16:04:35.491 INFO:tasks.workunit.client.1.smithi152.stdout:5/381: dwrite d6/d50/f5c [0,4194304] 0 2024-03-04T16:04:35.492 INFO:tasks.workunit.client.1.smithi152.stdout:5/382: stat d6/fa 0 2024-03-04T16:04:35.494 INFO:tasks.workunit.client.1.smithi152.stdout:5/383: read - d6/d8/d5a/d64/d76/f81 zero size 2024-03-04T16:04:35.496 INFO:tasks.workunit.client.1.smithi152.stdout:7/324: dwrite d14/d21/d47/d40/f43 [0,4194304] 0 2024-03-04T16:04:35.496 INFO:tasks.workunit.client.1.smithi152.stdout:5/384: stat d6/d8/d1f/f44 0 2024-03-04T16:04:35.499 INFO:tasks.workunit.client.1.smithi152.stdout:5/385: truncate d6/d8/d1f/d3d/f78 355802 0 2024-03-04T16:04:35.502 INFO:tasks.workunit.client.1.smithi152.stdout:5/386: write d6/d50/f7e [643714,125457] 0 2024-03-04T16:04:35.504 INFO:tasks.workunit.client.1.smithi152.stdout:6/403: dread d8/f28 [0,4194304] 0 2024-03-04T16:04:35.505 INFO:tasks.workunit.client.1.smithi152.stdout:1/351: write d19/d20/d2f/d6b/d38/f4c [2990379,107055] 0 2024-03-04T16:04:35.506 INFO:tasks.workunit.client.1.smithi152.stdout:5/387: mkdir d6/d8/d5a/d8b 0 2024-03-04T16:04:35.507 INFO:tasks.workunit.client.1.smithi152.stdout:1/352: truncate d19/d4b/d53/d57/f5e 4871178 0 2024-03-04T16:04:35.508 INFO:tasks.workunit.client.1.smithi152.stdout:6/404: read d8/d23/d3a/f3f [700451,8335] 0 2024-03-04T16:04:35.514 INFO:tasks.workunit.client.1.smithi152.stdout:6/405: dread d8/d38/f48 [0,4194304] 0 2024-03-04T16:04:35.516 INFO:tasks.workunit.client.1.smithi152.stdout:7/325: fdatasync d14/d21/d47/d40/f43 0 2024-03-04T16:04:35.516 INFO:tasks.workunit.client.1.smithi152.stdout:9/258: dwrite d2/d13/f15 [0,4194304] 0 2024-03-04T16:04:35.517 INFO:tasks.workunit.client.1.smithi152.stdout:6/406: write d8/d23/f5c [384986,68163] 0 2024-03-04T16:04:35.523 INFO:tasks.workunit.client.1.smithi152.stdout:6/407: read d8/d10/f24 [3600437,61564] 0 2024-03-04T16:04:35.523 INFO:tasks.workunit.client.1.smithi152.stdout:5/388: write d6/d8/fe [6096329,118838] 0 2024-03-04T16:04:35.524 INFO:tasks.workunit.client.1.smithi152.stdout:2/330: dwrite f6 [0,4194304] 0 2024-03-04T16:04:35.525 INFO:tasks.workunit.client.1.smithi152.stdout:7/326: creat d14/d2b/f63 x:0 0 0 2024-03-04T16:04:35.526 INFO:tasks.workunit.client.1.smithi152.stdout:2/331: chown da/d24/d47 1294 1 2024-03-04T16:04:35.531 INFO:tasks.workunit.client.1.smithi152.stdout:1/353: getdents d19/d4b/d53/d57/d62 0 2024-03-04T16:04:35.534 INFO:tasks.workunit.client.1.smithi152.stdout:7/327: dread d14/d21/d31/f33 [0,4194304] 0 2024-03-04T16:04:35.540 INFO:tasks.workunit.client.1.smithi152.stdout:9/259: rename d2/d13/c22 to d2/d13/d38/c51 0 2024-03-04T16:04:35.541 INFO:tasks.workunit.client.1.smithi152.stdout:7/328: dwrite d14/d21/d31/f44 [0,4194304] 0 2024-03-04T16:04:35.541 INFO:tasks.workunit.client.1.smithi152.stdout:1/354: unlink d19/d4b/d53/c54 0 2024-03-04T16:04:35.542 INFO:tasks.workunit.client.1.smithi152.stdout:7/329: read - d14/d2b/f63 zero size 2024-03-04T16:04:35.543 INFO:tasks.workunit.client.1.smithi152.stdout:1/355: write d19/d20/d2f/f59 [36454,10496] 0 2024-03-04T16:04:35.544 INFO:tasks.workunit.client.1.smithi152.stdout:7/330: truncate d14/f61 784326 0 2024-03-04T16:04:35.544 INFO:tasks.workunit.client.1.smithi152.stdout:1/356: chown d19/d20/d2f 59709 1 2024-03-04T16:04:35.545 INFO:tasks.workunit.client.1.smithi152.stdout:6/408: rename d8/d23/d3a/f6b to d8/d2a/f89 0 2024-03-04T16:04:35.546 INFO:tasks.workunit.client.1.smithi152.stdout:9/260: fdatasync d2/d13/d23/f4e 0 2024-03-04T16:04:35.547 INFO:tasks.workunit.client.1.smithi152.stdout:5/389: link d6/d8/d1f/d3d/d6f/d70/c77 d6/c8c 0 2024-03-04T16:04:35.548 INFO:tasks.workunit.client.1.smithi152.stdout:5/390: write d6/d8/d10/f43 [911158,20302] 0 2024-03-04T16:04:35.552 INFO:tasks.workunit.client.1.smithi152.stdout:1/357: creat d19/d4b/d53/d5c/f73 x:0 0 0 2024-03-04T16:04:35.553 INFO:tasks.workunit.client.1.smithi152.stdout:8/338: sync 2024-03-04T16:04:35.554 INFO:tasks.workunit.client.1.smithi152.stdout:8/339: read - d2/d8/dd/f54 zero size 2024-03-04T16:04:35.557 INFO:tasks.workunit.client.1.smithi152.stdout:6/409: chown d8/d10/c65 355466477 1 2024-03-04T16:04:35.557 INFO:tasks.workunit.client.1.smithi152.stdout:9/261: write d2/d13/f1a [2110612,9507] 0 2024-03-04T16:04:35.559 INFO:tasks.workunit.client.1.smithi152.stdout:6/410: read d8/d10/f1e [11325,98453] 0 2024-03-04T16:04:35.560 INFO:tasks.workunit.client.1.smithi152.stdout:5/391: fsync d6/d8/f30 0 2024-03-04T16:04:35.562 INFO:tasks.workunit.client.1.smithi152.stdout:8/340: mkdir d2/d3/d21/d2d/d31/d66 0 2024-03-04T16:04:35.564 INFO:tasks.workunit.client.1.smithi152.stdout:8/341: write d2/d3/d21/d2d/f2c [5565347,123528] 0 2024-03-04T16:04:35.564 INFO:tasks.workunit.client.1.smithi152.stdout:8/342: chown d2/d3/d21/d2d/l4d 1253 1 2024-03-04T16:04:35.571 INFO:tasks.workunit.client.1.smithi152.stdout:9/262: write d2/f19 [4808842,62122] 0 2024-03-04T16:04:35.573 INFO:tasks.workunit.client.1.smithi152.stdout:5/392: write d6/d8/d10/f2d [1093050,123823] 0 2024-03-04T16:04:35.574 INFO:tasks.workunit.client.1.smithi152.stdout:1/358: truncate d19/d4b/d53/d57/f5e 592742 0 2024-03-04T16:04:35.578 INFO:tasks.workunit.client.1.smithi152.stdout:1/359: symlink d19/d4b/d53/l74 0 2024-03-04T16:04:35.579 INFO:tasks.workunit.client.1.smithi152.stdout:9/263: dwrite d2/d13/d23/f4e [0,4194304] 0 2024-03-04T16:04:35.580 INFO:tasks.workunit.client.1.smithi152.stdout:5/393: symlink d6/d8/d5a/d64/d8a/l8d 0 2024-03-04T16:04:35.580 INFO:tasks.workunit.client.1.smithi152.stdout:1/360: fdatasync d19/d20/d2f/d6b/d38/f4c 0 2024-03-04T16:04:35.582 INFO:tasks.workunit.client.1.smithi152.stdout:1/361: write d19/f1c [862118,125336] 0 2024-03-04T16:04:35.588 INFO:tasks.workunit.client.1.smithi152.stdout:5/394: read d6/d50/f7e [506835,3910] 0 2024-03-04T16:04:35.588 INFO:tasks.workunit.client.1.smithi152.stdout:9/264: dwrite d2/fd [4194304,4194304] 0 2024-03-04T16:04:35.591 INFO:tasks.workunit.client.1.smithi152.stdout:1/362: mkdir d19/d75 0 2024-03-04T16:04:35.597 INFO:tasks.workunit.client.1.smithi152.stdout:5/395: symlink d6/d8/d1f/d3d/l8e 0 2024-03-04T16:04:35.600 INFO:tasks.workunit.client.1.smithi152.stdout:9/265: rmdir d2/d13/d35 39 2024-03-04T16:04:35.600 INFO:tasks.workunit.client.1.smithi152.stdout:1/363: creat d19/d20/d2f/f76 x:0 0 0 2024-03-04T16:04:35.601 INFO:tasks.workunit.client.1.smithi152.stdout:9/266: chown d2/d13/d23/l39 1862889956 1 2024-03-04T16:04:35.604 INFO:tasks.workunit.client.1.smithi152.stdout:5/396: rename d6/d8/d10/f5b to d6/d8/d5a/d8b/f8f 0 2024-03-04T16:04:35.608 INFO:tasks.workunit.client.1.smithi152.stdout:9/267: creat d2/d13/f52 x:0 0 0 2024-03-04T16:04:35.612 INFO:tasks.workunit.client.1.smithi152.stdout:9/268: getdents d2/d13/d38 0 2024-03-04T16:04:35.616 INFO:tasks.workunit.client.1.smithi152.stdout:9/269: symlink d2/d13/d35/l53 0 2024-03-04T16:04:35.618 INFO:tasks.workunit.client.1.smithi152.stdout:9/270: symlink d2/d13/d38/l54 0 2024-03-04T16:04:35.620 INFO:tasks.workunit.client.1.smithi152.stdout:9/271: rename d2/d13/d38/l54 to d2/d13/d35/l55 0 2024-03-04T16:04:35.621 INFO:tasks.workunit.client.1.smithi152.stdout:9/272: chown d2/d13/d35/l53 4829 1 2024-03-04T16:04:35.738 INFO:tasks.workunit.client.1.smithi152.stdout:9/273: creat d2/d13/d23/f56 x:0 0 0 2024-03-04T16:04:35.739 INFO:tasks.workunit.client.1.smithi152.stdout:9/274: readlink d2/d13/d23/l2a 0 2024-03-04T16:04:35.739 INFO:tasks.workunit.client.1.smithi152.stdout:5/397: truncate d6/d50/f5c 457459 0 2024-03-04T16:04:35.740 INFO:tasks.workunit.client.1.smithi152.stdout:5/398: fdatasync d6/d8/d1f/f44 0 2024-03-04T16:04:35.743 INFO:tasks.workunit.client.1.smithi152.stdout:5/399: unlink d6/d8/d10/f46 0 2024-03-04T16:04:35.745 INFO:tasks.workunit.client.1.smithi152.stdout:5/400: mkdir d6/d8/d5a/d90 0 2024-03-04T16:04:35.748 INFO:tasks.workunit.client.1.smithi152.stdout:5/401: getdents d6/d8/d1f/d88 0 2024-03-04T16:04:35.749 INFO:tasks.workunit.client.1.smithi152.stdout:5/402: write d6/d50/f83 [938256,96300] 0 2024-03-04T16:04:35.750 INFO:tasks.workunit.client.1.smithi152.stdout:5/403: chown d6/d8/d1f/d3d/l6b 264148154 1 2024-03-04T16:04:35.758 INFO:tasks.workunit.client.1.smithi152.stdout:9/275: dread d2/f19 [0,4194304] 0 2024-03-04T16:04:35.759 INFO:tasks.workunit.client.1.smithi152.stdout:9/276: chown d2/d13/d2e/f44 63480551 1 2024-03-04T16:04:35.761 INFO:tasks.workunit.client.1.smithi152.stdout:3/354: sync 2024-03-04T16:04:35.764 INFO:tasks.workunit.client.1.smithi152.stdout:5/404: rmdir d6/d50 39 2024-03-04T16:04:35.768 INFO:tasks.workunit.client.1.smithi152.stdout:3/355: mknod d6/d10/c78 0 2024-03-04T16:04:35.771 INFO:tasks.workunit.client.1.smithi152.stdout:5/405: mknod d6/d8/d1f/d88/c91 0 2024-03-04T16:04:35.774 INFO:tasks.workunit.client.1.smithi152.stdout:5/406: rmdir d6/d8/d5a/d64/d8a 39 2024-03-04T16:04:35.777 INFO:tasks.workunit.client.1.smithi152.stdout:3/356: link d6/d10/d2d/d4d/l73 d6/d10/d2d/d6e/l79 0 2024-03-04T16:04:35.778 INFO:tasks.workunit.client.1.smithi152.stdout:3/357: readlink d6/d10/d2d/d4d/l5d 0 2024-03-04T16:04:35.779 INFO:tasks.workunit.client.1.smithi152.stdout:5/407: link d6/d8/d1f/f2e d6/d8/d5a/d82/f92 0 2024-03-04T16:04:35.779 INFO:tasks.workunit.client.1.smithi152.stdout:9/277: truncate d2/f2d 2918054 0 2024-03-04T16:04:35.781 INFO:tasks.workunit.client.1.smithi152.stdout:3/358: creat d6/d14/f7a x:0 0 0 2024-03-04T16:04:35.782 INFO:tasks.workunit.client.1.smithi152.stdout:3/359: fdatasync d6/f3a 0 2024-03-04T16:04:35.782 INFO:tasks.workunit.client.1.smithi152.stdout:5/408: chown d6/d8/d1f/d3d/d61/c63 817564 1 2024-03-04T16:04:35.783 INFO:tasks.workunit.client.1.smithi152.stdout:5/409: dread - d6/d8/d5a/d64/d76/f7d zero size 2024-03-04T16:04:35.783 INFO:tasks.workunit.client.1.smithi152.stdout:3/360: read d6/d10/d2d/f3b [691464,107932] 0 2024-03-04T16:04:35.786 INFO:tasks.workunit.client.1.smithi152.stdout:3/361: truncate d6/d53/f72 102127 0 2024-03-04T16:04:35.787 INFO:tasks.workunit.client.1.smithi152.stdout:3/362: fsync d6/d14/f15 0 2024-03-04T16:04:35.788 INFO:tasks.workunit.client.1.smithi152.stdout:3/363: chown d6/f1c 2797324 1 2024-03-04T16:04:35.789 INFO:tasks.workunit.client.1.smithi152.stdout:9/278: dwrite d2/f19 [0,4194304] 0 2024-03-04T16:04:35.792 INFO:tasks.workunit.client.1.smithi152.stdout:5/410: dread d6/f39 [0,4194304] 0 2024-03-04T16:04:35.793 INFO:tasks.workunit.client.1.smithi152.stdout:3/364: rename d6/c31 to d6/d10/d2d/d6e/c7b 0 2024-03-04T16:04:35.793 INFO:tasks.workunit.client.1.smithi152.stdout:9/279: symlink d2/d13/d2e/l57 0 2024-03-04T16:04:35.794 INFO:tasks.workunit.client.1.smithi152.stdout:9/280: fdatasync d2/fd 0 2024-03-04T16:04:35.797 INFO:tasks.workunit.client.1.smithi152.stdout:3/365: link d6/l16 d6/d53/l7c 0 2024-03-04T16:04:35.800 INFO:tasks.workunit.client.1.smithi152.stdout:3/366: truncate d6/f4a 113672 0 2024-03-04T16:04:35.805 INFO:tasks.workunit.client.1.smithi152.stdout:3/367: link d6/f9 d6/d14/f7d 0 2024-03-04T16:04:35.809 INFO:tasks.workunit.client.1.smithi152.stdout:3/368: unlink d6/d14/d41/l75 0 2024-03-04T16:04:35.815 INFO:tasks.workunit.client.1.smithi152.stdout:3/369: dwrite d6/d10/d2d/d59/f77 [0,4194304] 0 2024-03-04T16:04:35.816 INFO:tasks.workunit.client.1.smithi152.stdout:3/370: readlink d6/d14/l50 0 2024-03-04T16:04:35.841 INFO:tasks.workunit.client.1.smithi152.stdout:3/371: dread d6/d10/d2d/f32 [0,4194304] 0 2024-03-04T16:04:35.847 INFO:tasks.workunit.client.1.smithi152.stdout:3/372: dwrite d6/d10/f21 [0,4194304] 0 2024-03-04T16:04:35.850 INFO:tasks.workunit.client.1.smithi152.stdout:3/373: link d6/f1c d6/d14/d58/f7e 0 2024-03-04T16:04:35.859 INFO:tasks.workunit.client.1.smithi152.stdout:3/374: truncate d6/f9 2708062 0 2024-03-04T16:04:35.861 INFO:tasks.workunit.client.1.smithi152.stdout:3/375: symlink d6/d60/l7f 0 2024-03-04T16:04:35.864 INFO:tasks.workunit.client.1.smithi152.stdout:3/376: stat d6/f4a 0 2024-03-04T16:04:35.868 INFO:tasks.workunit.client.1.smithi152.stdout:3/377: link d6/d10/d2d/d4d/l52 d6/d14/l80 0 2024-03-04T16:04:35.870 INFO:tasks.workunit.client.1.smithi152.stdout:3/378: rename d6/d14/f7a to d6/d60/f81 0 2024-03-04T16:04:35.872 INFO:tasks.workunit.client.1.smithi152.stdout:3/379: creat d6/d53/f82 x:0 0 0 2024-03-04T16:04:35.873 INFO:tasks.workunit.client.1.smithi152.stdout:3/380: fsync d6/d53/f72 0 2024-03-04T16:04:35.874 INFO:tasks.workunit.client.1.smithi152.stdout:3/381: chown d6/d14/d41/d49/f5a 0 1 2024-03-04T16:04:35.875 INFO:tasks.workunit.client.1.smithi152.stdout:2/332: sync 2024-03-04T16:04:35.876 INFO:tasks.workunit.client.1.smithi152.stdout:0/316: sync 2024-03-04T16:04:35.876 INFO:tasks.workunit.client.1.smithi152.stdout:4/241: sync 2024-03-04T16:04:35.876 INFO:tasks.workunit.client.1.smithi152.stdout:0/317: chown d9/c34 12 1 2024-03-04T16:04:35.878 INFO:tasks.workunit.client.1.smithi152.stdout:2/333: creat da/d2c/d5e/f6d x:0 0 0 2024-03-04T16:04:35.879 INFO:tasks.workunit.client.1.smithi152.stdout:0/318: chown d9/d2c/d2e/l43 838 1 2024-03-04T16:04:35.883 INFO:tasks.workunit.client.1.smithi152.stdout:0/319: creat d9/d2c/d2e/d38/d56/f63 x:0 0 0 2024-03-04T16:04:35.904 INFO:tasks.workunit.client.1.smithi152.stdout:0/320: dwrite d9/f26 [0,4194304] 0 2024-03-04T16:04:35.904 INFO:tasks.workunit.client.1.smithi152.stdout:4/242: truncate f0 5994210 0 2024-03-04T16:04:35.907 INFO:tasks.workunit.client.1.smithi152.stdout:4/243: write d8/f16 [3193830,109115] 0 2024-03-04T16:04:35.914 INFO:tasks.workunit.client.1.smithi152.stdout:0/321: dwrite d9/d2c/d2e/d38/d56/d5b/f60 [0,4194304] 0 2024-03-04T16:04:35.915 INFO:tasks.workunit.client.1.smithi152.stdout:4/244: creat d8/d3d/f4f x:0 0 0 2024-03-04T16:04:35.922 INFO:tasks.workunit.client.1.smithi152.stdout:0/322: dwrite d9/d2c/d2e/f41 [0,4194304] 0 2024-03-04T16:04:35.933 INFO:tasks.workunit.client.1.smithi152.stdout:0/323: symlink d9/d1b/d54/l64 0 2024-03-04T16:04:35.941 INFO:tasks.workunit.client.1.smithi152.stdout:0/324: dwrite d9/d1b/f23 [0,4194304] 0 2024-03-04T16:04:35.943 INFO:tasks.workunit.client.1.smithi152.stdout:0/325: mknod d9/d2c/d2e/d38/d56/c65 0 2024-03-04T16:04:35.954 INFO:tasks.workunit.client.1.smithi152.stdout:4/245: dread d8/f37 [0,4194304] 0 2024-03-04T16:04:35.954 INFO:tasks.workunit.client.1.smithi152.stdout:0/326: link d9/d2c/d2e/c44 d9/d2c/d2e/d38/d56/d5b/c66 0 2024-03-04T16:04:35.956 INFO:tasks.workunit.client.1.smithi152.stdout:0/327: symlink d9/d1b/d3c/l67 0 2024-03-04T16:04:35.958 INFO:tasks.workunit.client.1.smithi152.stdout:4/246: creat d8/d9/f50 x:0 0 0 2024-03-04T16:04:35.972 INFO:tasks.workunit.client.1.smithi152.stdout:4/247: dread f7 [0,4194304] 0 2024-03-04T16:04:35.972 INFO:tasks.workunit.client.1.smithi152.stdout:4/248: dread - d8/d2f/f33 zero size 2024-03-04T16:04:35.974 INFO:tasks.workunit.client.1.smithi152.stdout:4/249: mknod d8/d3d/c51 0 2024-03-04T16:04:35.975 INFO:tasks.workunit.client.1.smithi152.stdout:4/250: write d8/d2f/f33 [349212,85650] 0 2024-03-04T16:04:35.976 INFO:tasks.workunit.client.1.smithi152.stdout:4/251: fsync d8/d9/f28 0 2024-03-04T16:04:35.981 INFO:tasks.workunit.client.1.smithi152.stdout:4/252: dwrite d8/d9/d24/d26/f2e [0,4194304] 0 2024-03-04T16:04:35.985 INFO:tasks.workunit.client.1.smithi152.stdout:4/253: symlink d8/d9/d24/d26/d42/l52 0 2024-03-04T16:04:35.996 INFO:tasks.workunit.client.1.smithi152.stdout:4/254: rmdir d8 39 2024-03-04T16:04:36.004 INFO:tasks.workunit.client.1.smithi152.stdout:4/255: dwrite d8/d2f/f4a [0,4194304] 0 2024-03-04T16:04:36.006 INFO:tasks.workunit.client.1.smithi152.stdout:4/256: fsync d8/f23 0 2024-03-04T16:04:36.010 INFO:tasks.workunit.client.1.smithi152.stdout:4/257: mkdir d8/d53 0 2024-03-04T16:04:36.015 INFO:tasks.workunit.client.1.smithi152.stdout:4/258: dwrite d8/d3d/f4f [0,4194304] 0 2024-03-04T16:04:36.016 INFO:tasks.workunit.client.1.smithi152.stdout:4/259: chown d8/d3d/c3f 5741000 1 2024-03-04T16:04:36.018 INFO:tasks.workunit.client.1.smithi152.stdout:4/260: mknod d8/d53/c54 0 2024-03-04T16:04:36.027 INFO:tasks.workunit.client.1.smithi152.stdout:7/331: sync 2024-03-04T16:04:36.027 INFO:tasks.workunit.client.1.smithi152.stdout:4/261: write f7 [847062,35956] 0 2024-03-04T16:04:36.028 INFO:tasks.workunit.client.1.smithi152.stdout:7/332: fdatasync d14/d21/d47/d36/f5b 0 2024-03-04T16:04:36.028 INFO:tasks.workunit.client.1.smithi152.stdout:4/262: fsync d8/f25 0 2024-03-04T16:04:36.032 INFO:tasks.workunit.client.1.smithi152.stdout:4/263: mknod d8/c55 0 2024-03-04T16:04:36.032 INFO:tasks.workunit.client.1.smithi152.stdout:4/264: dread - d8/d3d/f4c zero size 2024-03-04T16:04:36.034 INFO:tasks.workunit.client.1.smithi152.stdout:4/265: fdatasync d8/d9/d24/d26/f48 0 2024-03-04T16:04:36.035 INFO:tasks.workunit.client.1.smithi152.stdout:7/333: dwrite d14/d2b/f3b [0,4194304] 0 2024-03-04T16:04:36.038 INFO:tasks.workunit.client.1.smithi152.stdout:4/266: symlink d8/d4e/l56 0 2024-03-04T16:04:36.042 INFO:tasks.workunit.client.1.smithi152.stdout:7/334: dwrite d14/d2b/f63 [0,4194304] 0 2024-03-04T16:04:36.042 INFO:tasks.workunit.client.1.smithi152.stdout:4/267: creat d8/d4e/f57 x:0 0 0 2024-03-04T16:04:36.100 INFO:tasks.workunit.client.1.smithi152.stdout:7/335: truncate d14/f61 245297 0 2024-03-04T16:04:36.102 INFO:tasks.workunit.client.1.smithi152.stdout:4/268: dread d8/f16 [4194304,4194304] 0 2024-03-04T16:04:36.106 INFO:tasks.workunit.client.1.smithi152.stdout:4/269: mkdir d8/d58 0 2024-03-04T16:04:36.106 INFO:tasks.workunit.client.1.smithi152.stdout:4/270: dread - d8/d2f/f47 zero size 2024-03-04T16:04:36.114 INFO:tasks.workunit.client.1.smithi152.stdout:7/336: dread d14/d21/f30 [0,4194304] 0 2024-03-04T16:04:36.115 INFO:tasks.workunit.client.1.smithi152.stdout:4/271: dread d8/d9/d24/d26/f2a [0,4194304] 0 2024-03-04T16:04:36.119 INFO:tasks.workunit.client.1.smithi152.stdout:7/337: creat d14/d21/d31/f64 x:0 0 0 2024-03-04T16:04:36.120 INFO:tasks.workunit.client.1.smithi152.stdout:4/272: symlink d8/d9/d24/d26/d42/l59 0 2024-03-04T16:04:36.121 INFO:tasks.workunit.client.1.smithi152.stdout:7/338: creat d14/d21/d47/f65 x:0 0 0 2024-03-04T16:04:36.126 INFO:tasks.workunit.client.1.smithi152.stdout:4/273: dwrite d8/f25 [0,4194304] 0 2024-03-04T16:04:36.145 INFO:tasks.workunit.client.1.smithi152.stdout:7/339: dread d14/d21/d48/f4b [0,4194304] 0 2024-03-04T16:04:36.147 INFO:tasks.workunit.client.1.smithi152.stdout:7/340: symlink d14/d21/d47/d36/l66 0 2024-03-04T16:04:36.148 INFO:tasks.workunit.client.1.smithi152.stdout:7/341: truncate d14/d21/d47/f51 63408 0 2024-03-04T16:04:36.150 INFO:tasks.workunit.client.1.smithi152.stdout:7/342: symlink d14/l67 0 2024-03-04T16:04:36.150 INFO:tasks.workunit.client.1.smithi152.stdout:7/343: chown d14/f16 149 1 2024-03-04T16:04:36.152 INFO:tasks.workunit.client.1.smithi152.stdout:7/344: read d14/f15 [170891,124172] 0 2024-03-04T16:04:36.154 INFO:tasks.workunit.client.1.smithi152.stdout:7/345: creat d14/d21/d47/d36/f68 x:0 0 0 2024-03-04T16:04:36.157 INFO:tasks.workunit.client.1.smithi152.stdout:7/346: mkdir d14/d69 0 2024-03-04T16:04:36.159 INFO:tasks.workunit.client.1.smithi152.stdout:6/411: sync 2024-03-04T16:04:36.168 INFO:tasks.workunit.client.1.smithi152.stdout:7/347: write d14/d2b/f4e [2234390,130885] 0 2024-03-04T16:04:36.173 INFO:tasks.workunit.client.1.smithi152.stdout:7/348: truncate d14/d21/d47/d40/f43 3187424 0 2024-03-04T16:04:36.174 INFO:tasks.workunit.client.1.smithi152.stdout:6/412: dwrite d8/d2a/f45 [0,4194304] 0 2024-03-04T16:04:36.183 INFO:tasks.workunit.client.1.smithi152.stdout:7/349: link fd d14/f6a 0 2024-03-04T16:04:36.185 INFO:tasks.workunit.client.1.smithi152.stdout:7/350: chown d14/d21/d31/f44 106038 1 2024-03-04T16:04:36.186 INFO:tasks.workunit.client.1.smithi152.stdout:7/351: dread - d14/d21/d47/d36/f68 zero size 2024-03-04T16:04:36.187 INFO:tasks.workunit.client.1.smithi152.stdout:6/413: dwrite d8/d10/f24 [0,4194304] 0 2024-03-04T16:04:36.188 INFO:tasks.workunit.client.1.smithi152.stdout:6/414: truncate d8/d10/f16 866632 0 2024-03-04T16:04:36.189 INFO:tasks.workunit.client.1.smithi152.stdout:6/415: chown d8/f56 1159220 1 2024-03-04T16:04:36.190 INFO:tasks.workunit.client.1.smithi152.stdout:7/352: unlink d14/d21/d31/f44 0 2024-03-04T16:04:36.192 INFO:tasks.workunit.client.1.smithi152.stdout:7/353: write d14/d21/d47/f35 [2295517,54929] 0 2024-03-04T16:04:36.193 INFO:tasks.workunit.client.1.smithi152.stdout:6/416: stat d8/c27 0 2024-03-04T16:04:36.197 INFO:tasks.workunit.client.1.smithi152.stdout:6/417: fsync d8/f2f 0 2024-03-04T16:04:36.198 INFO:tasks.workunit.client.1.smithi152.stdout:6/418: write d8/d10/f18 [2298565,40961] 0 2024-03-04T16:04:36.199 INFO:tasks.workunit.client.1.smithi152.stdout:6/419: fdatasync d8/d2a/f3e 0 2024-03-04T16:04:36.200 INFO:tasks.workunit.client.1.smithi152.stdout:7/354: dwrite d14/d21/d47/d36/f68 [0,4194304] 0 2024-03-04T16:04:36.201 INFO:tasks.workunit.client.1.smithi152.stdout:7/355: dread - d14/d21/d31/f3e zero size 2024-03-04T16:04:36.208 INFO:tasks.workunit.client.1.smithi152.stdout:7/356: truncate d14/d21/f30 5819223 0 2024-03-04T16:04:36.209 INFO:tasks.workunit.client.1.smithi152.stdout:6/420: getdents d8/d38/d59/d61/d57 0 2024-03-04T16:04:36.211 INFO:tasks.workunit.client.1.smithi152.stdout:7/357: chown d14/f6a 244 1 2024-03-04T16:04:36.213 INFO:tasks.workunit.client.1.smithi152.stdout:6/421: unlink d8/d23/l3d 0 2024-03-04T16:04:36.217 INFO:tasks.workunit.client.1.smithi152.stdout:6/422: creat d8/d38/d59/d61/d57/f8a x:0 0 0 2024-03-04T16:04:36.218 INFO:tasks.workunit.client.1.smithi152.stdout:7/358: dwrite d14/d21/d31/f57 [0,4194304] 0 2024-03-04T16:04:36.218 INFO:tasks.workunit.client.1.smithi152.stdout:6/423: chown d8/d2a/c37 9772 1 2024-03-04T16:04:36.219 INFO:tasks.workunit.client.1.smithi152.stdout:6/424: chown d8/c69 447 1 2024-03-04T16:04:36.220 INFO:tasks.workunit.client.1.smithi152.stdout:7/359: write d14/d21/d31/f64 [845354,129599] 0 2024-03-04T16:04:36.221 INFO:tasks.workunit.client.1.smithi152.stdout:6/425: truncate d8/d10/f44 4325578 0 2024-03-04T16:04:36.223 INFO:tasks.workunit.client.1.smithi152.stdout:7/360: rmdir d14/d21 39 2024-03-04T16:04:36.225 INFO:tasks.workunit.client.1.smithi152.stdout:6/426: chown d8/d10/l49 3785600 1 2024-03-04T16:04:36.225 INFO:tasks.workunit.client.1.smithi152.stdout:6/427: chown d8/lf 0 1 2024-03-04T16:04:36.226 INFO:tasks.workunit.client.1.smithi152.stdout:7/361: chown d14/d21/d47/d32/c3c 191715 1 2024-03-04T16:04:36.227 INFO:tasks.workunit.client.1.smithi152.stdout:6/428: creat d8/d23/d3a/f8b x:0 0 0 2024-03-04T16:04:36.228 INFO:tasks.workunit.client.1.smithi152.stdout:6/429: stat f7 0 2024-03-04T16:04:36.228 INFO:tasks.workunit.client.1.smithi152.stdout:6/430: chown d8/d2a/f3e 0 1 2024-03-04T16:04:36.240 INFO:tasks.workunit.client.1.smithi152.stdout:6/431: truncate d8/d2a/f3c 1330517 0 2024-03-04T16:04:36.240 INFO:tasks.workunit.client.1.smithi152.stdout:6/432: chown d8/d38/f48 227 1 2024-03-04T16:04:36.248 INFO:tasks.workunit.client.1.smithi152.stdout:7/362: dread d14/f6a [0,4194304] 0 2024-03-04T16:04:36.249 INFO:tasks.workunit.client.1.smithi152.stdout:7/363: readlink d14/d2b/l53 0 2024-03-04T16:04:36.249 INFO:tasks.workunit.client.1.smithi152.stdout:7/364: write d14/d21/f25 [2544041,93447] 0 2024-03-04T16:04:36.256 INFO:tasks.workunit.client.1.smithi152.stdout:7/365: dwrite d14/d2b/f60 [0,4194304] 0 2024-03-04T16:04:36.264 INFO:tasks.workunit.client.1.smithi152.stdout:7/366: creat d14/d21/f6b x:0 0 0 2024-03-04T16:04:36.266 INFO:tasks.workunit.client.1.smithi152.stdout:7/367: truncate d14/d21/f30 517770 0 2024-03-04T16:04:36.268 INFO:tasks.workunit.client.1.smithi152.stdout:7/368: creat d14/d21/d47/d40/f6c x:0 0 0 2024-03-04T16:04:36.270 INFO:tasks.workunit.client.1.smithi152.stdout:1/364: sync 2024-03-04T16:04:36.270 INFO:tasks.workunit.client.1.smithi152.stdout:8/343: sync 2024-03-04T16:04:36.270 INFO:tasks.workunit.client.1.smithi152.stdout:2/334: sync 2024-03-04T16:04:36.272 INFO:tasks.workunit.client.1.smithi152.stdout:1/365: truncate d19/d20/d2f/d5d/f67 640730 0 2024-03-04T16:04:36.273 INFO:tasks.workunit.client.1.smithi152.stdout:1/366: chown d19/d4b/f5f 49421 1 2024-03-04T16:04:36.274 INFO:tasks.workunit.client.1.smithi152.stdout:1/367: fdatasync d19/d20/d2f/f76 0 2024-03-04T16:04:36.275 INFO:tasks.workunit.client.1.smithi152.stdout:8/344: mkdir d2/d8/dd/d22/d67 0 2024-03-04T16:04:36.276 INFO:tasks.workunit.client.1.smithi152.stdout:7/369: getdents d14/d21/d47/d32 0 2024-03-04T16:04:36.278 INFO:tasks.workunit.client.1.smithi152.stdout:1/368: symlink d19/d20/d2f/d6b/l77 0 2024-03-04T16:04:36.281 INFO:tasks.workunit.client.1.smithi152.stdout:1/369: dread d19/d20/d2f/d5d/f67 [0,4194304] 0 2024-03-04T16:04:36.282 INFO:tasks.workunit.client.1.smithi152.stdout:8/345: mkdir d2/d3/d21/d2d/d52/d68 0 2024-03-04T16:04:36.284 INFO:tasks.workunit.client.1.smithi152.stdout:2/335: dread da/f56 [0,4194304] 0 2024-03-04T16:04:36.285 INFO:tasks.workunit.client.1.smithi152.stdout:1/370: dread d19/d3b/f49 [0,4194304] 0 2024-03-04T16:04:36.289 INFO:tasks.workunit.client.1.smithi152.stdout:2/336: dread da/df/d2e/f39 [0,4194304] 0 2024-03-04T16:04:36.291 INFO:tasks.workunit.client.1.smithi152.stdout:7/370: write d14/d21/d48/f4b [4135405,64312] 0 2024-03-04T16:04:36.291 INFO:tasks.workunit.client.1.smithi152.stdout:7/371: fsync d14/d21/d47/d40/f6c 0 2024-03-04T16:04:36.296 INFO:tasks.workunit.client.1.smithi152.stdout:2/337: mkdir da/df/d58/d6e 0 2024-03-04T16:04:36.299 INFO:tasks.workunit.client.1.smithi152.stdout:1/371: symlink d19/d75/l78 0 2024-03-04T16:04:36.301 INFO:tasks.workunit.client.1.smithi152.stdout:8/346: rename d2/d3/d15/l64 to d2/d3/l69 0 2024-03-04T16:04:36.304 INFO:tasks.workunit.client.1.smithi152.stdout:8/347: read d2/d3/d21/d2d/fc [820168,103411] 0 2024-03-04T16:04:36.305 INFO:tasks.workunit.client.1.smithi152.stdout:8/348: stat d2/d3/d21/f32 0 2024-03-04T16:04:36.307 INFO:tasks.workunit.client.1.smithi152.stdout:1/372: rename f17 to d19/d3b/d6e/f79 0 2024-03-04T16:04:36.308 INFO:tasks.workunit.client.1.smithi152.stdout:2/338: dread da/df/d35/d50/f5c [0,4194304] 0 2024-03-04T16:04:36.311 INFO:tasks.workunit.client.1.smithi152.stdout:8/349: creat d2/d3/d15/f6a x:0 0 0 2024-03-04T16:04:36.316 INFO:tasks.workunit.client.1.smithi152.stdout:7/372: getdents d14/d21/d47/d40 0 2024-03-04T16:04:36.324 INFO:tasks.workunit.client.1.smithi152.stdout:7/373: creat d14/d69/f6d x:0 0 0 2024-03-04T16:04:36.326 INFO:tasks.workunit.client.1.smithi152.stdout:1/373: getdents d19/d75 0 2024-03-04T16:04:36.330 INFO:tasks.workunit.client.1.smithi152.stdout:7/374: getdents d14/d21/d47/d36 0 2024-03-04T16:04:36.331 INFO:tasks.workunit.client.1.smithi152.stdout:8/350: dwrite d2/f53 [0,4194304] 0 2024-03-04T16:04:36.333 INFO:tasks.workunit.client.1.smithi152.stdout:1/374: truncate d19/d20/d2f/d6b/d38/f4c 2032324 0 2024-03-04T16:04:36.334 INFO:tasks.workunit.client.1.smithi152.stdout:8/351: truncate d2/d8/dd/d22/d26/f58 164663 0 2024-03-04T16:04:36.336 INFO:tasks.workunit.client.1.smithi152.stdout:8/352: truncate d2/d8/dd/f2f 1969837 0 2024-03-04T16:04:36.336 INFO:tasks.workunit.client.1.smithi152.stdout:8/353: chown d2/d8/l12 3 1 2024-03-04T16:04:36.336 INFO:tasks.workunit.client.1.smithi152.stdout:7/375: mkdir d14/d21/d47/d40/d6e 0 2024-03-04T16:04:36.338 INFO:tasks.workunit.client.1.smithi152.stdout:1/375: symlink d19/d20/d2f/l7a 0 2024-03-04T16:04:36.340 INFO:tasks.workunit.client.1.smithi152.stdout:8/354: creat d2/d3/d15/f6b x:0 0 0 2024-03-04T16:04:36.341 INFO:tasks.workunit.client.1.smithi152.stdout:7/376: symlink d14/d69/l6f 0 2024-03-04T16:04:36.343 INFO:tasks.workunit.client.1.smithi152.stdout:1/376: rename d19/l2b to d19/d20/d2f/d6b/d38/l7b 0 2024-03-04T16:04:36.347 INFO:tasks.workunit.client.1.smithi152.stdout:1/377: creat d19/f7c x:0 0 0 2024-03-04T16:04:36.347 INFO:tasks.workunit.client.1.smithi152.stdout:8/355: dwrite d2/d3/d15/f6b [0,4194304] 0 2024-03-04T16:04:36.348 INFO:tasks.workunit.client.1.smithi152.stdout:1/378: readlink d19/d20/d2f/l36 0 2024-03-04T16:04:36.349 INFO:tasks.workunit.client.1.smithi152.stdout:7/377: rename d14/l22 to d14/d21/d48/l70 0 2024-03-04T16:04:36.352 INFO:tasks.workunit.client.1.smithi152.stdout:1/379: read d19/d20/d2f/d5d/f67 [100066,31354] 0 2024-03-04T16:04:36.356 INFO:tasks.workunit.client.1.smithi152.stdout:1/380: mknod d19/c7d 0 2024-03-04T16:04:36.357 INFO:tasks.workunit.client.1.smithi152.stdout:1/381: stat d19/d20/f45 0 2024-03-04T16:04:36.358 INFO:tasks.workunit.client.1.smithi152.stdout:7/378: mkdir d14/d71 0 2024-03-04T16:04:36.359 INFO:tasks.workunit.client.1.smithi152.stdout:7/379: readlink d14/d21/d31/l62 0 2024-03-04T16:04:36.363 INFO:tasks.workunit.client.1.smithi152.stdout:7/380: symlink d14/d21/d47/d32/l72 0 2024-03-04T16:04:36.364 INFO:tasks.workunit.client.1.smithi152.stdout:1/382: dwrite f10 [0,4194304] 0 2024-03-04T16:04:36.368 INFO:tasks.workunit.client.1.smithi152.stdout:7/381: rmdir d14/d21/d47/d40 39 2024-03-04T16:04:36.370 INFO:tasks.workunit.client.1.smithi152.stdout:1/383: rmdir d19/d4b/d53/d57/d62 39 2024-03-04T16:04:36.370 INFO:tasks.workunit.client.1.smithi152.stdout:7/382: read d14/d21/d31/f64 [635117,1066] 0 2024-03-04T16:04:36.376 INFO:tasks.workunit.client.1.smithi152.stdout:7/383: link f4 d14/d21/d47/d36/f73 0 2024-03-04T16:04:36.377 INFO:tasks.workunit.client.1.smithi152.stdout:1/384: link c14 d19/d4b/d53/d5c/c7e 0 2024-03-04T16:04:36.379 INFO:tasks.workunit.client.1.smithi152.stdout:8/356: dread d2/f37 [0,4194304] 0 2024-03-04T16:04:36.381 INFO:tasks.workunit.client.1.smithi152.stdout:1/385: symlink d19/d3b/l7f 0 2024-03-04T16:04:36.384 INFO:tasks.workunit.client.1.smithi152.stdout:1/386: stat d19/d4b/d53/d57/f5e 0 2024-03-04T16:04:36.387 INFO:tasks.workunit.client.1.smithi152.stdout:8/357: mkdir d2/d8/dd/d22/d6c 0 2024-03-04T16:04:36.388 INFO:tasks.workunit.client.1.smithi152.stdout:1/387: symlink d19/d20/d2f/d6b/l80 0 2024-03-04T16:04:36.389 INFO:tasks.workunit.client.1.smithi152.stdout:1/388: write d19/d20/d2f/d6b/f37 [690598,70196] 0 2024-03-04T16:04:36.391 INFO:tasks.workunit.client.1.smithi152.stdout:8/358: fsync d2/f37 0 2024-03-04T16:04:36.393 INFO:tasks.workunit.client.1.smithi152.stdout:7/384: write d14/f20 [501117,47679] 0 2024-03-04T16:04:36.393 INFO:tasks.workunit.client.1.smithi152.stdout:7/385: chown d14/f20 264240417 1 2024-03-04T16:04:36.395 INFO:tasks.workunit.client.1.smithi152.stdout:1/389: write fa [2135901,19329] 0 2024-03-04T16:04:36.397 INFO:tasks.workunit.client.1.smithi152.stdout:1/390: write d19/d4b/f51 [465791,56054] 0 2024-03-04T16:04:36.399 INFO:tasks.workunit.client.1.smithi152.stdout:8/359: mknod d2/d3/d15/d1c/c6d 0 2024-03-04T16:04:36.401 INFO:tasks.workunit.client.1.smithi152.stdout:1/391: dread d19/d4b/f51 [0,4194304] 0 2024-03-04T16:04:36.401 INFO:tasks.workunit.client.1.smithi152.stdout:7/386: creat d14/d21/d47/d36/f74 x:0 0 0 2024-03-04T16:04:36.410 INFO:tasks.workunit.client.1.smithi152.stdout:7/387: unlink d14/d21/d47/f18 0 2024-03-04T16:04:36.418 INFO:tasks.workunit.client.1.smithi152.stdout:7/388: unlink d14/d21/d47/c5c 0 2024-03-04T16:04:36.418 INFO:tasks.workunit.client.1.smithi152.stdout:7/389: fdatasync d14/d2b/f60 0 2024-03-04T16:04:36.422 INFO:tasks.workunit.client.1.smithi152.stdout:1/392: creat d19/d4b/d53/d57/f81 x:0 0 0 2024-03-04T16:04:36.422 INFO:tasks.workunit.client.1.smithi152.stdout:1/393: chown d19/d4b 982647 1 2024-03-04T16:04:36.424 INFO:tasks.workunit.client.1.smithi152.stdout:1/394: chown d19/d3b/f49 378 1 2024-03-04T16:04:36.424 INFO:tasks.workunit.client.1.smithi152.stdout:8/360: dwrite d2/d8/dd/d22/d26/f38 [0,4194304] 0 2024-03-04T16:04:36.426 INFO:tasks.workunit.client.1.smithi152.stdout:1/395: write d19/d20/d2f/d6b/f4a [5156694,52254] 0 2024-03-04T16:04:36.431 INFO:tasks.workunit.client.1.smithi152.stdout:1/396: mkdir d19/d4b/d53/d57/d62/d82 0 2024-03-04T16:04:36.442 INFO:tasks.workunit.client.1.smithi152.stdout:1/397: truncate d19/d20/d2f/d6b/d38/f3a 3958689 0 2024-03-04T16:04:36.444 INFO:tasks.workunit.client.1.smithi152.stdout:4/274: sync 2024-03-04T16:04:36.444 INFO:tasks.workunit.client.1.smithi152.stdout:0/328: sync 2024-03-04T16:04:36.444 INFO:tasks.workunit.client.1.smithi152.stdout:3/382: sync 2024-03-04T16:04:36.444 INFO:tasks.workunit.client.1.smithi152.stdout:5/411: sync 2024-03-04T16:04:36.444 INFO:tasks.workunit.client.1.smithi152.stdout:9/281: sync 2024-03-04T16:04:36.445 INFO:tasks.workunit.client.1.smithi152.stdout:1/398: dread d19/f28 [0,4194304] 0 2024-03-04T16:04:36.447 INFO:tasks.workunit.client.1.smithi152.stdout:3/383: unlink d6/d10/l1b 0 2024-03-04T16:04:36.448 INFO:tasks.workunit.client.1.smithi152.stdout:9/282: creat d2/d13/d38/f58 x:0 0 0 2024-03-04T16:04:36.448 INFO:tasks.workunit.client.1.smithi152.stdout:3/384: chown d6/d14/f15 0 1 2024-03-04T16:04:36.449 INFO:tasks.workunit.client.1.smithi152.stdout:5/412: fsync d6/d50/f7e 0 2024-03-04T16:04:36.471 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:36 smithi116 ceph-mon[29120]: pgmap v34: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 49 MiB/s rd, 193 MiB/s wr, 438 op/s 2024-03-04T16:04:36.502 INFO:tasks.workunit.client.1.smithi152.stdout:1/399: creat d19/d75/f83 x:0 0 0 2024-03-04T16:04:36.502 INFO:tasks.workunit.client.1.smithi152.stdout:4/275: dwrite d8/d9/d24/d26/f2e [0,4194304] 0 2024-03-04T16:04:36.505 INFO:tasks.workunit.client.1.smithi152.stdout:9/283: symlink d2/l59 0 2024-03-04T16:04:36.507 INFO:tasks.workunit.client.1.smithi152.stdout:9/284: write d2/d13/d23/d42/f46 [513558,47026] 0 2024-03-04T16:04:36.508 INFO:tasks.workunit.client.1.smithi152.stdout:0/329: write d9/f45 [411267,46081] 0 2024-03-04T16:04:36.510 INFO:tasks.workunit.client.1.smithi152.stdout:5/413: symlink d6/d8/d5a/l93 0 2024-03-04T16:04:36.513 INFO:tasks.workunit.client.1.smithi152.stdout:4/276: symlink d8/d4e/l5a 0 2024-03-04T16:04:36.516 INFO:tasks.workunit.client.1.smithi152.stdout:9/285: mknod d2/d13/d23/c5a 0 2024-03-04T16:04:36.521 INFO:tasks.workunit.client.1.smithi152.stdout:5/414: symlink d6/d8/d1f/l94 0 2024-03-04T16:04:36.522 INFO:tasks.workunit.client.1.smithi152.stdout:5/415: chown d6/f68 124649173 1 2024-03-04T16:04:36.522 INFO:tasks.workunit.client.1.smithi152.stdout:1/400: mknod d19/d20/d2f/d65/c84 0 2024-03-04T16:04:36.523 INFO:tasks.workunit.client.1.smithi152.stdout:1/401: chown d19/d20/d2f/d5d 16 1 2024-03-04T16:04:36.523 INFO:tasks.workunit.client.1.smithi152.stdout:3/385: dread d6/fc [0,4194304] 0 2024-03-04T16:04:36.524 INFO:tasks.workunit.client.1.smithi152.stdout:4/277: mknod d8/d4e/c5b 0 2024-03-04T16:04:36.526 INFO:tasks.workunit.client.1.smithi152.stdout:9/286: truncate d2/fe 1485626 0 2024-03-04T16:04:36.531 INFO:tasks.workunit.client.1.smithi152.stdout:1/402: chown d19/d20/d2f/d6b/d38/f4c 25020479 1 2024-03-04T16:04:36.533 INFO:tasks.workunit.client.1.smithi152.stdout:4/278: chown d8/d9/l22 5 1 2024-03-04T16:04:36.540 INFO:tasks.workunit.client.1.smithi152.stdout:0/330: getdents d9/d1b/d54 0 2024-03-04T16:04:36.543 INFO:tasks.workunit.client.1.smithi152.stdout:1/403: unlink d19/d20/d2f/d6b/c2a 0 2024-03-04T16:04:36.544 INFO:tasks.workunit.client.1.smithi152.stdout:1/404: stat d19/d3b/f52 0 2024-03-04T16:04:36.544 INFO:tasks.workunit.client.1.smithi152.stdout:1/405: fdatasync f10 0 2024-03-04T16:04:36.545 INFO:tasks.workunit.client.1.smithi152.stdout:4/279: rename d8/d2f/f34 to d8/d2f/f5c 0 2024-03-04T16:04:36.546 INFO:tasks.workunit.client.1.smithi152.stdout:5/416: truncate d6/d50/f59 323297 0 2024-03-04T16:04:36.549 INFO:tasks.workunit.client.1.smithi152.stdout:0/331: rmdir d9/d2c/d2e/d38/d56/d5b 39 2024-03-04T16:04:36.549 INFO:tasks.workunit.client.1.smithi152.stdout:3/386: dwrite d6/d10/d2d/f32 [4194304,4194304] 0 2024-03-04T16:04:36.558 INFO:tasks.workunit.client.1.smithi152.stdout:9/287: write d2/f2d [269378,105137] 0 2024-03-04T16:04:36.565 INFO:tasks.workunit.client.1.smithi152.stdout:3/387: chown d6/d10/d2d/d4d/l52 176 1 2024-03-04T16:04:36.565 INFO:tasks.workunit.client.1.smithi152.stdout:3/388: stat d6/d14/c25 0 2024-03-04T16:04:36.566 INFO:tasks.workunit.client.1.smithi152.stdout:4/280: symlink d8/d58/l5d 0 2024-03-04T16:04:36.568 INFO:tasks.workunit.client.1.smithi152.stdout:4/281: write d8/d2f/f33 [744479,65871] 0 2024-03-04T16:04:36.571 INFO:tasks.workunit.client.1.smithi152.stdout:3/389: creat d6/d14/d41/d49/f83 x:0 0 0 2024-03-04T16:04:36.573 INFO:tasks.workunit.client.1.smithi152.stdout:9/288: link d2/d13/d2e/f37 d2/d13/d23/f5b 0 2024-03-04T16:04:36.574 INFO:tasks.workunit.client.1.smithi152.stdout:9/289: write d2/d13/d23/d42/f46 [237975,87542] 0 2024-03-04T16:04:36.576 INFO:tasks.workunit.client.1.smithi152.stdout:1/406: rename d19/d20/d2f/d6b/d38/d63/d64 to d19/d4b/d53/d57/d85 0 2024-03-04T16:04:36.580 INFO:tasks.workunit.client.1.smithi152.stdout:3/390: mkdir d6/d14/d41/d49/d84 0 2024-03-04T16:04:36.581 INFO:tasks.workunit.client.1.smithi152.stdout:0/332: dwrite d9/f49 [0,4194304] 0 2024-03-04T16:04:36.583 INFO:tasks.workunit.client.1.smithi152.stdout:9/290: creat d2/d13/d23/f5c x:0 0 0 2024-03-04T16:04:36.584 INFO:tasks.workunit.client.1.smithi152.stdout:0/333: chown d9/d1b/d3c/c5d 6797 1 2024-03-04T16:04:36.588 INFO:tasks.workunit.client.1.smithi152.stdout:5/417: rename d6/fa to d6/d8/d1f/d3d/d6f/f95 0 2024-03-04T16:04:36.595 INFO:tasks.workunit.client.1.smithi152.stdout:5/418: dwrite d6/d8/d1f/f44 [0,4194304] 0 2024-03-04T16:04:36.597 INFO:tasks.workunit.client.1.smithi152.stdout:3/391: read d6/d14/f7d [1559530,24888] 0 2024-03-04T16:04:36.601 INFO:tasks.workunit.client.1.smithi152.stdout:5/419: unlink d6/d50/f5c 0 2024-03-04T16:04:36.602 INFO:tasks.workunit.client.1.smithi152.stdout:4/282: write f0 [4539483,96189] 0 2024-03-04T16:04:36.603 INFO:tasks.workunit.client.1.smithi152.stdout:4/283: readlink d8/l1f 0 2024-03-04T16:04:36.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:36 smithi152 ceph-mon[36876]: pgmap v34: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 49 MiB/s rd, 193 MiB/s wr, 438 op/s 2024-03-04T16:04:36.608 INFO:tasks.workunit.client.1.smithi152.stdout:9/291: dread d2/d13/d23/f5b [0,4194304] 0 2024-03-04T16:04:36.611 INFO:tasks.workunit.client.1.smithi152.stdout:4/284: unlink d8/d2f/f5c 0 2024-03-04T16:04:36.616 INFO:tasks.workunit.client.1.smithi152.stdout:4/285: dwrite f7 [0,4194304] 0 2024-03-04T16:04:36.618 INFO:tasks.workunit.client.1.smithi152.stdout:4/286: write d8/d2f/f47 [484944,113150] 0 2024-03-04T16:04:36.619 INFO:tasks.workunit.client.1.smithi152.stdout:9/292: rmdir d2/d13 39 2024-03-04T16:04:36.625 INFO:tasks.workunit.client.1.smithi152.stdout:3/392: getdents d6/d14 0 2024-03-04T16:04:36.625 INFO:tasks.workunit.client.1.smithi152.stdout:4/287: fdatasync d8/f3e 0 2024-03-04T16:04:36.630 INFO:tasks.workunit.client.1.smithi152.stdout:5/420: dread d6/d8/d10/f54 [0,4194304] 0 2024-03-04T16:04:36.630 INFO:tasks.workunit.client.1.smithi152.stdout:3/393: creat d6/f85 x:0 0 0 2024-03-04T16:04:36.638 INFO:tasks.workunit.client.1.smithi152.stdout:4/288: dread f3 [0,4194304] 0 2024-03-04T16:04:36.639 INFO:tasks.workunit.client.1.smithi152.stdout:4/289: fsync d8/d9/d24/d26/d42/f4b 0 2024-03-04T16:04:36.640 INFO:tasks.workunit.client.1.smithi152.stdout:4/290: write d8/d9/d24/f4d [962025,39907] 0 2024-03-04T16:04:36.642 INFO:tasks.workunit.client.1.smithi152.stdout:4/291: dread d8/d2f/f47 [0,4194304] 0 2024-03-04T16:04:36.646 INFO:tasks.workunit.client.1.smithi152.stdout:4/292: dread d8/d9/d24/f4d [0,4194304] 0 2024-03-04T16:04:36.664 INFO:tasks.workunit.client.1.smithi152.stdout:6/433: sync 2024-03-04T16:04:36.664 INFO:tasks.workunit.client.1.smithi152.stdout:2/339: sync 2024-03-04T16:04:36.665 INFO:tasks.workunit.client.1.smithi152.stdout:3/394: link d6/d10/d2d/d59/f77 d6/d5c/f86 0 2024-03-04T16:04:36.667 INFO:tasks.workunit.client.1.smithi152.stdout:6/434: write d8/f22 [2046678,129288] 0 2024-03-04T16:04:36.673 INFO:tasks.workunit.client.1.smithi152.stdout:5/421: write d6/d8/f30 [4296543,92399] 0 2024-03-04T16:04:36.675 INFO:tasks.workunit.client.1.smithi152.stdout:2/340: symlink da/d2c/d5e/l6f 0 2024-03-04T16:04:36.681 INFO:tasks.workunit.client.1.smithi152.stdout:6/435: symlink d8/d42/l8c 0 2024-03-04T16:04:36.682 INFO:tasks.workunit.client.1.smithi152.stdout:4/293: symlink d8/d9/l5e 0 2024-03-04T16:04:36.683 INFO:tasks.workunit.client.1.smithi152.stdout:6/436: rmdir d8/d23/d3a/d6d 39 2024-03-04T16:04:36.686 INFO:tasks.workunit.client.1.smithi152.stdout:5/422: dwrite d6/fd [8388608,4194304] 0 2024-03-04T16:04:36.688 INFO:tasks.workunit.client.1.smithi152.stdout:5/423: read - d6/d8/d1f/d3d/d6f/d70/f71 zero size 2024-03-04T16:04:36.689 INFO:tasks.workunit.client.1.smithi152.stdout:4/294: creat d8/d3d/f5f x:0 0 0 2024-03-04T16:04:36.690 INFO:tasks.workunit.client.1.smithi152.stdout:3/395: dread d6/f1c [0,4194304] 0 2024-03-04T16:04:36.694 INFO:tasks.workunit.client.1.smithi152.stdout:6/437: mkdir d8/d23/d3a/d8d 0 2024-03-04T16:04:36.694 INFO:tasks.workunit.client.1.smithi152.stdout:3/396: dread d6/f3a [0,4194304] 0 2024-03-04T16:04:36.696 INFO:tasks.workunit.client.1.smithi152.stdout:5/424: creat d6/d8/d5a/d64/f96 x:0 0 0 2024-03-04T16:04:36.699 INFO:tasks.workunit.client.1.smithi152.stdout:4/295: creat d8/d9/f60 x:0 0 0 2024-03-04T16:04:36.699 INFO:tasks.workunit.client.1.smithi152.stdout:2/341: write da/df/d2e/f39 [35042,17064] 0 2024-03-04T16:04:36.700 INFO:tasks.workunit.client.1.smithi152.stdout:5/425: dread d6/f4e [0,4194304] 0 2024-03-04T16:04:36.700 INFO:tasks.workunit.client.1.smithi152.stdout:3/397: mknod d6/d10/d2d/d6e/c87 0 2024-03-04T16:04:36.703 INFO:tasks.workunit.client.1.smithi152.stdout:6/438: truncate d8/f12 1770391 0 2024-03-04T16:04:36.704 INFO:tasks.workunit.client.1.smithi152.stdout:4/296: symlink d8/d53/l61 0 2024-03-04T16:04:36.704 INFO:tasks.workunit.client.1.smithi152.stdout:5/426: mknod d6/d8/d5a/c97 0 2024-03-04T16:04:36.706 INFO:tasks.workunit.client.1.smithi152.stdout:2/342: mknod da/df/d2e/d61/c70 0 2024-03-04T16:04:36.707 INFO:tasks.workunit.client.1.smithi152.stdout:3/398: creat d6/d53/f88 x:0 0 0 2024-03-04T16:04:36.708 INFO:tasks.workunit.client.1.smithi152.stdout:2/343: truncate da/d24/f31 1121754 0 2024-03-04T16:04:36.708 INFO:tasks.workunit.client.1.smithi152.stdout:6/439: mknod d8/d52/d84/c8e 0 2024-03-04T16:04:36.709 INFO:tasks.workunit.client.1.smithi152.stdout:4/297: mknod d8/c62 0 2024-03-04T16:04:36.710 INFO:tasks.workunit.client.1.smithi152.stdout:4/298: dread - d8/d4e/f57 zero size 2024-03-04T16:04:36.710 INFO:tasks.workunit.client.1.smithi152.stdout:5/427: mkdir d6/d8/d1f/d3d/d6f/d70/d98 0 2024-03-04T16:04:36.712 INFO:tasks.workunit.client.1.smithi152.stdout:5/428: write d6/d8/f41 [528827,12201] 0 2024-03-04T16:04:36.712 INFO:tasks.workunit.client.1.smithi152.stdout:2/344: creat da/df/d35/d50/f71 x:0 0 0 2024-03-04T16:04:36.714 INFO:tasks.workunit.client.1.smithi152.stdout:5/429: dread - d6/d8/d5a/d64/d76/f81 zero size 2024-03-04T16:04:36.714 INFO:tasks.workunit.client.1.smithi152.stdout:4/299: symlink d8/d53/l63 0 2024-03-04T16:04:36.715 INFO:tasks.workunit.client.1.smithi152.stdout:5/430: readlink d6/d8/d1f/l79 0 2024-03-04T16:04:36.717 INFO:tasks.workunit.client.1.smithi152.stdout:5/431: chown d6/d8/d1f/l94 52 1 2024-03-04T16:04:36.717 INFO:tasks.workunit.client.1.smithi152.stdout:6/440: mkdir d8/d8f 0 2024-03-04T16:04:36.718 INFO:tasks.workunit.client.1.smithi152.stdout:3/399: dread d6/f9 [0,4194304] 0 2024-03-04T16:04:36.719 INFO:tasks.workunit.client.1.smithi152.stdout:3/400: chown d6/d14/d41/d49 153549588 1 2024-03-04T16:04:36.720 INFO:tasks.workunit.client.1.smithi152.stdout:4/300: rename d8/d3d/f4f to d8/d4e/f64 0 2024-03-04T16:04:36.723 INFO:tasks.workunit.client.1.smithi152.stdout:6/441: mknod d8/d10/c90 0 2024-03-04T16:04:36.727 INFO:tasks.workunit.client.1.smithi152.stdout:4/301: creat d8/f65 x:0 0 0 2024-03-04T16:04:36.730 INFO:tasks.workunit.client.1.smithi152.stdout:3/401: dwrite d6/d10/d2d/f32 [8388608,4194304] 0 2024-03-04T16:04:36.730 INFO:tasks.workunit.client.1.smithi152.stdout:2/345: dread da/f23 [0,4194304] 0 2024-03-04T16:04:36.731 INFO:tasks.workunit.client.1.smithi152.stdout:5/432: dwrite d6/f39 [0,4194304] 0 2024-03-04T16:04:36.732 INFO:tasks.workunit.client.1.smithi152.stdout:2/346: dread - da/d24/d47/f4d zero size 2024-03-04T16:04:36.738 INFO:tasks.workunit.client.1.smithi152.stdout:6/442: mkdir d8/d38/d59/d7d/d91 0 2024-03-04T16:04:36.743 INFO:tasks.workunit.client.1.smithi152.stdout:2/347: rename da/d2c/f41 to da/d2c/d3f/f72 0 2024-03-04T16:04:36.755 INFO:tasks.workunit.client.1.smithi152.stdout:5/433: creat d6/d8/d5a/d64/f99 x:0 0 0 2024-03-04T16:04:36.756 INFO:tasks.workunit.client.1.smithi152.stdout:5/434: dread - d6/d50/f7b zero size 2024-03-04T16:04:36.760 INFO:tasks.workunit.client.1.smithi152.stdout:5/435: dread d6/fd [8388608,4194304] 0 2024-03-04T16:04:36.764 INFO:tasks.workunit.client.1.smithi152.stdout:6/443: rename d8/d38 to d8/d92 0 2024-03-04T16:04:36.767 INFO:tasks.workunit.client.1.smithi152.stdout:2/348: getdents da/d2c 0 2024-03-04T16:04:36.773 INFO:tasks.workunit.client.1.smithi152.stdout:2/349: mkdir da/d24/d47/d73 0 2024-03-04T16:04:36.774 INFO:tasks.workunit.client.1.smithi152.stdout:6/444: dread d8/d2a/f3c [0,4194304] 0 2024-03-04T16:04:36.775 INFO:tasks.workunit.client.1.smithi152.stdout:6/445: chown d8/d52/d84/c8e 477061 1 2024-03-04T16:04:36.776 INFO:tasks.workunit.client.1.smithi152.stdout:6/446: mkdir d8/d23/d93 0 2024-03-04T16:04:36.777 INFO:tasks.workunit.client.1.smithi152.stdout:6/447: chown d8/d10/c3b 68071892 1 2024-03-04T16:04:36.786 INFO:tasks.workunit.client.1.smithi152.stdout:8/361: sync 2024-03-04T16:04:36.786 INFO:tasks.workunit.client.1.smithi152.stdout:7/390: sync 2024-03-04T16:04:36.788 INFO:tasks.workunit.client.1.smithi152.stdout:8/362: fsync d2/f37 0 2024-03-04T16:04:36.790 INFO:tasks.workunit.client.1.smithi152.stdout:6/448: dwrite d8/d2a/f3c [0,4194304] 0 2024-03-04T16:04:36.795 INFO:tasks.workunit.client.1.smithi152.stdout:7/391: truncate d14/d21/d31/f64 1854893 0 2024-03-04T16:04:36.798 INFO:tasks.workunit.client.1.smithi152.stdout:8/363: rename d2/d3/d21/d2d/d31/d66 to d2/d3/d21/d2d/d52/d68/d6e 0 2024-03-04T16:04:36.798 INFO:tasks.workunit.client.1.smithi152.stdout:8/364: fdatasync d2/d8/f1a 0 2024-03-04T16:04:36.799 INFO:tasks.workunit.client.1.smithi152.stdout:6/449: symlink d8/d42/l94 0 2024-03-04T16:04:36.800 INFO:tasks.workunit.client.1.smithi152.stdout:8/365: write d2/d8/dd/d22/d26/f38 [5014447,12483] 0 2024-03-04T16:04:36.806 INFO:tasks.workunit.client.1.smithi152.stdout:8/366: creat d2/d8/dd/d22/d49/f6f x:0 0 0 2024-03-04T16:04:36.809 INFO:tasks.workunit.client.1.smithi152.stdout:6/450: link f7 d8/d52/d84/f95 0 2024-03-04T16:04:36.811 INFO:tasks.workunit.client.1.smithi152.stdout:7/392: rename d14/c1a to d14/c75 0 2024-03-04T16:04:36.812 INFO:tasks.workunit.client.1.smithi152.stdout:8/367: dwrite d2/d3/d15/d1c/f1d [0,4194304] 0 2024-03-04T16:04:36.814 INFO:tasks.workunit.client.1.smithi152.stdout:8/368: write d2/d3/d21/f5f [4258721,18687] 0 2024-03-04T16:04:36.816 INFO:tasks.workunit.client.1.smithi152.stdout:7/393: dread d14/d21/d48/f4b [0,4194304] 0 2024-03-04T16:04:36.819 INFO:tasks.workunit.client.1.smithi152.stdout:6/451: write d8/d52/d84/f95 [8708124,43524] 0 2024-03-04T16:04:36.819 INFO:tasks.workunit.client.1.smithi152.stdout:6/452: stat d8/d5f/f77 0 2024-03-04T16:04:36.821 INFO:tasks.workunit.client.1.smithi152.stdout:8/369: dwrite d2/d3/d21/d2d/f1e [0,4194304] 0 2024-03-04T16:04:36.825 INFO:tasks.workunit.client.1.smithi152.stdout:6/453: dread d8/d2a/f3c [0,4194304] 0 2024-03-04T16:04:36.825 INFO:tasks.workunit.client.1.smithi152.stdout:6/454: fdatasync d8/d10/f18 0 2024-03-04T16:04:36.907 INFO:tasks.workunit.client.1.smithi152.stdout:7/394: dwrite d14/d21/d47/d40/f6c [0,4194304] 0 2024-03-04T16:04:36.911 INFO:tasks.workunit.client.1.smithi152.stdout:8/370: creat d2/d8/dd/d22/d6c/f70 x:0 0 0 2024-03-04T16:04:36.914 INFO:tasks.workunit.client.1.smithi152.stdout:7/395: fsync d14/f2a 0 2024-03-04T16:04:36.917 INFO:tasks.workunit.client.1.smithi152.stdout:7/396: dread d14/d21/d47/f51 [0,4194304] 0 2024-03-04T16:04:36.919 INFO:tasks.workunit.client.1.smithi152.stdout:8/371: dwrite d2/d8/dd/d22/d49/f6f [0,4194304] 0 2024-03-04T16:04:36.924 INFO:tasks.workunit.client.1.smithi152.stdout:6/455: write d8/d23/d3a/f3f [1361252,5297] 0 2024-03-04T16:04:36.931 INFO:tasks.workunit.client.1.smithi152.stdout:7/397: rename d14/d21/d31/f3e to d14/d21/d47/f76 0 2024-03-04T16:04:36.932 INFO:tasks.workunit.client.1.smithi152.stdout:6/456: mkdir d8/d23/d3a/d8d/d96 0 2024-03-04T16:04:36.933 INFO:tasks.workunit.client.1.smithi152.stdout:6/457: readlink d8/d42/l62 0 2024-03-04T16:04:36.935 INFO:tasks.workunit.client.1.smithi152.stdout:8/372: dread d2/d3/d21/d2d/fc [0,4194304] 0 2024-03-04T16:04:36.936 INFO:tasks.workunit.client.1.smithi152.stdout:7/398: symlink d14/d21/d47/d40/l77 0 2024-03-04T16:04:36.937 INFO:tasks.workunit.client.1.smithi152.stdout:8/373: write d2/d3/d15/f6a [67719,56157] 0 2024-03-04T16:04:36.938 INFO:tasks.workunit.client.1.smithi152.stdout:7/399: write d14/d21/d47/d40/f6c [1933112,122002] 0 2024-03-04T16:04:36.938 INFO:tasks.workunit.client.1.smithi152.stdout:6/458: creat d8/d23/d3a/d8d/d96/f97 x:0 0 0 2024-03-04T16:04:36.939 INFO:tasks.workunit.client.1.smithi152.stdout:8/374: chown d2/d3/d21/d2d/d48 7077314 1 2024-03-04T16:04:36.940 INFO:tasks.workunit.client.1.smithi152.stdout:6/459: write d8/d2a/f79 [923325,65517] 0 2024-03-04T16:04:36.941 INFO:tasks.workunit.client.1.smithi152.stdout:6/460: read - d8/d23/d3a/f8b zero size 2024-03-04T16:04:36.941 INFO:tasks.workunit.client.1.smithi152.stdout:6/461: fdatasync d8/d42/f51 0 2024-03-04T16:04:36.943 INFO:tasks.workunit.client.1.smithi152.stdout:8/375: unlink d2/d3/c62 0 2024-03-04T16:04:36.945 INFO:tasks.workunit.client.1.smithi152.stdout:8/376: write d2/d3/d15/d1c/f1d [492270,2457] 0 2024-03-04T16:04:36.948 INFO:tasks.workunit.client.1.smithi152.stdout:6/462: dwrite d8/d10/f44 [0,4194304] 0 2024-03-04T16:04:36.949 INFO:tasks.workunit.client.1.smithi152.stdout:6/463: chown d8/d2a 67661550 1 2024-03-04T16:04:36.950 INFO:tasks.workunit.client.1.smithi152.stdout:1/407: sync 2024-03-04T16:04:36.952 INFO:tasks.workunit.client.1.smithi152.stdout:6/464: mkdir d8/d23/d98 0 2024-03-04T16:04:36.953 INFO:tasks.workunit.client.1.smithi152.stdout:8/377: dwrite d2/d3/d21/d2d/fc [4194304,4194304] 0 2024-03-04T16:04:36.962 INFO:tasks.workunit.client.1.smithi152.stdout:6/465: mkdir d8/d23/d93/d99 0 2024-03-04T16:04:36.965 INFO:tasks.workunit.client.1.smithi152.stdout:6/466: mkdir d8/d10/d53/d9a 0 2024-03-04T16:04:36.966 INFO:tasks.workunit.client.1.smithi152.stdout:8/378: dread d2/f42 [0,4194304] 0 2024-03-04T16:04:36.967 INFO:tasks.workunit.client.1.smithi152.stdout:6/467: write d8/d23/d3a/d8d/d96/f97 [309895,106163] 0 2024-03-04T16:04:36.971 INFO:tasks.workunit.client.1.smithi152.stdout:8/379: symlink d2/d3/l71 0 2024-03-04T16:04:36.972 INFO:tasks.workunit.client.1.smithi152.stdout:6/468: creat d8/d23/d93/d99/f9b x:0 0 0 2024-03-04T16:04:36.973 INFO:tasks.workunit.client.1.smithi152.stdout:6/469: truncate d8/f39 1990213 0 2024-03-04T16:04:36.973 INFO:tasks.workunit.client.1.smithi152.stdout:8/380: creat d2/d3/d21/d2d/d52/d68/f72 x:0 0 0 2024-03-04T16:04:36.974 INFO:tasks.workunit.client.1.smithi152.stdout:7/400: write d14/d21/d47/d36/f73 [4559678,113498] 0 2024-03-04T16:04:36.974 INFO:tasks.workunit.client.1.smithi152.stdout:7/401: stat d14/f42 0 2024-03-04T16:04:36.978 INFO:tasks.workunit.client.1.smithi152.stdout:7/402: dread d14/f20 [0,4194304] 0 2024-03-04T16:04:36.986 INFO:tasks.workunit.client.1.smithi152.stdout:7/403: rename d14/f20 to d14/f78 0 2024-03-04T16:04:36.988 INFO:tasks.workunit.client.1.smithi152.stdout:7/404: mknod d14/d21/d31/c79 0 2024-03-04T16:04:36.996 INFO:tasks.workunit.client.1.smithi152.stdout:1/408: dwrite d19/d3b/f49 [0,4194304] 0 2024-03-04T16:04:37.001 INFO:tasks.workunit.client.1.smithi152.stdout:1/409: creat d19/d4b/d53/d57/d62/d82/f86 x:0 0 0 2024-03-04T16:04:37.006 INFO:tasks.workunit.client.1.smithi152.stdout:1/410: unlink d19/d3b/f52 0 2024-03-04T16:04:37.007 INFO:tasks.workunit.client.1.smithi152.stdout:6/470: write d8/d2a/f36 [5767349,7799] 0 2024-03-04T16:04:37.010 INFO:tasks.workunit.client.1.smithi152.stdout:6/471: symlink d8/d42/l9c 0 2024-03-04T16:04:37.011 INFO:tasks.workunit.client.1.smithi152.stdout:6/472: truncate d8/f56 513467 0 2024-03-04T16:04:37.014 INFO:tasks.workunit.client.1.smithi152.stdout:8/381: truncate d2/d8/f4e 2753591 0 2024-03-04T16:04:37.016 INFO:tasks.workunit.client.1.smithi152.stdout:6/473: mkdir d8/d23/d3a/d71/d9d 0 2024-03-04T16:04:37.019 INFO:tasks.workunit.client.1.smithi152.stdout:7/405: rmdir d14 39 2024-03-04T16:04:37.019 INFO:tasks.workunit.client.1.smithi152.stdout:8/382: creat d2/d8/f73 x:0 0 0 2024-03-04T16:04:37.021 INFO:tasks.workunit.client.1.smithi152.stdout:8/383: chown d2/d8/dd/d22/d26/f58 534937925 1 2024-03-04T16:04:37.030 INFO:tasks.workunit.client.1.smithi152.stdout:8/384: dwrite d2/d3/d15/d1c/f1d [4194304,4194304] 0 2024-03-04T16:04:37.031 INFO:tasks.workunit.client.1.smithi152.stdout:8/385: readlink d2/d3/l71 0 2024-03-04T16:04:37.038 INFO:tasks.workunit.client.1.smithi152.stdout:8/386: getdents d2/d3/d21/d2d/d48 0 2024-03-04T16:04:37.041 INFO:tasks.workunit.client.1.smithi152.stdout:8/387: creat d2/d8/dd/d22/d6c/f74 x:0 0 0 2024-03-04T16:04:37.049 INFO:tasks.workunit.client.1.smithi152.stdout:8/388: dwrite d2/d3/d15/f6b [0,4194304] 0 2024-03-04T16:04:37.058 INFO:tasks.workunit.client.1.smithi152.stdout:8/389: dwrite d2/d8/dd/d22/d26/f58 [0,4194304] 0 2024-03-04T16:04:37.080 INFO:tasks.workunit.client.1.smithi152.stdout:8/390: dwrite d2/f37 [0,4194304] 0 2024-03-04T16:04:37.085 INFO:tasks.workunit.client.1.smithi152.stdout:8/391: mkdir d2/d75 0 2024-03-04T16:04:37.087 INFO:tasks.workunit.client.1.smithi152.stdout:8/392: read d2/d8/dd/d22/d49/d5d/f44 [425253,37489] 0 2024-03-04T16:04:37.092 INFO:tasks.workunit.client.1.smithi152.stdout:8/393: dread d2/d8/f4e [0,4194304] 0 2024-03-04T16:04:37.094 INFO:tasks.workunit.client.1.smithi152.stdout:8/394: truncate d2/d3/d21/f1f 3578353 0 2024-03-04T16:04:37.096 INFO:tasks.workunit.client.1.smithi152.stdout:8/395: mknod d2/d8/c76 0 2024-03-04T16:04:37.097 INFO:tasks.workunit.client.1.smithi152.stdout:8/396: truncate d2/d3/d15/f6b 4402414 0 2024-03-04T16:04:37.099 INFO:tasks.workunit.client.1.smithi152.stdout:8/397: read - d2/d8/dd/d22/d49/f55 zero size 2024-03-04T16:04:37.099 INFO:tasks.workunit.client.1.smithi152.stdout:8/398: stat d2/d3/f4b 0 2024-03-04T16:04:37.099 INFO:tasks.workunit.client.1.smithi152.stdout:8/399: chown d2/d8/dd/d22/f3b 0 1 2024-03-04T16:04:37.100 INFO:tasks.workunit.client.1.smithi152.stdout:8/400: fdatasync d2/d8/f14 0 2024-03-04T16:04:37.114 INFO:tasks.workunit.client.1.smithi152.stdout:8/401: symlink d2/d3/d21/d2d/d31/l77 0 2024-03-04T16:04:37.118 INFO:tasks.workunit.client.1.smithi152.stdout:8/402: creat d2/d8/dd/d22/f78 x:0 0 0 2024-03-04T16:04:37.125 INFO:tasks.workunit.client.1.smithi152.stdout:8/403: dwrite d2/d3/f40 [0,4194304] 0 2024-03-04T16:04:37.126 INFO:tasks.workunit.client.1.smithi152.stdout:8/404: read - d2/d8/f73 zero size 2024-03-04T16:04:37.128 INFO:tasks.workunit.client.1.smithi152.stdout:0/334: sync 2024-03-04T16:04:37.130 INFO:tasks.workunit.client.1.smithi152.stdout:0/335: mknod d9/d1b/d54/c68 0 2024-03-04T16:04:37.131 INFO:tasks.workunit.client.1.smithi152.stdout:0/336: readlink d9/lc 0 2024-03-04T16:04:37.288 INFO:tasks.workunit.client.1.smithi152.stdout:4/302: sync 2024-03-04T16:04:37.288 INFO:tasks.workunit.client.1.smithi152.stdout:9/293: sync 2024-03-04T16:04:37.297 INFO:tasks.workunit.client.1.smithi152.stdout:9/294: creat d2/d13/d35/f5d x:0 0 0 2024-03-04T16:04:37.315 INFO:tasks.workunit.client.1.smithi152.stdout:4/303: creat d8/d9/d24/f66 x:0 0 0 2024-03-04T16:04:37.317 INFO:tasks.workunit.client.1.smithi152.stdout:4/304: mkdir d8/d2f/d67 0 2024-03-04T16:04:37.318 INFO:tasks.workunit.client.1.smithi152.stdout:9/295: dread d2/d13/d23/f4e [0,4194304] 0 2024-03-04T16:04:37.319 INFO:tasks.workunit.client.1.smithi152.stdout:4/305: creat d8/d3d/f68 x:0 0 0 2024-03-04T16:04:37.319 INFO:tasks.workunit.client.1.smithi152.stdout:9/296: read - d2/d13/d38/f58 zero size 2024-03-04T16:04:37.320 INFO:tasks.workunit.client.1.smithi152.stdout:4/306: mknod d8/d58/c69 0 2024-03-04T16:04:37.321 INFO:tasks.workunit.client.1.smithi152.stdout:4/307: stat d8/d53 0 2024-03-04T16:04:37.324 INFO:tasks.workunit.client.1.smithi152.stdout:4/308: symlink d8/d9/d24/l6a 0 2024-03-04T16:04:37.325 INFO:tasks.workunit.client.1.smithi152.stdout:4/309: fdatasync d8/f65 0 2024-03-04T16:04:37.326 INFO:tasks.workunit.client.1.smithi152.stdout:4/310: write d8/f45 [216040,54386] 0 2024-03-04T16:04:37.326 INFO:tasks.workunit.client.1.smithi152.stdout:9/297: dwrite d2/d13/d23/f4e [0,4194304] 0 2024-03-04T16:04:37.327 INFO:tasks.workunit.client.1.smithi152.stdout:4/311: chown d8/d9/d24/f4d 3442668 1 2024-03-04T16:04:37.335 INFO:tasks.workunit.client.1.smithi152.stdout:9/298: getdents d2/d13 0 2024-03-04T16:04:37.336 INFO:tasks.workunit.client.1.smithi152.stdout:9/299: write d2/d13/f1a [1936045,254] 0 2024-03-04T16:04:37.338 INFO:tasks.workunit.client.1.smithi152.stdout:9/300: truncate d2/d13/d38/f58 373708 0 2024-03-04T16:04:37.339 INFO:tasks.workunit.client.1.smithi152.stdout:9/301: chown d2/d13/d23/c32 0 1 2024-03-04T16:04:37.360 INFO:tasks.workunit.client.1.smithi152.stdout:4/312: getdents d8/d58 0 2024-03-04T16:04:37.364 INFO:tasks.workunit.client.1.smithi152.stdout:4/313: dread d8/f25 [0,4194304] 0 2024-03-04T16:04:37.369 INFO:tasks.workunit.client.1.smithi152.stdout:4/314: dwrite d8/d2f/f33 [0,4194304] 0 2024-03-04T16:04:37.370 INFO:tasks.workunit.client.1.smithi152.stdout:4/315: chown d8/d9/f28 3397177 1 2024-03-04T16:04:37.374 INFO:tasks.workunit.client.1.smithi152.stdout:4/316: creat d8/d9/d24/d26/d42/f6b x:0 0 0 2024-03-04T16:04:37.376 INFO:tasks.workunit.client.1.smithi152.stdout:4/317: mknod d8/d2f/d67/c6c 0 2024-03-04T16:04:37.378 INFO:tasks.workunit.client.1.smithi152.stdout:9/302: truncate d2/d13/d23/f4e 767143 0 2024-03-04T16:04:37.379 INFO:tasks.workunit.client.1.smithi152.stdout:9/303: write d2/f19 [1712491,77757] 0 2024-03-04T16:04:37.381 INFO:tasks.workunit.client.1.smithi152.stdout:4/318: dread d8/f25 [0,4194304] 0 2024-03-04T16:04:37.382 INFO:tasks.workunit.client.1.smithi152.stdout:4/319: write f0 [5630328,97769] 0 2024-03-04T16:04:37.385 INFO:tasks.workunit.client.1.smithi152.stdout:9/304: getdents d2 0 2024-03-04T16:04:37.389 INFO:tasks.workunit.client.1.smithi152.stdout:9/305: rename d2/d13/d35/l53 to d2/d13/d38/l5e 0 2024-03-04T16:04:37.390 INFO:tasks.workunit.client.1.smithi152.stdout:4/320: dwrite d8/d4e/f64 [0,4194304] 0 2024-03-04T16:04:37.396 INFO:tasks.workunit.client.1.smithi152.stdout:4/321: dwrite f5 [0,4194304] 0 2024-03-04T16:04:37.400 INFO:tasks.workunit.client.1.smithi152.stdout:3/402: sync 2024-03-04T16:04:37.400 INFO:tasks.workunit.client.1.smithi152.stdout:5/436: sync 2024-03-04T16:04:37.402 INFO:tasks.workunit.client.1.smithi152.stdout:3/403: stat d6/d10 0 2024-03-04T16:04:37.403 INFO:tasks.workunit.client.1.smithi152.stdout:3/404: fdatasync d6/f85 0 2024-03-04T16:04:37.403 INFO:tasks.workunit.client.1.smithi152.stdout:4/322: dread d8/f25 [0,4194304] 0 2024-03-04T16:04:37.406 INFO:tasks.workunit.client.1.smithi152.stdout:4/323: truncate d8/d9/d24/d26/d42/f6b 959054 0 2024-03-04T16:04:37.407 INFO:tasks.workunit.client.1.smithi152.stdout:4/324: chown d8/d53/c54 395541182 1 2024-03-04T16:04:37.407 INFO:tasks.workunit.client.1.smithi152.stdout:3/405: creat d6/d10/d2d/d59/f89 x:0 0 0 2024-03-04T16:04:37.408 INFO:tasks.workunit.client.1.smithi152.stdout:5/437: link d6/d8/d5a/d64/d76/f7d d6/d8/f9a 0 2024-03-04T16:04:37.409 INFO:tasks.workunit.client.1.smithi152.stdout:3/406: mkdir d6/d60/d8a 0 2024-03-04T16:04:37.412 INFO:tasks.workunit.client.1.smithi152.stdout:4/325: link d8/d3d/c51 d8/d9/d24/d26/d42/c6d 0 2024-03-04T16:04:37.413 INFO:tasks.workunit.client.1.smithi152.stdout:4/326: fdatasync d8/f10 0 2024-03-04T16:04:37.414 INFO:tasks.workunit.client.1.smithi152.stdout:4/327: readlink d8/d9/d24/d26/l31 0 2024-03-04T16:04:37.415 INFO:tasks.workunit.client.1.smithi152.stdout:3/407: read d6/d14/d58/f6b [2486671,101109] 0 2024-03-04T16:04:37.416 INFO:tasks.workunit.client.1.smithi152.stdout:3/408: chown d6/f1c 1783 1 2024-03-04T16:04:37.419 INFO:tasks.workunit.client.1.smithi152.stdout:3/409: mknod d6/d5c/c8b 0 2024-03-04T16:04:37.420 INFO:tasks.workunit.client.1.smithi152.stdout:4/328: dwrite f7 [0,4194304] 0 2024-03-04T16:04:37.421 INFO:tasks.workunit.client.1.smithi152.stdout:9/306: dread d2/f28 [0,4194304] 0 2024-03-04T16:04:37.424 INFO:tasks.workunit.client.1.smithi152.stdout:3/410: symlink d6/d5c/l8c 0 2024-03-04T16:04:37.430 INFO:tasks.workunit.client.1.smithi152.stdout:3/411: creat d6/d14/f8d x:0 0 0 2024-03-04T16:04:37.433 INFO:tasks.workunit.client.1.smithi152.stdout:9/307: truncate d2/fd 3642594 0 2024-03-04T16:04:37.435 INFO:tasks.workunit.client.1.smithi152.stdout:3/412: mkdir d6/d60/d8e 0 2024-03-04T16:04:37.436 INFO:tasks.workunit.client.1.smithi152.stdout:3/413: rmdir d6/d10/d2d/d6e 39 2024-03-04T16:04:37.437 INFO:tasks.workunit.client.1.smithi152.stdout:3/414: read - d6/d60/f67 zero size 2024-03-04T16:04:37.437 INFO:tasks.workunit.client.1.smithi152.stdout:5/438: mknod d6/d8/d1f/d88/c9b 0 2024-03-04T16:04:37.443 INFO:tasks.workunit.client.1.smithi152.stdout:5/439: write d6/d8/f4b [718398,55551] 0 2024-03-04T16:04:37.444 INFO:tasks.workunit.client.1.smithi152.stdout:3/415: dwrite d6/f68 [0,4194304] 0 2024-03-04T16:04:37.445 INFO:tasks.workunit.client.1.smithi152.stdout:5/440: stat d6/d8/d1f/f57 0 2024-03-04T16:04:37.446 INFO:tasks.workunit.client.1.smithi152.stdout:3/416: truncate d6/d53/f88 1029854 0 2024-03-04T16:04:37.446 INFO:tasks.workunit.client.1.smithi152.stdout:3/417: dread - d6/f85 zero size 2024-03-04T16:04:37.447 INFO:tasks.workunit.client.1.smithi152.stdout:3/418: stat d6/d10/d2d/c76 0 2024-03-04T16:04:37.451 INFO:tasks.workunit.client.1.smithi152.stdout:3/419: creat d6/d53/f8f x:0 0 0 2024-03-04T16:04:37.458 INFO:tasks.workunit.client.1.smithi152.stdout:4/329: dwrite f3 [0,4194304] 0 2024-03-04T16:04:37.464 INFO:tasks.workunit.client.1.smithi152.stdout:4/330: rename d8/l27 to d8/d9/d24/d26/d42/l6e 0 2024-03-04T16:04:37.465 INFO:tasks.workunit.client.1.smithi152.stdout:9/308: truncate d2/f19 2733174 0 2024-03-04T16:04:37.465 INFO:tasks.workunit.client.1.smithi152.stdout:3/420: dread d6/d10/d2d/d4d/f55 [0,4194304] 0 2024-03-04T16:04:37.466 INFO:tasks.workunit.client.1.smithi152.stdout:3/421: fsync d6/d53/f88 0 2024-03-04T16:04:37.467 INFO:tasks.workunit.client.1.smithi152.stdout:9/309: write d2/d13/d23/f33 [5000682,84749] 0 2024-03-04T16:04:37.468 INFO:tasks.workunit.client.1.smithi152.stdout:3/422: write d6/d14/d41/d49/f62 [2354585,99753] 0 2024-03-04T16:04:37.475 INFO:tasks.workunit.client.1.smithi152.stdout:5/441: dwrite d6/d50/f7e [0,4194304] 0 2024-03-04T16:04:37.479 INFO:tasks.workunit.client.1.smithi152.stdout:9/310: unlink d2/c16 0 2024-03-04T16:04:37.480 INFO:tasks.workunit.client.1.smithi152.stdout:9/311: readlink d2/d13/d23/l39 0 2024-03-04T16:04:37.492 INFO:tasks.workunit.client.1.smithi152.stdout:3/423: link d6/d10/f1e d6/d14/f90 0 2024-03-04T16:04:37.493 INFO:tasks.workunit.client.1.smithi152.stdout:4/331: dread d8/d3d/f44 [0,4194304] 0 2024-03-04T16:04:37.497 INFO:tasks.workunit.client.1.smithi152.stdout:9/312: rename d2/f1c to d2/d13/d23/f5f 0 2024-03-04T16:04:37.499 INFO:tasks.workunit.client.1.smithi152.stdout:5/442: write d6/d50/f52 [1173400,15971] 0 2024-03-04T16:04:37.501 INFO:tasks.workunit.client.1.smithi152.stdout:4/332: dwrite d8/d9/d24/d26/d42/f6b [0,4194304] 0 2024-03-04T16:04:37.501 INFO:tasks.workunit.client.1.smithi152.stdout:3/424: rmdir d6/d60 39 2024-03-04T16:04:37.503 INFO:tasks.workunit.client.1.smithi152.stdout:4/333: write d8/d9/ff [7651137,115402] 0 2024-03-04T16:04:37.512 INFO:tasks.workunit.client.1.smithi152.stdout:5/443: write d6/d8/f24 [4721085,57267] 0 2024-03-04T16:04:37.514 INFO:tasks.workunit.client.1.smithi152.stdout:4/334: mkdir d8/d2f/d6f 0 2024-03-04T16:04:37.515 INFO:tasks.workunit.client.1.smithi152.stdout:4/335: chown d8/d2f/f47 2 1 2024-03-04T16:04:37.518 INFO:tasks.workunit.client.1.smithi152.stdout:4/336: rename d8/c1e to d8/d9/d24/d26/d42/c70 0 2024-03-04T16:04:37.519 INFO:tasks.workunit.client.1.smithi152.stdout:3/425: link d6/c35 d6/d10/c91 0 2024-03-04T16:04:37.520 INFO:tasks.workunit.client.1.smithi152.stdout:3/426: chown d6/d10/c19 196930 1 2024-03-04T16:04:37.525 INFO:tasks.workunit.client.1.smithi152.stdout:3/427: dwrite d6/d10/f27 [0,4194304] 0 2024-03-04T16:04:37.530 INFO:tasks.workunit.client.1.smithi152.stdout:5/444: rename d6/d8/c1b to d6/d8/d5a/d90/c9c 0 2024-03-04T16:04:37.533 INFO:tasks.workunit.client.1.smithi152.stdout:3/428: mknod d6/d10/d2d/d59/c92 0 2024-03-04T16:04:37.533 INFO:tasks.workunit.client.1.smithi152.stdout:9/313: write d2/f19 [1496672,3426] 0 2024-03-04T16:04:37.536 INFO:tasks.workunit.client.1.smithi152.stdout:3/429: unlink d6/d14/c28 0 2024-03-04T16:04:37.537 INFO:tasks.workunit.client.1.smithi152.stdout:3/430: write d6/d10/d2d/f34 [4521715,98307] 0 2024-03-04T16:04:37.538 INFO:tasks.workunit.client.1.smithi152.stdout:2/350: sync 2024-03-04T16:04:37.538 INFO:tasks.workunit.client.1.smithi152.stdout:6/474: sync 2024-03-04T16:04:37.538 INFO:tasks.workunit.client.1.smithi152.stdout:1/411: sync 2024-03-04T16:04:37.540 INFO:tasks.workunit.client.1.smithi152.stdout:9/314: readlink d2/d13/d23/l39 0 2024-03-04T16:04:37.541 INFO:tasks.workunit.client.1.smithi152.stdout:9/315: fdatasync d2/d13/d2e/f44 0 2024-03-04T16:04:37.565 INFO:tasks.workunit.client.1.smithi152.stdout:1/412: rmdir d19/d20 39 2024-03-04T16:04:37.566 INFO:tasks.workunit.client.1.smithi152.stdout:2/351: symlink da/df/d2e/d61/l74 0 2024-03-04T16:04:37.567 INFO:tasks.workunit.client.1.smithi152.stdout:2/352: fsync da/d24/f65 0 2024-03-04T16:04:37.569 INFO:tasks.workunit.client.1.smithi152.stdout:3/431: rename d6/d10/d2d/d59 to d6/d14/d93 0 2024-03-04T16:04:37.572 INFO:tasks.workunit.client.1.smithi152.stdout:6/475: dwrite d8/d2a/f34 [0,4194304] 0 2024-03-04T16:04:37.574 INFO:tasks.workunit.client.1.smithi152.stdout:1/413: unlink d19/d4b/d53/l74 0 2024-03-04T16:04:37.577 INFO:tasks.workunit.client.1.smithi152.stdout:2/353: creat da/d2c/f75 x:0 0 0 2024-03-04T16:04:37.577 INFO:tasks.workunit.client.1.smithi152.stdout:2/354: dread - da/f3c zero size 2024-03-04T16:04:37.579 INFO:tasks.workunit.client.1.smithi152.stdout:3/432: unlink d6/d10/d2d/d4d/l5d 0 2024-03-04T16:04:37.580 INFO:tasks.workunit.client.1.smithi152.stdout:3/433: truncate d6/d53/f8f 386948 0 2024-03-04T16:04:37.581 INFO:tasks.workunit.client.1.smithi152.stdout:6/476: symlink d8/d23/d3a/d8d/d96/l9e 0 2024-03-04T16:04:37.582 INFO:tasks.workunit.client.1.smithi152.stdout:1/414: fsync d19/d4b/f51 0 2024-03-04T16:04:37.589 INFO:tasks.workunit.client.1.smithi152.stdout:9/316: write d2/fd [1710935,54618] 0 2024-03-04T16:04:37.589 INFO:tasks.workunit.client.1.smithi152.stdout:9/317: stat d2/d13/d2e 0 2024-03-04T16:04:37.591 INFO:tasks.workunit.client.1.smithi152.stdout:2/355: dwrite da/f23 [0,4194304] 0 2024-03-04T16:04:37.593 INFO:tasks.workunit.client.1.smithi152.stdout:6/477: rename d8/d5f to d8/d23/d98/d9f 0 2024-03-04T16:04:37.602 INFO:tasks.workunit.client.1.smithi152.stdout:6/478: mknod d8/d23/d3a/d71/ca0 0 2024-03-04T16:04:37.602 INFO:tasks.workunit.client.1.smithi152.stdout:6/479: fsync d8/f56 0 2024-03-04T16:04:37.604 INFO:tasks.workunit.client.1.smithi152.stdout:5/445: link d6/d8/d1f/l79 d6/d8/d5a/l9d 0 2024-03-04T16:04:37.608 INFO:tasks.workunit.client.1.smithi152.stdout:2/356: dread da/d24/f42 [0,4194304] 0 2024-03-04T16:04:37.609 INFO:tasks.workunit.client.1.smithi152.stdout:9/318: dread d2/f2d [0,4194304] 0 2024-03-04T16:04:37.610 INFO:tasks.workunit.client.1.smithi152.stdout:9/319: chown d2/l30 1576656729 1 2024-03-04T16:04:37.610 INFO:tasks.workunit.client.1.smithi152.stdout:2/357: write da/f45 [4561898,118220] 0 2024-03-04T16:04:37.610 INFO:tasks.workunit.client.1.smithi152.stdout:6/480: creat d8/d2a/fa1 x:0 0 0 2024-03-04T16:04:37.612 INFO:tasks.workunit.client.1.smithi152.stdout:3/434: link d6/d10/d2d/c2f d6/d60/c94 0 2024-03-04T16:04:37.612 INFO:tasks.workunit.client.1.smithi152.stdout:5/446: write d6/d8/f24 [1541234,122553] 0 2024-03-04T16:04:37.612 INFO:tasks.workunit.client.1.smithi152.stdout:6/481: write d8/d92/d59/f7c [104512,116771] 0 2024-03-04T16:04:37.613 INFO:tasks.workunit.client.1.smithi152.stdout:3/435: readlink d6/d60/l61 0 2024-03-04T16:04:37.614 INFO:tasks.workunit.client.1.smithi152.stdout:3/436: fsync d6/d10/f21 0 2024-03-04T16:04:37.614 INFO:tasks.workunit.client.1.smithi152.stdout:5/447: chown d6/d8/d1f/d88/c9b 9222071 1 2024-03-04T16:04:37.616 INFO:tasks.workunit.client.1.smithi152.stdout:2/358: unlink da/c6a 0 2024-03-04T16:04:37.622 INFO:tasks.workunit.client.1.smithi152.stdout:2/359: dwrite da/df/d2e/d51/f5b [0,4194304] 0 2024-03-04T16:04:37.623 INFO:tasks.workunit.client.1.smithi152.stdout:6/482: dread - d8/d23/d3a/d6d/f82 zero size 2024-03-04T16:04:37.626 INFO:tasks.workunit.client.1.smithi152.stdout:1/415: link d19/d20/l58 d19/d4b/l87 0 2024-03-04T16:04:37.629 INFO:tasks.workunit.client.1.smithi152.stdout:3/437: creat d6/d14/d41/d49/f95 x:0 0 0 2024-03-04T16:04:37.630 INFO:tasks.workunit.client.1.smithi152.stdout:3/438: chown d6/d53/f88 119 1 2024-03-04T16:04:37.630 INFO:tasks.workunit.client.1.smithi152.stdout:9/320: symlink d2/d13/d23/l60 0 2024-03-04T16:04:37.631 INFO:tasks.workunit.client.1.smithi152.stdout:9/321: fsync d2/d13/d35/f5d 0 2024-03-04T16:04:37.632 INFO:tasks.workunit.client.1.smithi152.stdout:9/322: write d2/d13/f52 [768109,50493] 0 2024-03-04T16:04:37.633 INFO:tasks.workunit.client.1.smithi152.stdout:9/323: readlink d2/l30 0 2024-03-04T16:04:37.639 INFO:tasks.workunit.client.1.smithi152.stdout:2/360: symlink da/df/d35/d50/l76 0 2024-03-04T16:04:37.639 INFO:tasks.workunit.client.1.smithi152.stdout:6/483: dread f7 [4194304,4194304] 0 2024-03-04T16:04:37.640 INFO:tasks.workunit.client.1.smithi152.stdout:6/484: chown d8/d10/f18 1660804208 1 2024-03-04T16:04:37.642 INFO:tasks.workunit.client.1.smithi152.stdout:1/416: chown d19/d4b/d53/d57/d85/l6f 38966 1 2024-03-04T16:04:37.643 INFO:tasks.workunit.client.1.smithi152.stdout:5/448: symlink d6/d8/d10/l9e 0 2024-03-04T16:04:37.645 INFO:tasks.workunit.client.1.smithi152.stdout:9/324: dread - d2/d13/d23/f5c zero size 2024-03-04T16:04:37.652 INFO:tasks.workunit.client.1.smithi152.stdout:6/485: dwrite d8/f28 [0,4194304] 0 2024-03-04T16:04:37.653 INFO:tasks.workunit.client.1.smithi152.stdout:6/486: chown d8/d23/d3a/d8d 13424602 1 2024-03-04T16:04:37.654 INFO:tasks.workunit.client.1.smithi152.stdout:6/487: chown d8/d23/l7b 5331 1 2024-03-04T16:04:37.657 INFO:tasks.workunit.client.1.smithi152.stdout:3/439: creat d6/f96 x:0 0 0 2024-03-04T16:04:37.657 INFO:tasks.workunit.client.1.smithi152.stdout:3/440: chown d6/d53/l69 993595 1 2024-03-04T16:04:37.657 INFO:tasks.workunit.client.1.smithi152.stdout:9/325: dwrite d2/f2d [0,4194304] 0 2024-03-04T16:04:37.659 INFO:tasks.workunit.client.1.smithi152.stdout:5/449: unlink d6/d8/d5a/d64/d8a/l8d 0 2024-03-04T16:04:37.660 INFO:tasks.workunit.client.1.smithi152.stdout:5/450: chown d6/d8/d1f/d3d/c89 59 1 2024-03-04T16:04:37.660 INFO:tasks.workunit.client.1.smithi152.stdout:2/361: creat da/f77 x:0 0 0 2024-03-04T16:04:37.661 INFO:tasks.workunit.client.1.smithi152.stdout:1/417: creat d19/d4b/d53/d57/f88 x:0 0 0 2024-03-04T16:04:37.662 INFO:tasks.workunit.client.1.smithi152.stdout:5/451: readlink d6/d8/d1f/d3d/l6b 0 2024-03-04T16:04:37.663 INFO:tasks.workunit.client.1.smithi152.stdout:9/326: creat d2/f61 x:0 0 0 2024-03-04T16:04:37.664 INFO:tasks.workunit.client.1.smithi152.stdout:6/488: symlink d8/d8f/la2 0 2024-03-04T16:04:37.667 INFO:tasks.workunit.client.1.smithi152.stdout:2/362: write da/d2c/d3f/f72 [970630,39091] 0 2024-03-04T16:04:37.667 INFO:tasks.workunit.client.1.smithi152.stdout:3/441: dwrite d6/d14/d41/d49/f83 [0,4194304] 0 2024-03-04T16:04:37.669 INFO:tasks.workunit.client.1.smithi152.stdout:3/442: chown d6/d10/c19 68253538 1 2024-03-04T16:04:37.671 INFO:tasks.workunit.client.1.smithi152.stdout:6/489: write d8/d2a/f34 [808284,74748] 0 2024-03-04T16:04:37.671 INFO:tasks.workunit.client.1.smithi152.stdout:9/327: mknod d2/d13/d2e/c62 0 2024-03-04T16:04:37.672 INFO:tasks.workunit.client.1.smithi152.stdout:6/490: chown d8/d23/c5a 68 1 2024-03-04T16:04:37.679 INFO:tasks.workunit.client.1.smithi152.stdout:9/328: mkdir d2/d63 0 2024-03-04T16:04:37.680 INFO:tasks.workunit.client.1.smithi152.stdout:9/329: write d2/d13/f18 [704138,82925] 0 2024-03-04T16:04:37.680 INFO:tasks.workunit.client.1.smithi152.stdout:5/452: rename d6/d8/d10 to d6/d8/d5a/d64/d76/d9f 0 2024-03-04T16:04:37.681 INFO:tasks.workunit.client.1.smithi152.stdout:2/363: symlink da/d24/l78 0 2024-03-04T16:04:37.682 INFO:tasks.workunit.client.1.smithi152.stdout:6/491: dread d8/d10/f16 [0,4194304] 0 2024-03-04T16:04:37.687 INFO:tasks.workunit.client.1.smithi152.stdout:9/330: symlink d2/l64 0 2024-03-04T16:04:37.687 INFO:tasks.workunit.client.1.smithi152.stdout:9/331: readlink d2/d13/l20 0 2024-03-04T16:04:37.689 INFO:tasks.workunit.client.1.smithi152.stdout:1/418: write d19/d20/d2f/d5d/f67 [674085,42223] 0 2024-03-04T16:04:37.691 INFO:tasks.workunit.client.1.smithi152.stdout:6/492: unlink d8/d10/f43 0 2024-03-04T16:04:37.694 INFO:tasks.workunit.client.1.smithi152.stdout:9/332: rename d2/d13/d2e/f37 to d2/d13/d35/f65 0 2024-03-04T16:04:37.698 INFO:tasks.workunit.client.1.smithi152.stdout:1/419: rmdir d19/d4b/d53 39 2024-03-04T16:04:37.708 INFO:tasks.workunit.client.1.smithi152.stdout:1/420: write d19/d4b/d53/d57/f60 [849260,33236] 0 2024-03-04T16:04:37.708 INFO:tasks.workunit.client.1.smithi152.stdout:2/364: dread da/df/f13 [0,4194304] 0 2024-03-04T16:04:37.711 INFO:tasks.workunit.client.1.smithi152.stdout:5/453: rmdir d6/d8/d5a 39 2024-03-04T16:04:37.713 INFO:tasks.workunit.client.1.smithi152.stdout:1/421: chown d19/d20/d2f/d6b/d38/f3a 713023001 1 2024-03-04T16:04:37.714 INFO:tasks.workunit.client.1.smithi152.stdout:9/333: truncate d2/d13/d38/f58 124150 0 2024-03-04T16:04:37.715 INFO:tasks.workunit.client.1.smithi152.stdout:2/365: creat da/df/d58/f79 x:0 0 0 2024-03-04T16:04:37.720 INFO:tasks.workunit.client.1.smithi152.stdout:1/422: creat d19/d4b/d53/d5c/f89 x:0 0 0 2024-03-04T16:04:37.721 INFO:tasks.workunit.client.1.smithi152.stdout:1/423: chown d19/d20/d2f/d6b/f42 0 1 2024-03-04T16:04:37.723 INFO:tasks.workunit.client.1.smithi152.stdout:6/493: getdents d8/d23/d93/d99 0 2024-03-04T16:04:37.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:37 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:37 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:37.725 INFO:tasks.workunit.client.1.smithi152.stdout:9/334: mkdir d2/d13/d23/d66 0 2024-03-04T16:04:37.727 INFO:tasks.workunit.client.1.smithi152.stdout:9/335: readlink d2/l30 0 2024-03-04T16:04:37.729 INFO:tasks.workunit.client.1.smithi152.stdout:9/336: write d2/fb [2603343,54508] 0 2024-03-04T16:04:37.730 INFO:tasks.workunit.client.1.smithi152.stdout:1/424: dwrite d19/d20/d2f/d6b/f5a [0,4194304] 0 2024-03-04T16:04:37.731 INFO:tasks.workunit.client.1.smithi152.stdout:5/454: dread d6/d8/d1f/f48 [0,4194304] 0 2024-03-04T16:04:37.731 INFO:tasks.workunit.client.1.smithi152.stdout:9/337: chown d2/d13/f1a 216649 1 2024-03-04T16:04:37.734 INFO:tasks.workunit.client.1.smithi152.stdout:5/455: chown d6/l47 20 1 2024-03-04T16:04:37.735 INFO:tasks.workunit.client.1.smithi152.stdout:2/366: dread da/f1a [0,4194304] 0 2024-03-04T16:04:37.736 INFO:tasks.workunit.client.1.smithi152.stdout:9/338: creat d2/f67 x:0 0 0 2024-03-04T16:04:37.739 INFO:tasks.workunit.client.1.smithi152.stdout:5/456: mknod d6/d8/d1f/d3d/d61/ca0 0 2024-03-04T16:04:37.744 INFO:tasks.workunit.client.1.smithi152.stdout:2/367: dwrite da/d24/d47/f6b [0,4194304] 0 2024-03-04T16:04:37.745 INFO:tasks.workunit.client.1.smithi152.stdout:6/494: dread d8/d23/f2c [0,4194304] 0 2024-03-04T16:04:37.747 INFO:tasks.workunit.client.1.smithi152.stdout:1/425: fdatasync d19/d20/d2f/d6b/f5a 0 2024-03-04T16:04:37.747 INFO:tasks.workunit.client.1.smithi152.stdout:9/339: link d2/d13/d23/l39 d2/d63/l68 0 2024-03-04T16:04:37.751 INFO:tasks.workunit.client.1.smithi152.stdout:2/368: creat da/d5f/f7a x:0 0 0 2024-03-04T16:04:37.751 INFO:tasks.workunit.client.1.smithi152.stdout:2/369: fdatasync da/d24/f65 0 2024-03-04T16:04:37.754 INFO:tasks.workunit.client.1.smithi152.stdout:6/495: mkdir d8/d8f/da3 0 2024-03-04T16:04:37.756 INFO:tasks.workunit.client.1.smithi152.stdout:9/340: rmdir d2/d13 39 2024-03-04T16:04:37.758 INFO:tasks.workunit.client.1.smithi152.stdout:1/426: creat d19/d20/d2f/f8a x:0 0 0 2024-03-04T16:04:37.760 INFO:tasks.workunit.client.1.smithi152.stdout:5/457: dwrite d6/f4e [0,4194304] 0 2024-03-04T16:04:37.762 INFO:tasks.workunit.client.1.smithi152.stdout:6/496: getdents d8/d8f/da3 0 2024-03-04T16:04:37.767 INFO:tasks.workunit.client.1.smithi152.stdout:9/341: mkdir d2/d63/d69 0 2024-03-04T16:04:37.770 INFO:tasks.workunit.client.1.smithi152.stdout:2/370: rename da/df/d35/d50 to da/d7b 0 2024-03-04T16:04:37.770 INFO:tasks.workunit.client.1.smithi152.stdout:8/405: sync 2024-03-04T16:04:37.770 INFO:tasks.workunit.client.1.smithi152.stdout:0/337: sync 2024-03-04T16:04:37.770 INFO:tasks.workunit.client.1.smithi152.stdout:7/406: sync 2024-03-04T16:04:37.771 INFO:tasks.workunit.client.1.smithi152.stdout:2/371: write da/d2c/d3f/f72 [275822,99555] 0 2024-03-04T16:04:37.773 INFO:tasks.workunit.client.1.smithi152.stdout:9/342: creat d2/d63/f6a x:0 0 0 2024-03-04T16:04:37.777 INFO:tasks.workunit.client.1.smithi152.stdout:5/458: write d6/d8/d5a/d64/d76/d9f/f49 [143876,2162] 0 2024-03-04T16:04:37.777 INFO:tasks.workunit.client.1.smithi152.stdout:0/338: mkdir d9/d1b/d54/d69 0 2024-03-04T16:04:37.779 INFO:tasks.workunit.client.1.smithi152.stdout:0/339: read d9/f26 [766234,28873] 0 2024-03-04T16:04:37.780 INFO:tasks.workunit.client.1.smithi152.stdout:0/340: write d9/f26 [1538122,49243] 0 2024-03-04T16:04:37.781 INFO:tasks.workunit.client.1.smithi152.stdout:0/341: read d9/d1b/f23 [2174900,3620] 0 2024-03-04T16:04:37.783 INFO:tasks.workunit.client.1.smithi152.stdout:1/427: rename d19/d20/l58 to d19/d75/l8b 0 2024-03-04T16:04:37.784 INFO:tasks.workunit.client.1.smithi152.stdout:7/407: creat d14/d21/d47/d40/f7a x:0 0 0 2024-03-04T16:04:37.789 INFO:tasks.workunit.client.1.smithi152.stdout:8/406: link d2/f37 d2/d3/d21/d2d/f79 0 2024-03-04T16:04:37.795 INFO:tasks.workunit.client.1.smithi152.stdout:6/497: rename d8/d2a/l64 to d8/d92/d75/la4 0 2024-03-04T16:04:37.799 INFO:tasks.workunit.client.1.smithi152.stdout:0/342: dwrite d9/d2c/d2e/f47 [0,4194304] 0 2024-03-04T16:04:37.801 INFO:tasks.workunit.client.1.smithi152.stdout:0/343: fdatasync d9/d2c/d2e/f32 0 2024-03-04T16:04:37.804 INFO:tasks.workunit.client.1.smithi152.stdout:9/343: mknod d2/d13/d23/d66/c6b 0 2024-03-04T16:04:37.807 INFO:tasks.workunit.client.1.smithi152.stdout:8/407: chown d2/d8/dd/d22/d26/l43 2510 1 2024-03-04T16:04:37.810 INFO:tasks.workunit.client.1.smithi152.stdout:5/459: dwrite d6/d8/d1f/d3d/d6f/f95 [0,4194304] 0 2024-03-04T16:04:37.811 INFO:tasks.workunit.client.1.smithi152.stdout:5/460: fdatasync d6/d8/d1f/d3d/d6f/f73 0 2024-03-04T16:04:37.813 INFO:tasks.workunit.client.1.smithi152.stdout:8/408: dread d2/d3/d15/d1c/f1d [0,4194304] 0 2024-03-04T16:04:37.815 INFO:tasks.workunit.client.1.smithi152.stdout:6/498: truncate d8/d2a/f3c 2030805 0 2024-03-04T16:04:37.815 INFO:tasks.workunit.client.1.smithi152.stdout:8/409: read - d2/d8/dd/d22/d26/f61 zero size 2024-03-04T16:04:37.819 INFO:tasks.workunit.client.1.smithi152.stdout:9/344: symlink d2/d13/d38/l6c 0 2024-03-04T16:04:37.824 INFO:tasks.workunit.client.1.smithi152.stdout:5/461: creat d6/d8/d5a/d64/d76/fa1 x:0 0 0 2024-03-04T16:04:37.826 INFO:tasks.workunit.client.1.smithi152.stdout:8/410: creat d2/d3/d21/d2d/d31/f7a x:0 0 0 2024-03-04T16:04:37.827 INFO:tasks.workunit.client.1.smithi152.stdout:8/411: read d2/f42 [2015586,43227] 0 2024-03-04T16:04:37.829 INFO:tasks.workunit.client.1.smithi152.stdout:9/345: mknod d2/d13/c6d 0 2024-03-04T16:04:37.831 INFO:tasks.workunit.client.1.smithi152.stdout:9/346: truncate d2/d13/f18 2496323 0 2024-03-04T16:04:37.833 INFO:tasks.workunit.client.1.smithi152.stdout:1/428: getdents d19/d4b/d53/d57 0 2024-03-04T16:04:37.834 INFO:tasks.workunit.client.1.smithi152.stdout:5/462: creat d6/d8/d1f/d3d/d6f/d70/fa2 x:0 0 0 2024-03-04T16:04:37.836 INFO:tasks.workunit.client.1.smithi152.stdout:5/463: dread - d6/d8/d5a/d64/f99 zero size 2024-03-04T16:04:37.837 INFO:tasks.workunit.client.1.smithi152.stdout:0/344: write d9/fa [978426,89605] 0 2024-03-04T16:04:37.840 INFO:tasks.workunit.client.1.smithi152.stdout:0/345: write d9/d2c/f2f [4886821,452] 0 2024-03-04T16:04:37.841 INFO:tasks.workunit.client.1.smithi152.stdout:2/372: rename da/df/l1b to da/l7c 0 2024-03-04T16:04:37.841 INFO:tasks.workunit.client.1.smithi152.stdout:6/499: dwrite d8/d10/f16 [0,4194304] 0 2024-03-04T16:04:37.842 INFO:tasks.workunit.client.1.smithi152.stdout:0/346: write d9/f37 [636963,54950] 0 2024-03-04T16:04:37.843 INFO:tasks.workunit.client.1.smithi152.stdout:6/500: stat d8/d2a/f36 0 2024-03-04T16:04:37.843 INFO:tasks.workunit.client.1.smithi152.stdout:1/429: unlink d19/d75/l78 0 2024-03-04T16:04:37.844 INFO:tasks.workunit.client.1.smithi152.stdout:5/464: truncate d6/d8/d1f/d3d/d6f/d70/f71 834192 0 2024-03-04T16:04:37.846 INFO:tasks.workunit.client.1.smithi152.stdout:6/501: dread - d8/d42/f74 zero size 2024-03-04T16:04:37.847 INFO:tasks.workunit.client.1.smithi152.stdout:8/412: link d2/d3/d21/d2d/d31/f7a d2/d8/dd/d22/d49/d5d/f7b 0 2024-03-04T16:04:37.847 INFO:tasks.workunit.client.1.smithi152.stdout:6/502: chown d8/f39 0 1 2024-03-04T16:04:37.852 INFO:tasks.workunit.client.1.smithi152.stdout:9/347: getdents d2/d63/d69 0 2024-03-04T16:04:37.853 INFO:tasks.workunit.client.1.smithi152.stdout:0/347: creat d9/d2c/d2e/f6a x:0 0 0 2024-03-04T16:04:37.856 INFO:tasks.workunit.client.1.smithi152.stdout:1/430: mknod d19/d20/d2f/d6b/d38/d63/c8c 0 2024-03-04T16:04:37.857 INFO:tasks.workunit.client.1.smithi152.stdout:1/431: fdatasync d19/d20/d2f/d6b/f4a 0 2024-03-04T16:04:37.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:37 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:37.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:37 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:37.858 INFO:tasks.workunit.client.1.smithi152.stdout:6/503: rmdir d8/d52 39 2024-03-04T16:04:37.864 INFO:tasks.workunit.client.1.smithi152.stdout:2/373: dread da/fb [0,4194304] 0 2024-03-04T16:04:37.866 INFO:tasks.workunit.client.1.smithi152.stdout:2/374: read da/d24/f2a [3882691,47827] 0 2024-03-04T16:04:37.867 INFO:tasks.workunit.client.1.smithi152.stdout:6/504: dread d8/f39 [0,4194304] 0 2024-03-04T16:04:37.869 INFO:tasks.workunit.client.1.smithi152.stdout:5/465: write d6/d8/f9a [443410,39544] 0 2024-03-04T16:04:37.872 INFO:tasks.workunit.client.1.smithi152.stdout:9/348: mkdir d2/d6e 0 2024-03-04T16:04:37.872 INFO:tasks.workunit.client.1.smithi152.stdout:9/349: chown d2/d13/d23 2 1 2024-03-04T16:04:37.873 INFO:tasks.workunit.client.1.smithi152.stdout:0/348: rename d9/d2c/d2e/f41 to d9/d2c/d2e/d38/d56/d5b/f6b 0 2024-03-04T16:04:37.875 INFO:tasks.workunit.client.1.smithi152.stdout:1/432: link d19/d4b/d53/d5c/f73 d19/d4b/d53/f8d 0 2024-03-04T16:04:37.876 INFO:tasks.workunit.client.1.smithi152.stdout:6/505: symlink d8/d23/d3a/d8d/la5 0 2024-03-04T16:04:37.877 INFO:tasks.workunit.client.1.smithi152.stdout:6/506: readlink d8/d23/d3a/d8d/d96/l9e 0 2024-03-04T16:04:37.879 INFO:tasks.workunit.client.1.smithi152.stdout:5/466: readlink d6/d8/l6d 0 2024-03-04T16:04:37.884 INFO:tasks.workunit.client.1.smithi152.stdout:6/507: dwrite d8/d2a/f45 [0,4194304] 0 2024-03-04T16:04:37.885 INFO:tasks.workunit.client.1.smithi152.stdout:6/508: chown d8/d92/c5d 201 1 2024-03-04T16:04:37.887 INFO:tasks.workunit.client.1.smithi152.stdout:2/375: dread da/d7b/f66 [0,4194304] 0 2024-03-04T16:04:37.892 INFO:tasks.workunit.client.1.smithi152.stdout:8/413: getdents d2/d8/dd/d22/d6c 0 2024-03-04T16:04:37.893 INFO:tasks.workunit.client.1.smithi152.stdout:8/414: fdatasync d2/d8/dd/d22/d6c/f70 0 2024-03-04T16:04:37.894 INFO:tasks.workunit.client.1.smithi152.stdout:1/433: write d19/d20/d2f/d6b/d38/f4c [2480887,59579] 0 2024-03-04T16:04:37.894 INFO:tasks.workunit.client.1.smithi152.stdout:9/350: mknod d2/d63/d69/c6f 0 2024-03-04T16:04:37.896 INFO:tasks.workunit.client.1.smithi152.stdout:0/349: mkdir d9/d1b/d54/d69/d6c 0 2024-03-04T16:04:37.898 INFO:tasks.workunit.client.1.smithi152.stdout:2/376: unlink da/df/d2e/d51/f5b 0 2024-03-04T16:04:37.899 INFO:tasks.workunit.client.1.smithi152.stdout:2/377: stat da/df/d2e/l3b 0 2024-03-04T16:04:37.900 INFO:tasks.workunit.client.1.smithi152.stdout:2/378: read da/d24/d47/f6b [3848902,82960] 0 2024-03-04T16:04:37.901 INFO:tasks.workunit.client.1.smithi152.stdout:2/379: chown da/d7b/f71 12626 1 2024-03-04T16:04:37.901 INFO:tasks.workunit.client.1.smithi152.stdout:2/380: stat da/d5f/f7a 0 2024-03-04T16:04:37.902 INFO:tasks.workunit.client.1.smithi152.stdout:2/381: truncate da/df/d2e/f38 374824 0 2024-03-04T16:04:37.905 INFO:tasks.workunit.client.1.smithi152.stdout:9/351: symlink d2/d13/d23/d66/l70 0 2024-03-04T16:04:37.906 INFO:tasks.workunit.client.1.smithi152.stdout:1/434: mknod d19/d4b/d53/d57/d85/c8e 0 2024-03-04T16:04:37.908 INFO:tasks.workunit.client.1.smithi152.stdout:0/350: unlink f6 0 2024-03-04T16:04:37.908 INFO:tasks.workunit.client.1.smithi152.stdout:2/382: dwrite da/df/d35/f4a [0,4194304] 0 2024-03-04T16:04:37.910 INFO:tasks.workunit.client.1.smithi152.stdout:0/351: read d9/f16 [273720,50205] 0 2024-03-04T16:04:37.911 INFO:tasks.workunit.client.1.smithi152.stdout:0/352: truncate d9/d1b/f1d 4359568 0 2024-03-04T16:04:37.912 INFO:tasks.workunit.client.1.smithi152.stdout:0/353: chown d9/d2c/d2e/l43 58260 1 2024-03-04T16:04:37.912 INFO:tasks.workunit.client.1.smithi152.stdout:6/509: rename d8/d2a/c31 to d8/d52/d58/ca6 0 2024-03-04T16:04:37.912 INFO:tasks.workunit.client.1.smithi152.stdout:0/354: stat d9/d1b/c48 0 2024-03-04T16:04:37.915 INFO:tasks.workunit.client.1.smithi152.stdout:5/467: creat d6/d8/fa3 x:0 0 0 2024-03-04T16:04:37.922 INFO:tasks.workunit.client.1.smithi152.stdout:1/435: symlink d19/d3b/d6e/l8f 0 2024-03-04T16:04:37.923 INFO:tasks.workunit.client.1.smithi152.stdout:9/352: dread d2/fe [0,4194304] 0 2024-03-04T16:04:37.923 INFO:tasks.workunit.client.1.smithi152.stdout:1/436: write f10 [2894773,100726] 0 2024-03-04T16:04:37.923 INFO:tasks.workunit.client.1.smithi152.stdout:9/353: stat d2/d13/c1e 0 2024-03-04T16:04:37.925 INFO:tasks.workunit.client.1.smithi152.stdout:9/354: write d2/d13/d23/f5c [324283,111523] 0 2024-03-04T16:04:37.927 INFO:tasks.workunit.client.1.smithi152.stdout:9/355: dread - d2/f61 zero size 2024-03-04T16:04:37.927 INFO:tasks.workunit.client.1.smithi152.stdout:2/383: rename da/d24/f2a to da/d7b/f7d 0 2024-03-04T16:04:37.928 INFO:tasks.workunit.client.1.smithi152.stdout:6/510: mknod d8/d23/d98/d9f/ca7 0 2024-03-04T16:04:37.929 INFO:tasks.workunit.client.1.smithi152.stdout:2/384: readlink da/d24/l67 0 2024-03-04T16:04:37.930 INFO:tasks.workunit.client.1.smithi152.stdout:9/356: write d2/d13/d2e/f3c [1334149,58665] 0 2024-03-04T16:04:37.933 INFO:tasks.workunit.client.1.smithi152.stdout:1/437: dwrite f10 [0,4194304] 0 2024-03-04T16:04:37.934 INFO:tasks.workunit.client.1.smithi152.stdout:1/438: chown d19/f1a 5 1 2024-03-04T16:04:37.936 INFO:tasks.workunit.client.1.smithi152.stdout:8/415: rmdir d2/d8/dd/d22/d67 0 2024-03-04T16:04:37.938 INFO:tasks.workunit.client.1.smithi152.stdout:8/416: read d2/d3/d15/f6b [1379775,76498] 0 2024-03-04T16:04:37.942 INFO:tasks.workunit.client.1.smithi152.stdout:1/439: dwrite fa [0,4194304] 0 2024-03-04T16:04:37.942 INFO:tasks.workunit.client.1.smithi152.stdout:2/385: mknod da/d5f/c7e 0 2024-03-04T16:04:37.948 INFO:tasks.workunit.client.1.smithi152.stdout:6/511: rename d8/d2a/f36 to d8/d23/d3a/d71/d9d/fa8 0 2024-03-04T16:04:37.950 INFO:tasks.workunit.client.1.smithi152.stdout:6/512: chown d8/d23/l41 160649 1 2024-03-04T16:04:37.951 INFO:tasks.workunit.client.1.smithi152.stdout:1/440: dwrite d19/d75/f83 [0,4194304] 0 2024-03-04T16:04:37.952 INFO:tasks.workunit.client.1.smithi152.stdout:1/441: readlink d19/d20/d2f/d6b/l25 0 2024-03-04T16:04:37.953 INFO:tasks.workunit.client.1.smithi152.stdout:8/417: truncate d2/d8/f4e 541490 0 2024-03-04T16:04:37.956 INFO:tasks.workunit.client.1.smithi152.stdout:9/357: mkdir d2/d13/d23/d71 0 2024-03-04T16:04:37.957 INFO:tasks.workunit.client.1.smithi152.stdout:2/386: fdatasync da/d24/f42 0 2024-03-04T16:04:37.962 INFO:tasks.workunit.client.1.smithi152.stdout:1/442: dwrite d19/f1c [0,4194304] 0 2024-03-04T16:04:37.964 INFO:tasks.workunit.client.1.smithi152.stdout:0/355: getdents d9/d2c/d2e/d38/d56 0 2024-03-04T16:04:37.966 INFO:tasks.workunit.client.1.smithi152.stdout:8/418: dwrite d2/d3/d21/f32 [0,4194304] 0 2024-03-04T16:04:37.966 INFO:tasks.workunit.client.1.smithi152.stdout:0/356: write d9/d2c/d2e/f31 [135651,70443] 0 2024-03-04T16:04:37.969 INFO:tasks.workunit.client.1.smithi152.stdout:2/387: chown da/df/c20 71994335 1 2024-03-04T16:04:37.970 INFO:tasks.workunit.client.1.smithi152.stdout:6/513: mkdir d8/da9 0 2024-03-04T16:04:37.972 INFO:tasks.workunit.client.1.smithi152.stdout:1/443: chown d19/d20/d2f/d6b/c29 15057379 1 2024-03-04T16:04:37.974 INFO:tasks.workunit.client.1.smithi152.stdout:0/357: creat d9/d1b/f6d x:0 0 0 2024-03-04T16:04:37.980 INFO:tasks.workunit.client.1.smithi152.stdout:2/388: dwrite da/df/f14 [0,4194304] 0 2024-03-04T16:04:37.980 INFO:tasks.workunit.client.1.smithi152.stdout:0/358: readlink d9/l2b 0 2024-03-04T16:04:37.980 INFO:tasks.workunit.client.1.smithi152.stdout:8/419: dwrite d2/d3/d21/f32 [0,4194304] 0 2024-03-04T16:04:37.981 INFO:tasks.workunit.client.1.smithi152.stdout:0/359: fsync d9/d2c/d2e/f32 0 2024-03-04T16:04:37.983 INFO:tasks.workunit.client.1.smithi152.stdout:0/360: readlink d9/d2c/l3b 0 2024-03-04T16:04:37.985 INFO:tasks.workunit.client.1.smithi152.stdout:1/444: unlink d19/d20/d2f/d5d/c6c 0 2024-03-04T16:04:37.987 INFO:tasks.workunit.client.1.smithi152.stdout:2/389: symlink da/df/d2e/l7f 0 2024-03-04T16:04:37.991 INFO:tasks.workunit.client.1.smithi152.stdout:8/420: symlink d2/d3/d21/d2d/d52/d68/d6e/l7c 0 2024-03-04T16:04:37.991 INFO:tasks.workunit.client.1.smithi152.stdout:0/361: dwrite d9/d2c/d2e/d38/d56/d5b/f60 [0,4194304] 0 2024-03-04T16:04:37.996 INFO:tasks.workunit.client.1.smithi152.stdout:1/445: rename d19/d20/d2f/d6b/l80 to d19/d20/d2f/l90 0 2024-03-04T16:04:37.997 INFO:tasks.workunit.client.1.smithi152.stdout:1/446: stat d19/d20/d2f/l7a 0 2024-03-04T16:04:38.000 INFO:tasks.workunit.client.1.smithi152.stdout:0/362: creat d9/d1b/f6e x:0 0 0 2024-03-04T16:04:38.003 INFO:tasks.workunit.client.1.smithi152.stdout:8/421: dread d2/d8/dd/d22/d49/d5d/f44 [0,4194304] 0 2024-03-04T16:04:38.005 INFO:tasks.workunit.client.1.smithi152.stdout:0/363: creat d9/d2c/f6f x:0 0 0 2024-03-04T16:04:38.006 INFO:tasks.workunit.client.1.smithi152.stdout:2/390: dwrite da/f1a [0,4194304] 0 2024-03-04T16:04:38.006 INFO:tasks.workunit.client.1.smithi152.stdout:5/468: write d6/f38 [1836418,49282] 0 2024-03-04T16:04:38.007 INFO:tasks.workunit.client.1.smithi152.stdout:1/447: read d19/d3b/d6e/f79 [319587,4622] 0 2024-03-04T16:04:38.009 INFO:tasks.workunit.client.1.smithi152.stdout:0/364: write d9/d2c/d2e/f32 [682087,98291] 0 2024-03-04T16:04:38.010 INFO:tasks.workunit.client.1.smithi152.stdout:5/469: read - d6/d8/d1f/d3d/d6f/d70/f75 zero size 2024-03-04T16:04:38.012 INFO:tasks.workunit.client.1.smithi152.stdout:8/422: dread d2/d3/d21/d2d/fc [0,4194304] 0 2024-03-04T16:04:38.018 INFO:tasks.workunit.client.1.smithi152.stdout:5/470: mkdir d6/d8/d1f/d3d/da4 0 2024-03-04T16:04:38.020 INFO:tasks.workunit.client.1.smithi152.stdout:1/448: dwrite d19/f1a [0,4194304] 0 2024-03-04T16:04:38.021 INFO:tasks.workunit.client.1.smithi152.stdout:8/423: mkdir d2/d3/d21/d2d/d52/d68/d6e/d7d 0 2024-03-04T16:04:38.024 INFO:tasks.workunit.client.1.smithi152.stdout:0/365: symlink d9/d1b/d62/l70 0 2024-03-04T16:04:38.027 INFO:tasks.workunit.client.1.smithi152.stdout:1/449: creat d19/d75/f91 x:0 0 0 2024-03-04T16:04:38.030 INFO:tasks.workunit.client.1.smithi152.stdout:0/366: rmdir d9/d1b/d54/d69 39 2024-03-04T16:04:38.031 INFO:tasks.workunit.client.1.smithi152.stdout:8/424: dwrite d2/d8/dd/d22/d49/f55 [0,4194304] 0 2024-03-04T16:04:38.039 INFO:tasks.workunit.client.1.smithi152.stdout:0/367: creat d9/d1b/f71 x:0 0 0 2024-03-04T16:04:38.043 INFO:tasks.workunit.client.1.smithi152.stdout:0/368: write f7 [14406448,86361] 0 2024-03-04T16:04:38.043 INFO:tasks.workunit.client.1.smithi152.stdout:8/425: rmdir d2/d3/d15/d1c 39 2024-03-04T16:04:38.045 INFO:tasks.workunit.client.1.smithi152.stdout:0/369: write d9/d2c/f2f [3188600,112474] 0 2024-03-04T16:04:38.047 INFO:tasks.workunit.client.1.smithi152.stdout:1/450: link d19/d75/f83 d19/d3b/f92 0 2024-03-04T16:04:38.048 INFO:tasks.workunit.client.1.smithi152.stdout:0/370: read f7 [8890105,125792] 0 2024-03-04T16:04:38.050 INFO:tasks.workunit.client.1.smithi152.stdout:5/471: dread d6/d8/fe [0,4194304] 0 2024-03-04T16:04:38.051 INFO:tasks.workunit.client.1.smithi152.stdout:8/426: creat d2/d3/d21/d2d/d48/f7e x:0 0 0 2024-03-04T16:04:38.053 INFO:tasks.workunit.client.1.smithi152.stdout:1/451: symlink d19/d20/d2f/d6b/d38/l93 0 2024-03-04T16:04:38.055 INFO:tasks.workunit.client.1.smithi152.stdout:0/371: symlink d9/d2c/d2e/d38/d56/d5b/l72 0 2024-03-04T16:04:38.061 INFO:tasks.workunit.client.1.smithi152.stdout:5/472: truncate d6/d8/d5a/d64/d76/d9f/f54 4413579 0 2024-03-04T16:04:38.063 INFO:tasks.workunit.client.1.smithi152.stdout:1/452: rename d19/d20/f45 to d19/d20/d2f/f94 0 2024-03-04T16:04:38.065 INFO:tasks.workunit.client.1.smithi152.stdout:1/453: chown d19/d4b/d53/d5c 501 1 2024-03-04T16:04:38.070 INFO:tasks.workunit.client.1.smithi152.stdout:6/514: write d8/d2a/f3c [2357384,69067] 0 2024-03-04T16:04:38.072 INFO:tasks.workunit.client.1.smithi152.stdout:5/473: creat d6/d8/d5a/d8b/fa5 x:0 0 0 2024-03-04T16:04:38.074 INFO:tasks.workunit.client.1.smithi152.stdout:5/474: write d6/d8/d1f/d3d/f85 [443239,86040] 0 2024-03-04T16:04:38.074 INFO:tasks.workunit.client.1.smithi152.stdout:4/337: sync 2024-03-04T16:04:38.076 INFO:tasks.workunit.client.1.smithi152.stdout:0/372: mknod d9/d1b/d54/c73 0 2024-03-04T16:04:38.077 INFO:tasks.workunit.client.1.smithi152.stdout:0/373: read - d9/f36 zero size 2024-03-04T16:04:38.078 INFO:tasks.workunit.client.1.smithi152.stdout:1/454: rename d19/d4b/d53/d5c/f73 to d19/d20/d2f/d5d/f95 0 2024-03-04T16:04:38.079 INFO:tasks.workunit.client.1.smithi152.stdout:0/374: rename d9 to d9/d2c/d74 22 2024-03-04T16:04:38.080 INFO:tasks.workunit.client.1.smithi152.stdout:0/375: write d9/fa [443438,94883] 0 2024-03-04T16:04:38.085 INFO:tasks.workunit.client.1.smithi152.stdout:5/475: chown d6/d8/d5a/d64/d76/d9f/c13 1171450965 1 2024-03-04T16:04:38.089 INFO:tasks.workunit.client.1.smithi152.stdout:4/338: rename d8/d4e/c5b to d8/d4e/c71 0 2024-03-04T16:04:38.091 INFO:tasks.workunit.client.1.smithi152.stdout:8/427: getdents d2/d8/dd 0 2024-03-04T16:04:38.094 INFO:tasks.workunit.client.1.smithi152.stdout:1/455: creat d19/d4b/d53/d5c/f96 x:0 0 0 2024-03-04T16:04:38.099 INFO:tasks.workunit.client.1.smithi152.stdout:0/376: mkdir d9/d2c/d2e/d38/d56/d5b/d75 0 2024-03-04T16:04:38.107 INFO:tasks.workunit.client.1.smithi152.stdout:5/476: creat d6/d8/d1f/d88/fa6 x:0 0 0 2024-03-04T16:04:38.108 INFO:tasks.workunit.client.1.smithi152.stdout:2/391: write da/f56 [1789632,49789] 0 2024-03-04T16:04:38.114 INFO:tasks.workunit.client.1.smithi152.stdout:5/477: dread d6/d8/d1f/d3d/f85 [0,4194304] 0 2024-03-04T16:04:38.116 INFO:tasks.workunit.client.1.smithi152.stdout:5/478: chown d6/d8/d5a/l93 3 1 2024-03-04T16:04:38.118 INFO:tasks.workunit.client.1.smithi152.stdout:2/392: dwrite da/df/d2e/f30 [0,4194304] 0 2024-03-04T16:04:38.119 INFO:tasks.workunit.client.1.smithi152.stdout:1/456: dread fe [4194304,4194304] 0 2024-03-04T16:04:38.123 INFO:tasks.workunit.client.1.smithi152.stdout:8/428: mkdir d2/d7f 0 2024-03-04T16:04:38.126 INFO:tasks.workunit.client.1.smithi152.stdout:2/393: dwrite da/f3c [0,4194304] 0 2024-03-04T16:04:38.128 INFO:tasks.workunit.client.1.smithi152.stdout:0/377: dread d9/f16 [0,4194304] 0 2024-03-04T16:04:38.132 INFO:tasks.workunit.client.1.smithi152.stdout:1/457: chown c4 26052 1 2024-03-04T16:04:38.135 INFO:tasks.workunit.client.1.smithi152.stdout:4/339: creat d8/d9/f72 x:0 0 0 2024-03-04T16:04:38.137 INFO:tasks.workunit.client.1.smithi152.stdout:2/394: dwrite da/f77 [0,4194304] 0 2024-03-04T16:04:38.140 INFO:tasks.workunit.client.1.smithi152.stdout:2/395: stat da/d24/f31 0 2024-03-04T16:04:38.140 INFO:tasks.workunit.client.1.smithi152.stdout:0/378: dwrite d9/d1b/f2a [0,4194304] 0 2024-03-04T16:04:38.142 INFO:tasks.workunit.client.1.smithi152.stdout:5/479: unlink d6/d8/l7a 0 2024-03-04T16:04:38.143 INFO:tasks.workunit.client.1.smithi152.stdout:0/379: write d9/f45 [4091213,39388] 0 2024-03-04T16:04:38.143 INFO:tasks.workunit.client.1.smithi152.stdout:2/396: dread da/df/d2e/f38 [0,4194304] 0 2024-03-04T16:04:38.145 INFO:tasks.workunit.client.1.smithi152.stdout:0/380: chown d9/d2c/d2e/f47 1391893 1 2024-03-04T16:04:38.150 INFO:tasks.workunit.client.1.smithi152.stdout:1/458: mkdir d19/d3b/d6e/d97 0 2024-03-04T16:04:38.156 INFO:tasks.workunit.client.1.smithi152.stdout:5/480: rmdir d6/d8/d5a/d64/d76/d9f 39 2024-03-04T16:04:38.159 INFO:tasks.workunit.client.1.smithi152.stdout:4/340: creat d8/d2f/d6f/f73 x:0 0 0 2024-03-04T16:04:38.163 INFO:tasks.workunit.client.1.smithi152.stdout:8/429: creat d2/f80 x:0 0 0 2024-03-04T16:04:38.164 INFO:tasks.workunit.client.1.smithi152.stdout:6/515: dwrite d8/d10/f1e [0,4194304] 0 2024-03-04T16:04:38.170 INFO:tasks.workunit.client.1.smithi152.stdout:4/341: rmdir d8/d3d 39 2024-03-04T16:04:38.176 INFO:tasks.workunit.client.1.smithi152.stdout:6/516: write d8/d23/d3a/d71/d9d/fa8 [6199454,88596] 0 2024-03-04T16:04:38.176 INFO:tasks.workunit.client.1.smithi152.stdout:6/517: chown d8/d2a/c30 53113453 1 2024-03-04T16:04:38.178 INFO:tasks.workunit.client.1.smithi152.stdout:6/518: chown d8/d23/d98/d9f/f73 993496 1 2024-03-04T16:04:38.179 INFO:tasks.workunit.client.1.smithi152.stdout:1/459: link d19/d20/d2f/l7a d19/d4b/d53/d57/d62/d82/l98 0 2024-03-04T16:04:38.180 INFO:tasks.workunit.client.1.smithi152.stdout:8/430: link d2/d8/dd/d22/d49/f6f d2/d3/d21/d2d/d48/f81 0 2024-03-04T16:04:38.182 INFO:tasks.workunit.client.1.smithi152.stdout:6/519: mknod d8/d23/caa 0 2024-03-04T16:04:38.185 INFO:tasks.workunit.client.1.smithi152.stdout:1/460: unlink d19/d4b/d53/d57/d62/l6a 0 2024-03-04T16:04:38.189 INFO:tasks.workunit.client.1.smithi152.stdout:5/481: link d6/d8/d5a/d64/d76/d9f/c31 d6/d8/d5a/d64/d76/ca7 0 2024-03-04T16:04:38.191 INFO:tasks.workunit.client.1.smithi152.stdout:1/461: dread d19/d20/d2f/f59 [0,4194304] 0 2024-03-04T16:04:38.193 INFO:tasks.workunit.client.1.smithi152.stdout:5/482: creat d6/d8/d1f/d3d/d6f/d70/fa8 x:0 0 0 2024-03-04T16:04:38.194 INFO:tasks.workunit.client.1.smithi152.stdout:5/483: readlink d6/d8/d1f/d3d/l5e 0 2024-03-04T16:04:38.196 INFO:tasks.workunit.client.1.smithi152.stdout:6/520: link d8/c6e d8/d23/d3a/cab 0 2024-03-04T16:04:38.198 INFO:tasks.workunit.client.1.smithi152.stdout:8/431: dread d2/d8/dd/d22/d49/f6f [0,4194304] 0 2024-03-04T16:04:38.201 INFO:tasks.workunit.client.1.smithi152.stdout:1/462: creat d19/d20/f99 x:0 0 0 2024-03-04T16:04:38.201 INFO:tasks.workunit.client.1.smithi152.stdout:2/397: write da/fc [887890,46662] 0 2024-03-04T16:04:38.204 INFO:tasks.workunit.client.1.smithi152.stdout:5/484: creat d6/d8/d1f/d3d/d6f/d70/d98/fa9 x:0 0 0 2024-03-04T16:04:38.205 INFO:tasks.workunit.client.1.smithi152.stdout:6/521: creat d8/d8f/da3/fac x:0 0 0 2024-03-04T16:04:38.208 INFO:tasks.workunit.client.1.smithi152.stdout:1/463: mkdir d19/d20/d2f/d6b/d38/d63/d9a 0 2024-03-04T16:04:38.209 INFO:tasks.workunit.client.1.smithi152.stdout:0/381: write d9/d1a/f46 [547233,107953] 0 2024-03-04T16:04:38.209 INFO:tasks.workunit.client.1.smithi152.stdout:1/464: readlink d19/d4b/d53/d57/d85/l6f 0 2024-03-04T16:04:38.210 INFO:tasks.workunit.client.1.smithi152.stdout:0/382: dread - d9/d1b/f6d zero size 2024-03-04T16:04:38.210 INFO:tasks.workunit.client.1.smithi152.stdout:1/465: stat d19/d20/d2f/d6b/l25 0 2024-03-04T16:04:38.211 INFO:tasks.workunit.client.1.smithi152.stdout:6/522: rmdir d8/d23/d3a/d8d/d96 39 2024-03-04T16:04:38.213 INFO:tasks.workunit.client.1.smithi152.stdout:6/523: write d8/d92/d59/f7c [127554,23628] 0 2024-03-04T16:04:38.214 INFO:tasks.workunit.client.1.smithi152.stdout:2/398: mkdir da/d24/d80 0 2024-03-04T16:04:38.216 INFO:tasks.workunit.client.1.smithi152.stdout:5/485: mknod d6/d8/d5a/d82/caa 0 2024-03-04T16:04:38.216 INFO:tasks.workunit.client.1.smithi152.stdout:4/342: truncate d8/f37 4416585 0 2024-03-04T16:04:38.218 INFO:tasks.workunit.client.1.smithi152.stdout:1/466: dwrite fa [0,4194304] 0 2024-03-04T16:04:38.219 INFO:tasks.workunit.client.1.smithi152.stdout:0/383: symlink d9/d2c/d2e/d38/d56/l76 0 2024-03-04T16:04:38.221 INFO:tasks.workunit.client.1.smithi152.stdout:0/384: dread - d9/d2c/d2e/f4a zero size 2024-03-04T16:04:38.223 INFO:tasks.workunit.client.1.smithi152.stdout:6/524: creat d8/d92/fad x:0 0 0 2024-03-04T16:04:38.223 INFO:tasks.workunit.client.1.smithi152.stdout:0/385: chown d9/d2c/d2e/f4a 444086 1 2024-03-04T16:04:38.226 INFO:tasks.workunit.client.1.smithi152.stdout:4/343: dwrite d8/d9/f72 [0,4194304] 0 2024-03-04T16:04:38.228 INFO:tasks.workunit.client.1.smithi152.stdout:5/486: rename d6/d8/d1f/d3d/l5e to d6/d8/d1f/d3d/d61/lab 0 2024-03-04T16:04:38.231 INFO:tasks.workunit.client.1.smithi152.stdout:2/399: mkdir da/d24/d81 0 2024-03-04T16:04:38.233 INFO:tasks.workunit.client.1.smithi152.stdout:0/386: chown d9/d2c/l40 737458 1 2024-03-04T16:04:38.236 INFO:tasks.workunit.client.1.smithi152.stdout:6/525: mknod d8/d8f/cae 0 2024-03-04T16:04:38.242 INFO:tasks.workunit.client.1.smithi152.stdout:5/487: unlink d6/d8/d5a/d82/caa 0 2024-03-04T16:04:38.243 INFO:tasks.workunit.client.1.smithi152.stdout:5/488: stat d6/d8/d1f/d3d/da4 0 2024-03-04T16:04:38.246 INFO:tasks.workunit.client.1.smithi152.stdout:1/467: creat d19/d20/f9b x:0 0 0 2024-03-04T16:04:38.247 INFO:tasks.workunit.client.1.smithi152.stdout:0/387: creat d9/d2c/d2e/d38/f77 x:0 0 0 2024-03-04T16:04:38.248 INFO:tasks.workunit.client.1.smithi152.stdout:0/388: stat d9/d2c/l3b 0 2024-03-04T16:04:38.250 INFO:tasks.workunit.client.1.smithi152.stdout:6/526: unlink d8/d23/caa 0 2024-03-04T16:04:38.253 INFO:tasks.workunit.client.1.smithi152.stdout:2/400: mkdir da/d24/d80/d82 0 2024-03-04T16:04:38.254 INFO:tasks.workunit.client.1.smithi152.stdout:6/527: dread d8/d10/d53/f5b [0,4194304] 0 2024-03-04T16:04:38.256 INFO:tasks.workunit.client.1.smithi152.stdout:1/468: mknod d19/d3b/d6e/c9c 0 2024-03-04T16:04:38.257 INFO:tasks.workunit.client.1.smithi152.stdout:0/389: mknod d9/d1b/d3c/c78 0 2024-03-04T16:04:38.258 INFO:tasks.workunit.client.1.smithi152.stdout:0/390: chown d9/l2b 1447291821 1 2024-03-04T16:04:38.259 INFO:tasks.workunit.client.1.smithi152.stdout:4/344: creat d8/d9/d24/f74 x:0 0 0 2024-03-04T16:04:38.261 INFO:tasks.workunit.client.1.smithi152.stdout:5/489: dwrite d6/d8/f1d [0,4194304] 0 2024-03-04T16:04:38.263 INFO:tasks.workunit.client.1.smithi152.stdout:0/391: dread d9/d2c/d2e/f32 [0,4194304] 0 2024-03-04T16:04:38.269 INFO:tasks.workunit.client.1.smithi152.stdout:2/401: mknod da/df/d58/d6e/c83 0 2024-03-04T16:04:38.270 INFO:tasks.workunit.client.1.smithi152.stdout:1/469: mkdir d19/d9d 0 2024-03-04T16:04:38.273 INFO:tasks.workunit.client.1.smithi152.stdout:4/345: link d8/f10 d8/d4e/f75 0 2024-03-04T16:04:38.273 INFO:tasks.workunit.client.1.smithi152.stdout:4/346: chown d8/d4e 322908374 1 2024-03-04T16:04:38.274 INFO:tasks.workunit.client.1.smithi152.stdout:2/402: creat da/df/d2e/d61/f84 x:0 0 0 2024-03-04T16:04:38.275 INFO:tasks.workunit.client.1.smithi152.stdout:2/403: fdatasync da/f77 0 2024-03-04T16:04:38.276 INFO:tasks.workunit.client.1.smithi152.stdout:2/404: read - da/d7b/f71 zero size 2024-03-04T16:04:38.277 INFO:tasks.workunit.client.1.smithi152.stdout:1/470: mknod d19/d20/d2f/d6b/d38/d63/c9e 0 2024-03-04T16:04:38.277 INFO:tasks.workunit.client.1.smithi152.stdout:1/471: truncate d19/f7c 983156 0 2024-03-04T16:04:38.278 INFO:tasks.workunit.client.1.smithi152.stdout:0/392: creat d9/d1a/f79 x:0 0 0 2024-03-04T16:04:38.279 INFO:tasks.workunit.client.1.smithi152.stdout:1/472: write d19/d4b/d53/d57/f81 [857528,2313] 0 2024-03-04T16:04:38.286 INFO:tasks.workunit.client.1.smithi152.stdout:0/393: dwrite d9/d1b/f71 [0,4194304] 0 2024-03-04T16:04:38.286 INFO:tasks.workunit.client.1.smithi152.stdout:1/473: dread d19/d20/f41 [0,4194304] 0 2024-03-04T16:04:38.288 INFO:tasks.workunit.client.1.smithi152.stdout:1/474: chown d19/d20/d2f/d6b/f40 988867502 1 2024-03-04T16:04:38.292 INFO:tasks.workunit.client.1.smithi152.stdout:4/347: getdents d8 0 2024-03-04T16:04:38.293 INFO:tasks.workunit.client.1.smithi152.stdout:4/348: write f0 [5525173,10268] 0 2024-03-04T16:04:38.294 INFO:tasks.workunit.client.1.smithi152.stdout:4/349: dread - d8/f30 zero size 2024-03-04T16:04:38.295 INFO:tasks.workunit.client.1.smithi152.stdout:1/475: rmdir d19/d75 39 2024-03-04T16:04:38.295 INFO:tasks.workunit.client.1.smithi152.stdout:4/350: read - d8/d9/d24/f66 zero size 2024-03-04T16:04:38.296 INFO:tasks.workunit.client.1.smithi152.stdout:4/351: write d8/d9/d24/d26/f2e [4277411,10703] 0 2024-03-04T16:04:38.298 INFO:tasks.workunit.client.1.smithi152.stdout:4/352: truncate d8/f3b 703953 0 2024-03-04T16:04:38.298 INFO:tasks.workunit.client.1.smithi152.stdout:0/394: write d9/d2c/d2e/d38/d56/d5b/f60 [3156412,68932] 0 2024-03-04T16:04:38.299 INFO:tasks.workunit.client.1.smithi152.stdout:4/353: write d8/d2f/f4a [5164364,121385] 0 2024-03-04T16:04:38.300 INFO:tasks.workunit.client.1.smithi152.stdout:0/395: chown d9/d2c/f6f 3573 1 2024-03-04T16:04:38.306 INFO:tasks.workunit.client.1.smithi152.stdout:4/354: mknod d8/d2f/d6f/c76 0 2024-03-04T16:04:38.310 INFO:tasks.workunit.client.1.smithi152.stdout:4/355: creat d8/d2f/d67/f77 x:0 0 0 2024-03-04T16:04:38.310 INFO:tasks.workunit.client.1.smithi152.stdout:2/405: rmdir da/df/d2e/d61 39 2024-03-04T16:04:38.315 INFO:tasks.workunit.client.1.smithi152.stdout:2/406: mkdir da/d5f/d85 0 2024-03-04T16:04:38.317 INFO:tasks.workunit.client.1.smithi152.stdout:0/396: getdents d9/d1b/d3c 0 2024-03-04T16:04:38.318 INFO:tasks.workunit.client.1.smithi152.stdout:4/356: dwrite d8/d9/d24/f66 [0,4194304] 0 2024-03-04T16:04:38.319 INFO:tasks.workunit.client.1.smithi152.stdout:2/407: dread da/f48 [0,4194304] 0 2024-03-04T16:04:38.322 INFO:tasks.workunit.client.1.smithi152.stdout:4/357: mkdir d8/d78 0 2024-03-04T16:04:38.322 INFO:tasks.workunit.client.1.smithi152.stdout:1/476: truncate d19/d20/d2f/d5d/f67 69870 0 2024-03-04T16:04:38.323 INFO:tasks.workunit.client.1.smithi152.stdout:2/408: creat da/d2c/d3f/f86 x:0 0 0 2024-03-04T16:04:38.326 INFO:tasks.workunit.client.1.smithi152.stdout:0/397: mkdir d9/d2c/d2e/d38/d56/d5b/d7a 0 2024-03-04T16:04:38.331 INFO:tasks.workunit.client.1.smithi152.stdout:1/477: symlink d19/d4b/d53/d5c/l9f 0 2024-03-04T16:04:38.333 INFO:tasks.workunit.client.1.smithi152.stdout:1/478: write d19/d20/d2f/d6b/d38/f4c [3175037,72899] 0 2024-03-04T16:04:38.334 INFO:tasks.workunit.client.1.smithi152.stdout:0/398: dwrite d9/d1b/f6d [0,4194304] 0 2024-03-04T16:04:38.335 INFO:tasks.workunit.client.1.smithi152.stdout:2/409: dread da/df/d2e/f39 [0,4194304] 0 2024-03-04T16:04:38.336 INFO:tasks.workunit.client.1.smithi152.stdout:0/399: write d9/d1b/f6d [489775,29918] 0 2024-03-04T16:04:38.348 INFO:tasks.workunit.client.1.smithi152.stdout:2/410: mknod da/df/c87 0 2024-03-04T16:04:38.350 INFO:tasks.workunit.client.1.smithi152.stdout:1/479: write d19/d20/d2f/d6b/d38/f3a [2287095,2576] 0 2024-03-04T16:04:38.353 INFO:tasks.workunit.client.1.smithi152.stdout:2/411: rename da/d24/l2b to da/d2c/l88 0 2024-03-04T16:04:38.354 INFO:tasks.workunit.client.1.smithi152.stdout:4/358: mkdir d8/d9/d79 0 2024-03-04T16:04:38.354 INFO:tasks.workunit.client.1.smithi152.stdout:2/412: truncate da/d7b/f71 857848 0 2024-03-04T16:04:38.358 INFO:tasks.workunit.client.1.smithi152.stdout:2/413: rmdir da/df/d35 39 2024-03-04T16:04:38.361 INFO:tasks.workunit.client.1.smithi152.stdout:2/414: creat da/df/d58/d6e/f89 x:0 0 0 2024-03-04T16:04:38.362 INFO:tasks.workunit.client.1.smithi152.stdout:4/359: rename d8/d53 to d8/d9/d7a 0 2024-03-04T16:04:38.367 INFO:tasks.workunit.client.1.smithi152.stdout:1/480: rename fe to d19/d75/fa0 0 2024-03-04T16:04:38.369 INFO:tasks.workunit.client.1.smithi152.stdout:2/415: dwrite da/d7b/f5c [4194304,4194304] 0 2024-03-04T16:04:38.374 INFO:tasks.workunit.client.1.smithi152.stdout:1/481: rename d19/d4b/d53/d5c/f89 to d19/d20/d2f/fa1 0 2024-03-04T16:04:38.374 INFO:tasks.workunit.client.1.smithi152.stdout:2/416: dread da/df/f13 [0,4194304] 0 2024-03-04T16:04:38.374 INFO:tasks.workunit.client.1.smithi152.stdout:2/417: chown da/d7b 58636 1 2024-03-04T16:04:38.375 INFO:tasks.workunit.client.1.smithi152.stdout:2/418: write da/d5f/f68 [998209,122980] 0 2024-03-04T16:04:38.387 INFO:tasks.workunit.client.1.smithi152.stdout:3/443: sync 2024-03-04T16:04:38.387 INFO:tasks.workunit.client.1.smithi152.stdout:3/444: chown d6/f68 313 1 2024-03-04T16:04:38.389 INFO:tasks.workunit.client.1.smithi152.stdout:1/482: creat d19/d20/d2f/d6b/d38/d63/d9a/fa2 x:0 0 0 2024-03-04T16:04:38.391 INFO:tasks.workunit.client.1.smithi152.stdout:2/419: unlink da/d7b/f66 0 2024-03-04T16:04:38.392 INFO:tasks.workunit.client.1.smithi152.stdout:2/420: stat da/df/d2e/d51/c53 0 2024-03-04T16:04:38.393 INFO:tasks.workunit.client.1.smithi152.stdout:3/445: dwrite d6/f68 [4194304,4194304] 0 2024-03-04T16:04:38.473 INFO:tasks.workunit.client.1.smithi152.stdout:1/483: creat d19/d4b/d53/d57/d85/fa3 x:0 0 0 2024-03-04T16:04:38.474 INFO:tasks.workunit.client.1.smithi152.stdout:3/446: write d6/f46 [4148329,108034] 0 2024-03-04T16:04:38.474 INFO:tasks.workunit.client.1.smithi152.stdout:2/421: read da/df/d35/f4a [1395185,71489] 0 2024-03-04T16:04:38.475 INFO:tasks.workunit.client.1.smithi152.stdout:3/447: stat d6/d14/c17 0 2024-03-04T16:04:38.477 INFO:tasks.workunit.client.1.smithi152.stdout:3/448: read - d6/d14/d93/f89 zero size 2024-03-04T16:04:38.480 INFO:tasks.workunit.client.1.smithi152.stdout:1/484: dwrite d19/f1d [4194304,4194304] 0 2024-03-04T16:04:38.488 INFO:tasks.workunit.client.1.smithi152.stdout:3/449: creat d6/d5c/f97 x:0 0 0 2024-03-04T16:04:38.495 INFO:tasks.workunit.client.1.smithi152.stdout:4/360: link d8/d2f/c3c d8/d9/c7b 0 2024-03-04T16:04:38.499 INFO:tasks.workunit.client.1.smithi152.stdout:4/361: mkdir d8/d2f/d7c 0 2024-03-04T16:04:38.501 INFO:tasks.workunit.client.1.smithi152.stdout:4/362: truncate d8/d9/d24/d26/d42/f4b 25115 0 2024-03-04T16:04:38.502 INFO:tasks.workunit.client.1.smithi152.stdout:1/485: getdents d19/d4b/d53/d5c 0 2024-03-04T16:04:38.505 INFO:tasks.workunit.client.1.smithi152.stdout:3/450: rename d6/d10/d2d/d4d/l73 to d6/l98 0 2024-03-04T16:04:38.506 INFO:tasks.workunit.client.1.smithi152.stdout:7/408: sync 2024-03-04T16:04:38.506 INFO:tasks.workunit.client.1.smithi152.stdout:9/358: sync 2024-03-04T16:04:38.508 INFO:tasks.workunit.client.1.smithi152.stdout:4/363: dwrite d8/d2f/d67/f77 [0,4194304] 0 2024-03-04T16:04:38.510 INFO:tasks.workunit.client.1.smithi152.stdout:4/364: write d8/d9/f72 [1536565,42269] 0 2024-03-04T16:04:38.512 INFO:tasks.workunit.client.1.smithi152.stdout:1/486: link d19/d20/d2f/d6b/f2d d19/d4b/d53/d57/d85/fa4 0 2024-03-04T16:04:38.516 INFO:tasks.workunit.client.1.smithi152.stdout:1/487: dread d19/f1c [0,4194304] 0 2024-03-04T16:04:38.517 INFO:tasks.workunit.client.1.smithi152.stdout:7/409: rmdir d14/d21/d47 39 2024-03-04T16:04:38.517 INFO:tasks.workunit.client.1.smithi152.stdout:1/488: readlink d19/d20/d2f/d6b/l4e 0 2024-03-04T16:04:38.518 INFO:tasks.workunit.client.1.smithi152.stdout:1/489: write d19/d75/f91 [890656,87857] 0 2024-03-04T16:04:38.519 INFO:tasks.workunit.client.1.smithi152.stdout:1/490: chown d19/d20/d2f/d6b/l44 2886434 1 2024-03-04T16:04:38.520 INFO:tasks.workunit.client.1.smithi152.stdout:3/451: fdatasync d6/d14/f90 0 2024-03-04T16:04:38.522 INFO:tasks.workunit.client.1.smithi152.stdout:4/365: read - d8/d3d/f68 zero size 2024-03-04T16:04:38.525 INFO:tasks.workunit.client.1.smithi152.stdout:7/410: rename d14/d21/f55 to d14/d71/f7b 0 2024-03-04T16:04:38.526 INFO:tasks.workunit.client.1.smithi152.stdout:9/359: mknod d2/d13/d23/c72 0 2024-03-04T16:04:38.528 INFO:tasks.workunit.client.1.smithi152.stdout:1/491: unlink d19/l27 0 2024-03-04T16:04:38.532 INFO:tasks.workunit.client.1.smithi152.stdout:4/366: creat d8/d9/d7a/f7d x:0 0 0 2024-03-04T16:04:38.535 INFO:tasks.workunit.client.1.smithi152.stdout:7/411: creat d14/d21/d31/f7c x:0 0 0 2024-03-04T16:04:38.536 INFO:tasks.workunit.client.1.smithi152.stdout:7/412: write f4 [811832,63801] 0 2024-03-04T16:04:38.537 INFO:tasks.workunit.client.1.smithi152.stdout:7/413: write d14/d2b/f60 [2548817,23472] 0 2024-03-04T16:04:38.538 INFO:tasks.workunit.client.1.smithi152.stdout:7/414: chown d14/c2e 393504 1 2024-03-04T16:04:38.538 INFO:tasks.workunit.client.1.smithi152.stdout:1/492: symlink d19/d20/d2f/d65/la5 0 2024-03-04T16:04:38.547 INFO:tasks.workunit.client.1.smithi152.stdout:4/367: dwrite d8/d4e/f64 [0,4194304] 0 2024-03-04T16:04:38.553 INFO:tasks.workunit.client.1.smithi152.stdout:3/452: dwrite d6/f9 [0,4194304] 0 2024-03-04T16:04:38.555 INFO:tasks.workunit.client.1.smithi152.stdout:7/415: write d14/d21/d48/f4b [3774068,76778] 0 2024-03-04T16:04:38.555 INFO:tasks.workunit.client.1.smithi152.stdout:3/453: write d6/d14/d41/d49/f62 [4681845,12320] 0 2024-03-04T16:04:38.560 INFO:tasks.workunit.client.1.smithi152.stdout:4/368: link d8/d9/l22 d8/d4e/l7e 0 2024-03-04T16:04:38.561 INFO:tasks.workunit.client.1.smithi152.stdout:9/360: dwrite d2/d13/d38/f58 [0,4194304] 0 2024-03-04T16:04:38.569 INFO:tasks.workunit.client.1.smithi152.stdout:4/369: rename d8/d4e/f75 to d8/d2f/d7c/f7f 0 2024-03-04T16:04:38.569 INFO:tasks.workunit.client.1.smithi152.stdout:4/370: dread - d8/d3d/f5f zero size 2024-03-04T16:04:38.572 INFO:tasks.workunit.client.1.smithi152.stdout:9/361: symlink d2/d13/d23/l73 0 2024-03-04T16:04:38.579 INFO:tasks.workunit.client.1.smithi152.stdout:9/362: dwrite d2/d13/f15 [0,4194304] 0 2024-03-04T16:04:38.580 INFO:tasks.workunit.client.1.smithi152.stdout:4/371: fdatasync d8/f37 0 2024-03-04T16:04:38.581 INFO:tasks.workunit.client.1.smithi152.stdout:3/454: dread d6/d14/d93/f77 [0,4194304] 0 2024-03-04T16:04:38.585 INFO:tasks.workunit.client.1.smithi152.stdout:9/363: fsync d2/d13/d23/f33 0 2024-03-04T16:04:38.586 INFO:tasks.workunit.client.1.smithi152.stdout:9/364: chown d2/fd 1629 1 2024-03-04T16:04:38.589 INFO:tasks.workunit.client.1.smithi152.stdout:4/372: dwrite d8/d9/d24/d26/f2e [0,4194304] 0 2024-03-04T16:04:38.590 INFO:tasks.workunit.client.1.smithi152.stdout:3/455: readlink d6/d10/d2d/d6e/l79 0 2024-03-04T16:04:38.591 INFO:tasks.workunit.client.1.smithi152.stdout:4/373: read - d8/d3d/f68 zero size 2024-03-04T16:04:38.597 INFO:tasks.workunit.client.1.smithi152.stdout:7/416: dwrite f7 [0,4194304] 0 2024-03-04T16:04:38.598 INFO:tasks.workunit.client.1.smithi152.stdout:3/456: creat d6/d53/f99 x:0 0 0 2024-03-04T16:04:38.600 INFO:tasks.workunit.client.1.smithi152.stdout:4/374: chown d8/d9/c7b 5441967 1 2024-03-04T16:04:38.601 INFO:tasks.workunit.client.1.smithi152.stdout:4/375: read d8/d4e/f64 [3252059,9126] 0 2024-03-04T16:04:38.603 INFO:tasks.workunit.client.1.smithi152.stdout:4/376: write d8/d3d/f4c [853335,10477] 0 2024-03-04T16:04:38.606 INFO:tasks.workunit.client.1.smithi152.stdout:7/417: creat d14/d71/f7d x:0 0 0 2024-03-04T16:04:38.606 INFO:tasks.workunit.client.1.smithi152.stdout:7/418: fdatasync d14/d21/f25 0 2024-03-04T16:04:38.607 INFO:tasks.workunit.client.1.smithi152.stdout:4/377: symlink d8/d9/d24/d26/d42/l80 0 2024-03-04T16:04:38.607 INFO:tasks.workunit.client.1.smithi152.stdout:3/457: dread d6/d53/f88 [0,4194304] 0 2024-03-04T16:04:38.610 INFO:tasks.workunit.client.1.smithi152.stdout:4/378: fsync d8/f3e 0 2024-03-04T16:04:38.611 INFO:tasks.workunit.client.1.smithi152.stdout:4/379: read - d8/d9/d7a/f7d zero size 2024-03-04T16:04:38.612 INFO:tasks.workunit.client.1.smithi152.stdout:3/458: creat d6/d14/d58/f9a x:0 0 0 2024-03-04T16:04:38.613 INFO:tasks.workunit.client.1.smithi152.stdout:3/459: read - d6/d14/d93/f89 zero size 2024-03-04T16:04:38.614 INFO:tasks.workunit.client.1.smithi152.stdout:3/460: truncate d6/d60/f67 272066 0 2024-03-04T16:04:38.618 INFO:tasks.workunit.client.1.smithi152.stdout:4/380: dwrite d8/d9/f28 [4194304,4194304] 0 2024-03-04T16:04:38.619 INFO:tasks.workunit.client.1.smithi152.stdout:3/461: creat d6/d10/d2d/d6e/f9b x:0 0 0 2024-03-04T16:04:38.620 INFO:tasks.workunit.client.1.smithi152.stdout:4/381: fsync d8/d9/d24/f40 0 2024-03-04T16:04:38.621 INFO:tasks.workunit.client.1.smithi152.stdout:9/365: dread d2/d13/d23/f33 [0,4194304] 0 2024-03-04T16:04:38.621 INFO:tasks.workunit.client.1.smithi152.stdout:3/462: chown d6/d14/d41/l48 392 1 2024-03-04T16:04:38.622 INFO:tasks.workunit.client.1.smithi152.stdout:3/463: write d6/f3a [1474659,40033] 0 2024-03-04T16:04:38.623 INFO:tasks.workunit.client.1.smithi152.stdout:4/382: truncate d8/d9/d24/f66 4258433 0 2024-03-04T16:04:38.627 INFO:tasks.workunit.client.1.smithi152.stdout:4/383: creat d8/d9/d24/d26/d42/f81 x:0 0 0 2024-03-04T16:04:38.630 INFO:tasks.workunit.client.1.smithi152.stdout:9/366: rename d2/d13/d23/l73 to d2/l74 0 2024-03-04T16:04:38.634 INFO:tasks.workunit.client.1.smithi152.stdout:4/384: creat d8/d78/f82 x:0 0 0 2024-03-04T16:04:38.635 INFO:tasks.workunit.client.1.smithi152.stdout:4/385: read f0 [2284668,31964] 0 2024-03-04T16:04:38.643 INFO:tasks.workunit.client.1.smithi152.stdout:4/386: symlink d8/d58/l83 0 2024-03-04T16:04:38.645 INFO:tasks.workunit.client.1.smithi152.stdout:9/367: link d2/d13/d23/c5a d2/d13/c75 0 2024-03-04T16:04:38.647 INFO:tasks.workunit.client.1.smithi152.stdout:4/387: mkdir d8/d2f/d6f/d84 0 2024-03-04T16:04:38.650 INFO:tasks.workunit.client.1.smithi152.stdout:3/464: dwrite d6/d10/d2d/f4c [0,4194304] 0 2024-03-04T16:04:38.652 INFO:tasks.workunit.client.1.smithi152.stdout:9/368: truncate d2/d13/d2e/f44 428245 0 2024-03-04T16:04:38.653 INFO:tasks.workunit.client.1.smithi152.stdout:9/369: write d2/fd [1284928,8802] 0 2024-03-04T16:04:38.655 INFO:tasks.workunit.client.1.smithi152.stdout:4/388: creat d8/d9/d79/f85 x:0 0 0 2024-03-04T16:04:38.659 INFO:tasks.workunit.client.1.smithi152.stdout:9/370: creat d2/d13/d2e/f76 x:0 0 0 2024-03-04T16:04:38.660 INFO:tasks.workunit.client.1.smithi152.stdout:4/389: mkdir d8/d4e/d86 0 2024-03-04T16:04:38.661 INFO:tasks.workunit.client.1.smithi152.stdout:4/390: readlink d8/d9/l3a 0 2024-03-04T16:04:38.662 INFO:tasks.workunit.client.1.smithi152.stdout:4/391: read d8/d9/d24/d26/f2e [585017,130324] 0 2024-03-04T16:04:38.668 INFO:tasks.workunit.client.1.smithi152.stdout:4/392: rmdir d8/d9/d24/d26/d42 39 2024-03-04T16:04:38.669 INFO:tasks.workunit.client.1.smithi152.stdout:3/465: link d6/f4a d6/f9c 0 2024-03-04T16:04:38.669 INFO:tasks.workunit.client.1.smithi152.stdout:3/466: chown d6/d60/d8a 4158 1 2024-03-04T16:04:38.672 INFO:tasks.workunit.client.1.smithi152.stdout:4/393: mkdir d8/d2f/d7c/d87 0 2024-03-04T16:04:38.674 INFO:tasks.workunit.client.1.smithi152.stdout:3/467: mknod d6/d10/d2d/d4d/c9d 0 2024-03-04T16:04:38.679 INFO:tasks.workunit.client.1.smithi152.stdout:3/468: dwrite d6/d14/d93/f89 [0,4194304] 0 2024-03-04T16:04:38.680 INFO:tasks.workunit.client.1.smithi152.stdout:3/469: write d6/ff [1098519,68781] 0 2024-03-04T16:04:38.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/371: write d2/d13/d23/f5f [885296,125108] 0 2024-03-04T16:04:38.684 INFO:tasks.workunit.client.1.smithi152.stdout:4/394: creat d8/d9/d24/f88 x:0 0 0 2024-03-04T16:04:38.687 INFO:tasks.workunit.client.1.smithi152.stdout:9/372: link d2/d13/c6d d2/d63/d69/c77 0 2024-03-04T16:04:38.689 INFO:tasks.workunit.client.1.smithi152.stdout:3/470: rename d6/d10/d2d/d6e/f9b to d6/d10/d2d/f9e 0 2024-03-04T16:04:38.691 INFO:tasks.workunit.client.1.smithi152.stdout:3/471: mknod d6/d60/c9f 0 2024-03-04T16:04:38.692 INFO:tasks.workunit.client.1.smithi152.stdout:3/472: fsync d6/d53/f8f 0 2024-03-04T16:04:38.697 INFO:tasks.workunit.client.1.smithi152.stdout:3/473: creat d6/d5c/fa0 x:0 0 0 2024-03-04T16:04:38.700 INFO:tasks.workunit.client.1.smithi152.stdout:9/373: getdents d2/d13/d23 0 2024-03-04T16:04:38.702 INFO:tasks.workunit.client.1.smithi152.stdout:9/374: write d2/f28 [4151523,42225] 0 2024-03-04T16:04:38.703 INFO:tasks.workunit.client.1.smithi152.stdout:9/375: truncate d2/d13/d2e/f4f 1012302 0 2024-03-04T16:04:38.704 INFO:tasks.workunit.client.1.smithi152.stdout:9/376: write d2/d13/d23/f33 [6589385,26432] 0 2024-03-04T16:04:38.705 INFO:tasks.workunit.client.1.smithi152.stdout:9/377: chown d2/d13/d23/d42/f46 43320998 1 2024-03-04T16:04:38.709 INFO:tasks.workunit.client.1.smithi152.stdout:9/378: creat d2/d13/d23/d42/f78 x:0 0 0 2024-03-04T16:04:38.710 INFO:tasks.workunit.client.1.smithi152.stdout:9/379: mkdir d2/d13/d38/d79 0 2024-03-04T16:04:38.713 INFO:tasks.workunit.client.1.smithi152.stdout:9/380: creat d2/d6e/f7a x:0 0 0 2024-03-04T16:04:38.714 INFO:tasks.workunit.client.1.smithi152.stdout:9/381: write d2/fb [904111,26096] 0 2024-03-04T16:04:38.714 INFO:tasks.workunit.client.1.smithi152.stdout:3/474: dread d6/d14/d41/d49/f83 [0,4194304] 0 2024-03-04T16:04:38.715 INFO:tasks.workunit.client.1.smithi152.stdout:3/475: fsync d6/d5c/f97 0 2024-03-04T16:04:38.715 INFO:tasks.workunit.client.1.smithi152.stdout:3/476: readlink d6/d60/l61 0 2024-03-04T16:04:38.717 INFO:tasks.workunit.client.1.smithi152.stdout:9/382: mknod d2/d63/d69/c7b 0 2024-03-04T16:04:38.718 INFO:tasks.workunit.client.1.smithi152.stdout:3/477: mkdir d6/d53/da1 0 2024-03-04T16:04:38.722 INFO:tasks.workunit.client.1.smithi152.stdout:9/383: truncate d2/d13/d23/f5b 1130439 0 2024-03-04T16:04:38.724 INFO:tasks.workunit.client.1.smithi152.stdout:3/478: dwrite d6/d53/f8f [0,4194304] 0 2024-03-04T16:04:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:38 smithi116 ceph-mon[29120]: pgmap v35: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 20 MiB/s rd, 114 MiB/s wr, 239 op/s 2024-03-04T16:04:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:38 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:38 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:38.727 INFO:tasks.workunit.client.1.smithi152.stdout:9/384: creat d2/d13/d35/f7c x:0 0 0 2024-03-04T16:04:38.731 INFO:tasks.workunit.client.1.smithi152.stdout:9/385: mkdir d2/d13/d23/d66/d7d 0 2024-03-04T16:04:38.731 INFO:tasks.workunit.client.1.smithi152.stdout:3/479: dwrite d6/d10/d2d/f4c [4194304,4194304] 0 2024-03-04T16:04:38.741 INFO:tasks.workunit.client.1.smithi152.stdout:9/386: symlink d2/d13/d23/d66/d7d/l7e 0 2024-03-04T16:04:38.742 INFO:tasks.workunit.client.1.smithi152.stdout:3/480: link d6/d10/d2d/f64 d6/d53/fa2 0 2024-03-04T16:04:38.743 INFO:tasks.workunit.client.1.smithi152.stdout:3/481: chown d6/f42 9 1 2024-03-04T16:04:38.746 INFO:tasks.workunit.client.1.smithi152.stdout:9/387: unlink d2/l30 0 2024-03-04T16:04:38.749 INFO:tasks.workunit.client.1.smithi152.stdout:9/388: creat d2/d13/d35/f7f x:0 0 0 2024-03-04T16:04:38.751 INFO:tasks.workunit.client.1.smithi152.stdout:9/389: write d2/fb [2887033,91607] 0 2024-03-04T16:04:38.752 INFO:tasks.workunit.client.1.smithi152.stdout:3/482: dwrite d6/d60/f81 [0,4194304] 0 2024-03-04T16:04:38.753 INFO:tasks.workunit.client.1.smithi152.stdout:3/483: chown d6/d5c 298791 1 2024-03-04T16:04:38.767 INFO:tasks.workunit.client.1.smithi152.stdout:3/484: getdents d6/d60 0 2024-03-04T16:04:38.773 INFO:tasks.workunit.client.1.smithi152.stdout:9/390: truncate d2/d13/f18 511329 0 2024-03-04T16:04:38.776 INFO:tasks.workunit.client.1.smithi152.stdout:9/391: mkdir d2/d13/d23/d71/d80 0 2024-03-04T16:04:38.778 INFO:tasks.workunit.client.1.smithi152.stdout:9/392: read d2/d13/f1a [471579,51127] 0 2024-03-04T16:04:38.781 INFO:tasks.workunit.client.1.smithi152.stdout:9/393: rename d2/c14 to d2/d13/d23/d66/d7d/c81 0 2024-03-04T16:04:38.783 INFO:tasks.workunit.client.1.smithi152.stdout:3/485: dread d6/d14/f2e [0,4194304] 0 2024-03-04T16:04:38.786 INFO:tasks.workunit.client.1.smithi152.stdout:3/486: creat d6/d14/d41/fa3 x:0 0 0 2024-03-04T16:04:38.788 INFO:tasks.workunit.client.1.smithi152.stdout:9/394: read d2/d13/f18 [197472,105158] 0 2024-03-04T16:04:38.789 INFO:tasks.workunit.client.1.smithi152.stdout:3/487: creat d6/d60/d8e/fa4 x:0 0 0 2024-03-04T16:04:38.792 INFO:tasks.workunit.client.1.smithi152.stdout:9/395: mknod d2/d13/d38/c82 0 2024-03-04T16:04:38.800 INFO:tasks.workunit.client.1.smithi152.stdout:9/396: write d2/d13/d2e/f44 [7877,112967] 0 2024-03-04T16:04:38.803 INFO:tasks.workunit.client.1.smithi152.stdout:9/397: creat d2/d13/d23/d66/d7d/f83 x:0 0 0 2024-03-04T16:04:38.808 INFO:tasks.workunit.client.1.smithi152.stdout:9/398: link d2/d13/c1e d2/d63/d69/c84 0 2024-03-04T16:04:38.808 INFO:tasks.workunit.client.1.smithi152.stdout:9/399: chown d2/d13/d38/f58 252 1 2024-03-04T16:04:38.813 INFO:tasks.workunit.client.1.smithi152.stdout:9/400: rename d2/d63/d69/c84 to d2/d13/d23/d71/c85 0 2024-03-04T16:04:38.823 INFO:tasks.workunit.client.1.smithi152.stdout:9/401: truncate d2/d13/d23/f5f 380823 0 2024-03-04T16:04:38.827 INFO:tasks.workunit.client.1.smithi152.stdout:9/402: dread d2/d13/d23/f4c [0,4194304] 0 2024-03-04T16:04:38.831 INFO:tasks.workunit.client.1.smithi152.stdout:9/403: link d2/d13/f18 d2/d13/d23/d71/f86 0 2024-03-04T16:04:38.833 INFO:tasks.workunit.client.1.smithi152.stdout:9/404: symlink d2/d63/d69/l87 0 2024-03-04T16:04:38.834 INFO:tasks.workunit.client.1.smithi152.stdout:9/405: write d2/d13/d23/f40 [1004036,95572] 0 2024-03-04T16:04:38.835 INFO:tasks.workunit.client.1.smithi152.stdout:9/406: readlink d2/d13/d23/l60 0 2024-03-04T16:04:38.836 INFO:tasks.workunit.client.1.smithi152.stdout:9/407: chown d2/d13/d23/d66 93282 1 2024-03-04T16:04:38.841 INFO:tasks.workunit.client.1.smithi152.stdout:9/408: link d2/d13/d35/l55 d2/d13/d23/d42/l88 0 2024-03-04T16:04:38.843 INFO:tasks.workunit.client.1.smithi152.stdout:9/409: chown d2/d13/d35/c47 19068681 1 2024-03-04T16:04:38.846 INFO:tasks.workunit.client.1.smithi152.stdout:9/410: creat d2/d13/d38/d79/f89 x:0 0 0 2024-03-04T16:04:38.851 INFO:tasks.workunit.client.1.smithi152.stdout:9/411: dwrite d2/f2d [4194304,4194304] 0 2024-03-04T16:04:38.852 INFO:tasks.workunit.client.1.smithi152.stdout:9/412: truncate d2/f67 42584 0 2024-03-04T16:04:38.852 INFO:tasks.workunit.client.1.smithi152.stdout:9/413: stat d2/fd 0 2024-03-04T16:04:38.853 INFO:tasks.workunit.client.1.smithi152.stdout:9/414: dread - d2/f61 zero size 2024-03-04T16:04:38.855 INFO:tasks.workunit.client.1.smithi152.stdout:9/415: symlink d2/d63/d69/l8a 0 2024-03-04T16:04:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:38 smithi152 ceph-mon[36876]: pgmap v35: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 20 MiB/s rd, 114 MiB/s wr, 239 op/s 2024-03-04T16:04:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:38 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:38 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:38.859 INFO:tasks.workunit.client.1.smithi152.stdout:9/416: creat d2/d63/d69/f8b x:0 0 0 2024-03-04T16:04:38.860 INFO:tasks.workunit.client.1.smithi152.stdout:9/417: dread - d2/d13/d23/d66/d7d/f83 zero size 2024-03-04T16:04:38.861 INFO:tasks.workunit.client.1.smithi152.stdout:9/418: creat d2/d13/d35/f8c x:0 0 0 2024-03-04T16:04:38.864 INFO:tasks.workunit.client.1.smithi152.stdout:9/419: creat d2/f8d x:0 0 0 2024-03-04T16:04:38.866 INFO:tasks.workunit.client.1.smithi152.stdout:9/420: read - d2/d13/d35/f34 zero size 2024-03-04T16:04:38.867 INFO:tasks.workunit.client.1.smithi152.stdout:9/421: write d2/d13/d35/f8c [910001,95439] 0 2024-03-04T16:04:38.867 INFO:tasks.workunit.client.1.smithi152.stdout:9/422: stat d2/d13/d2e 0 2024-03-04T16:04:38.869 INFO:tasks.workunit.client.1.smithi152.stdout:9/423: rmdir d2/d13/d2e 39 2024-03-04T16:04:38.872 INFO:tasks.workunit.client.1.smithi152.stdout:9/424: dread d2/d13/d23/f40 [0,4194304] 0 2024-03-04T16:04:38.873 INFO:tasks.workunit.client.1.smithi152.stdout:9/425: chown d2/d13/d23/c72 31576822 1 2024-03-04T16:04:38.877 INFO:tasks.workunit.client.1.smithi152.stdout:9/426: dread d2/d13/f52 [0,4194304] 0 2024-03-04T16:04:38.881 INFO:tasks.workunit.client.1.smithi152.stdout:9/427: dread d2/d13/d23/f5b [0,4194304] 0 2024-03-04T16:04:38.882 INFO:tasks.workunit.client.1.smithi152.stdout:9/428: chown d2/d13/d23/f5c 212487017 1 2024-03-04T16:04:38.884 INFO:tasks.workunit.client.1.smithi152.stdout:9/429: mknod d2/d63/c8e 0 2024-03-04T16:04:38.885 INFO:tasks.workunit.client.1.smithi152.stdout:9/430: truncate d2/f19 3650459 0 2024-03-04T16:04:38.890 INFO:tasks.workunit.client.1.smithi152.stdout:9/431: dwrite d2/f67 [0,4194304] 0 2024-03-04T16:04:38.892 INFO:tasks.workunit.client.1.smithi152.stdout:9/432: read d2/d13/d23/f4e [242743,51108] 0 2024-03-04T16:04:38.895 INFO:tasks.workunit.client.1.smithi152.stdout:9/433: mknod d2/d13/d23/d66/d7d/c8f 0 2024-03-04T16:04:38.896 INFO:tasks.workunit.client.1.smithi152.stdout:9/434: dread d2/d13/d23/f5b [0,4194304] 0 2024-03-04T16:04:38.908 INFO:tasks.workunit.client.1.smithi152.stdout:9/435: write d2/d13/d23/f5b [1339896,93104] 0 2024-03-04T16:04:38.922 INFO:tasks.workunit.client.1.smithi152.stdout:9/436: dread d2/f19 [0,4194304] 0 2024-03-04T16:04:38.923 INFO:tasks.workunit.client.1.smithi152.stdout:9/437: write d2/f28 [3134704,90444] 0 2024-03-04T16:04:38.929 INFO:tasks.workunit.client.1.smithi152.stdout:9/438: read d2/d13/f1a [571358,125525] 0 2024-03-04T16:04:38.938 INFO:tasks.workunit.client.1.smithi152.stdout:9/439: rename d2/d13/d23/d66/d7d/f83 to d2/d63/d69/f90 0 2024-03-04T16:04:38.944 INFO:tasks.workunit.client.1.smithi152.stdout:9/440: creat d2/f91 x:0 0 0 2024-03-04T16:04:38.954 INFO:tasks.workunit.client.1.smithi152.stdout:9/441: dwrite d2/d13/d2e/f76 [0,4194304] 0 2024-03-04T16:04:38.955 INFO:tasks.workunit.client.1.smithi152.stdout:9/442: chown d2/d63/f6a 39784 1 2024-03-04T16:04:38.959 INFO:tasks.workunit.client.1.smithi152.stdout:9/443: symlink d2/d13/d23/d71/d80/l92 0 2024-03-04T16:04:38.963 INFO:tasks.workunit.client.1.smithi152.stdout:9/444: creat d2/d13/f93 x:0 0 0 2024-03-04T16:04:38.964 INFO:tasks.workunit.client.1.smithi152.stdout:9/445: read - d2/d13/d35/f7c zero size 2024-03-04T16:04:38.964 INFO:tasks.workunit.client.1.smithi152.stdout:9/446: truncate d2/f67 5159631 0 2024-03-04T16:04:38.967 INFO:tasks.workunit.client.1.smithi152.stdout:9/447: rename d2/d13/d35/f65 to d2/d63/f94 0 2024-03-04T16:04:38.973 INFO:tasks.workunit.client.1.smithi152.stdout:9/448: dwrite d2/d13/d23/f29 [0,4194304] 0 2024-03-04T16:04:38.974 INFO:tasks.workunit.client.1.smithi152.stdout:8/432: sync 2024-03-04T16:04:38.974 INFO:tasks.workunit.client.1.smithi152.stdout:6/528: sync 2024-03-04T16:04:38.974 INFO:tasks.workunit.client.1.smithi152.stdout:2/422: sync 2024-03-04T16:04:38.974 INFO:tasks.workunit.client.1.smithi152.stdout:5/490: sync 2024-03-04T16:04:38.974 INFO:tasks.workunit.client.1.smithi152.stdout:0/400: sync 2024-03-04T16:04:38.975 INFO:tasks.workunit.client.1.smithi152.stdout:6/529: chown d8/lc 3 1 2024-03-04T16:04:38.979 INFO:tasks.workunit.client.1.smithi152.stdout:0/401: stat d9/d2c/d2e/c3f 0 2024-03-04T16:04:38.979 INFO:tasks.workunit.client.1.smithi152.stdout:0/402: dread - d9/d1b/f4f zero size 2024-03-04T16:04:38.980 INFO:tasks.workunit.client.1.smithi152.stdout:6/530: mknod d8/d4b/caf 0 2024-03-04T16:04:38.981 INFO:tasks.workunit.client.1.smithi152.stdout:6/531: write d8/d23/d3a/d71/d9d/fa8 [4796963,54057] 0 2024-03-04T16:04:38.982 INFO:tasks.workunit.client.1.smithi152.stdout:8/433: mknod d2/d3/d21/c82 0 2024-03-04T16:04:38.983 INFO:tasks.workunit.client.1.smithi152.stdout:2/423: symlink da/df/d2e/d51/l8a 0 2024-03-04T16:04:38.984 INFO:tasks.workunit.client.1.smithi152.stdout:8/434: truncate d2/d8/dd/d22/f78 203952 0 2024-03-04T16:04:38.984 INFO:tasks.workunit.client.1.smithi152.stdout:8/435: chown d2/d3/f4b 3821894 1 2024-03-04T16:04:38.986 INFO:tasks.workunit.client.1.smithi152.stdout:8/436: write d2/d8/dd/d22/f3b [5869107,128969] 0 2024-03-04T16:04:38.989 INFO:tasks.workunit.client.1.smithi152.stdout:2/424: write da/d24/f32 [4918679,76929] 0 2024-03-04T16:04:38.991 INFO:tasks.workunit.client.1.smithi152.stdout:6/532: symlink d8/d92/d59/d7d/d91/lb0 0 2024-03-04T16:04:38.992 INFO:tasks.workunit.client.1.smithi152.stdout:0/403: dwrite d9/d2c/d2e/d38/d56/d5b/f6b [4194304,4194304] 0 2024-03-04T16:04:38.992 INFO:tasks.workunit.client.1.smithi152.stdout:8/437: dread - d2/d3/d15/d1c/f59 zero size 2024-03-04T16:04:38.998 INFO:tasks.workunit.client.1.smithi152.stdout:0/404: rmdir d9/d1b/d62 39 2024-03-04T16:04:38.999 INFO:tasks.workunit.client.1.smithi152.stdout:8/438: rmdir d2/d8/dd/d22/d26 39 2024-03-04T16:04:39.001 INFO:tasks.workunit.client.1.smithi152.stdout:6/533: dwrite d8/f56 [0,4194304] 0 2024-03-04T16:04:39.001 INFO:tasks.workunit.client.1.smithi152.stdout:2/425: creat da/d2c/d5e/f8b x:0 0 0 2024-03-04T16:04:39.002 INFO:tasks.workunit.client.1.smithi152.stdout:8/439: readlink d2/d3/d21/d2d/d48/l4c 0 2024-03-04T16:04:39.004 INFO:tasks.workunit.client.1.smithi152.stdout:2/426: dread - da/df/d2e/d61/f84 zero size 2024-03-04T16:04:39.005 INFO:tasks.workunit.client.1.smithi152.stdout:6/534: rename d8/d52/d84 to d8/d92/db1 0 2024-03-04T16:04:39.010 INFO:tasks.workunit.client.1.smithi152.stdout:6/535: mkdir d8/d8f/db2 0 2024-03-04T16:04:39.011 INFO:tasks.workunit.client.1.smithi152.stdout:6/536: rename d8/d92 to d8/d92/d59/d7d/db3 22 2024-03-04T16:04:39.011 INFO:tasks.workunit.client.1.smithi152.stdout:6/537: readlink d8/lf 0 2024-03-04T16:04:39.015 INFO:tasks.workunit.client.1.smithi152.stdout:2/427: mkdir da/d5f/d85/d8c 0 2024-03-04T16:04:39.018 INFO:tasks.workunit.client.1.smithi152.stdout:2/428: mkdir da/df/d2e/d51/d8d 0 2024-03-04T16:04:39.019 INFO:tasks.workunit.client.1.smithi152.stdout:2/429: fsync da/d2c/d5e/f69 0 2024-03-04T16:04:39.021 INFO:tasks.workunit.client.1.smithi152.stdout:6/538: getdents d8 0 2024-03-04T16:04:39.022 INFO:tasks.workunit.client.1.smithi152.stdout:2/430: mkdir da/df/d2e/d61/d8e 0 2024-03-04T16:04:39.023 INFO:tasks.workunit.client.1.smithi152.stdout:5/491: dread d6/f38 [0,4194304] 0 2024-03-04T16:04:39.023 INFO:tasks.workunit.client.1.smithi152.stdout:6/539: truncate d8/d92/fad 832810 0 2024-03-04T16:04:39.027 INFO:tasks.workunit.client.1.smithi152.stdout:2/431: dread da/df/d35/f46 [0,4194304] 0 2024-03-04T16:04:39.027 INFO:tasks.workunit.client.1.smithi152.stdout:6/540: symlink d8/d10/lb4 0 2024-03-04T16:04:39.028 INFO:tasks.workunit.client.1.smithi152.stdout:6/541: write d8/d2a/fa1 [662110,107246] 0 2024-03-04T16:04:39.036 INFO:tasks.workunit.client.1.smithi152.stdout:8/440: dwrite d2/d3/d21/f1f [0,4194304] 0 2024-03-04T16:04:39.038 INFO:tasks.workunit.client.1.smithi152.stdout:6/542: creat d8/d23/d3a/d8d/d96/fb5 x:0 0 0 2024-03-04T16:04:39.042 INFO:tasks.workunit.client.1.smithi152.stdout:2/432: creat da/df/d2e/f8f x:0 0 0 2024-03-04T16:04:39.044 INFO:tasks.workunit.client.1.smithi152.stdout:5/492: dwrite d6/d8/f2f [0,4194304] 0 2024-03-04T16:04:39.047 INFO:tasks.workunit.client.1.smithi152.stdout:8/441: dwrite d2/d8/dd/d22/d49/f55 [0,4194304] 0 2024-03-04T16:04:39.047 INFO:tasks.workunit.client.1.smithi152.stdout:6/543: mknod d8/d23/d93/d99/cb6 0 2024-03-04T16:04:39.049 INFO:tasks.workunit.client.1.smithi152.stdout:6/544: write d8/d10/f44 [3963521,7204] 0 2024-03-04T16:04:39.051 INFO:tasks.workunit.client.1.smithi152.stdout:5/493: write d6/d8/f30 [903766,125320] 0 2024-03-04T16:04:39.058 INFO:tasks.workunit.client.1.smithi152.stdout:8/442: write d2/d8/dd/d22/d49/d5d/f44 [397149,11570] 0 2024-03-04T16:04:39.059 INFO:tasks.workunit.client.1.smithi152.stdout:6/545: mknod d8/d92/d59/cb7 0 2024-03-04T16:04:39.059 INFO:tasks.workunit.client.1.smithi152.stdout:8/443: chown d2/f37 116041 1 2024-03-04T16:04:39.060 INFO:tasks.workunit.client.1.smithi152.stdout:8/444: stat d2/d3/d21/d2d/l4d 0 2024-03-04T16:04:39.062 INFO:tasks.workunit.client.1.smithi152.stdout:8/445: truncate d2/d8/dd/d22/d6c/f74 456591 0 2024-03-04T16:04:39.065 INFO:tasks.workunit.client.1.smithi152.stdout:5/494: rmdir d6/d8 39 2024-03-04T16:04:39.066 INFO:tasks.workunit.client.1.smithi152.stdout:2/433: creat da/df/d2e/f90 x:0 0 0 2024-03-04T16:04:39.068 INFO:tasks.workunit.client.1.smithi152.stdout:6/546: symlink d8/d92/d59/d7d/d91/lb8 0 2024-03-04T16:04:39.069 INFO:tasks.workunit.client.1.smithi152.stdout:5/495: readlink d6/d8/d1f/l67 0 2024-03-04T16:04:39.073 INFO:tasks.workunit.client.1.smithi152.stdout:2/434: write da/d24/d47/f5a [1365772,26193] 0 2024-03-04T16:04:39.074 INFO:tasks.workunit.client.1.smithi152.stdout:8/446: mkdir d2/d8/dd/d22/d83 0 2024-03-04T16:04:39.076 INFO:tasks.workunit.client.1.smithi152.stdout:5/496: dread d6/d50/f83 [0,4194304] 0 2024-03-04T16:04:39.080 INFO:tasks.workunit.client.1.smithi152.stdout:6/547: rename d8/d23/d3a/d8d/d96/f97 to d8/d23/d3a/d8d/d96/fb9 0 2024-03-04T16:04:39.082 INFO:tasks.workunit.client.1.smithi152.stdout:2/435: write da/f23 [682806,10085] 0 2024-03-04T16:04:39.083 INFO:tasks.workunit.client.1.smithi152.stdout:8/447: chown d2/d8/dd/d22/d26/l43 9 1 2024-03-04T16:04:39.084 INFO:tasks.workunit.client.1.smithi152.stdout:5/497: dwrite d6/d8/d1f/f57 [0,4194304] 0 2024-03-04T16:04:39.085 INFO:tasks.workunit.client.1.smithi152.stdout:8/448: stat d2/d8/dd/d22/d26/l3e 0 2024-03-04T16:04:39.089 INFO:tasks.workunit.client.1.smithi152.stdout:6/548: creat d8/d8f/da3/fba x:0 0 0 2024-03-04T16:04:39.091 INFO:tasks.workunit.client.1.smithi152.stdout:8/449: rmdir d2/d8/dd/d22/d6c 39 2024-03-04T16:04:39.092 INFO:tasks.workunit.client.1.smithi152.stdout:5/498: truncate d6/d8/f74 845519 0 2024-03-04T16:04:39.094 INFO:tasks.workunit.client.1.smithi152.stdout:2/436: dwrite da/df/d2e/f90 [0,4194304] 0 2024-03-04T16:04:39.095 INFO:tasks.workunit.client.1.smithi152.stdout:6/549: mknod d8/d23/d98/cbb 0 2024-03-04T16:04:39.096 INFO:tasks.workunit.client.1.smithi152.stdout:5/499: fdatasync d6/d8/f32 0 2024-03-04T16:04:39.096 INFO:tasks.workunit.client.1.smithi152.stdout:6/550: stat d8/d23/d93 0 2024-03-04T16:04:39.097 INFO:tasks.workunit.client.1.smithi152.stdout:6/551: chown d8/d23/d3a/d71/c86 0 1 2024-03-04T16:04:39.099 INFO:tasks.workunit.client.1.smithi152.stdout:5/500: truncate d6/d8/d5a/d64/d76/d9f/f49 1716121 0 2024-03-04T16:04:39.100 INFO:tasks.workunit.client.1.smithi152.stdout:8/450: mkdir d2/d3/d15/d1c/d84 0 2024-03-04T16:04:39.100 INFO:tasks.workunit.client.1.smithi152.stdout:2/437: symlink da/df/d58/d6e/l91 0 2024-03-04T16:04:39.101 INFO:tasks.workunit.client.1.smithi152.stdout:5/501: chown d6/d8/f9a 368458778 1 2024-03-04T16:04:39.101 INFO:tasks.workunit.client.1.smithi152.stdout:5/502: chown d6/d8/d1f/f2e 590565 1 2024-03-04T16:04:39.102 INFO:tasks.workunit.client.1.smithi152.stdout:6/552: creat d8/d23/d93/fbc x:0 0 0 2024-03-04T16:04:39.104 INFO:tasks.workunit.client.1.smithi152.stdout:8/451: symlink d2/d8/dd/l85 0 2024-03-04T16:04:39.105 INFO:tasks.workunit.client.1.smithi152.stdout:5/503: unlink d6/d8/d1f/d88/c91 0 2024-03-04T16:04:39.106 INFO:tasks.workunit.client.1.smithi152.stdout:6/553: mknod d8/d23/d93/cbd 0 2024-03-04T16:04:39.112 INFO:tasks.workunit.client.1.smithi152.stdout:6/554: mknod d8/d52/d58/cbe 0 2024-03-04T16:04:39.123 INFO:tasks.workunit.client.1.smithi152.stdout:2/438: dread da/f56 [0,4194304] 0 2024-03-04T16:04:39.133 INFO:tasks.workunit.client.1.smithi152.stdout:6/555: rename d8/d92/d59/d61/l7e to d8/d23/d98/d9f/lbf 0 2024-03-04T16:04:39.134 INFO:tasks.workunit.client.1.smithi152.stdout:6/556: write d8/d2a/fa1 [1178169,48294] 0 2024-03-04T16:04:39.135 INFO:tasks.workunit.client.1.smithi152.stdout:6/557: chown d8/d92 17993 1 2024-03-04T16:04:39.135 INFO:tasks.workunit.client.1.smithi152.stdout:6/558: chown d8/d10/l35 1716 1 2024-03-04T16:04:39.137 INFO:tasks.workunit.client.1.smithi152.stdout:5/504: link d6/d8/d5a/d64/d76/d9f/f12 d6/d8/d1f/d88/fac 0 2024-03-04T16:04:39.167 INFO:tasks.workunit.client.1.smithi152.stdout:1/493: sync 2024-03-04T16:04:39.167 INFO:tasks.workunit.client.1.smithi152.stdout:7/419: sync 2024-03-04T16:04:39.168 INFO:tasks.workunit.client.1.smithi152.stdout:6/559: symlink d8/d23/d3a/d71/lc0 0 2024-03-04T16:04:39.168 INFO:tasks.workunit.client.1.smithi152.stdout:5/505: mkdir d6/d8/d5a/d8b/dad 0 2024-03-04T16:04:39.171 INFO:tasks.workunit.client.1.smithi152.stdout:8/452: dread d2/d8/dd/d22/d49/f55 [0,4194304] 0 2024-03-04T16:04:39.172 INFO:tasks.workunit.client.1.smithi152.stdout:1/494: mkdir d19/d20/d2f/d6b/d38/da6 0 2024-03-04T16:04:39.175 INFO:tasks.workunit.client.1.smithi152.stdout:5/506: symlink d6/d8/d5a/d64/d76/d9f/lae 0 2024-03-04T16:04:39.175 INFO:tasks.workunit.client.1.smithi152.stdout:2/439: dwrite da/fb [0,4194304] 0 2024-03-04T16:04:39.180 INFO:tasks.workunit.client.1.smithi152.stdout:7/420: dwrite d14/d21/d47/d36/f5b [0,4194304] 0 2024-03-04T16:04:39.180 INFO:tasks.workunit.client.1.smithi152.stdout:1/495: creat d19/d75/fa7 x:0 0 0 2024-03-04T16:04:39.181 INFO:tasks.workunit.client.1.smithi152.stdout:7/421: chown d14/d21/d47/d36/f74 73908775 1 2024-03-04T16:04:39.183 INFO:tasks.workunit.client.1.smithi152.stdout:8/453: mkdir d2/d8/dd/d22/d6c/d86 0 2024-03-04T16:04:39.184 INFO:tasks.workunit.client.1.smithi152.stdout:6/560: link d8/c69 d8/d52/d58/cc1 0 2024-03-04T16:04:39.186 INFO:tasks.workunit.client.1.smithi152.stdout:2/440: getdents da/df/d2e/d61/d8e 0 2024-03-04T16:04:39.190 INFO:tasks.workunit.client.1.smithi152.stdout:1/496: dwrite d19/d4b/d53/d5c/f6d [0,4194304] 0 2024-03-04T16:04:39.199 INFO:tasks.workunit.client.1.smithi152.stdout:2/441: mkdir da/df/d35/d92 0 2024-03-04T16:04:39.199 INFO:tasks.workunit.client.1.smithi152.stdout:8/454: getdents d2/d75 0 2024-03-04T16:04:39.201 INFO:tasks.workunit.client.1.smithi152.stdout:8/455: write d2/f80 [122534,51591] 0 2024-03-04T16:04:39.202 INFO:tasks.workunit.client.1.smithi152.stdout:1/497: mkdir d19/d75/da8 0 2024-03-04T16:04:39.203 INFO:tasks.workunit.client.1.smithi152.stdout:7/422: link d14/d21/d47/d40/f43 d14/d2b/f7e 0 2024-03-04T16:04:39.203 INFO:tasks.workunit.client.1.smithi152.stdout:2/442: truncate da/df/d2e/d61/f84 148012 0 2024-03-04T16:04:39.203 INFO:tasks.workunit.client.1.smithi152.stdout:8/456: fsync d2/d8/dd/d22/d26/f38 0 2024-03-04T16:04:39.205 INFO:tasks.workunit.client.1.smithi152.stdout:6/561: creat d8/d92/db1/d87/fc2 x:0 0 0 2024-03-04T16:04:39.208 INFO:tasks.workunit.client.1.smithi152.stdout:1/498: mkdir d19/d20/d2f/d6b/d38/d63/d9a/da9 0 2024-03-04T16:04:39.209 INFO:tasks.workunit.client.1.smithi152.stdout:1/499: dread - d19/d4b/d53/d57/d62/d82/f86 zero size 2024-03-04T16:04:39.210 INFO:tasks.workunit.client.1.smithi152.stdout:8/457: rename d2/d8/dd/d22/d49 to d2/d3/d15/d1c/d84/d87 0 2024-03-04T16:04:39.211 INFO:tasks.workunit.client.1.smithi152.stdout:8/458: chown d2/f37 316994 1 2024-03-04T16:04:39.212 INFO:tasks.workunit.client.1.smithi152.stdout:2/443: creat da/df/d2e/d61/d8e/f93 x:0 0 0 2024-03-04T16:04:39.212 INFO:tasks.workunit.client.1.smithi152.stdout:6/562: truncate d8/d23/f55 209841 0 2024-03-04T16:04:39.220 INFO:tasks.workunit.client.1.smithi152.stdout:1/500: dwrite d19/d20/d2f/f59 [0,4194304] 0 2024-03-04T16:04:39.220 INFO:tasks.workunit.client.1.smithi152.stdout:8/459: symlink d2/d8/dd/l88 0 2024-03-04T16:04:39.222 INFO:tasks.workunit.client.1.smithi152.stdout:8/460: dread - d2/d3/d21/d2d/d52/d68/f72 zero size 2024-03-04T16:04:39.229 INFO:tasks.workunit.client.1.smithi152.stdout:1/501: creat d19/d4b/d53/d57/d62/d82/faa x:0 0 0 2024-03-04T16:04:39.230 INFO:tasks.workunit.client.1.smithi152.stdout:2/444: creat da/d24/f94 x:0 0 0 2024-03-04T16:04:39.231 INFO:tasks.workunit.client.1.smithi152.stdout:6/563: dwrite d8/f9 [0,4194304] 0 2024-03-04T16:04:39.233 INFO:tasks.workunit.client.1.smithi152.stdout:8/461: rmdir d2/d3 39 2024-03-04T16:04:39.237 INFO:tasks.workunit.client.1.smithi152.stdout:6/564: creat d8/d42/fc3 x:0 0 0 2024-03-04T16:04:39.242 INFO:tasks.workunit.client.1.smithi152.stdout:2/445: rename da/d24/l3d to da/df/l95 0 2024-03-04T16:04:39.243 INFO:tasks.workunit.client.1.smithi152.stdout:1/502: rename d19/d4b/d53/d57/d62 to d19/d4b/d53/d57/d62/dab 22 2024-03-04T16:04:39.245 INFO:tasks.workunit.client.1.smithi152.stdout:1/503: chown d19/d20/d2f/f94 7932482 1 2024-03-04T16:04:39.245 INFO:tasks.workunit.client.1.smithi152.stdout:6/565: dwrite d8/d23/d3a/d6d/f82 [0,4194304] 0 2024-03-04T16:04:39.248 INFO:tasks.workunit.client.1.smithi152.stdout:2/446: link da/d5f/f68 da/d2c/f96 0 2024-03-04T16:04:39.251 INFO:tasks.workunit.client.1.smithi152.stdout:1/504: mkdir d19/d4b/d53/d57/d62/dac 0 2024-03-04T16:04:39.253 INFO:tasks.workunit.client.1.smithi152.stdout:6/566: mknod d8/d23/d3a/d8d/d96/cc4 0 2024-03-04T16:04:39.255 INFO:tasks.workunit.client.1.smithi152.stdout:8/462: write d2/d3/d21/d2d/d48/f81 [2556120,60791] 0 2024-03-04T16:04:39.256 INFO:tasks.workunit.client.1.smithi152.stdout:8/463: stat d2/d8 0 2024-03-04T16:04:39.256 INFO:tasks.workunit.client.1.smithi152.stdout:6/567: symlink d8/d10/d53/lc5 0 2024-03-04T16:04:39.261 INFO:tasks.workunit.client.1.smithi152.stdout:1/505: creat d19/d3b/d6e/d97/fad x:0 0 0 2024-03-04T16:04:39.264 INFO:tasks.workunit.client.1.smithi152.stdout:8/464: rename d2/d8 to d2/d75/d89 0 2024-03-04T16:04:39.266 INFO:tasks.workunit.client.1.smithi152.stdout:2/447: write da/f1c [5190061,21646] 0 2024-03-04T16:04:39.267 INFO:tasks.workunit.client.1.smithi152.stdout:1/506: creat d19/d20/d2f/d65/fae x:0 0 0 2024-03-04T16:04:39.268 INFO:tasks.workunit.client.1.smithi152.stdout:1/507: truncate d19/d20/f99 895265 0 2024-03-04T16:04:39.270 INFO:tasks.workunit.client.1.smithi152.stdout:1/508: fsync d19/d20/d2f/d6b/d38/f4c 0 2024-03-04T16:04:39.271 INFO:tasks.workunit.client.1.smithi152.stdout:2/448: chown da/f11 1 1 2024-03-04T16:04:39.276 INFO:tasks.workunit.client.1.smithi152.stdout:6/568: link d8/d52/l83 d8/d23/d93/d99/lc6 0 2024-03-04T16:04:39.278 INFO:tasks.workunit.client.1.smithi152.stdout:2/449: dwrite da/df/d2e/d61/f84 [0,4194304] 0 2024-03-04T16:04:39.280 INFO:tasks.workunit.client.1.smithi152.stdout:2/450: read - da/df/d58/d6e/f89 zero size 2024-03-04T16:04:39.282 INFO:tasks.workunit.client.1.smithi152.stdout:2/451: chown da/df/d2e/f90 1330 1 2024-03-04T16:04:39.284 INFO:tasks.workunit.client.1.smithi152.stdout:8/465: dwrite d2/d75/d89/f28 [0,4194304] 0 2024-03-04T16:04:39.291 INFO:tasks.workunit.client.1.smithi152.stdout:1/509: link d19/d4b/d53/d57/d85/c8e d19/d75/da8/caf 0 2024-03-04T16:04:39.296 INFO:tasks.workunit.client.1.smithi152.stdout:1/510: rename f2 to d19/d20/d2f/d6b/d38/d63/fb0 0 2024-03-04T16:04:39.297 INFO:tasks.workunit.client.1.smithi152.stdout:6/569: getdents d8/d92/d75 0 2024-03-04T16:04:39.297 INFO:tasks.workunit.client.1.smithi152.stdout:1/511: chown d19/f2c 1 1 2024-03-04T16:04:39.297 INFO:tasks.workunit.client.1.smithi152.stdout:6/570: fsync d8/d4b/f54 0 2024-03-04T16:04:39.298 INFO:tasks.workunit.client.1.smithi152.stdout:2/452: creat da/df/f97 x:0 0 0 2024-03-04T16:04:39.299 INFO:tasks.workunit.client.1.smithi152.stdout:8/466: creat d2/d75/d89/dd/d22/d6c/d86/f8a x:0 0 0 2024-03-04T16:04:39.301 INFO:tasks.workunit.client.1.smithi152.stdout:8/467: dread d2/f80 [0,4194304] 0 2024-03-04T16:04:39.301 INFO:tasks.workunit.client.1.smithi152.stdout:6/571: write d8/d23/d98/d9f/f77 [5995308,119566] 0 2024-03-04T16:04:39.306 INFO:tasks.workunit.client.1.smithi152.stdout:8/468: creat d2/d3/d15/d1c/d84/d87/d5d/f8b x:0 0 0 2024-03-04T16:04:39.307 INFO:tasks.workunit.client.1.smithi152.stdout:8/469: write d2/d75/d89/f28 [2671234,29407] 0 2024-03-04T16:04:39.316 INFO:tasks.workunit.client.1.smithi152.stdout:8/470: creat d2/d3/d21/d2d/d52/d68/d6e/d7d/f8c x:0 0 0 2024-03-04T16:04:39.317 INFO:tasks.workunit.client.1.smithi152.stdout:2/453: write da/d2c/f96 [535950,100299] 0 2024-03-04T16:04:39.318 INFO:tasks.workunit.client.1.smithi152.stdout:1/512: dread d19/f1a [0,4194304] 0 2024-03-04T16:04:39.318 INFO:tasks.workunit.client.1.smithi152.stdout:8/471: write d2/d3/d21/d2d/d52/d68/f72 [508448,23307] 0 2024-03-04T16:04:39.320 INFO:tasks.workunit.client.1.smithi152.stdout:8/472: readlink d2/d3/d21/d2d/d48/l4c 0 2024-03-04T16:04:39.321 INFO:tasks.workunit.client.1.smithi152.stdout:6/572: dread d8/d23/f2c [0,4194304] 0 2024-03-04T16:04:39.322 INFO:tasks.workunit.client.1.smithi152.stdout:6/573: write d8/f1f [4469078,116997] 0 2024-03-04T16:04:39.324 INFO:tasks.workunit.client.1.smithi152.stdout:2/454: symlink da/df/d2e/l98 0 2024-03-04T16:04:39.327 INFO:tasks.workunit.client.1.smithi152.stdout:8/473: dwrite d2/d3/d21/d2d/f1e [0,4194304] 0 2024-03-04T16:04:39.327 INFO:tasks.workunit.client.1.smithi152.stdout:1/513: mknod d19/d75/da8/cb1 0 2024-03-04T16:04:39.329 INFO:tasks.workunit.client.1.smithi152.stdout:4/395: sync 2024-03-04T16:04:39.330 INFO:tasks.workunit.client.1.smithi152.stdout:3/488: sync 2024-03-04T16:04:39.334 INFO:tasks.workunit.client.1.smithi152.stdout:2/455: creat da/d5f/d85/f99 x:0 0 0 2024-03-04T16:04:39.338 INFO:tasks.workunit.client.1.smithi152.stdout:8/474: creat d2/d3/d21/d2d/d52/d68/d6e/d7d/f8d x:0 0 0 2024-03-04T16:04:39.338 INFO:tasks.workunit.client.1.smithi152.stdout:4/396: dwrite d8/d4e/f57 [0,4194304] 0 2024-03-04T16:04:39.340 INFO:tasks.workunit.client.1.smithi152.stdout:4/397: write d8/d9/f28 [5344958,19800] 0 2024-03-04T16:04:39.340 INFO:tasks.workunit.client.1.smithi152.stdout:6/574: creat d8/d8f/db2/fc7 x:0 0 0 2024-03-04T16:04:39.345 INFO:tasks.workunit.client.1.smithi152.stdout:8/475: mknod d2/d3/d15/d1c/d84/d87/c8e 0 2024-03-04T16:04:39.347 INFO:tasks.workunit.client.1.smithi152.stdout:1/514: rmdir d19/d20/d2f/d6b/d38/d63/d9a/da9 0 2024-03-04T16:04:39.349 INFO:tasks.workunit.client.1.smithi152.stdout:2/456: mkdir da/df/d2e/d51/d8d/d9a 0 2024-03-04T16:04:39.349 INFO:tasks.workunit.client.1.smithi152.stdout:8/476: write d2/d75/d89/f73 [289484,24203] 0 2024-03-04T16:04:39.350 INFO:tasks.workunit.client.1.smithi152.stdout:2/457: chown da 2 1 2024-03-04T16:04:39.351 INFO:tasks.workunit.client.1.smithi152.stdout:4/398: dwrite d8/d9/d79/f85 [0,4194304] 0 2024-03-04T16:04:39.353 INFO:tasks.workunit.client.1.smithi152.stdout:8/477: chown d2/d3/d21/d2d/d48/f7e 215086768 1 2024-03-04T16:04:39.358 INFO:tasks.workunit.client.1.smithi152.stdout:2/458: unlink da/f22 0 2024-03-04T16:04:39.359 INFO:tasks.workunit.client.1.smithi152.stdout:3/489: dread d6/d10/d2d/f36 [0,4194304] 0 2024-03-04T16:04:39.359 INFO:tasks.workunit.client.1.smithi152.stdout:4/399: mkdir d8/d89 0 2024-03-04T16:04:39.363 INFO:tasks.workunit.client.1.smithi152.stdout:1/515: creat d19/d20/fb2 x:0 0 0 2024-03-04T16:04:39.364 INFO:tasks.workunit.client.1.smithi152.stdout:1/516: write d19/d75/f91 [163160,9562] 0 2024-03-04T16:04:39.367 INFO:tasks.workunit.client.1.smithi152.stdout:1/517: write d19/d3b/f49 [1449395,120609] 0 2024-03-04T16:04:39.367 INFO:tasks.workunit.client.1.smithi152.stdout:2/459: dwrite da/d5f/f7a [0,4194304] 0 2024-03-04T16:04:39.380 INFO:tasks.workunit.client.1.smithi152.stdout:3/490: write d6/d14/d58/f6b [5014439,84563] 0 2024-03-04T16:04:39.381 INFO:tasks.workunit.client.1.smithi152.stdout:1/518: dwrite f1 [0,4194304] 0 2024-03-04T16:04:39.381 INFO:tasks.workunit.client.1.smithi152.stdout:1/519: stat d19/d20/d2f/d6b/d38/f4c 0 2024-03-04T16:04:39.381 INFO:tasks.workunit.client.1.smithi152.stdout:1/520: fdatasync d19/d20/d2f/d6b/d38/d63/d9a/fa2 0 2024-03-04T16:04:39.382 INFO:tasks.workunit.client.1.smithi152.stdout:6/575: dread d8/d10/f18 [0,4194304] 0 2024-03-04T16:04:39.382 INFO:tasks.workunit.client.1.smithi152.stdout:1/521: truncate d19/d20/d2f/d6b/f4a 5319483 0 2024-03-04T16:04:39.382 INFO:tasks.workunit.client.1.smithi152.stdout:6/576: stat d8/d2a/f34 0 2024-03-04T16:04:39.383 INFO:tasks.workunit.client.1.smithi152.stdout:3/491: dread d6/d10/d2d/f3b [0,4194304] 0 2024-03-04T16:04:39.390 INFO:tasks.workunit.client.1.smithi152.stdout:8/478: truncate d2/d3/d21/f5f 1828946 0 2024-03-04T16:04:39.391 INFO:tasks.workunit.client.1.smithi152.stdout:4/400: fdatasync d8/d2f/f47 0 2024-03-04T16:04:39.391 INFO:tasks.workunit.client.1.smithi152.stdout:8/479: chown d2/d3/d21/f1b 0 1 2024-03-04T16:04:39.392 INFO:tasks.workunit.client.1.smithi152.stdout:4/401: chown d8/d2f/d7c/f7f 45665 1 2024-03-04T16:04:39.395 INFO:tasks.workunit.client.1.smithi152.stdout:3/492: symlink d6/d5c/la5 0 2024-03-04T16:04:39.402 INFO:tasks.workunit.client.1.smithi152.stdout:4/402: fdatasync d8/d9/f72 0 2024-03-04T16:04:39.409 INFO:tasks.workunit.client.1.smithi152.stdout:2/460: truncate da/df/d2e/d61/f84 1604974 0 2024-03-04T16:04:39.412 INFO:tasks.workunit.client.1.smithi152.stdout:4/403: mknod d8/d9/d24/d26/c8a 0 2024-03-04T16:04:39.414 INFO:tasks.workunit.client.1.smithi152.stdout:3/493: dread d6/d53/fa2 [0,4194304] 0 2024-03-04T16:04:39.416 INFO:tasks.workunit.client.1.smithi152.stdout:8/480: truncate d2/d75/d89/f2e 6616 0 2024-03-04T16:04:39.421 INFO:tasks.workunit.client.1.smithi152.stdout:8/481: creat d2/d75/f8f x:0 0 0 2024-03-04T16:04:39.423 INFO:tasks.workunit.client.1.smithi152.stdout:2/461: dread da/df/d2e/d61/f84 [0,4194304] 0 2024-03-04T16:04:39.427 INFO:tasks.workunit.client.1.smithi152.stdout:2/462: creat da/d24/d80/f9b x:0 0 0 2024-03-04T16:04:39.428 INFO:tasks.workunit.client.1.smithi152.stdout:8/482: dwrite d2/d75/d89/dd/d22/d6c/f74 [0,4194304] 0 2024-03-04T16:04:39.429 INFO:tasks.workunit.client.1.smithi152.stdout:2/463: chown da/df/d2e/l98 29989989 1 2024-03-04T16:04:39.430 INFO:tasks.workunit.client.1.smithi152.stdout:4/404: write d8/d4e/f64 [1512191,55045] 0 2024-03-04T16:04:39.436 INFO:tasks.workunit.client.1.smithi152.stdout:4/405: mkdir d8/d4e/d8b 0 2024-03-04T16:04:39.438 INFO:tasks.workunit.client.1.smithi152.stdout:8/483: rename d2/d75/d89/f4e to d2/d75/d89/f90 0 2024-03-04T16:04:39.438 INFO:tasks.workunit.client.1.smithi152.stdout:2/464: dwrite da/d24/d80/f9b [0,4194304] 0 2024-03-04T16:04:39.443 INFO:tasks.workunit.client.1.smithi152.stdout:4/406: symlink d8/d2f/d6f/l8c 0 2024-03-04T16:04:39.446 INFO:tasks.workunit.client.1.smithi152.stdout:8/484: mknod d2/d75/c91 0 2024-03-04T16:04:39.449 INFO:tasks.workunit.client.1.smithi152.stdout:4/407: mknod d8/d2f/c8d 0 2024-03-04T16:04:39.457 INFO:tasks.workunit.client.1.smithi152.stdout:2/465: link da/d24/f32 da/df/d2e/f9c 0 2024-03-04T16:04:39.458 INFO:tasks.workunit.client.1.smithi152.stdout:2/466: fdatasync da/d24/d47/f5a 0 2024-03-04T16:04:39.459 INFO:tasks.workunit.client.1.smithi152.stdout:2/467: write da/d5f/f7a [2501798,18600] 0 2024-03-04T16:04:39.466 INFO:tasks.workunit.client.1.smithi152.stdout:4/408: dwrite d8/f25 [0,4194304] 0 2024-03-04T16:04:39.469 INFO:tasks.workunit.client.1.smithi152.stdout:2/468: fsync da/f45 0 2024-03-04T16:04:39.469 INFO:tasks.workunit.client.1.smithi152.stdout:2/469: dread - da/d2c/f75 zero size 2024-03-04T16:04:39.474 INFO:tasks.workunit.client.1.smithi152.stdout:2/470: creat da/d2c/d3f/f9d x:0 0 0 2024-03-04T16:04:39.477 INFO:tasks.workunit.client.1.smithi152.stdout:2/471: read da/df/f1f [5801013,71691] 0 2024-03-04T16:04:39.477 INFO:tasks.workunit.client.1.smithi152.stdout:4/409: fdatasync d8/f10 0 2024-03-04T16:04:39.483 INFO:tasks.workunit.client.1.smithi152.stdout:2/472: creat da/d5f/f9e x:0 0 0 2024-03-04T16:04:39.483 INFO:tasks.workunit.client.1.smithi152.stdout:8/485: dwrite d2/d75/d89/f2e [0,4194304] 0 2024-03-04T16:04:39.485 INFO:tasks.workunit.client.1.smithi152.stdout:2/473: chown da/d5f/d85/f99 1313 1 2024-03-04T16:04:39.486 INFO:tasks.workunit.client.1.smithi152.stdout:4/410: creat d8/d2f/d6f/d84/f8e x:0 0 0 2024-03-04T16:04:39.487 INFO:tasks.workunit.client.1.smithi152.stdout:4/411: readlink d8/d9/d7a/l63 0 2024-03-04T16:04:39.489 INFO:tasks.workunit.client.1.smithi152.stdout:8/486: symlink d2/d3/l92 0 2024-03-04T16:04:39.491 INFO:tasks.workunit.client.1.smithi152.stdout:2/474: mknod da/df/d35/d92/c9f 0 2024-03-04T16:04:39.492 INFO:tasks.workunit.client.1.smithi152.stdout:4/412: creat d8/d9/d7a/f8f x:0 0 0 2024-03-04T16:04:39.493 INFO:tasks.workunit.client.1.smithi152.stdout:4/413: dread - d8/d9/f50 zero size 2024-03-04T16:04:39.496 INFO:tasks.workunit.client.1.smithi152.stdout:4/414: fdatasync d8/f10 0 2024-03-04T16:04:39.498 INFO:tasks.workunit.client.1.smithi152.stdout:2/475: symlink da/la0 0 2024-03-04T16:04:39.500 INFO:tasks.workunit.client.1.smithi152.stdout:4/415: mkdir d8/d2f/d6f/d90 0 2024-03-04T16:04:39.502 INFO:tasks.workunit.client.1.smithi152.stdout:8/487: rename d2/d75/d89/dd/f54 to d2/d75/d89/dd/f93 0 2024-03-04T16:04:39.506 INFO:tasks.workunit.client.1.smithi152.stdout:8/488: symlink d2/d3/d15/d1c/d84/d87/l94 0 2024-03-04T16:04:39.507 INFO:tasks.workunit.client.1.smithi152.stdout:4/416: symlink d8/d9/d24/l91 0 2024-03-04T16:04:39.509 INFO:tasks.workunit.client.1.smithi152.stdout:2/476: creat da/df/d2e/fa1 x:0 0 0 2024-03-04T16:04:39.512 INFO:tasks.workunit.client.1.smithi152.stdout:2/477: mkdir da/df/d58/da2 0 2024-03-04T16:04:39.513 INFO:tasks.workunit.client.1.smithi152.stdout:8/489: dwrite d2/d3/d15/f6a [0,4194304] 0 2024-03-04T16:04:39.514 INFO:tasks.workunit.client.1.smithi152.stdout:2/478: creat da/d2c/fa3 x:0 0 0 2024-03-04T16:04:39.515 INFO:tasks.workunit.client.1.smithi152.stdout:8/490: fdatasync d2/d75/d89/dd/d22/f78 0 2024-03-04T16:04:39.521 INFO:tasks.workunit.client.1.smithi152.stdout:2/479: dwrite da/f1a [0,4194304] 0 2024-03-04T16:04:39.521 INFO:tasks.workunit.client.1.smithi152.stdout:8/491: symlink d2/l95 0 2024-03-04T16:04:39.527 INFO:tasks.workunit.client.1.smithi152.stdout:4/417: dwrite d8/d9/d24/f4d [0,4194304] 0 2024-03-04T16:04:39.529 INFO:tasks.workunit.client.1.smithi152.stdout:4/418: readlink d8/d2f/d6f/l8c 0 2024-03-04T16:04:39.530 INFO:tasks.workunit.client.1.smithi152.stdout:2/480: rename da/df/d2e/l7f to da/df/d2e/d61/d8e/la4 0 2024-03-04T16:04:39.533 INFO:tasks.workunit.client.1.smithi152.stdout:8/492: mkdir d2/d75/d89/dd/d22/d83/d96 0 2024-03-04T16:04:39.534 INFO:tasks.workunit.client.1.smithi152.stdout:8/493: readlink d2/d75/d89/dd/l85 0 2024-03-04T16:04:39.535 INFO:tasks.workunit.client.1.smithi152.stdout:8/494: read - d2/d3/d21/d2d/d48/f7e zero size 2024-03-04T16:04:39.536 INFO:tasks.workunit.client.1.smithi152.stdout:4/419: creat d8/d2f/d6f/d90/f92 x:0 0 0 2024-03-04T16:04:39.537 INFO:tasks.workunit.client.1.smithi152.stdout:2/481: mknod da/df/d2e/d51/d8d/d9a/ca5 0 2024-03-04T16:04:39.539 INFO:tasks.workunit.client.1.smithi152.stdout:4/420: readlink d8/d9/d24/l6a 0 2024-03-04T16:04:39.541 INFO:tasks.workunit.client.1.smithi152.stdout:4/421: readlink d8/d9/l22 0 2024-03-04T16:04:39.542 INFO:tasks.workunit.client.1.smithi152.stdout:8/495: dwrite d2/d75/d89/f2e [0,4194304] 0 2024-03-04T16:04:39.546 INFO:tasks.workunit.client.1.smithi152.stdout:2/482: mkdir da/d24/d80/d82/da6 0 2024-03-04T16:04:39.548 INFO:tasks.workunit.client.1.smithi152.stdout:8/496: symlink d2/d3/d15/d1c/l97 0 2024-03-04T16:04:39.550 INFO:tasks.workunit.client.1.smithi152.stdout:2/483: read da/df/d2e/f38 [199424,76383] 0 2024-03-04T16:04:39.550 INFO:tasks.workunit.client.1.smithi152.stdout:8/497: write d2/d3/d21/d2d/d31/f35 [1212444,10326] 0 2024-03-04T16:04:39.551 INFO:tasks.workunit.client.1.smithi152.stdout:8/498: fsync d2/d3/d21/d2d/f50 0 2024-03-04T16:04:39.555 INFO:tasks.workunit.client.1.smithi152.stdout:9/449: sync 2024-03-04T16:04:39.558 INFO:tasks.workunit.client.1.smithi152.stdout:9/450: dread d2/d13/d23/f40 [0,4194304] 0 2024-03-04T16:04:39.559 INFO:tasks.workunit.client.1.smithi152.stdout:8/499: dwrite d2/d75/d89/f73 [0,4194304] 0 2024-03-04T16:04:39.560 INFO:tasks.workunit.client.1.smithi152.stdout:4/422: getdents d8/d78 0 2024-03-04T16:04:39.563 INFO:tasks.workunit.client.1.smithi152.stdout:4/423: dread d8/f25 [0,4194304] 0 2024-03-04T16:04:39.571 INFO:tasks.workunit.client.1.smithi152.stdout:8/500: rmdir d2/d3/d21/d2d/d48 39 2024-03-04T16:04:39.571 INFO:tasks.workunit.client.1.smithi152.stdout:2/484: write da/df/d35/f4a [3575590,119893] 0 2024-03-04T16:04:39.573 INFO:tasks.workunit.client.1.smithi152.stdout:2/485: chown da/d5f/c7e 114 1 2024-03-04T16:04:39.573 INFO:tasks.workunit.client.1.smithi152.stdout:2/486: fsync da/fb 0 2024-03-04T16:04:39.574 INFO:tasks.workunit.client.1.smithi152.stdout:9/451: dread d2/fe [0,4194304] 0 2024-03-04T16:04:39.577 INFO:tasks.workunit.client.1.smithi152.stdout:9/452: creat d2/d63/f95 x:0 0 0 2024-03-04T16:04:39.579 INFO:tasks.workunit.client.1.smithi152.stdout:4/424: dwrite d8/f23 [0,4194304] 0 2024-03-04T16:04:39.584 INFO:tasks.workunit.client.1.smithi152.stdout:4/425: truncate d8/d9/d24/d26/d42/f4b 305113 0 2024-03-04T16:04:39.585 INFO:tasks.workunit.client.1.smithi152.stdout:9/453: mkdir d2/d13/d96 0 2024-03-04T16:04:39.585 INFO:tasks.workunit.client.1.smithi152.stdout:4/426: chown d8/d2f/d6f/c76 84 1 2024-03-04T16:04:39.586 INFO:tasks.workunit.client.1.smithi152.stdout:9/454: truncate d2/f61 597432 0 2024-03-04T16:04:39.586 INFO:tasks.workunit.client.1.smithi152.stdout:4/427: write d8/f23 [3447742,125877] 0 2024-03-04T16:04:39.590 INFO:tasks.workunit.client.1.smithi152.stdout:8/501: dread d2/d3/d21/f32 [0,4194304] 0 2024-03-04T16:04:39.590 INFO:tasks.workunit.client.1.smithi152.stdout:9/455: creat d2/d63/d69/f97 x:0 0 0 2024-03-04T16:04:39.595 INFO:tasks.workunit.client.1.smithi152.stdout:8/502: rename d2/f53 to d2/d7f/f98 0 2024-03-04T16:04:39.597 INFO:tasks.workunit.client.1.smithi152.stdout:9/456: dwrite d2/d13/f15 [0,4194304] 0 2024-03-04T16:04:39.602 INFO:tasks.workunit.client.1.smithi152.stdout:8/503: mknod d2/d3/d15/c99 0 2024-03-04T16:04:39.607 INFO:tasks.workunit.client.1.smithi152.stdout:8/504: mkdir d2/d3/d21/d2d/d31/d9a 0 2024-03-04T16:04:39.617 INFO:tasks.workunit.client.1.smithi152.stdout:4/428: dread d8/d9/ff [0,4194304] 0 2024-03-04T16:04:39.636 INFO:tasks.workunit.client.1.smithi152.stdout:9/457: dwrite d2/d13/d23/f5f [0,4194304] 0 2024-03-04T16:04:39.636 INFO:tasks.workunit.client.1.smithi152.stdout:8/505: dread d2/d3/d21/d2d/d48/f81 [0,4194304] 0 2024-03-04T16:04:39.645 INFO:tasks.workunit.client.1.smithi152.stdout:9/458: rename d2/d13/d23/c27 to d2/d63/d69/c98 0 2024-03-04T16:04:39.651 INFO:tasks.workunit.client.1.smithi152.stdout:8/506: getdents d2/d75/d89 0 2024-03-04T16:04:39.652 INFO:tasks.workunit.client.1.smithi152.stdout:9/459: dwrite d2/d13/d38/f58 [0,4194304] 0 2024-03-04T16:04:39.653 INFO:tasks.workunit.client.1.smithi152.stdout:9/460: write d2/f2d [7922706,30756] 0 2024-03-04T16:04:39.653 INFO:tasks.workunit.client.1.smithi152.stdout:9/461: readlink d2/lc 0 2024-03-04T16:04:39.656 INFO:tasks.workunit.client.1.smithi152.stdout:9/462: dread d2/f61 [0,4194304] 0 2024-03-04T16:04:39.657 INFO:tasks.workunit.client.1.smithi152.stdout:8/507: creat d2/d3/d21/d2d/d52/d68/d6e/d7d/f9b x:0 0 0 2024-03-04T16:04:39.663 INFO:tasks.workunit.client.1.smithi152.stdout:8/508: mknod d2/d3/d21/d2d/c9c 0 2024-03-04T16:04:39.664 INFO:tasks.workunit.client.1.smithi152.stdout:9/463: dwrite d2/f8d [0,4194304] 0 2024-03-04T16:04:39.665 INFO:tasks.workunit.client.1.smithi152.stdout:9/464: readlink d2/d13/l20 0 2024-03-04T16:04:39.667 INFO:tasks.workunit.client.1.smithi152.stdout:8/509: unlink d2/d75/d89/dd/d22/d26/l43 0 2024-03-04T16:04:39.673 INFO:tasks.workunit.client.1.smithi152.stdout:8/510: read d2/d3/f40 [1576924,355] 0 2024-03-04T16:04:39.674 INFO:tasks.workunit.client.1.smithi152.stdout:8/511: chown d2/d3/d21/d2d/d52/d68/d6e/l7c 0 1 2024-03-04T16:04:39.674 INFO:tasks.workunit.client.1.smithi152.stdout:8/512: chown d2/d3/d21 1552 1 2024-03-04T16:04:39.676 INFO:tasks.workunit.client.1.smithi152.stdout:8/513: dread - d2/d3/d21/d2d/d52/d68/d6e/d7d/f8c zero size 2024-03-04T16:04:39.679 INFO:tasks.workunit.client.1.smithi152.stdout:8/514: unlink d2/d75/d89/dd/d22/d6c/d86/f8a 0 2024-03-04T16:04:39.681 INFO:tasks.workunit.client.1.smithi152.stdout:8/515: write d2/d3/d21/d2d/d52/d68/d6e/d7d/f8c [492935,90413] 0 2024-03-04T16:04:39.686 INFO:tasks.workunit.client.1.smithi152.stdout:8/516: rename d2/d3/d15/c99 to d2/d75/d89/dd/d22/d83/c9d 0 2024-03-04T16:04:39.686 INFO:tasks.workunit.client.1.smithi152.stdout:9/465: dwrite d2/f19 [0,4194304] 0 2024-03-04T16:04:39.688 INFO:tasks.workunit.client.1.smithi152.stdout:8/517: mkdir d2/d3/d9e 0 2024-03-04T16:04:39.689 INFO:tasks.workunit.client.1.smithi152.stdout:9/466: symlink d2/d13/d23/l99 0 2024-03-04T16:04:39.692 INFO:tasks.workunit.client.1.smithi152.stdout:8/518: truncate d2/d3/d21/d2d/f79 3769927 0 2024-03-04T16:04:39.693 INFO:tasks.workunit.client.1.smithi152.stdout:8/519: chown d2/d3/d21/c57 38491 1 2024-03-04T16:04:39.695 INFO:tasks.workunit.client.1.smithi152.stdout:8/520: creat d2/d75/f9f x:0 0 0 2024-03-04T16:04:39.697 INFO:tasks.workunit.client.1.smithi152.stdout:8/521: rmdir d2/d3/d15/d1c/d84/d87 39 2024-03-04T16:04:39.701 INFO:tasks.workunit.client.1.smithi152.stdout:8/522: link d2/d3/d21/d2d/d52/d68/d6e/d7d/f8d d2/d75/d89/dd/d22/fa0 0 2024-03-04T16:04:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:39 smithi116 ceph-mon[29120]: Standby manager daemon smithi116.jwdhop restarted 2024-03-04T16:04:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:39 smithi116 ceph-mon[29120]: Standby manager daemon smithi116.jwdhop started 2024-03-04T16:04:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:39 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.jwdhop/crt"}]: dispatch 2024-03-04T16:04:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:39 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:04:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:39 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.jwdhop/key"}]: dispatch 2024-03-04T16:04:39.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:39 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:04:39.812 INFO:tasks.workunit.client.1.smithi152.stdout:0/405: sync 2024-03-04T16:04:39.813 INFO:tasks.workunit.client.1.smithi152.stdout:8/523: unlink d2/l46 0 2024-03-04T16:04:39.815 INFO:tasks.workunit.client.1.smithi152.stdout:0/406: mknod d9/d2c/d2e/d38/d56/d5b/c7b 0 2024-03-04T16:04:39.819 INFO:tasks.workunit.client.1.smithi152.stdout:0/407: mkdir d9/d1a/d7c 0 2024-03-04T16:04:39.822 INFO:tasks.workunit.client.1.smithi152.stdout:0/408: mkdir d9/d2c/d7d 0 2024-03-04T16:04:39.822 INFO:tasks.workunit.client.1.smithi152.stdout:8/524: link d2/d3/d15/d1c/c2a d2/d3/d15/d1c/d84/d87/ca1 0 2024-03-04T16:04:39.824 INFO:tasks.workunit.client.1.smithi152.stdout:0/409: fdatasync d9/d2c/d2e/d38/d56/f63 0 2024-03-04T16:04:39.832 INFO:tasks.workunit.client.1.smithi152.stdout:0/410: dwrite d9/f26 [0,4194304] 0 2024-03-04T16:04:39.832 INFO:tasks.workunit.client.1.smithi152.stdout:0/411: readlink d9/d1b/l27 0 2024-03-04T16:04:39.838 INFO:tasks.workunit.client.1.smithi152.stdout:0/412: rename d9/fa to d9/d1a/d7c/f7e 0 2024-03-04T16:04:39.839 INFO:tasks.workunit.client.1.smithi152.stdout:0/413: fdatasync d9/d2c/f2f 0 2024-03-04T16:04:39.842 INFO:tasks.workunit.client.1.smithi152.stdout:8/525: write d2/d3/d15/f4f [763999,112942] 0 2024-03-04T16:04:39.843 INFO:tasks.workunit.client.1.smithi152.stdout:0/414: mkdir d9/d2c/d2e/d7f 0 2024-03-04T16:04:39.846 INFO:tasks.workunit.client.1.smithi152.stdout:8/526: creat d2/d3/d21/d2d/d52/d68/fa2 x:0 0 0 2024-03-04T16:04:39.848 INFO:tasks.workunit.client.1.smithi152.stdout:8/527: fdatasync d2/d7f/f98 0 2024-03-04T16:04:39.849 INFO:tasks.workunit.client.1.smithi152.stdout:0/415: dwrite d9/d2c/d2e/f47 [4194304,4194304] 0 2024-03-04T16:04:39.850 INFO:tasks.workunit.client.1.smithi152.stdout:8/528: rmdir d2/d3/d21/d2d/d31 39 2024-03-04T16:04:39.852 INFO:tasks.workunit.client.1.smithi152.stdout:8/529: dread - d2/d3/d21/d2d/d52/d68/fa2 zero size 2024-03-04T16:04:39.856 INFO:tasks.workunit.client.1.smithi152.stdout:0/416: mkdir d9/d1b/d54/d80 0 2024-03-04T16:04:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:39 smithi152 ceph-mon[36876]: Standby manager daemon smithi116.jwdhop restarted 2024-03-04T16:04:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:39 smithi152 ceph-mon[36876]: Standby manager daemon smithi116.jwdhop started 2024-03-04T16:04:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:39 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.jwdhop/crt"}]: dispatch 2024-03-04T16:04:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:39 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:04:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:39 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.jwdhop/key"}]: dispatch 2024-03-04T16:04:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:39 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:04:39.861 INFO:tasks.workunit.client.1.smithi152.stdout:0/417: rename d9/l4d to d9/d2c/d2e/d38/d56/d5b/d7a/l81 0 2024-03-04T16:04:39.862 INFO:tasks.workunit.client.1.smithi152.stdout:0/418: write d9/d1a/f46 [1931790,112359] 0 2024-03-04T16:04:39.865 INFO:tasks.workunit.client.1.smithi152.stdout:0/419: symlink d9/d2c/d2e/d38/d56/d5b/l82 0 2024-03-04T16:04:39.867 INFO:tasks.workunit.client.1.smithi152.stdout:8/530: truncate d2/d3/d21/f32 3296307 0 2024-03-04T16:04:39.867 INFO:tasks.workunit.client.1.smithi152.stdout:8/531: chown d2/d3/d15/d1c/d84 4788 1 2024-03-04T16:04:39.869 INFO:tasks.workunit.client.1.smithi152.stdout:0/420: mkdir d9/d1a/d83 0 2024-03-04T16:04:39.869 INFO:tasks.workunit.client.1.smithi152.stdout:0/421: write d9/d1a/f79 [637327,65721] 0 2024-03-04T16:04:39.873 INFO:tasks.workunit.client.1.smithi152.stdout:8/532: rename d2/d3/d21/d2d/c9c to d2/d3/d21/d2d/d31/d9a/ca3 0 2024-03-04T16:04:39.876 INFO:tasks.workunit.client.1.smithi152.stdout:8/533: creat d2/d75/d89/dd/d22/fa4 x:0 0 0 2024-03-04T16:04:39.879 INFO:tasks.workunit.client.1.smithi152.stdout:8/534: mknod d2/d75/d89/dd/d22/d83/d96/ca5 0 2024-03-04T16:04:39.880 INFO:tasks.workunit.client.1.smithi152.stdout:8/535: chown d2/d3/d15/d1c 88 1 2024-03-04T16:04:39.893 INFO:tasks.workunit.client.1.smithi152.stdout:8/536: dread d2/d3/d21/d2d/f2c [4194304,4194304] 0 2024-03-04T16:04:39.896 INFO:tasks.workunit.client.1.smithi152.stdout:8/537: getdents d2/d3/d21/d2d/d31/d9a 0 2024-03-04T16:04:39.898 INFO:tasks.workunit.client.1.smithi152.stdout:8/538: mkdir d2/d75/d89/dd/d22/d83/d96/da6 0 2024-03-04T16:04:39.904 INFO:tasks.workunit.client.1.smithi152.stdout:8/539: dwrite d2/d3/f4b [0,4194304] 0 2024-03-04T16:04:39.910 INFO:tasks.workunit.client.1.smithi152.stdout:8/540: dread d2/d3/d21/d2d/d52/d68/d6e/d7d/f8c [0,4194304] 0 2024-03-04T16:04:39.917 INFO:tasks.workunit.client.1.smithi152.stdout:8/541: symlink d2/d75/la7 0 2024-03-04T16:04:39.925 INFO:tasks.workunit.client.1.smithi152.stdout:8/542: creat d2/d3/d15/d1c/d84/fa8 x:0 0 0 2024-03-04T16:04:40.007 INFO:tasks.workunit.client.1.smithi152.stdout:5/507: sync 2024-03-04T16:04:40.012 INFO:tasks.workunit.client.1.smithi152.stdout:5/508: getdents d6/d8/d1f/d3d/d6f/d70 0 2024-03-04T16:04:40.015 INFO:tasks.workunit.client.1.smithi152.stdout:5/509: dread d6/d8/d5a/d64/d76/d9f/f43 [0,4194304] 0 2024-03-04T16:04:40.021 INFO:tasks.workunit.client.1.smithi152.stdout:5/510: mkdir d6/d8/d5a/d90/daf 0 2024-03-04T16:04:40.160 INFO:tasks.workunit.client.1.smithi152.stdout:1/522: sync 2024-03-04T16:04:40.161 INFO:tasks.workunit.client.1.smithi152.stdout:7/423: sync 2024-03-04T16:04:40.162 INFO:tasks.workunit.client.1.smithi152.stdout:1/523: dread d19/f7c [0,4194304] 0 2024-03-04T16:04:40.167 INFO:tasks.workunit.client.1.smithi152.stdout:7/424: write d14/f6a [224621,87874] 0 2024-03-04T16:04:40.170 INFO:tasks.workunit.client.1.smithi152.stdout:7/425: chown d14/d21/d47/d40/f43 10331 1 2024-03-04T16:04:40.171 INFO:tasks.workunit.client.1.smithi152.stdout:1/524: getdents d19/d3b/d6e/d97 0 2024-03-04T16:04:40.176 INFO:tasks.workunit.client.1.smithi152.stdout:7/426: dwrite d14/d21/f30 [0,4194304] 0 2024-03-04T16:04:40.182 INFO:tasks.workunit.client.1.smithi152.stdout:1/525: truncate fa 2484769 0 2024-03-04T16:04:40.186 INFO:tasks.workunit.client.1.smithi152.stdout:7/427: mknod d14/d71/c7f 0 2024-03-04T16:04:40.198 INFO:tasks.workunit.client.1.smithi152.stdout:1/526: mknod d19/d20/d2f/d6b/d38/da6/cb3 0 2024-03-04T16:04:40.204 INFO:tasks.workunit.client.1.smithi152.stdout:1/527: truncate d19/d20/d2f/d6b/f46 4954059 0 2024-03-04T16:04:40.207 INFO:tasks.workunit.client.1.smithi152.stdout:1/528: mknod d19/d20/d2f/d6b/d38/d63/d9a/cb4 0 2024-03-04T16:04:40.208 INFO:tasks.workunit.client.1.smithi152.stdout:7/428: dread fd [0,4194304] 0 2024-03-04T16:04:40.211 INFO:tasks.workunit.client.1.smithi152.stdout:7/429: dread - d14/d2b/f56 zero size 2024-03-04T16:04:40.211 INFO:tasks.workunit.client.1.smithi152.stdout:4/429: sync 2024-03-04T16:04:40.211 INFO:tasks.workunit.client.1.smithi152.stdout:8/543: sync 2024-03-04T16:04:40.211 INFO:tasks.workunit.client.1.smithi152.stdout:3/494: sync 2024-03-04T16:04:40.211 INFO:tasks.workunit.client.1.smithi152.stdout:6/577: sync 2024-03-04T16:04:40.211 INFO:tasks.workunit.client.1.smithi152.stdout:0/422: sync 2024-03-04T16:04:40.211 INFO:tasks.workunit.client.1.smithi152.stdout:2/487: sync 2024-03-04T16:04:40.211 INFO:tasks.workunit.client.1.smithi152.stdout:5/511: sync 2024-03-04T16:04:40.211 INFO:tasks.workunit.client.1.smithi152.stdout:9/467: sync 2024-03-04T16:04:40.214 INFO:tasks.workunit.client.1.smithi152.stdout:3/495: chown d6/d14/d58/f7e 1739 1 2024-03-04T16:04:40.214 INFO:tasks.workunit.client.1.smithi152.stdout:9/468: fsync d2/d13/d23/d42/f78 0 2024-03-04T16:04:40.214 INFO:tasks.workunit.client.1.smithi152.stdout:8/544: readlink d2/d75/d89/dd/d22/d26/l3e 0 2024-03-04T16:04:40.216 INFO:tasks.workunit.client.1.smithi152.stdout:2/488: truncate da/df/d2e/fa1 890445 0 2024-03-04T16:04:40.218 INFO:tasks.workunit.client.1.smithi152.stdout:5/512: truncate d6/d8/d5a/d64/f96 710276 0 2024-03-04T16:04:40.219 INFO:tasks.workunit.client.1.smithi152.stdout:6/578: creat d8/d92/d59/fc8 x:0 0 0 2024-03-04T16:04:40.219 INFO:tasks.workunit.client.1.smithi152.stdout:1/529: truncate d19/d20/d2f/d6b/f4a 3910336 0 2024-03-04T16:04:40.222 INFO:tasks.workunit.client.1.smithi152.stdout:0/423: dwrite d9/d1b/f6d [0,4194304] 0 2024-03-04T16:04:40.225 INFO:tasks.workunit.client.1.smithi152.stdout:0/424: write d9/d2c/f6f [727409,7111] 0 2024-03-04T16:04:40.225 INFO:tasks.workunit.client.1.smithi152.stdout:9/469: creat d2/d13/d23/d66/d7d/f9a x:0 0 0 2024-03-04T16:04:40.228 INFO:tasks.workunit.client.1.smithi152.stdout:0/425: write d9/d1b/f6e [867462,47905] 0 2024-03-04T16:04:40.229 INFO:tasks.workunit.client.1.smithi152.stdout:3/496: fsync d6/f42 0 2024-03-04T16:04:40.231 INFO:tasks.workunit.client.1.smithi152.stdout:4/430: dread d8/d2f/f33 [0,4194304] 0 2024-03-04T16:04:40.232 INFO:tasks.workunit.client.1.smithi152.stdout:8/545: symlink d2/d3/d21/d2d/d31/la9 0 2024-03-04T16:04:40.232 INFO:tasks.workunit.client.1.smithi152.stdout:4/431: truncate d8/d9/ff 8448319 0 2024-03-04T16:04:40.236 INFO:tasks.workunit.client.1.smithi152.stdout:2/489: mknod da/df/d2e/d51/d8d/d9a/ca7 0 2024-03-04T16:04:40.240 INFO:tasks.workunit.client.1.smithi152.stdout:1/530: write d19/d20/f41 [2277641,13334] 0 2024-03-04T16:04:40.241 INFO:tasks.workunit.client.1.smithi152.stdout:5/513: mkdir d6/d8/d5a/d64/d76/d9f/db0 0 2024-03-04T16:04:40.241 INFO:tasks.workunit.client.1.smithi152.stdout:1/531: chown d19/d20/d2f/d6b/l56 6910 1 2024-03-04T16:04:40.242 INFO:tasks.workunit.client.1.smithi152.stdout:1/532: write d19/d20/d2f/f59 [1654961,49309] 0 2024-03-04T16:04:40.244 INFO:tasks.workunit.client.1.smithi152.stdout:7/430: getdents d14/d21/d31 0 2024-03-04T16:04:40.246 INFO:tasks.workunit.client.1.smithi152.stdout:0/426: rename d9/d2c/d2e/d38 to d9/d1b/d3c/d84 0 2024-03-04T16:04:40.248 INFO:tasks.workunit.client.1.smithi152.stdout:3/497: fdatasync d6/d10/f39 0 2024-03-04T16:04:40.251 INFO:tasks.workunit.client.1.smithi152.stdout:9/470: dwrite d2/d13/d23/f4c [0,4194304] 0 2024-03-04T16:04:40.253 INFO:tasks.workunit.client.1.smithi152.stdout:9/471: truncate d2/d63/d69/f8b 610610 0 2024-03-04T16:04:40.254 INFO:tasks.workunit.client.1.smithi152.stdout:9/472: write d2/d6e/f7a [417575,13906] 0 2024-03-04T16:04:40.257 INFO:tasks.workunit.client.1.smithi152.stdout:1/533: chown d19/d75/l8b 0 1 2024-03-04T16:04:40.257 INFO:tasks.workunit.client.1.smithi152.stdout:1/534: stat d19/d4b/d53/d57/f5e 0 2024-03-04T16:04:40.258 INFO:tasks.workunit.client.1.smithi152.stdout:9/473: dread d2/d13/d2e/f44 [0,4194304] 0 2024-03-04T16:04:40.261 INFO:tasks.workunit.client.1.smithi152.stdout:9/474: dread - d2/d13/d23/d42/f78 zero size 2024-03-04T16:04:40.263 INFO:tasks.workunit.client.1.smithi152.stdout:0/427: creat d9/d2c/d2e/f85 x:0 0 0 2024-03-04T16:04:40.266 INFO:tasks.workunit.client.1.smithi152.stdout:0/428: fsync d9/d1b/f2a 0 2024-03-04T16:04:40.267 INFO:tasks.workunit.client.1.smithi152.stdout:6/579: dwrite d8/d92/d75/f88 [0,4194304] 0 2024-03-04T16:04:40.272 INFO:tasks.workunit.client.1.smithi152.stdout:4/432: dread d8/f10 [0,4194304] 0 2024-03-04T16:04:40.273 INFO:tasks.workunit.client.1.smithi152.stdout:9/475: dwrite d2/f67 [0,4194304] 0 2024-03-04T16:04:40.277 INFO:tasks.workunit.client.1.smithi152.stdout:4/433: write d8/d9/d7a/f8f [636316,83813] 0 2024-03-04T16:04:40.278 INFO:tasks.workunit.client.1.smithi152.stdout:0/429: dwrite d9/d1b/f71 [4194304,4194304] 0 2024-03-04T16:04:40.281 INFO:tasks.workunit.client.1.smithi152.stdout:3/498: symlink d6/d10/d2d/la6 0 2024-03-04T16:04:40.287 INFO:tasks.workunit.client.1.smithi152.stdout:1/535: symlink d19/d4b/d53/d57/d62/dac/lb5 0 2024-03-04T16:04:40.289 INFO:tasks.workunit.client.1.smithi152.stdout:8/546: rename d2/d3/d15/f6b to d2/faa 0 2024-03-04T16:04:40.295 INFO:tasks.workunit.client.1.smithi152.stdout:5/514: getdents d6/d8/d5a 0 2024-03-04T16:04:40.296 INFO:tasks.workunit.client.1.smithi152.stdout:1/536: unlink d19/d20/d2f/d6b/d38/c43 0 2024-03-04T16:04:40.302 INFO:tasks.workunit.client.1.smithi152.stdout:4/434: truncate d8/d9/d24/f40 195537 0 2024-03-04T16:04:40.305 INFO:tasks.workunit.client.1.smithi152.stdout:8/547: dwrite d2/d3/d21/d2d/d48/f81 [0,4194304] 0 2024-03-04T16:04:40.307 INFO:tasks.workunit.client.1.smithi152.stdout:5/515: creat d6/d8/d5a/d8b/fb1 x:0 0 0 2024-03-04T16:04:40.309 INFO:tasks.workunit.client.1.smithi152.stdout:8/548: truncate d2/d3/d15/d1c/d84/d87/d5d/f8b 696587 0 2024-03-04T16:04:40.312 INFO:tasks.workunit.client.1.smithi152.stdout:2/490: rename da/d7b to da/d24/d80/d82/da8 0 2024-03-04T16:04:40.313 INFO:tasks.workunit.client.1.smithi152.stdout:9/476: truncate d2/d13/d23/f4c 2552302 0 2024-03-04T16:04:40.314 INFO:tasks.workunit.client.1.smithi152.stdout:0/430: dread d9/d1b/f23 [0,4194304] 0 2024-03-04T16:04:40.315 INFO:tasks.workunit.client.1.smithi152.stdout:2/491: dread - da/d2c/d5e/f8b zero size 2024-03-04T16:04:40.315 INFO:tasks.workunit.client.1.smithi152.stdout:6/580: dwrite d8/d23/f55 [0,4194304] 0 2024-03-04T16:04:40.316 INFO:tasks.workunit.client.1.smithi152.stdout:0/431: chown d9/d2c/d2e/d7f 15625303 1 2024-03-04T16:04:40.317 INFO:tasks.workunit.client.1.smithi152.stdout:4/435: symlink d8/d2f/d6f/d84/l93 0 2024-03-04T16:04:40.318 INFO:tasks.workunit.client.1.smithi152.stdout:2/492: chown da/d24/d81 5 1 2024-03-04T16:04:40.326 INFO:tasks.workunit.client.1.smithi152.stdout:3/499: rename d6/d10/d2d/c56 to d6/d14/ca7 0 2024-03-04T16:04:40.327 INFO:tasks.workunit.client.1.smithi152.stdout:2/493: dwrite da/d24/d47/f5a [0,4194304] 0 2024-03-04T16:04:40.327 INFO:tasks.workunit.client.1.smithi152.stdout:3/500: stat d6/d14/f2e 0 2024-03-04T16:04:40.330 INFO:tasks.workunit.client.1.smithi152.stdout:6/581: creat d8/d23/d93/d99/fc9 x:0 0 0 2024-03-04T16:04:40.332 INFO:tasks.workunit.client.1.smithi152.stdout:6/582: write d8/d10/f1e [5169861,62401] 0 2024-03-04T16:04:40.334 INFO:tasks.workunit.client.1.smithi152.stdout:8/549: symlink d2/d3/lab 0 2024-03-04T16:04:40.340 INFO:tasks.workunit.client.1.smithi152.stdout:1/537: write d19/d20/d2f/d6b/d38/d63/fb0 [675429,112591] 0 2024-03-04T16:04:40.342 INFO:tasks.workunit.client.1.smithi152.stdout:3/501: unlink d6/d10/f1e 0 2024-03-04T16:04:40.344 INFO:tasks.workunit.client.1.smithi152.stdout:2/494: dwrite da/df/f13 [0,4194304] 0 2024-03-04T16:04:40.347 INFO:tasks.workunit.client.1.smithi152.stdout:0/432: symlink d9/d1a/l86 0 2024-03-04T16:04:40.351 INFO:tasks.workunit.client.1.smithi152.stdout:4/436: mknod d8/d4e/d86/c94 0 2024-03-04T16:04:40.354 INFO:tasks.workunit.client.1.smithi152.stdout:8/550: rmdir d2/d75 39 2024-03-04T16:04:40.354 INFO:tasks.workunit.client.1.smithi152.stdout:0/433: dwrite d9/d1b/f6d [0,4194304] 0 2024-03-04T16:04:40.360 INFO:tasks.workunit.client.1.smithi152.stdout:6/583: truncate d8/d23/d3a/f70 760169 0 2024-03-04T16:04:40.361 INFO:tasks.workunit.client.1.smithi152.stdout:1/538: symlink d19/d20/d2f/lb6 0 2024-03-04T16:04:40.364 INFO:tasks.workunit.client.1.smithi152.stdout:3/502: creat d6/d60/d8e/fa8 x:0 0 0 2024-03-04T16:04:40.364 INFO:tasks.workunit.client.1.smithi152.stdout:2/495: rmdir da/df/d2e/d51/d8d/d9a 39 2024-03-04T16:04:40.365 INFO:tasks.workunit.client.1.smithi152.stdout:2/496: readlink da/d2c/d5e/l6f 0 2024-03-04T16:04:40.366 INFO:tasks.workunit.client.1.smithi152.stdout:5/516: getdents d6/d8/d5a/d64/d76 0 2024-03-04T16:04:40.369 INFO:tasks.workunit.client.1.smithi152.stdout:4/437: creat d8/d78/f95 x:0 0 0 2024-03-04T16:04:40.369 INFO:tasks.workunit.client.1.smithi152.stdout:4/438: stat d8/f25 0 2024-03-04T16:04:40.374 INFO:tasks.workunit.client.1.smithi152.stdout:9/477: getdents d2 0 2024-03-04T16:04:40.374 INFO:tasks.workunit.client.1.smithi152.stdout:6/584: fdatasync d8/d23/d3a/d8d/d96/fb9 0 2024-03-04T16:04:40.376 INFO:tasks.workunit.client.1.smithi152.stdout:8/551: dwrite d2/d3/f40 [0,4194304] 0 2024-03-04T16:04:40.376 INFO:tasks.workunit.client.1.smithi152.stdout:9/478: truncate d2/d13/d23/f4e 1737396 0 2024-03-04T16:04:40.377 INFO:tasks.workunit.client.1.smithi152.stdout:9/479: write d2/f91 [275372,31410] 0 2024-03-04T16:04:40.381 INFO:tasks.workunit.client.1.smithi152.stdout:2/497: rmdir da/d5f/d85 39 2024-03-04T16:04:40.383 INFO:tasks.workunit.client.1.smithi152.stdout:5/517: rename d6/d8/d5a/d64/d76/d9f/c17 to d6/d8/d5a/d64/d76/cb2 0 2024-03-04T16:04:40.389 INFO:tasks.workunit.client.1.smithi152.stdout:1/539: truncate d19/d75/f83 2002821 0 2024-03-04T16:04:40.393 INFO:tasks.workunit.client.1.smithi152.stdout:9/480: creat d2/d6e/f9b x:0 0 0 2024-03-04T16:04:40.394 INFO:tasks.workunit.client.1.smithi152.stdout:3/503: truncate d6/d10/f21 3147123 0 2024-03-04T16:04:40.396 INFO:tasks.workunit.client.1.smithi152.stdout:2/498: fsync da/df/d2e/f9c 0 2024-03-04T16:04:40.398 INFO:tasks.workunit.client.1.smithi152.stdout:5/518: creat d6/d50/fb3 x:0 0 0 2024-03-04T16:04:40.401 INFO:tasks.workunit.client.1.smithi152.stdout:4/439: truncate d8/f30 199328 0 2024-03-04T16:04:40.402 INFO:tasks.workunit.client.1.smithi152.stdout:0/434: mknod d9/d1b/d54/d69/d6c/c87 0 2024-03-04T16:04:40.403 INFO:tasks.workunit.client.1.smithi152.stdout:1/540: write d19/d4b/d53/f55 [709100,11470] 0 2024-03-04T16:04:40.404 INFO:tasks.workunit.client.1.smithi152.stdout:8/552: getdents d2/d3/d21/d2d/d31/d9a 0 2024-03-04T16:04:40.408 INFO:tasks.workunit.client.1.smithi152.stdout:9/481: unlink d2/f19 0 2024-03-04T16:04:40.410 INFO:tasks.workunit.client.1.smithi152.stdout:3/504: symlink d6/d10/d2d/d4d/la9 0 2024-03-04T16:04:40.413 INFO:tasks.workunit.client.1.smithi152.stdout:2/499: symlink da/df/d35/la9 0 2024-03-04T16:04:40.415 INFO:tasks.workunit.client.1.smithi152.stdout:4/440: symlink d8/d78/l96 0 2024-03-04T16:04:40.420 INFO:tasks.workunit.client.1.smithi152.stdout:5/519: dwrite d6/d8/f4b [0,4194304] 0 2024-03-04T16:04:40.426 INFO:tasks.workunit.client.1.smithi152.stdout:4/441: dwrite d8/d9/d24/f4d [0,4194304] 0 2024-03-04T16:04:40.427 INFO:tasks.workunit.client.1.smithi152.stdout:4/442: fsync d8/d9/d24/f74 0 2024-03-04T16:04:40.433 INFO:tasks.workunit.client.1.smithi152.stdout:3/505: dwrite d6/f1c [0,4194304] 0 2024-03-04T16:04:40.435 INFO:tasks.workunit.client.1.smithi152.stdout:0/435: rename d9/d1a/l58 to d9/d1b/d54/d80/l88 0 2024-03-04T16:04:40.438 INFO:tasks.workunit.client.1.smithi152.stdout:8/553: mkdir d2/d3/d21/dac 0 2024-03-04T16:04:40.439 INFO:tasks.workunit.client.1.smithi152.stdout:5/520: mknod d6/d50/cb4 0 2024-03-04T16:04:40.440 INFO:tasks.workunit.client.1.smithi152.stdout:5/521: write d6/d8/d1f/d3d/d6f/d70/fa2 [825798,66058] 0 2024-03-04T16:04:40.442 INFO:tasks.workunit.client.1.smithi152.stdout:5/522: truncate d6/d8/d1f/d3d/d6f/f73 883080 0 2024-03-04T16:04:40.443 INFO:tasks.workunit.client.1.smithi152.stdout:2/500: symlink da/df/d2e/laa 0 2024-03-04T16:04:40.453 INFO:tasks.workunit.client.1.smithi152.stdout:2/501: mknod da/d24/d80/d82/cab 0 2024-03-04T16:04:40.455 INFO:tasks.workunit.client.1.smithi152.stdout:0/436: symlink d9/l89 0 2024-03-04T16:04:40.457 INFO:tasks.workunit.client.1.smithi152.stdout:9/482: mknod d2/d13/d35/c9c 0 2024-03-04T16:04:40.459 INFO:tasks.workunit.client.1.smithi152.stdout:8/554: mknod d2/d3/d15/cad 0 2024-03-04T16:04:40.461 INFO:tasks.workunit.client.1.smithi152.stdout:0/437: dwrite d9/d2c/d2e/f6a [0,4194304] 0 2024-03-04T16:04:40.461 INFO:tasks.workunit.client.1.smithi152.stdout:1/541: link d19/d4b/d53/d57/d62/d82/l98 d19/d4b/d53/d57/d85/lb7 0 2024-03-04T16:04:40.464 INFO:tasks.workunit.client.1.smithi152.stdout:5/523: symlink d6/d8/d5a/d8b/dad/lb5 0 2024-03-04T16:04:40.464 INFO:tasks.workunit.client.1.smithi152.stdout:3/506: write d6/d10/f65 [283479,103187] 0 2024-03-04T16:04:40.470 INFO:tasks.workunit.client.1.smithi152.stdout:0/438: dread d9/f37 [0,4194304] 0 2024-03-04T16:04:40.471 INFO:tasks.workunit.client.1.smithi152.stdout:0/439: read - d9/d1b/d3c/d84/d56/f63 zero size 2024-03-04T16:04:40.475 INFO:tasks.workunit.client.1.smithi152.stdout:1/542: mkdir d19/d75/da8/db8 0 2024-03-04T16:04:40.476 INFO:tasks.workunit.client.1.smithi152.stdout:1/543: chown d19/f1a 1968 1 2024-03-04T16:04:40.477 INFO:tasks.workunit.client.1.smithi152.stdout:1/544: chown d19/d20/d2f/d6b/d38/da6 18 1 2024-03-04T16:04:40.478 INFO:tasks.workunit.client.1.smithi152.stdout:1/545: fsync d19/d20/d2f/f8a 0 2024-03-04T16:04:40.479 INFO:tasks.workunit.client.1.smithi152.stdout:2/502: mknod da/d5f/cac 0 2024-03-04T16:04:40.480 INFO:tasks.workunit.client.1.smithi152.stdout:5/524: symlink d6/d8/d5a/d64/d8a/lb6 0 2024-03-04T16:04:40.480 INFO:tasks.workunit.client.1.smithi152.stdout:2/503: fsync da/d2c/f96 0 2024-03-04T16:04:40.481 INFO:tasks.workunit.client.1.smithi152.stdout:9/483: mknod d2/d13/d96/c9d 0 2024-03-04T16:04:40.484 INFO:tasks.workunit.client.1.smithi152.stdout:8/555: dwrite d2/d3/d15/d1c/f56 [0,4194304] 0 2024-03-04T16:04:40.486 INFO:tasks.workunit.client.1.smithi152.stdout:1/546: unlink c7 0 2024-03-04T16:04:40.492 INFO:tasks.workunit.client.1.smithi152.stdout:0/440: dwrite d9/d1b/f35 [0,4194304] 0 2024-03-04T16:04:40.493 INFO:tasks.workunit.client.1.smithi152.stdout:7/431: sync 2024-03-04T16:04:40.498 INFO:tasks.workunit.client.1.smithi152.stdout:3/507: dwrite d6/d14/d41/d49/f95 [0,4194304] 0 2024-03-04T16:04:40.499 INFO:tasks.workunit.client.1.smithi152.stdout:5/525: readlink d6/d8/d5a/d64/d76/d9f/lae 0 2024-03-04T16:04:40.500 INFO:tasks.workunit.client.1.smithi152.stdout:9/484: unlink d2/d13/c6d 0 2024-03-04T16:04:40.505 INFO:tasks.workunit.client.1.smithi152.stdout:2/504: dwrite da/d5f/d85/f99 [0,4194304] 0 2024-03-04T16:04:40.511 INFO:tasks.workunit.client.1.smithi152.stdout:7/432: mkdir d14/d21/d48/d80 0 2024-03-04T16:04:40.512 INFO:tasks.workunit.client.1.smithi152.stdout:3/508: dwrite f5 [0,4194304] 0 2024-03-04T16:04:40.517 INFO:tasks.workunit.client.1.smithi152.stdout:9/485: mknod d2/d13/d35/c9e 0 2024-03-04T16:04:40.519 INFO:tasks.workunit.client.1.smithi152.stdout:5/526: dread d6/d8/d1f/d3d/f4c [0,4194304] 0 2024-03-04T16:04:40.520 INFO:tasks.workunit.client.1.smithi152.stdout:5/527: chown d6/d8/d5a/d64/d76/d9f 19771 1 2024-03-04T16:04:40.521 INFO:tasks.workunit.client.1.smithi152.stdout:7/433: rmdir d14/d21/d47/d36 39 2024-03-04T16:04:40.523 INFO:tasks.workunit.client.1.smithi152.stdout:3/509: symlink d6/d60/d8e/laa 0 2024-03-04T16:04:40.531 INFO:tasks.workunit.client.1.smithi152.stdout:2/505: unlink da/d24/c5d 0 2024-03-04T16:04:40.532 INFO:tasks.workunit.client.1.smithi152.stdout:2/506: write da/d5f/d85/f99 [1655196,69303] 0 2024-03-04T16:04:40.533 INFO:tasks.workunit.client.1.smithi152.stdout:2/507: dread - da/df/d2e/f8f zero size 2024-03-04T16:04:40.534 INFO:tasks.workunit.client.1.smithi152.stdout:2/508: dread - da/df/d58/f79 zero size 2024-03-04T16:04:40.536 INFO:tasks.workunit.client.1.smithi152.stdout:0/441: rmdir d9/d1a/d83 0 2024-03-04T16:04:40.537 INFO:tasks.workunit.client.1.smithi152.stdout:8/556: getdents d2/d75/d89 0 2024-03-04T16:04:40.537 INFO:tasks.workunit.client.1.smithi152.stdout:0/442: read d9/d1b/f6d [4161157,33614] 0 2024-03-04T16:04:40.537 INFO:tasks.workunit.client.1.smithi152.stdout:0/443: fsync d9/d1b/f6e 0 2024-03-04T16:04:40.541 INFO:tasks.workunit.client.1.smithi152.stdout:0/444: dread d9/d1b/f71 [4194304,4194304] 0 2024-03-04T16:04:40.541 INFO:tasks.workunit.client.1.smithi152.stdout:5/528: creat d6/d8/d1f/d3d/d6f/d70/d98/fb7 x:0 0 0 2024-03-04T16:04:40.542 INFO:tasks.workunit.client.1.smithi152.stdout:5/529: readlink d6/l47 0 2024-03-04T16:04:40.544 INFO:tasks.workunit.client.1.smithi152.stdout:7/434: rename d14/d21/d47/d40/f7a to d14/d69/f81 0 2024-03-04T16:04:40.557 INFO:tasks.workunit.client.1.smithi152.stdout:8/557: creat d2/d3/d21/d2d/d52/d68/fae x:0 0 0 2024-03-04T16:04:40.563 INFO:tasks.workunit.client.1.smithi152.stdout:0/445: read d9/d2c/d2e/f32 [146128,48958] 0 2024-03-04T16:04:40.563 INFO:tasks.workunit.client.1.smithi152.stdout:0/446: chown d9/d1b/d3c/d84 10397 1 2024-03-04T16:04:40.564 INFO:tasks.workunit.client.1.smithi152.stdout:0/447: write d9/d1b/d3c/d84/f77 [379652,12327] 0 2024-03-04T16:04:40.566 INFO:tasks.workunit.client.1.smithi152.stdout:5/530: mkdir d6/d8/d1f/d3d/d6f/db8 0 2024-03-04T16:04:40.567 INFO:tasks.workunit.client.1.smithi152.stdout:2/509: rename da/d24/d80/d82/cab to da/df/d2e/d61/d8e/cad 0 2024-03-04T16:04:40.569 INFO:tasks.workunit.client.1.smithi152.stdout:7/435: truncate d14/d21/d31/f33 55253 0 2024-03-04T16:04:40.569 INFO:tasks.workunit.client.1.smithi152.stdout:7/436: chown l3 143 1 2024-03-04T16:04:40.570 INFO:tasks.workunit.client.1.smithi152.stdout:3/510: creat d6/d14/d41/d49/d84/fab x:0 0 0 2024-03-04T16:04:40.571 INFO:tasks.workunit.client.1.smithi152.stdout:9/486: link d2/d13/d2e/c3a d2/d13/d2e/c9f 0 2024-03-04T16:04:40.578 INFO:tasks.workunit.client.1.smithi152.stdout:5/531: dread d6/d8/f1d [0,4194304] 0 2024-03-04T16:04:40.578 INFO:tasks.workunit.client.1.smithi152.stdout:8/558: rename d2/d3/d21/d2d/d48/l65 to d2/d75/d89/dd/d22/d6c/laf 0 2024-03-04T16:04:40.582 INFO:tasks.workunit.client.1.smithi152.stdout:3/511: creat d6/d5c/fac x:0 0 0 2024-03-04T16:04:40.583 INFO:tasks.workunit.client.1.smithi152.stdout:9/487: rename d2/d13/d23/f4e to d2/d13/d23/fa0 0 2024-03-04T16:04:40.591 INFO:tasks.workunit.client.1.smithi152.stdout:5/532: rename d6/d8/fe to d6/d8/d5a/d64/d8a/fb9 0 2024-03-04T16:04:40.593 INFO:tasks.workunit.client.1.smithi152.stdout:0/448: truncate f7 4785168 0 2024-03-04T16:04:40.594 INFO:tasks.workunit.client.1.smithi152.stdout:0/449: write d9/d1b/d3c/d84/f77 [35144,129151] 0 2024-03-04T16:04:40.595 INFO:tasks.workunit.client.1.smithi152.stdout:9/488: creat d2/d13/d23/d71/fa1 x:0 0 0 2024-03-04T16:04:40.596 INFO:tasks.workunit.client.1.smithi152.stdout:9/489: write d2/d13/d23/d42/f78 [421516,12535] 0 2024-03-04T16:04:40.596 INFO:tasks.workunit.client.1.smithi152.stdout:2/510: creat da/df/fae x:0 0 0 2024-03-04T16:04:40.597 INFO:tasks.workunit.client.1.smithi152.stdout:8/559: mknod d2/cb0 0 2024-03-04T16:04:40.600 INFO:tasks.workunit.client.1.smithi152.stdout:8/560: dread d2/d3/d15/d1c/d84/d87/d5d/f8b [0,4194304] 0 2024-03-04T16:04:40.602 INFO:tasks.workunit.client.1.smithi152.stdout:3/512: mkdir d6/d14/d58/dad 0 2024-03-04T16:04:40.605 INFO:tasks.workunit.client.1.smithi152.stdout:9/490: creat d2/d13/d23/d42/fa2 x:0 0 0 2024-03-04T16:04:40.606 INFO:tasks.workunit.client.1.smithi152.stdout:9/491: write d2/d13/d2e/f4f [1858516,56625] 0 2024-03-04T16:04:40.608 INFO:tasks.workunit.client.1.smithi152.stdout:2/511: symlink da/d24/d47/d4f/laf 0 2024-03-04T16:04:40.610 INFO:tasks.workunit.client.1.smithi152.stdout:7/437: rename d14/d71/f7b to d14/d21/d47/d36/f82 0 2024-03-04T16:04:40.610 INFO:tasks.workunit.client.1.smithi152.stdout:5/533: creat d6/d8/d1f/d3d/da4/fba x:0 0 0 2024-03-04T16:04:40.612 INFO:tasks.workunit.client.1.smithi152.stdout:8/561: mkdir d2/d3/d21/d2d/d52/d68/d6e/db1 0 2024-03-04T16:04:40.613 INFO:tasks.workunit.client.1.smithi152.stdout:8/562: write d2/d3/f4b [4830636,103676] 0 2024-03-04T16:04:40.619 INFO:tasks.workunit.client.1.smithi152.stdout:2/512: fdatasync f6 0 2024-03-04T16:04:40.623 INFO:tasks.workunit.client.1.smithi152.stdout:7/438: dread d14/f61 [0,4194304] 0 2024-03-04T16:04:40.624 INFO:tasks.workunit.client.1.smithi152.stdout:3/513: rename d6/d53/f82 to d6/d14/d41/d49/fae 0 2024-03-04T16:04:40.624 INFO:tasks.workunit.client.1.smithi152.stdout:9/492: dwrite d2/d13/f52 [0,4194304] 0 2024-03-04T16:04:40.630 INFO:tasks.workunit.client.1.smithi152.stdout:6/585: sync 2024-03-04T16:04:40.634 INFO:tasks.workunit.client.1.smithi152.stdout:6/586: read d8/d10/f1e [3771142,23707] 0 2024-03-04T16:04:40.634 INFO:tasks.workunit.client.1.smithi152.stdout:3/514: dwrite d6/f96 [0,4194304] 0 2024-03-04T16:04:40.634 INFO:tasks.workunit.client.1.smithi152.stdout:0/450: creat d9/d1a/f8a x:0 0 0 2024-03-04T16:04:40.636 INFO:tasks.workunit.client.1.smithi152.stdout:5/534: dread d6/d8/f41 [0,4194304] 0 2024-03-04T16:04:40.639 INFO:tasks.workunit.client.1.smithi152.stdout:5/535: write d6/d8/d1f/d3d/d6f/d70/fa8 [438102,102922] 0 2024-03-04T16:04:40.645 INFO:tasks.workunit.client.1.smithi152.stdout:9/493: readlink d2/l1d 0 2024-03-04T16:04:40.647 INFO:tasks.workunit.client.1.smithi152.stdout:5/536: dwrite d6/d8/f9a [0,4194304] 0 2024-03-04T16:04:40.651 INFO:tasks.workunit.client.1.smithi152.stdout:3/515: creat d6/d53/faf x:0 0 0 2024-03-04T16:04:40.652 INFO:tasks.workunit.client.1.smithi152.stdout:3/516: readlink d6/d53/l69 0 2024-03-04T16:04:40.653 INFO:tasks.workunit.client.1.smithi152.stdout:6/587: mkdir d8/d92/db1/d87/dca 0 2024-03-04T16:04:40.653 INFO:tasks.workunit.client.1.smithi152.stdout:3/517: chown d6/d10/d2d/d6e 17 1 2024-03-04T16:04:40.654 INFO:tasks.workunit.client.1.smithi152.stdout:3/518: write d6/d60/d8e/fa4 [415719,81411] 0 2024-03-04T16:04:40.655 INFO:tasks.workunit.client.1.smithi152.stdout:3/519: chown d6/d14/d41/d49/f62 0 1 2024-03-04T16:04:40.655 INFO:tasks.workunit.client.1.smithi152.stdout:8/563: link d2/d75/d89/f28 d2/d3/d21/d2d/d48/fb2 0 2024-03-04T16:04:40.656 INFO:tasks.workunit.client.1.smithi152.stdout:8/564: chown d2/d3/d21/l51 1222 1 2024-03-04T16:04:40.657 INFO:tasks.workunit.client.1.smithi152.stdout:8/565: truncate d2/d75/f9f 321896 0 2024-03-04T16:04:40.659 INFO:tasks.workunit.client.1.smithi152.stdout:2/513: mkdir da/d5f/db0 0 2024-03-04T16:04:40.660 INFO:tasks.workunit.client.1.smithi152.stdout:2/514: fsync da/d24/d80/d82/da8/f71 0 2024-03-04T16:04:40.660 INFO:tasks.workunit.client.1.smithi152.stdout:7/439: truncate d14/d21/d47/d36/f82 753387 0 2024-03-04T16:04:40.661 INFO:tasks.workunit.client.1.smithi152.stdout:7/440: readlink d14/d21/l24 0 2024-03-04T16:04:40.661 INFO:tasks.workunit.client.1.smithi152.stdout:7/441: stat d14/d21/d47/f5e 0 2024-03-04T16:04:40.662 INFO:tasks.workunit.client.1.smithi152.stdout:7/442: fsync d14/d69/f6d 0 2024-03-04T16:04:40.663 INFO:tasks.workunit.client.1.smithi152.stdout:9/494: mkdir d2/d13/d23/d71/da3 0 2024-03-04T16:04:40.663 INFO:tasks.workunit.client.1.smithi152.stdout:7/443: write d14/d21/f25 [2314275,93703] 0 2024-03-04T16:04:40.664 INFO:tasks.workunit.client.1.smithi152.stdout:9/495: write d2/d13/d35/f5d [881309,6324] 0 2024-03-04T16:04:40.666 INFO:tasks.workunit.client.1.smithi152.stdout:6/588: creat d8/d92/d59/d61/fcb x:0 0 0 2024-03-04T16:04:40.667 INFO:tasks.workunit.client.1.smithi152.stdout:3/520: creat d6/d14/d58/fb0 x:0 0 0 2024-03-04T16:04:40.667 INFO:tasks.workunit.client.1.smithi152.stdout:6/589: readlink d8/d42/l9c 0 2024-03-04T16:04:40.668 INFO:tasks.workunit.client.1.smithi152.stdout:3/521: readlink d6/d14/d41/d49/l5e 0 2024-03-04T16:04:40.669 INFO:tasks.workunit.client.1.smithi152.stdout:3/522: stat d6/l4f 0 2024-03-04T16:04:40.669 INFO:tasks.workunit.client.1.smithi152.stdout:3/523: fsync d6/d10/f65 0 2024-03-04T16:04:40.671 INFO:tasks.workunit.client.1.smithi152.stdout:2/515: unlink da/df/d2e/f39 0 2024-03-04T16:04:40.672 INFO:tasks.workunit.client.1.smithi152.stdout:2/516: chown da/df/c20 127329668 1 2024-03-04T16:04:40.673 INFO:tasks.workunit.client.1.smithi152.stdout:7/444: read d14/f2f [705509,92298] 0 2024-03-04T16:04:40.676 INFO:tasks.workunit.client.1.smithi152.stdout:2/517: dread da/d5f/d85/f99 [0,4194304] 0 2024-03-04T16:04:40.678 INFO:tasks.workunit.client.1.smithi152.stdout:3/524: rmdir d6/d10/d2d/d4d 39 2024-03-04T16:04:40.679 INFO:tasks.workunit.client.1.smithi152.stdout:3/525: chown d6/d10/d2d/f36 394 1 2024-03-04T16:04:40.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/496: write d2/d63/f94 [2093324,92230] 0 2024-03-04T16:04:40.682 INFO:tasks.workunit.client.1.smithi152.stdout:7/445: mkdir d14/d2b/d83 0 2024-03-04T16:04:40.683 INFO:tasks.workunit.client.1.smithi152.stdout:6/590: truncate d8/d10/f1e 4463024 0 2024-03-04T16:04:40.689 INFO:tasks.workunit.client.1.smithi152.stdout:8/566: dwrite d2/d3/d15/d1c/d84/d87/d5d/f8b [0,4194304] 0 2024-03-04T16:04:40.693 INFO:tasks.workunit.client.1.smithi152.stdout:7/446: rmdir d14/d21 39 2024-03-04T16:04:40.695 INFO:tasks.workunit.client.1.smithi152.stdout:6/591: mkdir d8/d23/d3a/d71/d9d/dcc 0 2024-03-04T16:04:40.696 INFO:tasks.workunit.client.1.smithi152.stdout:2/518: mknod da/d24/d47/d73/cb1 0 2024-03-04T16:04:40.698 INFO:tasks.workunit.client.1.smithi152.stdout:6/592: chown d8/d2a/l4d 2 1 2024-03-04T16:04:40.698 INFO:tasks.workunit.client.1.smithi152.stdout:9/497: dwrite d2/fe [0,4194304] 0 2024-03-04T16:04:40.699 INFO:tasks.workunit.client.1.smithi152.stdout:6/593: readlink d8/d52/l68 0 2024-03-04T16:04:40.703 INFO:tasks.workunit.client.1.smithi152.stdout:8/567: mknod d2/d7f/cb3 0 2024-03-04T16:04:40.710 INFO:tasks.workunit.client.1.smithi152.stdout:8/568: dwrite d2/d75/d89/f73 [0,4194304] 0 2024-03-04T16:04:40.715 INFO:tasks.workunit.client.1.smithi152.stdout:6/594: fsync d8/d10/f44 0 2024-03-04T16:04:40.722 INFO:tasks.workunit.client.1.smithi152.stdout:8/569: mknod d2/d75/d89/cb4 0 2024-03-04T16:04:40.724 INFO:tasks.workunit.client.1.smithi152.stdout:7/447: write d14/d2b/f58 [1030156,102429] 0 2024-03-04T16:04:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:40 smithi116 ceph-mon[29120]: pgmap v36: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 27 MiB/s rd, 137 MiB/s wr, 287 op/s 2024-03-04T16:04:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:40 smithi116 ceph-mon[29120]: mgrmap e25: smithi152.aawieo(active, since 66s), standbys: smithi116.jwdhop 2024-03-04T16:04:40.726 INFO:tasks.workunit.client.1.smithi152.stdout:2/519: write da/df/f37 [1858519,130581] 0 2024-03-04T16:04:40.727 INFO:tasks.workunit.client.1.smithi152.stdout:3/526: getdents d6/d10/d2d 0 2024-03-04T16:04:40.728 INFO:tasks.workunit.client.1.smithi152.stdout:2/520: dread - da/d24/d47/f4d zero size 2024-03-04T16:04:40.731 INFO:tasks.workunit.client.1.smithi152.stdout:7/448: dwrite f7 [0,4194304] 0 2024-03-04T16:04:40.732 INFO:tasks.workunit.client.1.smithi152.stdout:6/595: creat d8/d92/d59/d7d/d91/fcd x:0 0 0 2024-03-04T16:04:40.734 INFO:tasks.workunit.client.1.smithi152.stdout:8/570: creat d2/d3/d15/d1c/fb5 x:0 0 0 2024-03-04T16:04:40.742 INFO:tasks.workunit.client.1.smithi152.stdout:9/498: truncate d2/d13/d35/f8c 373053 0 2024-03-04T16:04:40.744 INFO:tasks.workunit.client.1.smithi152.stdout:7/449: chown d14/d21/d47/d36/f68 8677240 1 2024-03-04T16:04:40.747 INFO:tasks.workunit.client.1.smithi152.stdout:8/571: fdatasync d2/d75/d89/dd/f93 0 2024-03-04T16:04:40.747 INFO:tasks.workunit.client.1.smithi152.stdout:2/521: mknod da/d5f/db0/cb2 0 2024-03-04T16:04:40.748 INFO:tasks.workunit.client.1.smithi152.stdout:3/527: dread d6/d14/d93/f77 [0,4194304] 0 2024-03-04T16:04:40.750 INFO:tasks.workunit.client.1.smithi152.stdout:8/572: dread - d2/d75/d89/dd/d22/fa4 zero size 2024-03-04T16:04:40.750 INFO:tasks.workunit.client.1.smithi152.stdout:2/522: chown da/d24/d47 324 1 2024-03-04T16:04:40.753 INFO:tasks.workunit.client.1.smithi152.stdout:7/450: dwrite d14/f2a [0,4194304] 0 2024-03-04T16:04:40.754 INFO:tasks.workunit.client.1.smithi152.stdout:7/451: readlink d14/d21/d47/l54 0 2024-03-04T16:04:40.755 INFO:tasks.workunit.client.1.smithi152.stdout:9/499: rename d2/f91 to d2/d13/d96/fa4 0 2024-03-04T16:04:40.756 INFO:tasks.workunit.client.1.smithi152.stdout:2/523: read da/f3c [1606322,1816] 0 2024-03-04T16:04:40.757 INFO:tasks.workunit.client.1.smithi152.stdout:6/596: dread d8/f22 [0,4194304] 0 2024-03-04T16:04:40.757 INFO:tasks.workunit.client.1.smithi152.stdout:2/524: readlink da/d24/d47/d4f/l54 0 2024-03-04T16:04:40.760 INFO:tasks.workunit.client.1.smithi152.stdout:8/573: unlink d2/d75/d89/dd/d22/d83/d96/ca5 0 2024-03-04T16:04:40.761 INFO:tasks.workunit.client.1.smithi152.stdout:3/528: mkdir d6/db1 0 2024-03-04T16:04:40.764 INFO:tasks.workunit.client.1.smithi152.stdout:3/529: dread d6/d53/f88 [0,4194304] 0 2024-03-04T16:04:40.764 INFO:tasks.workunit.client.1.smithi152.stdout:3/530: fsync d6/f3a 0 2024-03-04T16:04:40.767 INFO:tasks.workunit.client.1.smithi152.stdout:6/597: fsync f1 0 2024-03-04T16:04:40.769 INFO:tasks.workunit.client.1.smithi152.stdout:6/598: write d8/d92/db1/d87/fc2 [993770,12922] 0 2024-03-04T16:04:40.772 INFO:tasks.workunit.client.1.smithi152.stdout:7/452: truncate d14/f78 336239 0 2024-03-04T16:04:40.772 INFO:tasks.workunit.client.1.smithi152.stdout:1/547: sync 2024-03-04T16:04:40.772 INFO:tasks.workunit.client.1.smithi152.stdout:4/443: sync 2024-03-04T16:04:40.774 INFO:tasks.workunit.client.1.smithi152.stdout:1/548: write f10 [4375422,117927] 0 2024-03-04T16:04:40.775 INFO:tasks.workunit.client.1.smithi152.stdout:1/549: fsync f16 0 2024-03-04T16:04:40.776 INFO:tasks.workunit.client.1.smithi152.stdout:1/550: stat l13 0 2024-03-04T16:04:40.777 INFO:tasks.workunit.client.1.smithi152.stdout:6/599: dwrite d8/d92/db1/f95 [4194304,4194304] 0 2024-03-04T16:04:40.780 INFO:tasks.workunit.client.1.smithi152.stdout:6/600: write f7 [5182560,66471] 0 2024-03-04T16:04:40.780 INFO:tasks.workunit.client.1.smithi152.stdout:1/551: read d19/f1a [565697,121369] 0 2024-03-04T16:04:40.782 INFO:tasks.workunit.client.1.smithi152.stdout:7/453: dwrite d14/d2b/f3b [4194304,4194304] 0 2024-03-04T16:04:40.785 INFO:tasks.workunit.client.1.smithi152.stdout:2/525: creat da/df/d58/da2/fb3 x:0 0 0 2024-03-04T16:04:40.785 INFO:tasks.workunit.client.1.smithi152.stdout:7/454: write d14/d71/f7d [236311,54217] 0 2024-03-04T16:04:40.790 INFO:tasks.workunit.client.1.smithi152.stdout:1/552: dread d19/d4b/d53/f55 [0,4194304] 0 2024-03-04T16:04:40.792 INFO:tasks.workunit.client.1.smithi152.stdout:9/500: write d2/d13/f52 [2069599,77391] 0 2024-03-04T16:04:40.793 INFO:tasks.workunit.client.1.smithi152.stdout:3/531: symlink d6/d60/d8a/lb2 0 2024-03-04T16:04:40.793 INFO:tasks.workunit.client.1.smithi152.stdout:8/574: write d2/d3/d15/d1c/d84/d87/d5d/f7b [335152,3115] 0 2024-03-04T16:04:40.796 INFO:tasks.workunit.client.1.smithi152.stdout:6/601: mknod d8/d42/cce 0 2024-03-04T16:04:40.799 INFO:tasks.workunit.client.1.smithi152.stdout:7/455: unlink fd 0 2024-03-04T16:04:40.799 INFO:tasks.workunit.client.1.smithi152.stdout:8/575: dread d2/d3/f40 [0,4194304] 0 2024-03-04T16:04:40.801 INFO:tasks.workunit.client.1.smithi152.stdout:2/526: creat da/df/d2e/d51/fb4 x:0 0 0 2024-03-04T16:04:40.804 INFO:tasks.workunit.client.1.smithi152.stdout:6/602: dwrite d8/d10/f16 [4194304,4194304] 0 2024-03-04T16:04:40.806 INFO:tasks.workunit.client.1.smithi152.stdout:9/501: chown d2/d13/c1e 298 1 2024-03-04T16:04:40.809 INFO:tasks.workunit.client.1.smithi152.stdout:7/456: symlink d14/d21/d47/d40/l84 0 2024-03-04T16:04:40.810 INFO:tasks.workunit.client.1.smithi152.stdout:2/527: mkdir da/df/d58/da2/db5 0 2024-03-04T16:04:40.812 INFO:tasks.workunit.client.1.smithi152.stdout:6/603: stat d8/d10/f1e 0 2024-03-04T16:04:40.813 INFO:tasks.workunit.client.1.smithi152.stdout:6/604: write d8/d23/f5c [750167,34098] 0 2024-03-04T16:04:40.816 INFO:tasks.workunit.client.1.smithi152.stdout:3/532: creat d6/db1/fb3 x:0 0 0 2024-03-04T16:04:40.818 INFO:tasks.workunit.client.1.smithi152.stdout:9/502: read d2/f2d [552281,4255] 0 2024-03-04T16:04:40.818 INFO:tasks.workunit.client.1.smithi152.stdout:8/576: symlink d2/d3/lb6 0 2024-03-04T16:04:40.821 INFO:tasks.workunit.client.1.smithi152.stdout:7/457: rename d14/c3d to d14/d2b/c85 0 2024-03-04T16:04:40.823 INFO:tasks.workunit.client.1.smithi152.stdout:2/528: chown da/df/c18 95 1 2024-03-04T16:04:40.823 INFO:tasks.workunit.client.1.smithi152.stdout:8/577: dread d2/d3/d15/d1c/f56 [0,4194304] 0 2024-03-04T16:04:40.824 INFO:tasks.workunit.client.1.smithi152.stdout:2/529: dread - da/d2c/d5e/f69 zero size 2024-03-04T16:04:40.825 INFO:tasks.workunit.client.1.smithi152.stdout:4/444: truncate d8/d4e/f57 1643453 0 2024-03-04T16:04:40.830 INFO:tasks.workunit.client.1.smithi152.stdout:2/530: dwrite da/df/f37 [0,4194304] 0 2024-03-04T16:04:40.832 INFO:tasks.workunit.client.1.smithi152.stdout:2/531: write da/d2c/d5e/f6d [910151,12232] 0 2024-03-04T16:04:40.840 INFO:tasks.workunit.client.1.smithi152.stdout:1/553: rename d19/f7c to d19/d20/d2f/d5d/fb9 0 2024-03-04T16:04:40.840 INFO:tasks.workunit.client.1.smithi152.stdout:4/445: dread f7 [0,4194304] 0 2024-03-04T16:04:40.850 INFO:tasks.workunit.client.1.smithi152.stdout:3/533: symlink d6/d53/da1/lb4 0 2024-03-04T16:04:40.851 INFO:tasks.workunit.client.1.smithi152.stdout:2/532: mknod da/df/d2e/d61/cb6 0 2024-03-04T16:04:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:40 smithi152 ceph-mon[36876]: pgmap v36: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 27 MiB/s rd, 137 MiB/s wr, 287 op/s 2024-03-04T16:04:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:40 smithi152 ceph-mon[36876]: mgrmap e25: smithi152.aawieo(active, since 66s), standbys: smithi116.jwdhop 2024-03-04T16:04:40.861 INFO:tasks.workunit.client.1.smithi152.stdout:8/578: dwrite d2/d3/f40 [0,4194304] 0 2024-03-04T16:04:40.865 INFO:tasks.workunit.client.1.smithi152.stdout:4/446: creat d8/d2f/d7c/d87/f97 x:0 0 0 2024-03-04T16:04:40.868 INFO:tasks.workunit.client.1.smithi152.stdout:9/503: getdents d2/d13/d23/d66/d7d 0 2024-03-04T16:04:40.870 INFO:tasks.workunit.client.1.smithi152.stdout:8/579: creat d2/d3/d21/d2d/d31/fb7 x:0 0 0 2024-03-04T16:04:40.873 INFO:tasks.workunit.client.1.smithi152.stdout:3/534: write d6/d14/d93/f77 [4040747,125931] 0 2024-03-04T16:04:40.874 INFO:tasks.workunit.client.1.smithi152.stdout:1/554: dwrite d19/f5b [0,4194304] 0 2024-03-04T16:04:40.879 INFO:tasks.workunit.client.1.smithi152.stdout:8/580: mkdir d2/d3/d21/d2d/d48/db8 0 2024-03-04T16:04:40.880 INFO:tasks.workunit.client.1.smithi152.stdout:2/533: creat da/df/fb7 x:0 0 0 2024-03-04T16:04:40.881 INFO:tasks.workunit.client.1.smithi152.stdout:3/535: mkdir d6/d53/db5 0 2024-03-04T16:04:40.883 INFO:tasks.workunit.client.1.smithi152.stdout:1/555: mknod d19/d75/da8/cba 0 2024-03-04T16:04:40.889 INFO:tasks.workunit.client.1.smithi152.stdout:8/581: dwrite d2/d3/d21/d2d/d52/d68/f72 [0,4194304] 0 2024-03-04T16:04:40.891 INFO:tasks.workunit.client.1.smithi152.stdout:8/582: chown d2/d3/d15 36495475 1 2024-03-04T16:04:40.891 INFO:tasks.workunit.client.1.smithi152.stdout:8/583: stat d2/d75/f8f 0 2024-03-04T16:04:40.893 INFO:tasks.workunit.client.1.smithi152.stdout:8/584: dread - d2/d3/d21/d2d/d52/d68/fae zero size 2024-03-04T16:04:40.896 INFO:tasks.workunit.client.1.smithi152.stdout:1/556: symlink d19/d3b/d6e/d97/lbb 0 2024-03-04T16:04:40.897 INFO:tasks.workunit.client.1.smithi152.stdout:9/504: dread d2/d13/d23/f29 [0,4194304] 0 2024-03-04T16:04:40.898 INFO:tasks.workunit.client.1.smithi152.stdout:2/534: truncate da/d24/f42 829551 0 2024-03-04T16:04:40.900 INFO:tasks.workunit.client.1.smithi152.stdout:4/447: rename d8/c55 to d8/d9/c98 0 2024-03-04T16:04:40.910 INFO:tasks.workunit.client.1.smithi152.stdout:2/535: fsync da/d2c/f4b 0 2024-03-04T16:04:40.912 INFO:tasks.workunit.client.1.smithi152.stdout:3/536: truncate d6/d14/d93/f89 1717244 0 2024-03-04T16:04:40.913 INFO:tasks.workunit.client.1.smithi152.stdout:8/585: creat d2/d3/d21/d2d/d48/db8/fb9 x:0 0 0 2024-03-04T16:04:40.914 INFO:tasks.workunit.client.1.smithi152.stdout:9/505: dwrite d2/d63/f94 [0,4194304] 0 2024-03-04T16:04:40.915 INFO:tasks.workunit.client.1.smithi152.stdout:8/586: dread - d2/d3/d21/d2d/d52/d68/d6e/d7d/f8d zero size 2024-03-04T16:04:40.918 INFO:tasks.workunit.client.1.smithi152.stdout:8/587: write d2/d3/d21/d2d/d52/d68/fa2 [389726,106829] 0 2024-03-04T16:04:40.920 INFO:tasks.workunit.client.1.smithi152.stdout:8/588: write d2/d75/d89/dd/d22/fa4 [695317,111696] 0 2024-03-04T16:04:40.923 INFO:tasks.workunit.client.1.smithi152.stdout:4/448: write d8/d9/d24/d26/f2e [3619438,67430] 0 2024-03-04T16:04:40.928 INFO:tasks.workunit.client.1.smithi152.stdout:3/537: mknod d6/d14/d41/d49/cb6 0 2024-03-04T16:04:40.943 INFO:tasks.workunit.client.1.smithi152.stdout:1/557: getdents d19/d20/d2f 0 2024-03-04T16:04:40.943 INFO:tasks.workunit.client.1.smithi152.stdout:8/589: dread d2/d75/d89/dd/d22/d6c/f74 [0,4194304] 0 2024-03-04T16:04:40.947 INFO:tasks.workunit.client.1.smithi152.stdout:4/449: dread d8/d9/ff [4194304,4194304] 0 2024-03-04T16:04:40.947 INFO:tasks.workunit.client.1.smithi152.stdout:0/451: sync 2024-03-04T16:04:40.950 INFO:tasks.workunit.client.1.smithi152.stdout:1/558: mknod d19/d4b/d53/cbc 0 2024-03-04T16:04:40.961 INFO:tasks.workunit.client.1.smithi152.stdout:8/590: mkdir d2/d3/d21/d2d/d52/d68/d6e/d7d/dba 0 2024-03-04T16:04:40.962 INFO:tasks.workunit.client.1.smithi152.stdout:2/536: getdents da/d24/d47 0 2024-03-04T16:04:40.963 INFO:tasks.workunit.client.1.smithi152.stdout:2/537: chown da/df/d2e/l98 3 1 2024-03-04T16:04:40.963 INFO:tasks.workunit.client.1.smithi152.stdout:8/591: truncate d2/d3/d21/d2d/d31/f7a 867837 0 2024-03-04T16:04:40.969 INFO:tasks.workunit.client.1.smithi152.stdout:9/506: getdents d2/d13/d23/d71 0 2024-03-04T16:04:40.969 INFO:tasks.workunit.client.1.smithi152.stdout:2/538: dwrite da/d24/f94 [0,4194304] 0 2024-03-04T16:04:40.970 INFO:tasks.workunit.client.1.smithi152.stdout:3/538: write d6/d14/d41/d49/f5a [299048,34616] 0 2024-03-04T16:04:40.970 INFO:tasks.workunit.client.1.smithi152.stdout:1/559: creat d19/d20/d2f/d65/fbd x:0 0 0 2024-03-04T16:04:40.970 INFO:tasks.workunit.client.1.smithi152.stdout:2/539: stat da/df 0 2024-03-04T16:04:40.971 INFO:tasks.workunit.client.1.smithi152.stdout:1/560: write d19/d20/d2f/d65/fae [542045,9166] 0 2024-03-04T16:04:40.973 INFO:tasks.workunit.client.1.smithi152.stdout:1/561: dread - d19/d20/d2f/f8a zero size 2024-03-04T16:04:40.981 INFO:tasks.workunit.client.1.smithi152.stdout:3/539: chown d6/d10/d2d/c2f 0 1 2024-03-04T16:04:40.983 INFO:tasks.workunit.client.1.smithi152.stdout:8/592: unlink d2/c3c 0 2024-03-04T16:04:40.985 INFO:tasks.workunit.client.1.smithi152.stdout:1/562: creat d19/d4b/d53/d57/d62/d82/fbe x:0 0 0 2024-03-04T16:04:40.986 INFO:tasks.workunit.client.1.smithi152.stdout:0/452: rmdir d9/d2c/d7d 0 2024-03-04T16:04:40.987 INFO:tasks.workunit.client.1.smithi152.stdout:0/453: fsync d9/d1b/f4f 0 2024-03-04T16:04:40.988 INFO:tasks.workunit.client.1.smithi152.stdout:9/507: symlink d2/d6e/la5 0 2024-03-04T16:04:40.989 INFO:tasks.workunit.client.1.smithi152.stdout:9/508: stat d2/d13/d23/f5c 0 2024-03-04T16:04:40.995 INFO:tasks.workunit.client.1.smithi152.stdout:2/540: dread da/d2c/d3f/f72 [0,4194304] 0 2024-03-04T16:04:40.997 INFO:tasks.workunit.client.1.smithi152.stdout:1/563: dread d19/d4b/f5f [0,4194304] 0 2024-03-04T16:04:40.998 INFO:tasks.workunit.client.1.smithi152.stdout:9/509: dwrite d2/d13/d23/d71/fa1 [0,4194304] 0 2024-03-04T16:04:41.000 INFO:tasks.workunit.client.1.smithi152.stdout:1/564: chown d19/d20/d2f/d6b/l4e 127 1 2024-03-04T16:04:41.001 INFO:tasks.workunit.client.1.smithi152.stdout:4/450: dwrite d8/f10 [0,4194304] 0 2024-03-04T16:04:41.004 INFO:tasks.workunit.client.1.smithi152.stdout:8/593: creat d2/d3/d15/d1c/d84/fbb x:0 0 0 2024-03-04T16:04:41.005 INFO:tasks.workunit.client.1.smithi152.stdout:8/594: dread - d2/d3/d21/d2d/d52/d68/d6e/d7d/f8d zero size 2024-03-04T16:04:41.006 INFO:tasks.workunit.client.1.smithi152.stdout:2/541: symlink da/d2c/lb8 0 2024-03-04T16:04:41.007 INFO:tasks.workunit.client.1.smithi152.stdout:9/510: stat d2/lc 0 2024-03-04T16:04:41.011 INFO:tasks.workunit.client.1.smithi152.stdout:0/454: rename d9/d1b/d3c/d84/d56/d5b/f60 to d9/d1b/d3c/d84/d56/f8b 0 2024-03-04T16:04:41.014 INFO:tasks.workunit.client.1.smithi152.stdout:8/595: dwrite d2/d75/d89/dd/d22/d26/f38 [0,4194304] 0 2024-03-04T16:04:41.016 INFO:tasks.workunit.client.1.smithi152.stdout:8/596: dread - d2/d3/d21/d2d/d52/d68/d6e/d7d/f9b zero size 2024-03-04T16:04:41.028 INFO:tasks.workunit.client.1.smithi152.stdout:0/455: mknod d9/d1b/d3c/d84/c8c 0 2024-03-04T16:04:41.030 INFO:tasks.workunit.client.1.smithi152.stdout:8/597: symlink d2/d3/d21/d2d/d31/lbc 0 2024-03-04T16:04:41.034 INFO:tasks.workunit.client.1.smithi152.stdout:1/565: rmdir d19/d9d 0 2024-03-04T16:04:41.047 INFO:tasks.workunit.client.1.smithi152.stdout:0/456: symlink d9/d1b/d54/l8d 0 2024-03-04T16:04:41.047 INFO:tasks.workunit.client.1.smithi152.stdout:0/457: readlink d9/l2b 0 2024-03-04T16:04:41.047 INFO:tasks.workunit.client.1.smithi152.stdout:0/458: readlink d9/l2b 0 2024-03-04T16:04:41.053 INFO:tasks.workunit.client.1.smithi152.stdout:0/459: creat d9/d1b/d54/d80/f8e x:0 0 0 2024-03-04T16:04:41.055 INFO:tasks.workunit.client.1.smithi152.stdout:9/511: link d2/d63/d69/l8a d2/d13/d35/la6 0 2024-03-04T16:04:41.056 INFO:tasks.workunit.client.1.smithi152.stdout:9/512: readlink d2/d13/d23/d66/l70 0 2024-03-04T16:04:41.058 INFO:tasks.workunit.client.1.smithi152.stdout:9/513: write d2/d13/d23/d71/fa1 [1583899,43448] 0 2024-03-04T16:04:41.059 INFO:tasks.workunit.client.1.smithi152.stdout:9/514: fsync d2/d13/d23/f5b 0 2024-03-04T16:04:41.065 INFO:tasks.workunit.client.1.smithi152.stdout:9/515: mkdir d2/d13/d23/d66/d7d/da7 0 2024-03-04T16:04:41.068 INFO:tasks.workunit.client.1.smithi152.stdout:1/566: dread d19/d20/d2f/d6b/f42 [0,4194304] 0 2024-03-04T16:04:41.071 INFO:tasks.workunit.client.1.smithi152.stdout:9/516: creat d2/d13/d23/d71/d80/fa8 x:0 0 0 2024-03-04T16:04:41.072 INFO:tasks.workunit.client.1.smithi152.stdout:4/451: dwrite f7 [0,4194304] 0 2024-03-04T16:04:41.080 INFO:tasks.workunit.client.1.smithi152.stdout:2/542: dwrite da/f3c [0,4194304] 0 2024-03-04T16:04:41.085 INFO:tasks.workunit.client.1.smithi152.stdout:1/567: read d19/d20/d2f/d6b/f4a [14002,47478] 0 2024-03-04T16:04:41.086 INFO:tasks.workunit.client.1.smithi152.stdout:1/568: write d19/f1d [5388220,44850] 0 2024-03-04T16:04:41.096 INFO:tasks.workunit.client.1.smithi152.stdout:0/460: dread d9/f49 [0,4194304] 0 2024-03-04T16:04:41.098 INFO:tasks.workunit.client.1.smithi152.stdout:1/569: mknod d19/d20/d2f/d6b/d38/da6/cbf 0 2024-03-04T16:04:41.100 INFO:tasks.workunit.client.1.smithi152.stdout:8/598: write d2/d3/d21/d2d/d52/d68/fa2 [84112,75233] 0 2024-03-04T16:04:41.100 INFO:tasks.workunit.client.1.smithi152.stdout:1/570: fdatasync d19/d3b/d6e/d97/fad 0 2024-03-04T16:04:41.104 INFO:tasks.workunit.client.1.smithi152.stdout:2/543: rename da/df/d2e/d61/d8e/cad to da/df/d35/d92/cb9 0 2024-03-04T16:04:41.104 INFO:tasks.workunit.client.1.smithi152.stdout:2/544: chown da/d5f/d85/d8c 2508 1 2024-03-04T16:04:41.105 INFO:tasks.workunit.client.1.smithi152.stdout:4/452: creat d8/d9/f99 x:0 0 0 2024-03-04T16:04:41.107 INFO:tasks.workunit.client.1.smithi152.stdout:2/545: dread da/df/d2e/fa1 [0,4194304] 0 2024-03-04T16:04:41.110 INFO:tasks.workunit.client.1.smithi152.stdout:0/461: creat d9/d1b/d3c/d84/d56/d5b/d7a/f8f x:0 0 0 2024-03-04T16:04:41.110 INFO:tasks.workunit.client.1.smithi152.stdout:9/517: rmdir d2/d13 39 2024-03-04T16:04:41.120 INFO:tasks.workunit.client.1.smithi152.stdout:1/571: write d19/d20/f41 [2767793,125904] 0 2024-03-04T16:04:41.121 INFO:tasks.workunit.client.1.smithi152.stdout:4/453: dread d8/f30 [0,4194304] 0 2024-03-04T16:04:41.121 INFO:tasks.workunit.client.1.smithi152.stdout:1/572: truncate d19/d4b/d53/d5c/f96 1038003 0 2024-03-04T16:04:41.121 INFO:tasks.workunit.client.1.smithi152.stdout:6/605: sync 2024-03-04T16:04:41.121 INFO:tasks.workunit.client.1.smithi152.stdout:5/537: sync 2024-03-04T16:04:41.123 INFO:tasks.workunit.client.1.smithi152.stdout:4/454: chown d8/d9/d24/d26/f2a 1240 1 2024-03-04T16:04:41.123 INFO:tasks.workunit.client.1.smithi152.stdout:1/573: truncate d19/d75/fa7 91335 0 2024-03-04T16:04:41.123 INFO:tasks.workunit.client.1.smithi152.stdout:2/546: rmdir da/df 39 2024-03-04T16:04:41.126 INFO:tasks.workunit.client.1.smithi152.stdout:8/599: creat d2/d75/d89/dd/d22/d83/d96/da6/fbd x:0 0 0 2024-03-04T16:04:41.131 INFO:tasks.workunit.client.1.smithi152.stdout:4/455: dread d8/f10 [0,4194304] 0 2024-03-04T16:04:41.131 INFO:tasks.workunit.client.1.smithi152.stdout:5/538: dwrite d6/d8/d1f/d3d/d6f/d70/f71 [0,4194304] 0 2024-03-04T16:04:41.132 INFO:tasks.workunit.client.1.smithi152.stdout:5/539: chown d6/d8/d1f/d3d/f78 63 1 2024-03-04T16:04:41.146 INFO:tasks.workunit.client.1.smithi152.stdout:2/547: dread da/f56 [0,4194304] 0 2024-03-04T16:04:41.149 INFO:tasks.workunit.client.1.smithi152.stdout:7/458: sync 2024-03-04T16:04:41.150 INFO:tasks.workunit.client.1.smithi152.stdout:3/540: sync 2024-03-04T16:04:41.160 INFO:tasks.workunit.client.1.smithi152.stdout:6/606: creat d8/d42/fcf x:0 0 0 2024-03-04T16:04:41.163 INFO:tasks.workunit.client.1.smithi152.stdout:1/574: fdatasync d19/d20/d2f/d6b/f40 0 2024-03-04T16:04:41.168 INFO:tasks.workunit.client.1.smithi152.stdout:8/600: dread d2/d3/d21/d2d/d52/d68/f72 [0,4194304] 0 2024-03-04T16:04:41.168 INFO:tasks.workunit.client.1.smithi152.stdout:5/540: stat d6/d8/c35 0 2024-03-04T16:04:41.172 INFO:tasks.workunit.client.1.smithi152.stdout:2/548: creat da/d2c/fba x:0 0 0 2024-03-04T16:04:41.175 INFO:tasks.workunit.client.1.smithi152.stdout:6/607: mkdir d8/d52/d58/dd0 0 2024-03-04T16:04:41.179 INFO:tasks.workunit.client.1.smithi152.stdout:1/575: truncate d19/d20/f9b 468711 0 2024-03-04T16:04:41.180 INFO:tasks.workunit.client.1.smithi152.stdout:4/456: unlink d8/d3d/f68 0 2024-03-04T16:04:41.180 INFO:tasks.workunit.client.1.smithi152.stdout:4/457: fdatasync d8/d78/f95 0 2024-03-04T16:04:41.182 INFO:tasks.workunit.client.1.smithi152.stdout:8/601: symlink d2/d75/d89/lbe 0 2024-03-04T16:04:41.184 INFO:tasks.workunit.client.1.smithi152.stdout:5/541: mkdir d6/d8/d1f/d3d/d6f/dbb 0 2024-03-04T16:04:41.186 INFO:tasks.workunit.client.1.smithi152.stdout:3/541: getdents d6/d14/d58/dad 0 2024-03-04T16:04:41.187 INFO:tasks.workunit.client.1.smithi152.stdout:3/542: stat d6/d60/d8e 0 2024-03-04T16:04:41.189 INFO:tasks.workunit.client.1.smithi152.stdout:0/462: getdents d9/d2c/d2e 0 2024-03-04T16:04:41.190 INFO:tasks.workunit.client.1.smithi152.stdout:6/608: mknod d8/d52/d58/cd1 0 2024-03-04T16:04:41.194 INFO:tasks.workunit.client.1.smithi152.stdout:3/543: dwrite d6/d14/d58/f7e [0,4194304] 0 2024-03-04T16:04:41.201 INFO:tasks.workunit.client.1.smithi152.stdout:7/459: mkdir d14/d2b/d86 0 2024-03-04T16:04:41.204 INFO:tasks.workunit.client.1.smithi152.stdout:0/463: symlink d9/d1b/d3c/d84/d56/d5b/d7a/l90 0 2024-03-04T16:04:41.206 INFO:tasks.workunit.client.1.smithi152.stdout:6/609: creat d8/d8f/db2/fd2 x:0 0 0 2024-03-04T16:04:41.207 INFO:tasks.workunit.client.1.smithi152.stdout:4/458: truncate d8/d9/d24/f40 618547 0 2024-03-04T16:04:41.210 INFO:tasks.workunit.client.1.smithi152.stdout:3/544: mkdir d6/d14/d93/db7 0 2024-03-04T16:04:41.210 INFO:tasks.workunit.client.1.smithi152.stdout:4/459: fdatasync d8/d9/d24/d26/d42/f4b 0 2024-03-04T16:04:41.212 INFO:tasks.workunit.client.1.smithi152.stdout:2/549: symlink da/df/d2e/d51/d8d/d9a/lbb 0 2024-03-04T16:04:41.213 INFO:tasks.workunit.client.1.smithi152.stdout:8/602: dwrite d2/d3/d21/d2d/d52/d68/d6e/d7d/f8d [0,4194304] 0 2024-03-04T16:04:41.217 INFO:tasks.workunit.client.1.smithi152.stdout:8/603: read d2/d75/d89/dd/d22/fa0 [3866648,110053] 0 2024-03-04T16:04:41.218 INFO:tasks.workunit.client.1.smithi152.stdout:4/460: symlink d8/d78/l9a 0 2024-03-04T16:04:41.219 INFO:tasks.workunit.client.1.smithi152.stdout:5/542: dwrite d6/d8/d1f/d3d/f85 [0,4194304] 0 2024-03-04T16:04:41.222 INFO:tasks.workunit.client.1.smithi152.stdout:9/518: rename d2/d63/d69/c77 to d2/d13/ca9 0 2024-03-04T16:04:41.224 INFO:tasks.workunit.client.1.smithi152.stdout:1/576: rename d19/d20/d2f to d19/d20/d2f/d6b/d38/dc0 22 2024-03-04T16:04:41.225 INFO:tasks.workunit.client.1.smithi152.stdout:8/604: read d2/d3/d21/d2d/f2c [3570415,34721] 0 2024-03-04T16:04:41.225 INFO:tasks.workunit.client.1.smithi152.stdout:5/543: rename d6/d8/d5a/d8b/dad to d6/d8/d5a/d8b/dad/dbc 22 2024-03-04T16:04:41.225 INFO:tasks.workunit.client.1.smithi152.stdout:7/460: dwrite d14/d21/f30 [0,4194304] 0 2024-03-04T16:04:41.226 INFO:tasks.workunit.client.1.smithi152.stdout:1/577: readlink d19/d20/d2f/l33 0 2024-03-04T16:04:41.229 INFO:tasks.workunit.client.1.smithi152.stdout:2/550: readlink da/df/d2e/d61/d8e/la4 0 2024-03-04T16:04:41.229 INFO:tasks.workunit.client.1.smithi152.stdout:7/461: dread - d14/d21/f6b zero size 2024-03-04T16:04:41.231 INFO:tasks.workunit.client.1.smithi152.stdout:2/551: chown da/df/d2e/f38 83733 1 2024-03-04T16:04:41.232 INFO:tasks.workunit.client.1.smithi152.stdout:5/544: dread d6/d8/d5a/d64/f96 [0,4194304] 0 2024-03-04T16:04:41.233 INFO:tasks.workunit.client.1.smithi152.stdout:5/545: readlink d6/d8/d1f/l36 0 2024-03-04T16:04:41.238 INFO:tasks.workunit.client.1.smithi152.stdout:0/464: link d9/l5a d9/d1b/d54/d80/l91 0 2024-03-04T16:04:41.239 INFO:tasks.workunit.client.1.smithi152.stdout:0/465: write d9/f36 [126074,8735] 0 2024-03-04T16:04:41.241 INFO:tasks.workunit.client.1.smithi152.stdout:9/519: mknod d2/d13/d23/d71/d80/caa 0 2024-03-04T16:04:41.243 INFO:tasks.workunit.client.1.smithi152.stdout:9/520: stat d2/d13/d35/c3d 0 2024-03-04T16:04:41.245 INFO:tasks.workunit.client.1.smithi152.stdout:8/605: unlink d2/d3/d21/f1f 0 2024-03-04T16:04:41.246 INFO:tasks.workunit.client.1.smithi152.stdout:0/466: dwrite d9/d1a/f79 [0,4194304] 0 2024-03-04T16:04:41.249 INFO:tasks.workunit.client.1.smithi152.stdout:1/578: rmdir d19/d3b/d6e/d97 39 2024-03-04T16:04:41.251 INFO:tasks.workunit.client.1.smithi152.stdout:6/610: link d8/d10/l13 d8/d92/db1/d87/dca/ld3 0 2024-03-04T16:04:41.253 INFO:tasks.workunit.client.1.smithi152.stdout:6/611: write d8/f28 [3540348,98765] 0 2024-03-04T16:04:41.254 INFO:tasks.workunit.client.1.smithi152.stdout:6/612: fdatasync d8/d23/d98/d9f/f77 0 2024-03-04T16:04:41.256 INFO:tasks.workunit.client.1.smithi152.stdout:9/521: unlink d2/d13/d23/d66/d7d/c8f 0 2024-03-04T16:04:41.265 INFO:tasks.workunit.client.1.smithi152.stdout:8/606: dwrite d2/d3/d15/f4f [0,4194304] 0 2024-03-04T16:04:41.271 INFO:tasks.workunit.client.1.smithi152.stdout:8/607: truncate d2/d75/f8f 21758 0 2024-03-04T16:04:41.276 INFO:tasks.workunit.client.1.smithi152.stdout:5/546: dwrite d6/d50/f83 [0,4194304] 0 2024-03-04T16:04:41.279 INFO:tasks.workunit.client.1.smithi152.stdout:9/522: write d2/d13/d23/fa0 [749615,82027] 0 2024-03-04T16:04:41.281 INFO:tasks.workunit.client.1.smithi152.stdout:7/462: dwrite d14/f78 [0,4194304] 0 2024-03-04T16:04:41.286 INFO:tasks.workunit.client.1.smithi152.stdout:4/461: getdents d8/d4e/d86 0 2024-03-04T16:04:41.286 INFO:tasks.workunit.client.1.smithi152.stdout:1/579: dwrite d19/d4b/d53/d5c/f71 [0,4194304] 0 2024-03-04T16:04:41.291 INFO:tasks.workunit.client.1.smithi152.stdout:8/608: dread d2/d3/d21/d2d/d31/f35 [0,4194304] 0 2024-03-04T16:04:41.295 INFO:tasks.workunit.client.1.smithi152.stdout:2/552: getdents da/d24/d47/d73 0 2024-03-04T16:04:41.298 INFO:tasks.workunit.client.1.smithi152.stdout:2/553: fdatasync da/df/d58/d6e/f89 0 2024-03-04T16:04:41.299 INFO:tasks.workunit.client.1.smithi152.stdout:4/462: creat d8/d4e/f9b x:0 0 0 2024-03-04T16:04:41.300 INFO:tasks.workunit.client.1.smithi152.stdout:6/613: dwrite d8/d42/f4a [0,4194304] 0 2024-03-04T16:04:41.300 INFO:tasks.workunit.client.1.smithi152.stdout:2/554: truncate da/f23 5051756 0 2024-03-04T16:04:41.301 INFO:tasks.workunit.client.1.smithi152.stdout:2/555: readlink da/d24/d47/d4f/l6c 0 2024-03-04T16:04:41.302 INFO:tasks.workunit.client.1.smithi152.stdout:5/547: mknod d6/d8/d1f/d3d/d6f/db8/cbd 0 2024-03-04T16:04:41.302 INFO:tasks.workunit.client.1.smithi152.stdout:0/467: getdents d9/d1b/d54/d80 0 2024-03-04T16:04:41.306 INFO:tasks.workunit.client.1.smithi152.stdout:1/580: write d19/f28 [650116,55502] 0 2024-03-04T16:04:41.309 INFO:tasks.workunit.client.1.smithi152.stdout:9/523: truncate d2/d13/d23/f4c 1646425 0 2024-03-04T16:04:41.312 INFO:tasks.workunit.client.1.smithi152.stdout:6/614: chown d8/c6e 874822269 1 2024-03-04T16:04:41.318 INFO:tasks.workunit.client.1.smithi152.stdout:2/556: mkdir da/d5f/dbc 0 2024-03-04T16:04:41.318 INFO:tasks.workunit.client.1.smithi152.stdout:2/557: fdatasync da/df/d35/f4a 0 2024-03-04T16:04:41.322 INFO:tasks.workunit.client.1.smithi152.stdout:7/463: dread d14/f6a [0,4194304] 0 2024-03-04T16:04:41.322 INFO:tasks.workunit.client.1.smithi152.stdout:5/548: write d6/d8/d5a/f87 [597971,104777] 0 2024-03-04T16:04:41.324 INFO:tasks.workunit.client.1.smithi152.stdout:9/524: creat d2/d13/d23/d71/da3/fab x:0 0 0 2024-03-04T16:04:41.325 INFO:tasks.workunit.client.1.smithi152.stdout:7/464: chown d14/d21/f49 14867 1 2024-03-04T16:04:41.325 INFO:tasks.workunit.client.1.smithi152.stdout:0/468: truncate d9/f49 3658103 0 2024-03-04T16:04:41.326 INFO:tasks.workunit.client.1.smithi152.stdout:5/549: write d6/d8/d1f/d3d/f85 [4288221,7348] 0 2024-03-04T16:04:41.327 INFO:tasks.workunit.client.1.smithi152.stdout:9/525: dread - d2/d63/f6a zero size 2024-03-04T16:04:41.327 INFO:tasks.workunit.client.1.smithi152.stdout:6/615: dread d8/d23/d3a/f50 [0,4194304] 0 2024-03-04T16:04:41.329 INFO:tasks.workunit.client.1.smithi152.stdout:1/581: truncate d19/f1d 701634 0 2024-03-04T16:04:41.330 INFO:tasks.workunit.client.1.smithi152.stdout:8/609: dwrite d2/d3/d21/d2d/f79 [0,4194304] 0 2024-03-04T16:04:41.336 INFO:tasks.workunit.client.1.smithi152.stdout:4/463: getdents d8/d2f 0 2024-03-04T16:04:41.340 INFO:tasks.workunit.client.1.smithi152.stdout:0/469: rmdir d9/d1b/d54/d69/d6c 39 2024-03-04T16:04:41.341 INFO:tasks.workunit.client.1.smithi152.stdout:8/610: dread d2/d3/d21/d2d/d52/d68/d6e/d7d/f8c [0,4194304] 0 2024-03-04T16:04:41.344 INFO:tasks.workunit.client.1.smithi152.stdout:5/550: rmdir d6/d8/d1f/d3d/d61 39 2024-03-04T16:04:41.344 INFO:tasks.workunit.client.1.smithi152.stdout:5/551: read - d6/d8/fa3 zero size 2024-03-04T16:04:41.349 INFO:tasks.workunit.client.1.smithi152.stdout:9/526: mknod d2/cac 0 2024-03-04T16:04:41.351 INFO:tasks.workunit.client.1.smithi152.stdout:9/527: stat d2/d13/d38/d79/f89 0 2024-03-04T16:04:41.353 INFO:tasks.workunit.client.1.smithi152.stdout:5/552: rmdir d6/d8/d5a 39 2024-03-04T16:04:41.354 INFO:tasks.workunit.client.1.smithi152.stdout:1/582: dwrite d19/d20/d2f/fa1 [0,4194304] 0 2024-03-04T16:04:41.355 INFO:tasks.workunit.client.1.smithi152.stdout:5/553: write d6/d8/f30 [4118150,4976] 0 2024-03-04T16:04:41.358 INFO:tasks.workunit.client.1.smithi152.stdout:0/470: mkdir d9/d92 0 2024-03-04T16:04:41.360 INFO:tasks.workunit.client.1.smithi152.stdout:4/464: dwrite d8/d9/d24/d26/f2a [4194304,4194304] 0 2024-03-04T16:04:41.363 INFO:tasks.workunit.client.1.smithi152.stdout:4/465: stat d8/d2f/d6f/d84/l93 0 2024-03-04T16:04:41.364 INFO:tasks.workunit.client.1.smithi152.stdout:9/528: write d2/f8d [3764847,119051] 0 2024-03-04T16:04:41.366 INFO:tasks.workunit.client.1.smithi152.stdout:7/465: dwrite d14/d21/d47/d36/f82 [0,4194304] 0 2024-03-04T16:04:41.367 INFO:tasks.workunit.client.1.smithi152.stdout:1/583: creat d19/d4b/d53/fc1 x:0 0 0 2024-03-04T16:04:41.393 INFO:tasks.workunit.client.1.smithi152.stdout:1/584: write d19/d20/d2f/f76 [506806,5583] 0 2024-03-04T16:04:41.393 INFO:tasks.workunit.client.1.smithi152.stdout:6/616: getdents d8/d92/d59 0 2024-03-04T16:04:41.394 INFO:tasks.workunit.client.1.smithi152.stdout:0/471: creat d9/d1b/d3c/d84/d56/d5b/f93 x:0 0 0 2024-03-04T16:04:41.394 INFO:tasks.workunit.client.1.smithi152.stdout:4/466: creat d8/d2f/f9c x:0 0 0 2024-03-04T16:04:41.396 INFO:tasks.workunit.client.1.smithi152.stdout:5/554: dread d6/d8/d1f/d3d/f4f [0,4194304] 0 2024-03-04T16:04:41.398 INFO:tasks.workunit.client.1.smithi152.stdout:8/611: getdents d2/d75 0 2024-03-04T16:04:41.400 INFO:tasks.workunit.client.1.smithi152.stdout:0/472: mkdir d9/d2c/d2e/d94 0 2024-03-04T16:04:41.400 INFO:tasks.workunit.client.1.smithi152.stdout:1/585: write d19/d75/fa0 [4467801,47072] 0 2024-03-04T16:04:41.403 INFO:tasks.workunit.client.1.smithi152.stdout:1/586: stat l11 0 2024-03-04T16:04:41.403 INFO:tasks.workunit.client.1.smithi152.stdout:6/617: creat d8/da9/fd4 x:0 0 0 2024-03-04T16:04:41.404 INFO:tasks.workunit.client.1.smithi152.stdout:5/555: mkdir d6/d8/dbe 0 2024-03-04T16:04:41.406 INFO:tasks.workunit.client.1.smithi152.stdout:8/612: write d2/d3/d15/d1c/f56 [3927991,91508] 0 2024-03-04T16:04:41.410 INFO:tasks.workunit.client.1.smithi152.stdout:7/466: dwrite d14/d2b/f63 [4194304,4194304] 0 2024-03-04T16:04:41.411 INFO:tasks.workunit.client.1.smithi152.stdout:4/467: dwrite d8/d2f/f9c [0,4194304] 0 2024-03-04T16:04:41.412 INFO:tasks.workunit.client.1.smithi152.stdout:0/473: mkdir d9/d1b/d54/d69/d95 0 2024-03-04T16:04:41.416 INFO:tasks.workunit.client.1.smithi152.stdout:1/587: creat d19/d4b/d53/d57/d62/d82/fc2 x:0 0 0 2024-03-04T16:04:41.417 INFO:tasks.workunit.client.1.smithi152.stdout:0/474: truncate d9/d2c/d2e/f4a 782647 0 2024-03-04T16:04:41.420 INFO:tasks.workunit.client.1.smithi152.stdout:9/529: dread d2/fb [0,4194304] 0 2024-03-04T16:04:41.426 INFO:tasks.workunit.client.1.smithi152.stdout:4/468: chown d8/d3d/c51 125587 1 2024-03-04T16:04:41.427 INFO:tasks.workunit.client.1.smithi152.stdout:7/467: truncate d14/d69/f81 618977 0 2024-03-04T16:04:41.427 INFO:tasks.workunit.client.1.smithi152.stdout:9/530: fsync d2/d13/f1a 0 2024-03-04T16:04:41.433 INFO:tasks.workunit.client.1.smithi152.stdout:0/475: dwrite d9/d1b/f2a [0,4194304] 0 2024-03-04T16:04:41.433 INFO:tasks.workunit.client.1.smithi152.stdout:4/469: chown d8/c2d 18101 1 2024-03-04T16:04:41.433 INFO:tasks.workunit.client.1.smithi152.stdout:9/531: chown d2/d13/d23/d66/d7d/l7e 561 1 2024-03-04T16:04:41.434 INFO:tasks.workunit.client.1.smithi152.stdout:1/588: dwrite d19/d20/d2f/d6b/d38/d63/d9a/fa2 [0,4194304] 0 2024-03-04T16:04:41.436 INFO:tasks.workunit.client.1.smithi152.stdout:5/556: dread d6/d8/d1f/d3d/d6f/f95 [0,4194304] 0 2024-03-04T16:04:41.438 INFO:tasks.workunit.client.1.smithi152.stdout:6/618: write d8/d23/d3a/f70 [1540727,92232] 0 2024-03-04T16:04:41.440 INFO:tasks.workunit.client.1.smithi152.stdout:0/476: write d9/d1b/d3c/d84/d56/f63 [724359,26066] 0 2024-03-04T16:04:41.441 INFO:tasks.workunit.client.1.smithi152.stdout:8/613: dread d2/d75/d89/f28 [0,4194304] 0 2024-03-04T16:04:41.442 INFO:tasks.workunit.client.1.smithi152.stdout:0/477: readlink d9/d1b/d54/l64 0 2024-03-04T16:04:41.443 INFO:tasks.workunit.client.1.smithi152.stdout:0/478: read - d9/d1b/f4f zero size 2024-03-04T16:04:41.443 INFO:tasks.workunit.client.1.smithi152.stdout:1/589: fdatasync d19/f1c 0 2024-03-04T16:04:41.446 INFO:tasks.workunit.client.1.smithi152.stdout:6/619: creat d8/d92/db1/fd5 x:0 0 0 2024-03-04T16:04:41.450 INFO:tasks.workunit.client.1.smithi152.stdout:0/479: creat d9/d1b/d54/d80/f96 x:0 0 0 2024-03-04T16:04:41.451 INFO:tasks.workunit.client.1.smithi152.stdout:4/470: rename d8/l1f to d8/d3d/l9d 0 2024-03-04T16:04:41.455 INFO:tasks.workunit.client.1.smithi152.stdout:6/620: creat d8/d23/d3a/d6d/fd6 x:0 0 0 2024-03-04T16:04:41.456 INFO:tasks.workunit.client.1.smithi152.stdout:8/614: symlink d2/lbf 0 2024-03-04T16:04:41.456 INFO:tasks.workunit.client.1.smithi152.stdout:0/480: dwrite d9/f26 [0,4194304] 0 2024-03-04T16:04:41.460 INFO:tasks.workunit.client.1.smithi152.stdout:7/468: getdents d14/d21 0 2024-03-04T16:04:41.463 INFO:tasks.workunit.client.1.smithi152.stdout:9/532: getdents d2/d63 0 2024-03-04T16:04:41.465 INFO:tasks.workunit.client.1.smithi152.stdout:7/469: chown d14/c2e 59378591 1 2024-03-04T16:04:41.465 INFO:tasks.workunit.client.1.smithi152.stdout:4/471: creat d8/d9/f9e x:0 0 0 2024-03-04T16:04:41.467 INFO:tasks.workunit.client.1.smithi152.stdout:0/481: link d9/f45 d9/d1b/d54/d80/f97 0 2024-03-04T16:04:41.469 INFO:tasks.workunit.client.1.smithi152.stdout:9/533: mkdir d2/d13/dad 0 2024-03-04T16:04:41.470 INFO:tasks.workunit.client.1.smithi152.stdout:7/470: symlink d14/d21/d47/d40/l87 0 2024-03-04T16:04:41.471 INFO:tasks.workunit.client.1.smithi152.stdout:8/615: dwrite d2/d75/d89/dd/d22/f3b [0,4194304] 0 2024-03-04T16:04:41.476 INFO:tasks.workunit.client.1.smithi152.stdout:4/472: dwrite d8/d3d/f5f [0,4194304] 0 2024-03-04T16:04:41.483 INFO:tasks.workunit.client.1.smithi152.stdout:6/621: dread d8/d23/f55 [0,4194304] 0 2024-03-04T16:04:41.485 INFO:tasks.workunit.client.1.smithi152.stdout:8/616: dwrite d2/d75/d89/dd/d22/d26/f61 [0,4194304] 0 2024-03-04T16:04:41.486 INFO:tasks.workunit.client.1.smithi152.stdout:6/622: dread - d8/d23/d3a/d6d/fd6 zero size 2024-03-04T16:04:41.487 INFO:tasks.workunit.client.1.smithi152.stdout:6/623: fdatasync d8/da9/fd4 0 2024-03-04T16:04:41.496 INFO:tasks.workunit.client.1.smithi152.stdout:9/534: link d2/d13/d38/l5e d2/d13/d23/d71/lae 0 2024-03-04T16:04:41.497 INFO:tasks.workunit.client.1.smithi152.stdout:8/617: mkdir d2/d3/d21/dc0 0 2024-03-04T16:04:41.498 INFO:tasks.workunit.client.1.smithi152.stdout:8/618: fsync d2/d75/d89/dd/d22/d26/f38 0 2024-03-04T16:04:41.500 INFO:tasks.workunit.client.1.smithi152.stdout:4/473: getdents d8/d4e 0 2024-03-04T16:04:41.504 INFO:tasks.workunit.client.1.smithi152.stdout:9/535: mknod d2/d6e/caf 0 2024-03-04T16:04:41.504 INFO:tasks.workunit.client.1.smithi152.stdout:4/474: read d8/d4e/f57 [988688,126202] 0 2024-03-04T16:04:41.505 INFO:tasks.workunit.client.1.smithi152.stdout:9/536: truncate d2/d6e/f7a 838099 0 2024-03-04T16:04:41.507 INFO:tasks.workunit.client.1.smithi152.stdout:4/475: symlink d8/d78/l9f 0 2024-03-04T16:04:41.508 INFO:tasks.workunit.client.1.smithi152.stdout:4/476: fdatasync d8/d2f/f9c 0 2024-03-04T16:04:41.515 INFO:tasks.workunit.client.1.smithi152.stdout:6/624: dread d8/d10/f16 [0,4194304] 0 2024-03-04T16:04:41.515 INFO:tasks.workunit.client.1.smithi152.stdout:4/477: mkdir d8/d9/d24/da0 0 2024-03-04T16:04:41.516 INFO:tasks.workunit.client.1.smithi152.stdout:6/625: chown d8/la 27 1 2024-03-04T16:04:41.519 INFO:tasks.workunit.client.1.smithi152.stdout:6/626: mkdir d8/d10/d53/dd7 0 2024-03-04T16:04:41.519 INFO:tasks.workunit.client.1.smithi152.stdout:9/537: write d2/d13/f18 [247997,56084] 0 2024-03-04T16:04:41.520 INFO:tasks.workunit.client.1.smithi152.stdout:8/619: dread d2/d3/d21/f1b [0,4194304] 0 2024-03-04T16:04:41.522 INFO:tasks.workunit.client.1.smithi152.stdout:8/620: chown d2/cb0 189 1 2024-03-04T16:04:41.522 INFO:tasks.workunit.client.1.smithi152.stdout:6/627: symlink d8/d23/ld8 0 2024-03-04T16:04:41.523 INFO:tasks.workunit.client.1.smithi152.stdout:6/628: stat d8/d23/d3a/d8d/la5 0 2024-03-04T16:04:41.523 INFO:tasks.workunit.client.1.smithi152.stdout:9/538: mknod d2/d13/d35/cb0 0 2024-03-04T16:04:41.524 INFO:tasks.workunit.client.1.smithi152.stdout:6/629: chown d8/d23/l7b 0 1 2024-03-04T16:04:41.526 INFO:tasks.workunit.client.1.smithi152.stdout:9/539: mkdir d2/d13/d23/d42/db1 0 2024-03-04T16:04:41.527 INFO:tasks.workunit.client.1.smithi152.stdout:6/630: creat d8/d23/d93/fd9 x:0 0 0 2024-03-04T16:04:41.527 INFO:tasks.workunit.client.1.smithi152.stdout:6/631: chown d8/d23/c5a 1 1 2024-03-04T16:04:41.529 INFO:tasks.workunit.client.1.smithi152.stdout:6/632: symlink d8/d23/lda 0 2024-03-04T16:04:41.531 INFO:tasks.workunit.client.1.smithi152.stdout:8/621: creat d2/d3/d15/fc1 x:0 0 0 2024-03-04T16:04:41.534 INFO:tasks.workunit.client.1.smithi152.stdout:6/633: unlink d8/d42/f74 0 2024-03-04T16:04:41.534 INFO:tasks.workunit.client.1.smithi152.stdout:4/478: dread d8/f16 [0,4194304] 0 2024-03-04T16:04:41.535 INFO:tasks.workunit.client.1.smithi152.stdout:9/540: dwrite d2/d13/d23/d71/da3/fab [0,4194304] 0 2024-03-04T16:04:41.540 INFO:tasks.workunit.client.1.smithi152.stdout:4/479: creat d8/d3d/fa1 x:0 0 0 2024-03-04T16:04:41.546 INFO:tasks.workunit.client.1.smithi152.stdout:9/541: creat d2/d13/d23/d42/db1/fb2 x:0 0 0 2024-03-04T16:04:41.552 INFO:tasks.workunit.client.1.smithi152.stdout:4/480: mkdir d8/d9/da2 0 2024-03-04T16:04:41.554 INFO:tasks.workunit.client.1.smithi152.stdout:9/542: symlink d2/d13/d23/d42/lb3 0 2024-03-04T16:04:41.556 INFO:tasks.workunit.client.1.smithi152.stdout:4/481: fsync d8/d2f/f47 0 2024-03-04T16:04:41.557 INFO:tasks.workunit.client.1.smithi152.stdout:6/634: dread d8/d4b/f54 [0,4194304] 0 2024-03-04T16:04:41.560 INFO:tasks.workunit.client.1.smithi152.stdout:4/482: symlink d8/d2f/d7c/d87/la3 0 2024-03-04T16:04:41.561 INFO:tasks.workunit.client.1.smithi152.stdout:6/635: rmdir d8/d8f 39 2024-03-04T16:04:41.562 INFO:tasks.workunit.client.1.smithi152.stdout:6/636: stat d8/d23/d3a/d8d/d96/l9e 0 2024-03-04T16:04:41.563 INFO:tasks.workunit.client.1.smithi152.stdout:6/637: creat d8/d2a/fdb x:0 0 0 2024-03-04T16:04:41.564 INFO:tasks.workunit.client.1.smithi152.stdout:8/622: dread d2/d3/d15/f6a [0,4194304] 0 2024-03-04T16:04:41.565 INFO:tasks.workunit.client.1.smithi152.stdout:6/638: truncate d8/d42/fcf 347318 0 2024-03-04T16:04:41.566 INFO:tasks.workunit.client.1.smithi152.stdout:8/623: creat d2/d7f/fc2 x:0 0 0 2024-03-04T16:04:41.567 INFO:tasks.workunit.client.1.smithi152.stdout:6/639: creat d8/d92/db1/d87/fdc x:0 0 0 2024-03-04T16:04:41.572 INFO:tasks.workunit.client.1.smithi152.stdout:3/545: sync 2024-03-04T16:04:41.574 INFO:tasks.workunit.client.1.smithi152.stdout:3/546: read d6/d10/d2d/f34 [2839581,26899] 0 2024-03-04T16:04:41.576 INFO:tasks.workunit.client.1.smithi152.stdout:4/483: dread d8/d9/d24/f66 [0,4194304] 0 2024-03-04T16:04:41.576 INFO:tasks.workunit.client.1.smithi152.stdout:3/547: truncate d6/db1/fb3 1000994 0 2024-03-04T16:04:41.576 INFO:tasks.workunit.client.1.smithi152.stdout:6/640: dread d8/f22 [0,4194304] 0 2024-03-04T16:04:41.581 INFO:tasks.workunit.client.1.smithi152.stdout:3/548: creat d6/d60/fb8 x:0 0 0 2024-03-04T16:04:41.581 INFO:tasks.workunit.client.1.smithi152.stdout:6/641: symlink d8/d92/d59/d7d/d91/ldd 0 2024-03-04T16:04:41.584 INFO:tasks.workunit.client.1.smithi152.stdout:8/624: dwrite d2/d75/d89/f45 [0,4194304] 0 2024-03-04T16:04:41.592 INFO:tasks.workunit.client.1.smithi152.stdout:6/642: rename d8/d23/d98/d9f to d8/d42/dde 0 2024-03-04T16:04:41.593 INFO:tasks.workunit.client.1.smithi152.stdout:3/549: write d6/d53/f5b [171450,53844] 0 2024-03-04T16:04:41.594 INFO:tasks.workunit.client.1.smithi152.stdout:4/484: dwrite d8/d2f/f33 [0,4194304] 0 2024-03-04T16:04:41.597 INFO:tasks.workunit.client.1.smithi152.stdout:4/485: write d8/d9/f50 [581548,72142] 0 2024-03-04T16:04:41.604 INFO:tasks.workunit.client.1.smithi152.stdout:8/625: creat d2/d3/d21/d2d/d52/d68/fc3 x:0 0 0 2024-03-04T16:04:41.604 INFO:tasks.workunit.client.1.smithi152.stdout:3/550: dread d6/d14/d58/f7e [0,4194304] 0 2024-03-04T16:04:41.605 INFO:tasks.workunit.client.1.smithi152.stdout:3/551: chown d6/d10/d2d/f40 6593 1 2024-03-04T16:04:41.605 INFO:tasks.workunit.client.1.smithi152.stdout:3/552: readlink d6/d10/l3d 0 2024-03-04T16:04:41.699 INFO:tasks.workunit.client.1.smithi152.stdout:6/643: rename d8/d23/d3a/d71/c86 to d8/d2a/cdf 0 2024-03-04T16:04:41.701 INFO:tasks.workunit.client.1.smithi152.stdout:3/553: truncate d6/d10/d2d/f34 1665608 0 2024-03-04T16:04:41.703 INFO:tasks.workunit.client.1.smithi152.stdout:6/644: write d8/d92/f5e [4750057,67697] 0 2024-03-04T16:04:41.704 INFO:tasks.workunit.client.1.smithi152.stdout:3/554: dread - d6/d14/f51 zero size 2024-03-04T16:04:41.705 INFO:tasks.workunit.client.1.smithi152.stdout:6/645: write d8/d42/dde/f77 [6465026,79441] 0 2024-03-04T16:04:41.708 INFO:tasks.workunit.client.1.smithi152.stdout:8/626: dread d2/d75/d89/dd/f2f [0,4194304] 0 2024-03-04T16:04:41.710 INFO:tasks.workunit.client.1.smithi152.stdout:6/646: dread - d8/d8f/db2/fd2 zero size 2024-03-04T16:04:41.711 INFO:tasks.workunit.client.1.smithi152.stdout:3/555: truncate d6/f42 2877082 0 2024-03-04T16:04:41.716 INFO:tasks.workunit.client.1.smithi152.stdout:8/627: dwrite d2/d3/d15/f4f [0,4194304] 0 2024-03-04T16:04:41.719 INFO:tasks.workunit.client.1.smithi152.stdout:8/628: write d2/d75/d89/dd/f93 [1002886,8340] 0 2024-03-04T16:04:41.722 INFO:tasks.workunit.client.1.smithi152.stdout:8/629: dread - d2/d3/d21/d2d/d52/d68/fae zero size 2024-03-04T16:04:41.722 INFO:tasks.workunit.client.1.smithi152.stdout:6/647: dwrite d8/d23/d3a/d8d/d96/fb9 [0,4194304] 0 2024-03-04T16:04:41.724 INFO:tasks.workunit.client.1.smithi152.stdout:8/630: dread - d2/d75/d89/dd/d22/d6c/f70 zero size 2024-03-04T16:04:41.728 INFO:tasks.workunit.client.1.smithi152.stdout:6/648: mkdir d8/d92/de0 0 2024-03-04T16:04:41.733 INFO:tasks.workunit.client.1.smithi152.stdout:8/631: dwrite d2/d3/d15/d1c/d84/d87/d5d/f44 [0,4194304] 0 2024-03-04T16:04:41.733 INFO:tasks.workunit.client.1.smithi152.stdout:2/558: sync 2024-03-04T16:04:41.738 INFO:tasks.workunit.client.1.smithi152.stdout:6/649: dread d8/d23/d3a/d8d/d96/fb9 [0,4194304] 0 2024-03-04T16:04:41.740 INFO:tasks.workunit.client.1.smithi152.stdout:8/632: unlink d2/d3/d15/d1c/f59 0 2024-03-04T16:04:41.741 INFO:tasks.workunit.client.1.smithi152.stdout:2/559: truncate da/df/d2e/d61/f84 443089 0 2024-03-04T16:04:41.745 INFO:tasks.workunit.client.1.smithi152.stdout:6/650: dwrite d8/d42/fc3 [0,4194304] 0 2024-03-04T16:04:41.755 INFO:tasks.workunit.client.1.smithi152.stdout:2/560: symlink da/d24/d47/d73/lbd 0 2024-03-04T16:04:41.756 INFO:tasks.workunit.client.1.smithi152.stdout:3/556: dread d6/d10/d2d/f64 [0,4194304] 0 2024-03-04T16:04:41.762 INFO:tasks.workunit.client.1.smithi152.stdout:3/557: rename d6/c57 to d6/d10/d2d/d6e/cb9 0 2024-03-04T16:04:41.766 INFO:tasks.workunit.client.1.smithi152.stdout:8/633: link d2/d75/d89/f90 d2/d3/d21/dc0/fc4 0 2024-03-04T16:04:41.768 INFO:tasks.workunit.client.1.smithi152.stdout:3/558: creat d6/d14/d41/d49/d84/fba x:0 0 0 2024-03-04T16:04:41.770 INFO:tasks.workunit.client.1.smithi152.stdout:3/559: chown d6/d14/d41/d49/d84/fab 3524953 1 2024-03-04T16:04:41.772 INFO:tasks.workunit.client.1.smithi152.stdout:2/561: getdents da/df/d2e/d61 0 2024-03-04T16:04:41.774 INFO:tasks.workunit.client.1.smithi152.stdout:3/560: creat d6/d60/fbb x:0 0 0 2024-03-04T16:04:41.775 INFO:tasks.workunit.client.1.smithi152.stdout:8/634: mknod d2/d3/d15/cc5 0 2024-03-04T16:04:41.776 INFO:tasks.workunit.client.1.smithi152.stdout:6/651: dread d8/d23/f25 [0,4194304] 0 2024-03-04T16:04:41.777 INFO:tasks.workunit.client.1.smithi152.stdout:8/635: fsync d2/d75/d89/dd/d22/fa0 0 2024-03-04T16:04:41.778 INFO:tasks.workunit.client.1.smithi152.stdout:6/652: write d8/d10/f18 [2633924,88256] 0 2024-03-04T16:04:41.779 INFO:tasks.workunit.client.1.smithi152.stdout:3/561: creat d6/d14/fbc x:0 0 0 2024-03-04T16:04:41.781 INFO:tasks.workunit.client.1.smithi152.stdout:6/653: symlink d8/d92/db1/le1 0 2024-03-04T16:04:41.783 INFO:tasks.workunit.client.1.smithi152.stdout:6/654: rmdir d8/d92/db1/d87/dca 39 2024-03-04T16:04:41.788 INFO:tasks.workunit.client.1.smithi152.stdout:3/562: link d6/d5c/f97 d6/d60/d8e/fbd 0 2024-03-04T16:04:41.789 INFO:tasks.workunit.client.1.smithi152.stdout:8/636: getdents d2/d7f 0 2024-03-04T16:04:41.791 INFO:tasks.workunit.client.1.smithi152.stdout:6/655: symlink d8/d52/d58/dd0/le2 0 2024-03-04T16:04:41.791 INFO:tasks.workunit.client.1.smithi152.stdout:8/637: truncate d2/d75/d89/f1a 4468607 0 2024-03-04T16:04:41.792 INFO:tasks.workunit.client.1.smithi152.stdout:3/563: dwrite d6/d14/d58/f6b [4194304,4194304] 0 2024-03-04T16:04:41.794 INFO:tasks.workunit.client.1.smithi152.stdout:3/564: mknod d6/db1/cbe 0 2024-03-04T16:04:41.796 INFO:tasks.workunit.client.1.smithi152.stdout:6/656: dread d8/d10/f16 [0,4194304] 0 2024-03-04T16:04:41.797 INFO:tasks.workunit.client.1.smithi152.stdout:6/657: truncate d8/d10/f24 5094078 0 2024-03-04T16:04:41.798 INFO:tasks.workunit.client.1.smithi152.stdout:6/658: readlink d8/d42/l62 0 2024-03-04T16:04:41.799 INFO:tasks.workunit.client.1.smithi152.stdout:8/638: rmdir d2/d3/d21/d2d/d52/d68/d6e/db1 0 2024-03-04T16:04:41.800 INFO:tasks.workunit.client.1.smithi152.stdout:6/659: chown d8/d8f/la2 5986491 1 2024-03-04T16:04:41.802 INFO:tasks.workunit.client.1.smithi152.stdout:3/565: dwrite d6/d5c/fa0 [0,4194304] 0 2024-03-04T16:04:41.803 INFO:tasks.workunit.client.1.smithi152.stdout:6/660: mkdir d8/d8f/da3/de3 0 2024-03-04T16:04:41.805 INFO:tasks.workunit.client.1.smithi152.stdout:8/639: creat d2/d3/d15/fc6 x:0 0 0 2024-03-04T16:04:41.806 INFO:tasks.workunit.client.1.smithi152.stdout:6/661: symlink d8/d10/le4 0 2024-03-04T16:04:41.806 INFO:tasks.workunit.client.1.smithi152.stdout:8/640: dread - d2/d3/d21/d2d/d52/d68/fae zero size 2024-03-04T16:04:41.808 INFO:tasks.workunit.client.1.smithi152.stdout:8/641: chown d2/d3/d15/d1c/f1d 1095 1 2024-03-04T16:04:41.811 INFO:tasks.workunit.client.1.smithi152.stdout:8/642: symlink d2/d3/d21/d2d/d52/d68/d6e/lc7 0 2024-03-04T16:04:41.811 INFO:tasks.workunit.client.1.smithi152.stdout:6/662: link d8/d92/d59/d7d/d91/ldd d8/d42/dde/le5 0 2024-03-04T16:04:41.814 INFO:tasks.workunit.client.1.smithi152.stdout:6/663: dread d8/d2a/f89 [0,4194304] 0 2024-03-04T16:04:41.815 INFO:tasks.workunit.client.1.smithi152.stdout:3/566: dread d6/d10/f65 [0,4194304] 0 2024-03-04T16:04:41.815 INFO:tasks.workunit.client.1.smithi152.stdout:8/643: mknod d2/d75/d89/dd/d22/d26/cc8 0 2024-03-04T16:04:41.816 INFO:tasks.workunit.client.1.smithi152.stdout:6/664: dread - d8/d8f/da3/fba zero size 2024-03-04T16:04:41.818 INFO:tasks.workunit.client.1.smithi152.stdout:3/567: creat d6/d60/d8a/fbf x:0 0 0 2024-03-04T16:04:41.819 INFO:tasks.workunit.client.1.smithi152.stdout:3/568: chown d6/d10 351 1 2024-03-04T16:04:41.819 INFO:tasks.workunit.client.1.smithi152.stdout:3/569: readlink d6/d10/d2d/d6e/l79 0 2024-03-04T16:04:41.820 INFO:tasks.workunit.client.1.smithi152.stdout:8/644: symlink d2/d3/d21/d2d/d31/lc9 0 2024-03-04T16:04:41.821 INFO:tasks.workunit.client.1.smithi152.stdout:8/645: dread - d2/d7f/fc2 zero size 2024-03-04T16:04:41.822 INFO:tasks.workunit.client.1.smithi152.stdout:8/646: truncate d2/d75/f9f 1255244 0 2024-03-04T16:04:41.825 INFO:tasks.workunit.client.1.smithi152.stdout:3/570: rename d6/d53 to d6/d60/d8e/dc0 0 2024-03-04T16:04:41.827 INFO:tasks.workunit.client.1.smithi152.stdout:8/647: mknod d2/d3/d15/d1c/cca 0 2024-03-04T16:04:41.830 INFO:tasks.workunit.client.1.smithi152.stdout:3/571: creat d6/d60/d8e/dc0/fc1 x:0 0 0 2024-03-04T16:04:41.832 INFO:tasks.workunit.client.1.smithi152.stdout:8/648: creat d2/d75/d89/dd/d22/d26/fcb x:0 0 0 2024-03-04T16:04:41.834 INFO:tasks.workunit.client.1.smithi152.stdout:6/665: getdents d8/d23/d3a/d8d 0 2024-03-04T16:04:41.836 INFO:tasks.workunit.client.1.smithi152.stdout:3/572: dwrite d6/d10/f27 [0,4194304] 0 2024-03-04T16:04:41.838 INFO:tasks.workunit.client.1.smithi152.stdout:3/573: chown d6/f46 240701500 1 2024-03-04T16:04:41.838 INFO:tasks.workunit.client.1.smithi152.stdout:8/649: write d2/d75/d89/f28 [1874758,59708] 0 2024-03-04T16:04:41.840 INFO:tasks.workunit.client.1.smithi152.stdout:6/666: creat d8/d92/d59/d61/d57/fe6 x:0 0 0 2024-03-04T16:04:41.843 INFO:tasks.workunit.client.1.smithi152.stdout:6/667: chown d8/d52/d58/c76 3714564 1 2024-03-04T16:04:41.843 INFO:tasks.workunit.client.1.smithi152.stdout:8/650: mkdir d2/d3/d21/d2d/d52/dcc 0 2024-03-04T16:04:41.843 INFO:tasks.workunit.client.1.smithi152.stdout:6/668: stat d8/d23/d3a/d6d 0 2024-03-04T16:04:41.843 INFO:tasks.workunit.client.1.smithi152.stdout:6/669: read d8/d92/d75/f88 [1975030,9032] 0 2024-03-04T16:04:41.845 INFO:tasks.workunit.client.1.smithi152.stdout:3/574: rename d6/d60/d8a/fbf to d6/d60/d8e/dc0/da1/fc2 0 2024-03-04T16:04:41.845 INFO:tasks.workunit.client.1.smithi152.stdout:6/670: dread - d8/d23/d3a/d6d/fd6 zero size 2024-03-04T16:04:41.847 INFO:tasks.workunit.client.1.smithi152.stdout:8/651: dread d2/d75/d89/dd/d22/fa4 [0,4194304] 0 2024-03-04T16:04:41.849 INFO:tasks.workunit.client.1.smithi152.stdout:3/575: rmdir d6/d14/d41 39 2024-03-04T16:04:41.856 INFO:tasks.workunit.client.1.smithi152.stdout:6/671: creat d8/d10/d53/d9a/fe7 x:0 0 0 2024-03-04T16:04:41.866 INFO:tasks.workunit.client.1.smithi152.stdout:3/576: dwrite d6/d10/d2d/f36 [4194304,4194304] 0 2024-03-04T16:04:41.872 INFO:tasks.workunit.client.1.smithi152.stdout:8/652: dread d2/d75/d89/f2e [0,4194304] 0 2024-03-04T16:04:41.877 INFO:tasks.workunit.client.1.smithi152.stdout:3/577: dread d6/f2c [0,4194304] 0 2024-03-04T16:04:41.877 INFO:tasks.workunit.client.1.smithi152.stdout:6/672: dwrite d8/d52/f67 [0,4194304] 0 2024-03-04T16:04:41.880 INFO:tasks.workunit.client.1.smithi152.stdout:8/653: creat d2/d75/d89/dd/d22/d83/d96/da6/fcd x:0 0 0 2024-03-04T16:04:41.881 INFO:tasks.workunit.client.1.smithi152.stdout:3/578: creat d6/d10/d2d/d4d/fc3 x:0 0 0 2024-03-04T16:04:41.882 INFO:tasks.workunit.client.1.smithi152.stdout:6/673: creat d8/d10/d53/d9a/fe8 x:0 0 0 2024-03-04T16:04:41.885 INFO:tasks.workunit.client.1.smithi152.stdout:3/579: creat d6/d14/fc4 x:0 0 0 2024-03-04T16:04:41.886 INFO:tasks.workunit.client.1.smithi152.stdout:6/674: creat d8/d23/fe9 x:0 0 0 2024-03-04T16:04:41.889 INFO:tasks.workunit.client.1.smithi152.stdout:8/654: dwrite d2/d3/d15/d1c/d84/fa8 [0,4194304] 0 2024-03-04T16:04:41.889 INFO:tasks.workunit.client.1.smithi152.stdout:3/580: creat d6/d14/d93/db7/fc5 x:0 0 0 2024-03-04T16:04:41.895 INFO:tasks.workunit.client.1.smithi152.stdout:3/581: creat d6/d5c/fc6 x:0 0 0 2024-03-04T16:04:41.898 INFO:tasks.workunit.client.1.smithi152.stdout:6/675: link d8/d10/f16 d8/d92/d59/fea 0 2024-03-04T16:04:41.902 INFO:tasks.workunit.client.1.smithi152.stdout:3/582: dwrite d6/d14/d93/f77 [0,4194304] 0 2024-03-04T16:04:41.902 INFO:tasks.workunit.client.1.smithi152.stdout:6/676: rename d8/f56 to d8/d92/d59/d61/d57/feb 0 2024-03-04T16:04:41.907 INFO:tasks.workunit.client.1.smithi152.stdout:6/677: rename d8/d23/d93/fd9 to d8/d10/d53/dd7/fec 0 2024-03-04T16:04:41.910 INFO:tasks.workunit.client.1.smithi152.stdout:3/583: link d6/d5c/fa0 d6/d60/d8e/dc0/fc7 0 2024-03-04T16:04:41.913 INFO:tasks.workunit.client.1.smithi152.stdout:5/557: sync 2024-03-04T16:04:41.916 INFO:tasks.workunit.client.1.smithi152.stdout:6/678: mknod d8/d10/d53/d9a/ced 0 2024-03-04T16:04:41.916 INFO:tasks.workunit.client.1.smithi152.stdout:6/679: chown d8/c6e 41807 1 2024-03-04T16:04:41.916 INFO:tasks.workunit.client.1.smithi152.stdout:3/584: creat d6/d60/d8a/fc8 x:0 0 0 2024-03-04T16:04:41.924 INFO:tasks.workunit.client.1.smithi152.stdout:8/655: dread d2/d7f/f98 [0,4194304] 0 2024-03-04T16:04:41.931 INFO:tasks.workunit.client.1.smithi152.stdout:6/680: mkdir d8/d42/dde/dee 0 2024-03-04T16:04:41.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.931+0000 7f3aabfff700 1 -- 172.21.15.116:0/1472636610 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3aa40054b0 con 0x7f3ab41082d0 2024-03-04T16:04:41.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.932+0000 7f3abb429700 1 -- 172.21.15.116:0/1472636610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab41082d0 msgr2=0x7f3ab4108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:41.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.932+0000 7f3abb429700 1 --2- 172.21.15.116:0/1472636610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab41082d0 0x7f3ab4108750 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009910 tx=0x7f3aa4009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:41.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.932+0000 7f3abb429700 1 -- 172.21.15.116:0/1472636610 shutdown_connections 2024-03-04T16:04:41.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.932+0000 7f3abb429700 1 --2- 172.21.15.116:0/1472636610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab41082d0 0x7f3ab4108750 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:41.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.932+0000 7f3abb429700 1 --2- 172.21.15.116:0/1472636610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ab4072a30 0x7f3ab4107d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:41.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.932+0000 7f3abb429700 1 -- 172.21.15.116:0/1472636610 >> 172.21.15.116:0/1472636610 conn(0x7f3ab406d9f0 msgr2=0x7f3ab406fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:41.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.932+0000 7f3abb429700 1 -- 172.21.15.116:0/1472636610 shutdown_connections 2024-03-04T16:04:41.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.932+0000 7f3abb429700 1 -- 172.21.15.116:0/1472636610 wait complete. 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.933+0000 7f3abb429700 1 Processor -- start 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.933+0000 7f3abb429700 1 -- start start 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.933+0000 7f3abb429700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab4072a30 0x7f3ab410aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.933+0000 7f3abb429700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ab41082d0 0x7f3ab4109060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.933+0000 7f3abb429700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab41a2780 con 0x7f3ab4072a30 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.933+0000 7f3abb429700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab41a28c0 con 0x7f3ab41082d0 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.933+0000 7f3ab91c5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab4072a30 0x7f3ab410aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.933+0000 7f3ab91c5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab4072a30 0x7f3ab410aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53686/0 (socket says 172.21.15.116:53686) 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.933+0000 7f3ab91c5700 1 -- 172.21.15.116:0/1377867517 learned_addr learned my addr 172.21.15.116:0/1377867517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.933+0000 7f3ab89c4700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ab41082d0 0x7f3ab4109060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:41.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.934+0000 7f3ab91c5700 1 -- 172.21.15.116:0/1377867517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ab41082d0 msgr2=0x7f3ab4109060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:41.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.934+0000 7f3ab91c5700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ab41082d0 0x7f3ab4109060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:41.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.934+0000 7f3ab91c5700 1 -- 172.21.15.116:0/1377867517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa40095f0 con 0x7f3ab4072a30 2024-03-04T16:04:41.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.934+0000 7f3ab91c5700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab4072a30 0x7f3ab410aa10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000d8d0 tx=0x7f3ab000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:41.936 INFO:tasks.workunit.client.1.smithi152.stdout:3/585: mknod d6/cc9 0 2024-03-04T16:04:41.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.935+0000 7f3aaa7fc700 1 -- 172.21.15.116:0/1377867517 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ab00098a0 con 0x7f3ab4072a30 2024-03-04T16:04:41.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.935+0000 7f3abb429700 1 -- 172.21.15.116:0/1377867517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab41095a0 con 0x7f3ab4072a30 2024-03-04T16:04:41.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.935+0000 7f3abb429700 1 -- 172.21.15.116:0/1377867517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab4109ac0 con 0x7f3ab4072a30 2024-03-04T16:04:41.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.936+0000 7f3aaa7fc700 1 -- 172.21.15.116:0/1377867517 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3ab0010460 con 0x7f3ab4072a30 2024-03-04T16:04:41.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.936+0000 7f3aaa7fc700 1 -- 172.21.15.116:0/1377867517 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ab000f5f0 con 0x7f3ab4072a30 2024-03-04T16:04:41.938 INFO:tasks.workunit.client.1.smithi152.stdout:6/681: dread d8/d23/d3a/f70 [0,4194304] 0 2024-03-04T16:04:41.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.937+0000 7f3aaa7fc700 1 -- 172.21.15.116:0/1377867517 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f3ab000f750 con 0x7f3ab4072a30 2024-03-04T16:04:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.937+0000 7f3aaa7fc700 1 -- 172.21.15.116:0/1377867517 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f3ab004d780 con 0x7f3ab4072a30 2024-03-04T16:04:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.937+0000 7f3abb429700 1 -- 172.21.15.116:0/1377867517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a98005320 con 0x7f3ab4072a30 2024-03-04T16:04:41.941 INFO:tasks.workunit.client.1.smithi152.stdout:5/558: creat d6/d8/d5a/d90/daf/fbf x:0 0 0 2024-03-04T16:04:41.942 INFO:tasks.workunit.client.1.smithi152.stdout:5/559: truncate d6/d50/f7b 523182 0 2024-03-04T16:04:41.942 INFO:tasks.workunit.client.1.smithi152.stdout:5/560: chown d6/d8/d1f/d3d/d6f/d70/d98 129745 1 2024-03-04T16:04:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:41.944+0000 7f3aaa7fc700 1 -- 172.21.15.116:0/1377867517 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f3ab00228f0 con 0x7f3ab4072a30 2024-03-04T16:04:41.947 INFO:tasks.workunit.client.1.smithi152.stdout:3/586: rename d6/fc to d6/d60/d8a/fca 0 2024-03-04T16:04:41.950 INFO:tasks.workunit.client.1.smithi152.stdout:8/656: mkdir d2/d3/d21/dce 0 2024-03-04T16:04:41.952 INFO:tasks.workunit.client.1.smithi152.stdout:8/657: chown d2/d3/d21/d2d/d52/d68/d6e/lc7 0 1 2024-03-04T16:04:41.952 INFO:tasks.workunit.client.1.smithi152.stdout:6/682: unlink d8/d23/d3a/d71/ca0 0 2024-03-04T16:04:41.959 INFO:tasks.workunit.client.1.smithi152.stdout:6/683: dwrite d8/d92/db1/d87/fc2 [0,4194304] 0 2024-03-04T16:04:41.965 INFO:tasks.workunit.client.1.smithi152.stdout:5/561: mkdir d6/d8/d1f/d3d/dc0 0 2024-03-04T16:04:41.967 INFO:tasks.workunit.client.1.smithi152.stdout:5/562: truncate d6/d8/d1f/f57 4544017 0 2024-03-04T16:04:41.968 INFO:tasks.workunit.client.1.smithi152.stdout:6/684: creat d8/d92/d59/fef x:0 0 0 2024-03-04T16:04:41.977 INFO:tasks.workunit.client.1.smithi152.stdout:5/563: fdatasync d6/d8/d5a/d64/d76/d9f/f51 0 2024-03-04T16:04:41.984 INFO:tasks.workunit.client.1.smithi152.stdout:8/658: creat d2/d3/fcf x:0 0 0 2024-03-04T16:04:41.987 INFO:tasks.workunit.client.1.smithi152.stdout:6/685: dread d8/d10/f24 [0,4194304] 0 2024-03-04T16:04:41.989 INFO:tasks.workunit.client.1.smithi152.stdout:3/587: link d6/d14/f90 d6/d14/d58/dad/fcb 0 2024-03-04T16:04:41.992 INFO:tasks.workunit.client.1.smithi152.stdout:8/659: unlink d2/d75/d89/dd/d22/fa0 0 2024-03-04T16:04:41.997 INFO:tasks.workunit.client.1.smithi152.stdout:3/588: stat d6/d14/d41/d49/f62 0 2024-03-04T16:04:42.001 INFO:tasks.workunit.client.1.smithi152.stdout:8/660: truncate d2/d75/d89/dd/f2f 2115270 0 2024-03-04T16:04:42.002 INFO:tasks.workunit.client.1.smithi152.stdout:0/482: sync 2024-03-04T16:04:42.002 INFO:tasks.workunit.client.1.smithi152.stdout:1/590: sync 2024-03-04T16:04:42.003 INFO:tasks.workunit.client.1.smithi152.stdout:5/564: link d6/d8/d1f/d3d/f4c d6/d8/d1f/d3d/fc1 0 2024-03-04T16:04:42.006 INFO:tasks.workunit.client.1.smithi152.stdout:5/565: read d6/d8/d5a/d64/d76/d9f/f12 [8342528,21662] 0 2024-03-04T16:04:42.007 INFO:tasks.workunit.client.1.smithi152.stdout:3/589: rename d6/d5c/fac to d6/d60/d8e/dc0/db5/fcc 0 2024-03-04T16:04:42.009 INFO:tasks.workunit.client.1.smithi152.stdout:1/591: mkdir d19/d75/dc3 0 2024-03-04T16:04:42.011 INFO:tasks.workunit.client.1.smithi152.stdout:8/661: dwrite d2/d75/f8f [0,4194304] 0 2024-03-04T16:04:42.017 INFO:tasks.workunit.client.1.smithi152.stdout:1/592: dwrite d19/f5b [4194304,4194304] 0 2024-03-04T16:04:42.024 INFO:tasks.workunit.client.1.smithi152.stdout:3/590: symlink d6/d60/d8e/dc0/da1/lcd 0 2024-03-04T16:04:42.025 INFO:tasks.workunit.client.1.smithi152.stdout:8/662: rmdir d2/d3/d21/dc0 39 2024-03-04T16:04:42.030 INFO:tasks.workunit.client.1.smithi152.stdout:3/591: write d6/d14/f8d [589183,117504] 0 2024-03-04T16:04:42.031 INFO:tasks.workunit.client.1.smithi152.stdout:3/592: fsync d6/d14/d93/f77 0 2024-03-04T16:04:42.036 INFO:tasks.workunit.client.1.smithi152.stdout:8/663: symlink d2/d3/d21/d2d/d31/ld0 0 2024-03-04T16:04:42.037 INFO:tasks.workunit.client.1.smithi152.stdout:1/593: mknod d19/d75/da8/db8/cc4 0 2024-03-04T16:04:42.038 INFO:tasks.workunit.client.1.smithi152.stdout:8/664: write d2/d75/d89/dd/d22/d26/f61 [3059124,29109] 0 2024-03-04T16:04:42.043 INFO:tasks.workunit.client.1.smithi152.stdout:6/686: truncate d8/d2a/f3c 1741007 0 2024-03-04T16:04:42.044 INFO:tasks.workunit.client.1.smithi152.stdout:3/593: mkdir d6/db1/dce 0 2024-03-04T16:04:42.045 INFO:tasks.workunit.client.1.smithi152.stdout:5/566: link d6/d8/d5a/d64/d76/d9f/c40 d6/d8/d1f/d3d/da4/cc2 0 2024-03-04T16:04:42.045 INFO:tasks.workunit.client.1.smithi152.stdout:1/594: dwrite d19/d4b/d53/d57/f88 [0,4194304] 0 2024-03-04T16:04:42.047 INFO:tasks.workunit.client.1.smithi152.stdout:1/595: chown d19/d20/d2f/d6b/f46 132317 1 2024-03-04T16:04:42.049 INFO:tasks.workunit.client.1.smithi152.stdout:0/483: rename d9/d1b/d62 to d9/d1b/d54/d69/d98 0 2024-03-04T16:04:42.052 INFO:tasks.workunit.client.1.smithi152.stdout:8/665: readlink d2/d75/d89/dd/d22/l36 0 2024-03-04T16:04:42.054 INFO:tasks.workunit.client.1.smithi152.stdout:3/594: dread d6/f3a [0,4194304] 0 2024-03-04T16:04:42.058 INFO:tasks.workunit.client.1.smithi152.stdout:8/666: dwrite d2/d75/d89/f28 [0,4194304] 0 2024-03-04T16:04:42.145 INFO:tasks.workunit.client.1.smithi152.stdout:3/595: creat d6/d10/d2d/fcf x:0 0 0 2024-03-04T16:04:42.147 INFO:tasks.workunit.client.1.smithi152.stdout:8/667: creat d2/d3/d15/d1c/d84/d87/fd1 x:0 0 0 2024-03-04T16:04:42.150 INFO:tasks.workunit.client.1.smithi152.stdout:8/668: dread d2/d75/d89/dd/d22/f78 [0,4194304] 0 2024-03-04T16:04:42.151 INFO:tasks.workunit.client.1.smithi152.stdout:3/596: dwrite d6/d60/f81 [0,4194304] 0 2024-03-04T16:04:42.186 INFO:tasks.workunit.client.1.smithi152.stdout:0/484: truncate d9/d2c/f2f 2648055 0 2024-03-04T16:04:42.187 INFO:tasks.workunit.client.1.smithi152.stdout:0/485: chown d9/d1b/d3c/d84/d56 422 1 2024-03-04T16:04:42.194 INFO:tasks.workunit.client.1.smithi152.stdout:0/486: dwrite d9/d1b/f1d [0,4194304] 0 2024-03-04T16:04:42.196 INFO:tasks.workunit.client.1.smithi152.stdout:0/487: truncate d9/d2c/d2e/f31 972296 0 2024-03-04T16:04:42.197 INFO:tasks.workunit.client.1.smithi152.stdout:3/597: mknod d6/d14/d93/db7/cd0 0 2024-03-04T16:04:42.211 INFO:tasks.workunit.client.1.smithi152.stdout:0/488: unlink d9/d1a/c51 0 2024-03-04T16:04:42.217 INFO:tasks.workunit.client.1.smithi152.stdout:8/669: getdents d2/d3/d21/d2d/d52 0 2024-03-04T16:04:42.222 INFO:tasks.workunit.client.1.smithi152.stdout:3/598: rmdir d6/db1/dce 0 2024-03-04T16:04:42.223 INFO:tasks.workunit.client.1.smithi152.stdout:5/567: write d6/d8/d5a/d64/d76/d9f/f2d [961223,121480] 0 2024-03-04T16:04:42.224 INFO:tasks.workunit.client.1.smithi152.stdout:3/599: dread - d6/d14/d58/f9a zero size 2024-03-04T16:04:42.226 INFO:tasks.workunit.client.1.smithi152.stdout:3/600: chown d6/d14/d93 34288 1 2024-03-04T16:04:42.226 INFO:tasks.workunit.client.1.smithi152.stdout:0/489: mknod d9/d1b/d54/c99 0 2024-03-04T16:04:42.229 INFO:tasks.workunit.client.1.smithi152.stdout:8/670: link d2/d3/d15/d1c/d84/fbb d2/d3/d15/d1c/d84/d87/fd2 0 2024-03-04T16:04:42.229 INFO:tasks.workunit.client.1.smithi152.stdout:1/596: dwrite d19/d20/d2f/d6b/f4a [0,4194304] 0 2024-03-04T16:04:42.231 INFO:tasks.workunit.client.1.smithi152.stdout:3/601: fdatasync d6/d14/d41/d49/fae 0 2024-03-04T16:04:42.233 INFO:tasks.workunit.client.1.smithi152.stdout:4/486: sync 2024-03-04T16:04:42.233 INFO:tasks.workunit.client.1.smithi152.stdout:7/471: sync 2024-03-04T16:04:42.233 INFO:tasks.workunit.client.1.smithi152.stdout:9/543: sync 2024-03-04T16:04:42.233 INFO:tasks.workunit.client.1.smithi152.stdout:2/562: sync 2024-03-04T16:04:42.239 INFO:tasks.workunit.client.1.smithi152.stdout:9/544: stat d2/d13/d23/d66/c6b 0 2024-03-04T16:04:42.249 INFO:tasks.workunit.client.1.smithi152.stdout:5/568: dwrite d6/d8/d1f/d3d/d6f/f95 [0,4194304] 0 2024-03-04T16:04:42.252 INFO:tasks.workunit.client.1.smithi152.stdout:4/487: creat d8/d2f/d6f/d84/fa4 x:0 0 0 2024-03-04T16:04:42.253 INFO:tasks.workunit.client.1.smithi152.stdout:4/488: fsync d8/d2f/d6f/d90/f92 0 2024-03-04T16:04:42.255 INFO:tasks.workunit.client.1.smithi152.stdout:9/545: dwrite d2/d13/f52 [0,4194304] 0 2024-03-04T16:04:42.259 INFO:tasks.workunit.client.1.smithi152.stdout:8/671: link d2/d3/d15/fc6 d2/d75/d89/dd/d22/d6c/d86/fd3 0 2024-03-04T16:04:42.262 INFO:tasks.workunit.client.1.smithi152.stdout:4/489: dwrite d8/f23 [0,4194304] 0 2024-03-04T16:04:42.282 INFO:tasks.workunit.client.1.smithi152.stdout:7/472: unlink d14/f2a 0 2024-03-04T16:04:42.301 INFO:tasks.workunit.client.1.smithi152.stdout:0/490: creat d9/d1b/d54/d69/f9a x:0 0 0 2024-03-04T16:04:42.302 INFO:tasks.workunit.client.1.smithi152.stdout:9/546: stat d2/fe 0 2024-03-04T16:04:42.305 INFO:tasks.workunit.client.1.smithi152.stdout:4/490: creat d8/d2f/d6f/d90/fa5 x:0 0 0 2024-03-04T16:04:42.311 INFO:tasks.workunit.client.1.smithi152.stdout:2/563: symlink da/d5f/d85/d8c/lbe 0 2024-03-04T16:04:42.316 INFO:tasks.workunit.client.1.smithi152.stdout:3/602: link d6/f3f d6/d14/d58/dad/fd1 0 2024-03-04T16:04:42.321 INFO:tasks.workunit.client.1.smithi152.stdout:9/547: symlink d2/d13/d23/lb4 0 2024-03-04T16:04:42.325 INFO:tasks.workunit.client.1.smithi152.stdout:8/672: link d2/d3/d15/f6a d2/d75/d89/fd4 0 2024-03-04T16:04:42.333 INFO:tasks.workunit.client.1.smithi152.stdout:3/603: unlink d6/d60/l61 0 2024-03-04T16:04:42.334 INFO:tasks.workunit.client.1.smithi152.stdout:3/604: stat d6/d14/d93/db7 0 2024-03-04T16:04:42.339 INFO:tasks.workunit.client.1.smithi152.stdout:7/473: rmdir d14/d21/d48/d80 0 2024-03-04T16:04:42.341 INFO:tasks.workunit.client.1.smithi152.stdout:3/605: unlink d6/d10/d2d/l3e 0 2024-03-04T16:04:42.345 INFO:tasks.workunit.client.1.smithi152.stdout:8/673: truncate d2/d75/d89/dd/d22/d26/f38 369102 0 2024-03-04T16:04:42.352 INFO:tasks.workunit.client.1.smithi152.stdout:8/674: symlink d2/d75/ld5 0 2024-03-04T16:04:42.353 INFO:tasks.workunit.client.1.smithi152.stdout:7/474: truncate d14/f61 1211749 0 2024-03-04T16:04:42.357 INFO:tasks.workunit.client.1.smithi152.stdout:8/675: fsync d2/f80 0 2024-03-04T16:04:42.363 INFO:tasks.workunit.client.1.smithi152.stdout:8/676: dread d2/d3/d21/d2d/d31/f35 [0,4194304] 0 2024-03-04T16:04:42.365 INFO:tasks.workunit.client.1.smithi152.stdout:7/475: dread d14/d21/d47/d40/f43 [0,4194304] 0 2024-03-04T16:04:42.366 INFO:tasks.workunit.client.1.smithi152.stdout:3/606: getdents d6/d5c 0 2024-03-04T16:04:42.369 INFO:tasks.workunit.client.1.smithi152.stdout:3/607: dread d6/d60/d8e/dc0/f5b [0,4194304] 0 2024-03-04T16:04:42.374 INFO:tasks.workunit.client.1.smithi152.stdout:3/608: getdents d6/d14/d58 0 2024-03-04T16:04:42.375 INFO:tasks.workunit.client.1.smithi152.stdout:3/609: chown d6/d14/d41/d49/d84 7 1 2024-03-04T16:04:42.377 INFO:tasks.workunit.client.1.smithi152.stdout:3/610: creat d6/db1/fd2 x:0 0 0 2024-03-04T16:04:42.378 INFO:tasks.workunit.client.1.smithi152.stdout:3/611: write d6/d60/f81 [4358636,105169] 0 2024-03-04T16:04:42.378 INFO:tasks.workunit.client.1.smithi152.stdout:3/612: dread - d6/d60/fb8 zero size 2024-03-04T16:04:42.381 INFO:tasks.workunit.client.1.smithi152.stdout:3/613: stat d6/l13 0 2024-03-04T16:04:42.383 INFO:tasks.workunit.client.1.smithi152.stdout:7/476: dread d14/d21/d47/d40/f6c [0,4194304] 0 2024-03-04T16:04:42.384 INFO:tasks.workunit.client.1.smithi152.stdout:3/614: unlink d6/d10/d2d/d6e/l79 0 2024-03-04T16:04:42.385 INFO:tasks.workunit.client.1.smithi152.stdout:7/477: creat d14/d69/f88 x:0 0 0 2024-03-04T16:04:42.386 INFO:tasks.workunit.client.1.smithi152.stdout:7/478: dread - d14/d21/f6b zero size 2024-03-04T16:04:42.388 INFO:tasks.workunit.client.1.smithi152.stdout:7/479: read f7 [1149846,118587] 0 2024-03-04T16:04:42.388 INFO:tasks.workunit.client.1.smithi152.stdout:7/480: stat d14/d21/d47/d40/l84 0 2024-03-04T16:04:42.394 INFO:tasks.workunit.client.1.smithi152.stdout:4/491: write d8/f10 [4526187,31964] 0 2024-03-04T16:04:42.396 INFO:tasks.workunit.client.1.smithi152.stdout:5/569: creat d6/d8/fc3 x:0 0 0 2024-03-04T16:04:42.400 INFO:tasks.workunit.client.1.smithi152.stdout:0/491: rename d9/d1b/d54/d80/l91 to d9/d1b/d3c/d84/l9b 0 2024-03-04T16:04:42.403 INFO:tasks.workunit.client.1.smithi152.stdout:9/548: rename d2/d13/d38/l50 to d2/d63/lb5 0 2024-03-04T16:04:42.405 INFO:tasks.workunit.client.1.smithi152.stdout:9/549: write d2/d13/f15 [3551217,1169] 0 2024-03-04T16:04:42.407 INFO:tasks.workunit.client.1.smithi152.stdout:8/677: rename d2/d3/d21/d2d/d52/d68/d6e/d7d/dba to d2/d3/d21/d2d/d52/dcc/dd6 0 2024-03-04T16:04:42.411 INFO:tasks.workunit.client.1.smithi152.stdout:4/492: rename d8/d9/f72 to d8/d9/d7a/fa6 0 2024-03-04T16:04:42.413 INFO:tasks.workunit.client.1.smithi152.stdout:8/678: mknod d2/d3/d21/d2d/d31/d9a/cd7 0 2024-03-04T16:04:42.415 INFO:tasks.workunit.client.1.smithi152.stdout:4/493: rmdir d8/d4e 39 2024-03-04T16:04:42.417 INFO:tasks.workunit.client.1.smithi152.stdout:5/570: rename d6/d8/d1f/d3d/d6f/db8 to d6/d8/d1f/d3d/d6f/d70/dc4 0 2024-03-04T16:04:42.419 INFO:tasks.workunit.client.1.smithi152.stdout:8/679: mknod d2/d3/d21/d2d/d31/d9a/cd8 0 2024-03-04T16:04:42.427 INFO:tasks.workunit.client.1.smithi152.stdout:4/494: dread d8/d9/f28 [4194304,4194304] 0 2024-03-04T16:04:42.431 INFO:tasks.workunit.client.1.smithi152.stdout:8/680: getdents d2/d3/d15 0 2024-03-04T16:04:42.432 INFO:tasks.workunit.client.1.smithi152.stdout:8/681: chown d2/d75/d89/c25 17334813 1 2024-03-04T16:04:42.433 INFO:tasks.workunit.client.1.smithi152.stdout:4/495: creat d8/d9/d24/d26/d42/fa7 x:0 0 0 2024-03-04T16:04:42.440 INFO:tasks.workunit.client.1.smithi152.stdout:8/682: unlink d2/d3/d21/c82 0 2024-03-04T16:04:42.441 INFO:tasks.workunit.client.1.smithi152.stdout:8/683: chown d2/d3/d21/d2d/d31/d9a/ca3 270563 1 2024-03-04T16:04:42.442 INFO:tasks.workunit.client.1.smithi152.stdout:7/481: write d14/d21/d31/f64 [464797,56910] 0 2024-03-04T16:04:42.443 INFO:tasks.workunit.client.1.smithi152.stdout:4/496: symlink d8/d9/d7a/la8 0 2024-03-04T16:04:42.443 INFO:tasks.workunit.client.1.smithi152.stdout:0/492: link d9/d1b/d3c/d84/d56/l76 d9/d1b/l9c 0 2024-03-04T16:04:42.445 INFO:tasks.workunit.client.1.smithi152.stdout:8/684: mkdir d2/d75/d89/dd/d22/d83/dd9 0 2024-03-04T16:04:42.445 INFO:tasks.workunit.client.1.smithi152.stdout:8/685: readlink d2/l95 0 2024-03-04T16:04:42.447 INFO:tasks.workunit.client.1.smithi152.stdout:0/493: truncate d9/d1b/d3c/d84/d56/d5b/d7a/f8f 1033339 0 2024-03-04T16:04:42.449 INFO:tasks.workunit.client.1.smithi152.stdout:4/497: rename d8/d9/d24/f88 to d8/d4e/d8b/fa9 0 2024-03-04T16:04:42.455 INFO:tasks.workunit.client.1.smithi152.stdout:0/494: dwrite d9/d1b/d54/d69/f9a [0,4194304] 0 2024-03-04T16:04:42.459 INFO:tasks.workunit.client.1.smithi152.stdout:8/686: rename d2/d3/d21/d2d/d48/fb2 to d2/d3/d9e/fda 0 2024-03-04T16:04:42.460 INFO:tasks.workunit.client.1.smithi152.stdout:8/687: readlink d2/d3/d21/d2d/d48/l4c 0 2024-03-04T16:04:42.461 INFO:tasks.workunit.client.1.smithi152.stdout:4/498: symlink d8/d58/laa 0 2024-03-04T16:04:42.469 INFO:tasks.workunit.client.1.smithi152.stdout:4/499: symlink d8/d4e/d86/lab 0 2024-03-04T16:04:42.470 INFO:tasks.workunit.client.1.smithi152.stdout:4/500: write d8/d9/d24/d26/f2a [7858578,15345] 0 2024-03-04T16:04:42.472 INFO:tasks.workunit.client.1.smithi152.stdout:8/688: creat d2/d3/d21/dac/fdb x:0 0 0 2024-03-04T16:04:42.478 INFO:tasks.workunit.client.1.smithi152.stdout:4/501: rmdir d8/d2f/d7c 39 2024-03-04T16:04:42.480 INFO:tasks.workunit.client.1.smithi152.stdout:8/689: mknod d2/d3/d9e/cdc 0 2024-03-04T16:04:42.481 INFO:tasks.workunit.client.1.smithi152.stdout:8/690: stat d2/d7f 0 2024-03-04T16:04:42.482 INFO:tasks.workunit.client.1.smithi152.stdout:0/495: getdents d9/d1b/d54/d80 0 2024-03-04T16:04:42.485 INFO:tasks.workunit.client.1.smithi152.stdout:4/502: dwrite d8/d4e/f64 [0,4194304] 0 2024-03-04T16:04:42.487 INFO:tasks.workunit.client.1.smithi152.stdout:4/503: chown d8/d4e/d86/lab 23028 1 2024-03-04T16:04:42.492 INFO:tasks.workunit.client.1.smithi152.stdout:6/687: sync 2024-03-04T16:04:42.493 INFO:tasks.workunit.client.1.smithi152.stdout:4/504: dwrite d8/d9/d24/d26/f2a [0,4194304] 0 2024-03-04T16:04:42.504 INFO:tasks.workunit.client.1.smithi152.stdout:0/496: dread d9/d1a/f79 [0,4194304] 0 2024-03-04T16:04:42.504 INFO:tasks.workunit.client.1.smithi152.stdout:4/505: dwrite d8/d9/d24/f74 [0,4194304] 0 2024-03-04T16:04:42.534 INFO:tasks.workunit.client.1.smithi152.stdout:0/497: dwrite d9/d1b/f4f [0,4194304] 0 2024-03-04T16:04:42.535 INFO:tasks.workunit.client.1.smithi152.stdout:0/498: fdatasync d9/d2c/d2e/f31 0 2024-03-04T16:04:42.536 INFO:tasks.workunit.client.1.smithi152.stdout:0/499: truncate d9/d1a/d7c/f7e 1576203 0 2024-03-04T16:04:42.550 INFO:tasks.workunit.client.1.smithi152.stdout:6/688: unlink d8/d2a/fdb 0 2024-03-04T16:04:42.551 INFO:tasks.workunit.client.1.smithi152.stdout:6/689: stat d8/d52/f67 0 2024-03-04T16:04:42.551 INFO:tasks.workunit.client.1.smithi152.stdout:6/690: chown d8/d10/c90 536690971 1 2024-03-04T16:04:42.579 INFO:tasks.workunit.client.1.smithi152.stdout:4/506: mknod d8/d78/cac 0 2024-03-04T16:04:42.579 INFO:tasks.workunit.client.1.smithi152.stdout:8/691: write d2/d3/d21/f1b [4801094,50879] 0 2024-03-04T16:04:42.581 INFO:tasks.workunit.client.1.smithi152.stdout:4/507: rmdir d8 39 2024-03-04T16:04:42.581 INFO:tasks.workunit.client.1.smithi152.stdout:0/500: mkdir d9/d2c/d2e/d7f/d9d 0 2024-03-04T16:04:42.583 INFO:tasks.workunit.client.1.smithi152.stdout:8/692: dread d2/d3/d21/d2d/d31/f35 [0,4194304] 0 2024-03-04T16:04:42.583 INFO:tasks.workunit.client.1.smithi152.stdout:0/501: write d9/d1b/f1d [2031059,127649] 0 2024-03-04T16:04:42.584 INFO:tasks.workunit.client.1.smithi152.stdout:6/691: creat d8/d92/d59/d7d/d91/ff0 x:0 0 0 2024-03-04T16:04:42.587 INFO:tasks.workunit.client.1.smithi152.stdout:6/692: write d8/d92/db1/fd5 [314371,65985] 0 2024-03-04T16:04:42.588 INFO:tasks.workunit.client.1.smithi152.stdout:8/693: write d2/d3/d15/d1c/d84/d87/d5d/f7b [1775206,130408] 0 2024-03-04T16:04:42.590 INFO:tasks.workunit.client.1.smithi152.stdout:6/693: truncate d8/d10/d53/d9a/fe7 670506 0 2024-03-04T16:04:42.593 INFO:tasks.workunit.client.1.smithi152.stdout:4/508: symlink d8/d9/d24/lad 0 2024-03-04T16:04:42.599 INFO:tasks.workunit.client.1.smithi152.stdout:6/694: dwrite d8/d42/f4a [4194304,4194304] 0 2024-03-04T16:04:42.607 INFO:tasks.workunit.client.1.smithi152.stdout:0/502: dread d9/d1b/f35 [0,4194304] 0 2024-03-04T16:04:42.612 INFO:tasks.workunit.client.1.smithi152.stdout:0/503: creat d9/d1b/d54/d80/f9e x:0 0 0 2024-03-04T16:04:42.614 INFO:tasks.workunit.client.1.smithi152.stdout:4/509: dwrite d8/f16 [4194304,4194304] 0 2024-03-04T16:04:42.616 INFO:tasks.workunit.client.1.smithi152.stdout:0/504: dread - d9/d1b/d54/d80/f9e zero size 2024-03-04T16:04:42.617 INFO:tasks.workunit.client.1.smithi152.stdout:6/695: creat d8/d23/d3a/ff1 x:0 0 0 2024-03-04T16:04:42.618 INFO:tasks.workunit.client.1.smithi152.stdout:8/694: dread d2/d3/d21/f5f [0,4194304] 0 2024-03-04T16:04:42.626 INFO:tasks.workunit.client.1.smithi152.stdout:4/510: rename d8/d9/d24/f66 to d8/d9/fae 0 2024-03-04T16:04:42.628 INFO:tasks.workunit.client.1.smithi152.stdout:8/695: dwrite d2/d3/f40 [0,4194304] 0 2024-03-04T16:04:42.631 INFO:tasks.workunit.client.1.smithi152.stdout:0/505: link d9/d1b/d54/d80/f9e d9/d2c/d2e/d7f/f9f 0 2024-03-04T16:04:42.631 INFO:tasks.workunit.client.1.smithi152.stdout:6/696: fsync d8/d10/d53/f5b 0 2024-03-04T16:04:42.633 INFO:tasks.workunit.client.1.smithi152.stdout:8/696: write d2/d3/d15/d1c/d84/d87/f6f [4092245,35476] 0 2024-03-04T16:04:42.635 INFO:tasks.workunit.client.1.smithi152.stdout:8/697: truncate d2/d75/d89/f73 5058341 0 2024-03-04T16:04:42.645 INFO:tasks.workunit.client.1.smithi152.stdout:4/511: dread f5 [0,4194304] 0 2024-03-04T16:04:42.646 INFO:tasks.workunit.client.1.smithi152.stdout:4/512: chown d8/d9/d7a/f7d 1 1 2024-03-04T16:04:42.647 INFO:tasks.workunit.client.1.smithi152.stdout:6/697: dread - d8/d23/d3a/f8b zero size 2024-03-04T16:04:42.647 INFO:tasks.workunit.client.1.smithi152.stdout:6/698: fdatasync d8/d23/f5c 0 2024-03-04T16:04:42.651 INFO:tasks.workunit.client.1.smithi152.stdout:8/698: symlink d2/d3/d21/d2d/d52/d68/d6e/ldd 0 2024-03-04T16:04:42.654 INFO:tasks.workunit.client.1.smithi152.stdout:8/699: dread d2/d75/f9f [0,4194304] 0 2024-03-04T16:04:42.656 INFO:tasks.workunit.client.1.smithi152.stdout:4/513: mkdir d8/d2f/d6f/d90/daf 0 2024-03-04T16:04:42.657 INFO:tasks.workunit.client.1.smithi152.stdout:4/514: write d8/d9/d24/f4d [2304822,51545] 0 2024-03-04T16:04:42.658 INFO:tasks.workunit.client.1.smithi152.stdout:4/515: dread - d8/d2f/d6f/f73 zero size 2024-03-04T16:04:42.659 INFO:tasks.workunit.client.1.smithi152.stdout:4/516: chown d8/d9/d24/d26/d42/f81 55 1 2024-03-04T16:04:42.659 INFO:tasks.workunit.client.1.smithi152.stdout:6/699: symlink d8/d2a/lf2 0 2024-03-04T16:04:42.667 INFO:tasks.workunit.client.1.smithi152.stdout:0/506: dwrite f4 [0,4194304] 0 2024-03-04T16:04:42.669 INFO:tasks.workunit.client.1.smithi152.stdout:6/700: truncate d8/f12 1530399 0 2024-03-04T16:04:42.669 INFO:tasks.workunit.client.1.smithi152.stdout:0/507: stat d9/d1b/d3c/d84/d56 0 2024-03-04T16:04:42.669 INFO:tasks.workunit.client.1.smithi152.stdout:6/701: write d8/d92/d59/d7d/d91/ff0 [417819,19070] 0 2024-03-04T16:04:42.672 INFO:tasks.workunit.client.1.smithi152.stdout:8/700: mkdir d2/d3/d15/d1c/d84/dde 0 2024-03-04T16:04:42.677 INFO:tasks.workunit.client.1.smithi152.stdout:6/702: symlink d8/d92/d75/lf3 0 2024-03-04T16:04:42.678 INFO:tasks.workunit.client.1.smithi152.stdout:0/508: dwrite d9/d1b/d3c/d84/d56/d5b/f6b [0,4194304] 0 2024-03-04T16:04:42.679 INFO:tasks.workunit.client.1.smithi152.stdout:6/703: chown d8/d8f/db2/fc7 0 1 2024-03-04T16:04:42.680 INFO:tasks.workunit.client.1.smithi152.stdout:0/509: readlink d9/d1b/l27 0 2024-03-04T16:04:42.684 INFO:tasks.workunit.client.1.smithi152.stdout:8/701: mkdir d2/d75/d89/dd/d22/ddf 0 2024-03-04T16:04:42.690 INFO:tasks.workunit.client.1.smithi152.stdout:4/517: dwrite d8/d9/d24/d26/f32 [0,4194304] 0 2024-03-04T16:04:42.693 INFO:tasks.workunit.client.1.smithi152.stdout:0/510: creat d9/d2c/d2e/fa0 x:0 0 0 2024-03-04T16:04:42.695 INFO:tasks.workunit.client.1.smithi152.stdout:6/704: dwrite d8/d92/d59/d61/d57/feb [4194304,4194304] 0 2024-03-04T16:04:42.697 INFO:tasks.workunit.client.1.smithi152.stdout:0/511: creat d9/d2c/d2e/fa1 x:0 0 0 2024-03-04T16:04:42.700 INFO:tasks.workunit.client.1.smithi152.stdout:6/705: read d8/d2a/fa1 [766705,120420] 0 2024-03-04T16:04:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: pgmap v37: 65 pgs: 65 active+clean; 2.4 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 42 MiB/s rd, 160 MiB/s wr, 375 op/s 2024-03-04T16:04:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:04:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:04:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mgr fail", "who": "smithi152.aawieo"}]: dispatch 2024-03-04T16:04:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mgr fail", "who": "smithi152.aawieo"}]: dispatch 2024-03-04T16:04:42.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: osdmap e49: 6 total, 6 up, 6 in 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd='[{"prefix": "mgr fail", "who": "smithi152.aawieo"}]': finished 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: mgrmap e26: smithi116.jwdhop(active, starting, since 0.00589507s) 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi116.jwdhop", "id": "smithi116.jwdhop"}]: dispatch 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:04:42.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: Manager daemon smithi116.jwdhop is now available 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/trash_purge_schedule"}]: dispatch 2024-03-04T16:04:42.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:42 smithi116 ceph-mon[29120]: [04/Mar/2024:16:04:42] ENGINE Bus STARTING 2024-03-04T16:04:42.744 INFO:tasks.workunit.client.1.smithi152.stdout:6/706: dread - d8/d42/f51 zero size 2024-03-04T16:04:42.749 INFO:tasks.workunit.client.1.smithi152.stdout:8/702: getdents d2/d3/d15/d1c/d84/d87/d5d 0 2024-03-04T16:04:42.751 INFO:tasks.workunit.client.1.smithi152.stdout:8/703: write d2/d3/d21/f1b [667347,27717] 0 2024-03-04T16:04:42.751 INFO:tasks.workunit.client.1.smithi152.stdout:6/707: dwrite d8/d92/d59/fef [0,4194304] 0 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: pgmap v37: 65 pgs: 65 active+clean; 2.4 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 42 MiB/s rd, 160 MiB/s wr, 375 op/s 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.24387 172.21.15.152:0/3627442898' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mgr fail", "who": "smithi152.aawieo"}]: dispatch 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "mgr fail", "who": "smithi152.aawieo"}]: dispatch 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: osdmap e49: 6 total, 6 up, 6 in 2024-03-04T16:04:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.24387 ' entity='mgr.smithi152.aawieo' cmd='[{"prefix": "mgr fail", "who": "smithi152.aawieo"}]': finished 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: mgrmap e26: smithi116.jwdhop(active, starting, since 0.00589507s) 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi116.jwdhop", "id": "smithi116.jwdhop"}]: dispatch 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:04:42.766 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: Manager daemon smithi116.jwdhop is now available 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/trash_purge_schedule"}]: dispatch 2024-03-04T16:04:42.767 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:42 smithi152 ceph-mon[36876]: [04/Mar/2024:16:04:42] ENGINE Bus STARTING 2024-03-04T16:04:42.808 INFO:tasks.workunit.client.1.smithi152.stdout:1/597: sync 2024-03-04T16:04:42.808 INFO:tasks.workunit.client.1.smithi152.stdout:1/598: chown d19/d20/f9b 15653 1 2024-03-04T16:04:42.809 INFO:tasks.workunit.client.1.smithi152.stdout:0/512: dwrite d9/d1b/f6d [0,4194304] 0 2024-03-04T16:04:42.813 INFO:tasks.workunit.client.1.smithi152.stdout:8/704: creat d2/d3/d21/d2d/d52/d68/d6e/fe0 x:0 0 0 2024-03-04T16:04:42.816 INFO:tasks.workunit.client.1.smithi152.stdout:8/705: dread - d2/d3/d15/d1c/d84/d87/fd1 zero size 2024-03-04T16:04:42.816 INFO:tasks.workunit.client.1.smithi152.stdout:1/599: rename d19/c1e to d19/d4b/d53/d57/d62/cc5 0 2024-03-04T16:04:42.818 INFO:tasks.workunit.client.1.smithi152.stdout:6/708: dwrite d8/d10/d53/dd7/fec [0,4194304] 0 2024-03-04T16:04:42.820 INFO:tasks.workunit.client.1.smithi152.stdout:0/513: dread d9/d1b/d3c/d84/d56/d5b/f6b [0,4194304] 0 2024-03-04T16:04:42.826 INFO:tasks.workunit.client.1.smithi152.stdout:1/600: mkdir d19/d3b/d6e/dc6 0 2024-03-04T16:04:42.829 INFO:tasks.workunit.client.1.smithi152.stdout:0/514: fsync d9/d1b/d3c/d84/d56/d5b/f6b 0 2024-03-04T16:04:42.848 INFO:tasks.workunit.client.1.smithi152.stdout:6/709: dwrite d8/d92/d59/fea [8388608,4194304] 0 2024-03-04T16:04:42.851 INFO:tasks.workunit.client.1.smithi152.stdout:6/710: fdatasync f7 0 2024-03-04T16:04:42.852 INFO:tasks.workunit.client.1.smithi152.stdout:8/706: dwrite d2/d3/d21/d2d/d52/d68/d6e/d7d/f8d [0,4194304] 0 2024-03-04T16:04:42.854 INFO:tasks.workunit.client.1.smithi152.stdout:8/707: dread - d2/d3/d21/d2d/d48/db8/fb9 zero size 2024-03-04T16:04:42.858 INFO:tasks.workunit.client.1.smithi152.stdout:1/601: dread fa [0,4194304] 0 2024-03-04T16:04:42.859 INFO:tasks.workunit.client.1.smithi152.stdout:8/708: write d2/d3/d21/d2d/d52/d68/d6e/fe0 [1009165,43909] 0 2024-03-04T16:04:42.860 INFO:tasks.workunit.client.1.smithi152.stdout:6/711: mknod d8/cf4 0 2024-03-04T16:04:42.861 INFO:tasks.workunit.client.1.smithi152.stdout:0/515: dread d9/d1b/f23 [0,4194304] 0 2024-03-04T16:04:42.862 INFO:tasks.workunit.client.1.smithi152.stdout:6/712: fdatasync d8/d42/f4a 0 2024-03-04T16:04:42.866 INFO:tasks.workunit.client.1.smithi152.stdout:6/713: mknod d8/d92/d75/cf5 0 2024-03-04T16:04:42.868 INFO:tasks.workunit.client.1.smithi152.stdout:8/709: unlink d2/d75/d89/dd/d22/d6c/laf 0 2024-03-04T16:04:42.870 INFO:tasks.workunit.client.1.smithi152.stdout:0/516: creat d9/d92/fa2 x:0 0 0 2024-03-04T16:04:42.871 INFO:tasks.workunit.client.1.smithi152.stdout:0/517: write d9/d1a/f8a [664947,41813] 0 2024-03-04T16:04:42.872 INFO:tasks.workunit.client.1.smithi152.stdout:6/714: rename d8/d4b to d8/d23/d93/d99/df6 0 2024-03-04T16:04:42.880 INFO:tasks.workunit.client.1.smithi152.stdout:0/518: rmdir d9/d2c 39 2024-03-04T16:04:42.884 INFO:tasks.workunit.client.1.smithi152.stdout:0/519: dread d9/d1b/d3c/d84/d56/d5b/f6b [0,4194304] 0 2024-03-04T16:04:42.885 INFO:tasks.workunit.client.1.smithi152.stdout:6/715: mkdir d8/d23/d3a/d71/df7 0 2024-03-04T16:04:42.886 INFO:tasks.workunit.client.1.smithi152.stdout:0/520: write d9/d1b/f1d [5081458,39189] 0 2024-03-04T16:04:42.886 INFO:tasks.workunit.client.1.smithi152.stdout:6/716: stat d8/d23/d3a/ff1 0 2024-03-04T16:04:42.886 INFO:tasks.workunit.client.1.smithi152.stdout:1/602: link d19/d20/d2f/d6b/d38/d63/c9e d19/d4b/d53/d57/cc7 0 2024-03-04T16:04:42.888 INFO:tasks.workunit.client.1.smithi152.stdout:6/717: write d8/d92/d59/fef [4445972,73678] 0 2024-03-04T16:04:42.894 INFO:tasks.workunit.client.1.smithi152.stdout:1/603: mknod d19/d20/d2f/d6b/d38/cc8 0 2024-03-04T16:04:42.899 INFO:tasks.workunit.client.1.smithi152.stdout:0/521: mkdir d9/d2c/d2e/d7f/d9d/da3 0 2024-03-04T16:04:42.903 INFO:tasks.workunit.client.1.smithi152.stdout:6/718: truncate d8/d92/f5e 1392136 0 2024-03-04T16:04:42.904 INFO:tasks.workunit.client.1.smithi152.stdout:6/719: truncate d8/d10/f16 12831694 0 2024-03-04T16:04:42.910 INFO:tasks.workunit.client.1.smithi152.stdout:6/720: rename d8/d92/d75/f88 to d8/d23/d93/ff8 0 2024-03-04T16:04:42.910 INFO:tasks.workunit.client.1.smithi152.stdout:1/604: dwrite d19/f1d [0,4194304] 0 2024-03-04T16:04:42.911 INFO:tasks.workunit.client.1.smithi152.stdout:1/605: chown d19/d75 6852 1 2024-03-04T16:04:42.914 INFO:tasks.workunit.client.1.smithi152.stdout:6/721: symlink d8/d52/d58/lf9 0 2024-03-04T16:04:42.919 INFO:tasks.workunit.client.1.smithi152.stdout:6/722: creat d8/d10/ffa x:0 0 0 2024-03-04T16:04:42.920 INFO:tasks.workunit.client.1.smithi152.stdout:1/606: mknod d19/d3b/d6e/d97/cc9 0 2024-03-04T16:04:42.922 INFO:tasks.workunit.client.1.smithi152.stdout:1/607: truncate d19/d20/d2f/f76 1102692 0 2024-03-04T16:04:42.922 INFO:tasks.workunit.client.1.smithi152.stdout:6/723: mkdir d8/d23/dfb 0 2024-03-04T16:04:42.925 INFO:tasks.workunit.client.1.smithi152.stdout:6/724: creat d8/d92/d75/ffc x:0 0 0 2024-03-04T16:04:42.933 INFO:tasks.workunit.client.1.smithi152.stdout:1/608: getdents d19 0 2024-03-04T16:04:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.932+0000 7f3aaa7fc700 1 -- 172.21.15.116:0/1377867517 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 27) v1 ==== 45716+0+0 (secure 0 0 0) 0x7f3ab0023440 con 0x7f3ab4072a30 2024-03-04T16:04:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.932+0000 7f3aaa7fc700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f3aa003de50 0x7f3aa0040240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.932+0000 7f3aaa7fc700 1 -- 172.21.15.116:0/1377867517 --> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3aa0040780 con 0x7f3aa003de50 2024-03-04T16:04:42.934 INFO:tasks.workunit.client.1.smithi152.stdout:1/609: write d19/d20/d2f/d6b/f5a [730671,4022] 0 2024-03-04T16:04:42.936 INFO:tasks.workunit.client.1.smithi152.stdout:1/610: chown d19/d4b/d53/d5c/f6d 26 1 2024-03-04T16:04:42.938 INFO:tasks.workunit.client.1.smithi152.stdout:6/725: dwrite d8/d23/f25 [0,4194304] 0 2024-03-04T16:04:42.940 INFO:tasks.workunit.client.1.smithi152.stdout:6/726: truncate d8/d10/ffa 818581 0 2024-03-04T16:04:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.940+0000 7f3ab89c4700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f3aa003de50 0x7f3aa0040240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:42.941 INFO:tasks.workunit.client.1.smithi152.stdout:1/611: creat d19/fca x:0 0 0 2024-03-04T16:04:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.942+0000 7f3ab89c4700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f3aa003de50 0x7f3aa0040240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4006010 tx=0x7f3aa400b560 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:42.943 INFO:tasks.workunit.client.1.smithi152.stdout:6/727: chown d8/d52/l83 3064886 1 2024-03-04T16:04:42.944 INFO:tasks.workunit.client.1.smithi152.stdout:6/728: write d8/d23/fe9 [1026802,82207] 0 2024-03-04T16:04:42.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.944+0000 7f3aaa7fc700 1 -- 172.21.15.116:0/1377867517 <== mgr.14738 v2:172.21.15.116:6800/2966163948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f3aa0040780 con 0x7f3aa003de50 2024-03-04T16:04:42.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.949+0000 7f3abb429700 1 -- 172.21.15.116:0/1377867517 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f3aa003de50 msgr2=0x7f3aa0040240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:42.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.949+0000 7f3abb429700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f3aa003de50 0x7f3aa0040240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4006010 tx=0x7f3aa400b560 comp rx=0 tx=0).stop 2024-03-04T16:04:42.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.949+0000 7f3abb429700 1 -- 172.21.15.116:0/1377867517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab4072a30 msgr2=0x7f3ab410aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:42.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.949+0000 7f3abb429700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab4072a30 0x7f3ab410aa10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000d8d0 tx=0x7f3ab000dbe0 comp rx=0 tx=0).stop 2024-03-04T16:04:42.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.949+0000 7f3abb429700 1 -- 172.21.15.116:0/1377867517 shutdown_connections 2024-03-04T16:04:42.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.949+0000 7f3abb429700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab4072a30 0x7f3ab410aa10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:42.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.949+0000 7f3abb429700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f3aa003de50 0x7f3aa0040240 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:42.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.949+0000 7f3abb429700 1 --2- 172.21.15.116:0/1377867517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ab41082d0 0x7f3ab4109060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:42.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.949+0000 7f3abb429700 1 -- 172.21.15.116:0/1377867517 >> 172.21.15.116:0/1377867517 conn(0x7f3ab406d9f0 msgr2=0x7f3ab410d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:42.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.951+0000 7f3abb429700 1 -- 172.21.15.116:0/1377867517 shutdown_connections 2024-03-04T16:04:42.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:42.951+0000 7f3abb429700 1 -- 172.21.15.116:0/1377867517 wait complete. 2024-03-04T16:04:42.957 INFO:tasks.workunit.client.1.smithi152.stdout:1/612: write d19/d20/d2f/fa1 [3035997,76712] 0 2024-03-04T16:04:42.964 INFO:tasks.workunit.client.1.smithi152.stdout:6/729: truncate d8/f28 1787863 0 2024-03-04T16:04:42.965 INFO:tasks.workunit.client.1.smithi152.stdout:6/730: chown d8/d23/d3a/d8d/la5 265 1 2024-03-04T16:04:42.965 INFO:tasks.workunit.client.1.smithi152.stdout:1/613: dread - d19/d4b/d53/d57/d62/d82/fc2 zero size 2024-03-04T16:04:42.967 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:04:42.973 INFO:tasks.workunit.client.1.smithi152.stdout:1/614: dwrite d19/d20/d2f/d6b/f46 [0,4194304] 0 2024-03-04T16:04:42.975 INFO:tasks.workunit.client.1.smithi152.stdout:6/731: rename d8/d42/dde/c60 to d8/d10/d53/d9a/cfd 0 2024-03-04T16:04:42.979 INFO:tasks.workunit.client.1.smithi152.stdout:1/615: fsync d19/d20/d2f/f76 0 2024-03-04T16:04:42.981 INFO:tasks.workunit.client.1.smithi152.stdout:6/732: truncate d8/f1f 5169598 0 2024-03-04T16:04:42.990 INFO:tasks.workunit.client.1.smithi152.stdout:1/616: write d19/f5b [1931597,128839] 0 2024-03-04T16:04:42.991 INFO:tasks.workunit.client.1.smithi152.stdout:1/617: read - d19/fca zero size 2024-03-04T16:04:42.993 INFO:tasks.workunit.client.1.smithi152.stdout:6/733: getdents d8/d23/d3a/d71/df7 0 2024-03-04T16:04:43.004 INFO:tasks.workunit.client.1.smithi152.stdout:6/734: creat d8/d23/d3a/ffe x:0 0 0 2024-03-04T16:04:43.011 INFO:tasks.workunit.client.1.smithi152.stdout:0/522: dread d9/d2c/d2e/f6a [0,4194304] 0 2024-03-04T16:04:43.012 INFO:tasks.workunit.client.1.smithi152.stdout:8/710: dread d2/d75/d89/dd/d22/f3b [4194304,4194304] 0 2024-03-04T16:04:43.019 INFO:tasks.workunit.client.1.smithi152.stdout:0/523: mknod d9/d1b/d3c/d84/d56/d5b/ca4 0 2024-03-04T16:04:43.023 INFO:tasks.workunit.client.1.smithi152.stdout:8/711: mkdir d2/d75/d89/dd/d22/ddf/de1 0 2024-03-04T16:04:43.026 INFO:tasks.workunit.client.1.smithi152.stdout:0/524: unlink d9/d1b/d3c/d84/l5f 0 2024-03-04T16:04:43.029 INFO:tasks.workunit.client.1.smithi152.stdout:8/712: rename d2/d3/d21/d2d/d52/d68/d6e/fe0 to d2/d75/d89/fe2 0 2024-03-04T16:04:43.036 INFO:tasks.workunit.client.1.smithi152.stdout:8/713: rename d2/d75/d89/dd/l85 to d2/d3/d21/d2d/d52/le3 0 2024-03-04T16:04:43.038 INFO:tasks.workunit.client.1.smithi152.stdout:0/525: creat d9/d1b/d54/d69/d95/fa5 x:0 0 0 2024-03-04T16:04:43.041 INFO:tasks.workunit.client.1.smithi152.stdout:1/618: write d19/d20/d2f/d6b/d38/f3e [4385420,84966] 0 2024-03-04T16:04:43.043 INFO:tasks.workunit.client.1.smithi152.stdout:0/526: symlink d9/d2c/la6 0 2024-03-04T16:04:43.048 INFO:tasks.workunit.client.1.smithi152.stdout:8/714: mkdir d2/d3/d15/d1c/d84/dde/de4 0 2024-03-04T16:04:43.052 INFO:tasks.workunit.client.1.smithi152.stdout:8/715: dread d2/d3/d21/d2d/d52/d68/d6e/d7d/f8d [0,4194304] 0 2024-03-04T16:04:43.053 INFO:tasks.workunit.client.1.smithi152.stdout:8/716: chown d2/d3/d21/d2d/d52/dcc/dd6 16477804 1 2024-03-04T16:04:43.058 INFO:tasks.workunit.client.1.smithi152.stdout:8/717: stat d2/d3/l34 0 2024-03-04T16:04:43.059 INFO:tasks.workunit.client.1.smithi152.stdout:8/718: chown d2/d75/d89/f73 15 1 2024-03-04T16:04:43.062 INFO:tasks.workunit.client.1.smithi152.stdout:6/735: write d8/d2a/f79 [496920,39743] 0 2024-03-04T16:04:43.064 INFO:tasks.workunit.client.1.smithi152.stdout:2/564: sync 2024-03-04T16:04:43.073 INFO:tasks.workunit.client.1.smithi152.stdout:6/736: unlink d8/d42/dde/lbf 0 2024-03-04T16:04:43.073 INFO:tasks.workunit.client.1.smithi152.stdout:6/737: readlink d8/lf 0 2024-03-04T16:04:43.075 INFO:tasks.workunit.client.1.smithi152.stdout:8/719: rename d2/d75/d89/c25 to d2/d3/d21/ce5 0 2024-03-04T16:04:43.077 INFO:tasks.workunit.client.1.smithi152.stdout:6/738: symlink d8/da9/lff 0 2024-03-04T16:04:43.078 INFO:tasks.workunit.client.1.smithi152.stdout:1/619: truncate d19/d20/f41 1924171 0 2024-03-04T16:04:43.079 INFO:tasks.workunit.client.1.smithi152.stdout:8/720: unlink d2/d3/d15/cc5 0 2024-03-04T16:04:43.081 INFO:tasks.workunit.client.1.smithi152.stdout:8/721: dread - d2/d3/fcf zero size 2024-03-04T16:04:43.081 INFO:tasks.workunit.client.1.smithi152.stdout:2/565: creat da/d5f/fbf x:0 0 0 2024-03-04T16:04:43.083 INFO:tasks.workunit.client.1.smithi152.stdout:2/566: dread - da/d2c/d5e/f69 zero size 2024-03-04T16:04:43.087 INFO:tasks.workunit.client.1.smithi152.stdout:0/527: dwrite d9/d2c/f2f [0,4194304] 0 2024-03-04T16:04:43.088 INFO:tasks.workunit.client.1.smithi152.stdout:8/722: mknod d2/d75/d89/dd/d22/d83/d96/ce6 0 2024-03-04T16:04:43.089 INFO:tasks.workunit.client.1.smithi152.stdout:0/528: stat d9/d1b/f71 0 2024-03-04T16:04:43.093 INFO:tasks.workunit.client.1.smithi152.stdout:2/567: mknod da/df/d2e/d51/d8d/d9a/cc0 0 2024-03-04T16:04:43.096 INFO:tasks.workunit.client.1.smithi152.stdout:8/723: mkdir d2/d3/d21/d2d/d52/dcc/de7 0 2024-03-04T16:04:43.098 INFO:tasks.workunit.client.1.smithi152.stdout:8/724: chown d2/d7f/f98 13 1 2024-03-04T16:04:43.098 INFO:tasks.workunit.client.1.smithi152.stdout:1/620: dread d19/d20/d2f/d6b/f2d [0,4194304] 0 2024-03-04T16:04:43.099 INFO:tasks.workunit.client.1.smithi152.stdout:8/725: read d2/d7f/f98 [723759,66159] 0 2024-03-04T16:04:43.100 INFO:tasks.workunit.client.1.smithi152.stdout:0/529: write d9/d1b/f35 [741349,116483] 0 2024-03-04T16:04:43.103 INFO:tasks.workunit.client.1.smithi152.stdout:0/530: chown d9/d1b/d3c/d84/d56/d5b 22 1 2024-03-04T16:04:43.103 INFO:tasks.workunit.client.1.smithi152.stdout:1/621: dread d19/d20/f9b [0,4194304] 0 2024-03-04T16:04:43.104 INFO:tasks.workunit.client.1.smithi152.stdout:3/615: sync 2024-03-04T16:04:43.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.108+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/3863051875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc1010d0 msgr2=0x7fe1dc1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:43.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.108+0000 7fe1e3bdb700 1 --2- 172.21.15.116:0/3863051875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc1010d0 0x7fe1dc1014f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009910 tx=0x7fe1d8009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:43.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.108+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/3863051875 shutdown_connections 2024-03-04T16:04:43.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.108+0000 7fe1e3bdb700 1 --2- 172.21.15.116:0/3863051875 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc0ffd30 0x7fe1dc1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.108+0000 7fe1e3bdb700 1 --2- 172.21.15.116:0/3863051875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc1010d0 0x7fe1dc1014f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.108+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/3863051875 >> 172.21.15.116:0/3863051875 conn(0x7fe1dc0fa7c0 msgr2=0x7fe1dc0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:43.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.108+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/3863051875 shutdown_connections 2024-03-04T16:04:43.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.108+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/3863051875 wait complete. 2024-03-04T16:04:43.110 INFO:tasks.workunit.client.1.smithi152.stdout:6/739: dwrite d8/d23/d3a/d6d/f7a [0,4194304] 0 2024-03-04T16:04:43.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.109+0000 7fe1e3bdb700 1 Processor -- start 2024-03-04T16:04:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.109+0000 7fe1e3bdb700 1 -- start start 2024-03-04T16:04:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.109+0000 7fe1e3bdb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc0ffd30 0x7fe1dc103ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.109+0000 7fe1e3bdb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc1010d0 0x7fe1dc102120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.109+0000 7fe1e3bdb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc102660 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.109+0000 7fe1e3bdb700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc1027a0 con 0x7fe1dc1010d0 2024-03-04T16:04:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.109+0000 7fe1e1977700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc0ffd30 0x7fe1dc103ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.109+0000 7fe1e1977700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc0ffd30 0x7fe1dc103ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53772/0 (socket says 172.21.15.116:53772) 2024-03-04T16:04:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.109+0000 7fe1e1977700 1 -- 172.21.15.116:0/2430672950 learned_addr learned my addr 172.21.15.116:0/2430672950 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.109+0000 7fe1e1176700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc1010d0 0x7fe1dc102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.110+0000 7fe1e1977700 1 -- 172.21.15.116:0/2430672950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc1010d0 msgr2=0x7fe1dc102120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.110+0000 7fe1e1977700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc1010d0 0x7fe1dc102120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.110+0000 7fe1e1977700 1 -- 172.21.15.116:0/2430672950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d80095f0 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.110+0000 7fe1e1977700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc0ffd30 0x7fe1dc103ad0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8005b40 tx=0x7fe1d800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.110+0000 7fe1d2ffd700 1 -- 172.21.15.116:0/2430672950 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1d801d070 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.110+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/2430672950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc102a20 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.110+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/2430672950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc102f10 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.110+0000 7fe1d2ffd700 1 -- 172.21.15.116:0/2430672950 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe1d8005dc0 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.110+0000 7fe1d2ffd700 1 -- 172.21.15.116:0/2430672950 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1d8022470 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.114 INFO:tasks.workunit.client.1.smithi152.stdout:8/726: rename d2/d75/d89/lbe to d2/d75/d89/le8 0 2024-03-04T16:04:43.114 INFO:tasks.workunit.client.1.smithi152.stdout:2/568: mknod da/d24/cc1 0 2024-03-04T16:04:43.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.111+0000 7fe1d2ffd700 1 -- 172.21.15.116:0/2430672950 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 45716+0+0 (secure 0 0 0) 0x7fe1d80225d0 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.112+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/2430672950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c0005320 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.112+0000 7fe1d2ffd700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fe1c8038ab0 0x7fe1c803af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:43.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.112+0000 7fe1d2ffd700 1 -- 172.21.15.116:0/2430672950 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fe1d804df50 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.112+0000 7fe1e1176700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fe1c8038ab0 0x7fe1c803af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:43.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.113+0000 7fe1e1176700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fe1c8038ab0 0x7fe1c803af70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009c60 tx=0x7fe1cc009260 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:43.116 INFO:tasks.workunit.client.1.smithi152.stdout:0/531: creat d9/d2c/d2e/d7f/d9d/fa7 x:0 0 0 2024-03-04T16:04:43.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.119+0000 7fe1d2ffd700 1 -- 172.21.15.116:0/2430672950 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fe1d8020de0 con 0x7fe1dc0ffd30 2024-03-04T16:04:43.120 INFO:tasks.workunit.client.1.smithi152.stdout:1/622: truncate d19/d20/d2f/d6b/f37 4654893 0 2024-03-04T16:04:43.127 INFO:tasks.workunit.client.1.smithi152.stdout:6/740: mknod d8/d8f/da3/c100 0 2024-03-04T16:04:43.128 INFO:tasks.workunit.client.1.smithi152.stdout:2/569: rmdir da/df/d2e 39 2024-03-04T16:04:43.130 INFO:tasks.workunit.client.1.smithi152.stdout:3/616: dwrite d6/d14/d58/f7e [0,4194304] 0 2024-03-04T16:04:43.130 INFO:tasks.workunit.client.1.smithi152.stdout:6/741: write d8/d23/d3a/d6d/f7a [1841147,102656] 0 2024-03-04T16:04:43.132 INFO:tasks.workunit.client.1.smithi152.stdout:3/617: dread - d6/d10/d2d/f9e zero size 2024-03-04T16:04:43.134 INFO:tasks.workunit.client.1.smithi152.stdout:1/623: dread - d19/d20/d2f/d6b/f40 zero size 2024-03-04T16:04:43.145 INFO:tasks.workunit.client.1.smithi152.stdout:3/618: dread d6/f1a [0,4194304] 0 2024-03-04T16:04:43.165 INFO:tasks.workunit.client.1.smithi152.stdout:3/619: dread d6/d10/f27 [4194304,4194304] 0 2024-03-04T16:04:43.165 INFO:tasks.workunit.client.1.smithi152.stdout:3/620: chown d6/d10/d2d 3728099 1 2024-03-04T16:04:43.174 INFO:tasks.workunit.client.1.smithi152.stdout:8/727: chown d2/d3/d15/fc1 1 1 2024-03-04T16:04:43.178 INFO:tasks.workunit.client.1.smithi152.stdout:6/742: unlink d8/d10/f16 0 2024-03-04T16:04:43.179 INFO:tasks.workunit.client.1.smithi152.stdout:6/743: chown d8/f9 2 1 2024-03-04T16:04:43.182 INFO:tasks.workunit.client.1.smithi152.stdout:0/532: unlink f7 0 2024-03-04T16:04:43.185 INFO:tasks.workunit.client.1.smithi152.stdout:5/571: sync 2024-03-04T16:04:43.185 INFO:tasks.workunit.client.1.smithi152.stdout:9/550: sync 2024-03-04T16:04:43.185 INFO:tasks.workunit.client.1.smithi152.stdout:4/518: sync 2024-03-04T16:04:43.186 INFO:tasks.workunit.client.1.smithi152.stdout:7/482: sync 2024-03-04T16:04:43.187 INFO:tasks.workunit.client.1.smithi152.stdout:7/483: fsync d14/f78 0 2024-03-04T16:04:43.187 INFO:tasks.workunit.client.1.smithi152.stdout:6/744: dwrite d8/d92/db1/fd5 [0,4194304] 0 2024-03-04T16:04:43.189 INFO:tasks.workunit.client.1.smithi152.stdout:4/519: dread - d8/d2f/d6f/d84/f8e zero size 2024-03-04T16:04:43.191 INFO:tasks.workunit.client.1.smithi152.stdout:2/570: mknod da/df/d2e/d51/cc2 0 2024-03-04T16:04:43.192 INFO:tasks.workunit.client.1.smithi152.stdout:2/571: stat da/d2c/d5e/l6f 0 2024-03-04T16:04:43.193 INFO:tasks.workunit.client.1.smithi152.stdout:2/572: fdatasync da/f3c 0 2024-03-04T16:04:43.198 INFO:tasks.workunit.client.1.smithi152.stdout:8/728: write d2/d3/d15/f4f [977405,79036] 0 2024-03-04T16:04:43.200 INFO:tasks.workunit.client.1.smithi152.stdout:0/533: dread d9/f26 [0,4194304] 0 2024-03-04T16:04:43.209 INFO:tasks.workunit.client.1.smithi152.stdout:3/621: rename d6/d14/l80 to d6/d14/d93/ld3 0 2024-03-04T16:04:43.210 INFO:tasks.workunit.client.1.smithi152.stdout:3/622: chown d6/d10/c3c 744 1 2024-03-04T16:04:43.218 INFO:tasks.workunit.client.1.smithi152.stdout:2/573: symlink da/d24/d47/d73/lc3 0 2024-03-04T16:04:43.223 INFO:tasks.workunit.client.1.smithi152.stdout:0/534: mkdir d9/d1b/d54/d80/da8 0 2024-03-04T16:04:43.227 INFO:tasks.workunit.client.1.smithi152.stdout:4/520: dread d8/d9/d24/f4d [0,4194304] 0 2024-03-04T16:04:43.229 INFO:tasks.workunit.client.1.smithi152.stdout:1/624: write d19/d75/f83 [1669320,27582] 0 2024-03-04T16:04:43.230 INFO:tasks.workunit.client.1.smithi152.stdout:9/551: rename d2/d13/d35/c9c to d2/d13/d23/d42/cb6 0 2024-03-04T16:04:43.231 INFO:tasks.workunit.client.1.smithi152.stdout:8/729: dwrite d2/d75/d89/dd/d22/d6c/f74 [0,4194304] 0 2024-03-04T16:04:43.235 INFO:tasks.workunit.client.1.smithi152.stdout:6/745: unlink d8/d92/f5e 0 2024-03-04T16:04:43.240 INFO:tasks.workunit.client.1.smithi152.stdout:5/572: symlink d6/d8/d1f/d3d/dc0/lc5 0 2024-03-04T16:04:43.249 INFO:tasks.workunit.client.1.smithi152.stdout:0/535: creat d9/d1b/d3c/fa9 x:0 0 0 2024-03-04T16:04:43.250 INFO:tasks.workunit.client.1.smithi152.stdout:0/536: read d9/d2c/f2f [3010728,31140] 0 2024-03-04T16:04:43.250 INFO:tasks.workunit.client.1.smithi152.stdout:7/484: rmdir d14/d2b/d86 0 2024-03-04T16:04:43.252 INFO:tasks.workunit.client.1.smithi152.stdout:0/537: chown d9/d1b/d3c/c78 26750037 1 2024-03-04T16:04:43.253 INFO:tasks.workunit.client.1.smithi152.stdout:4/521: rename d8/d9/d7a/c54 to d8/d9/da2/cb0 0 2024-03-04T16:04:43.254 INFO:tasks.workunit.client.1.smithi152.stdout:0/538: readlink d9/d2c/d2e/l4e 0 2024-03-04T16:04:43.254 INFO:tasks.workunit.client.1.smithi152.stdout:8/730: chown d2/d3/d15/d1c/d84 6 1 2024-03-04T16:04:43.256 INFO:tasks.workunit.client.1.smithi152.stdout:4/522: write d8/d78/f95 [747633,6217] 0 2024-03-04T16:04:43.262 INFO:tasks.workunit.client.1.smithi152.stdout:7/485: rmdir d14 39 2024-03-04T16:04:43.265 INFO:tasks.workunit.client.1.smithi152.stdout:9/552: rename d2/d13/d38/c51 to d2/d13/d35/cb7 0 2024-03-04T16:04:43.268 INFO:tasks.workunit.client.1.smithi152.stdout:3/623: creat d6/fd4 x:0 0 0 2024-03-04T16:04:43.273 INFO:tasks.workunit.client.1.smithi152.stdout:2/574: dread da/f16 [0,4194304] 0 2024-03-04T16:04:43.274 INFO:tasks.workunit.client.1.smithi152.stdout:4/523: dwrite d8/f23 [4194304,4194304] 0 2024-03-04T16:04:43.275 INFO:tasks.workunit.client.1.smithi152.stdout:8/731: dwrite d2/d3/d15/d1c/d84/d87/f6f [4194304,4194304] 0 2024-03-04T16:04:43.277 INFO:tasks.workunit.client.1.smithi152.stdout:2/575: fdatasync da/df/fae 0 2024-03-04T16:04:43.278 INFO:tasks.workunit.client.1.smithi152.stdout:2/576: chown da/df/d2e/f30 0 1 2024-03-04T16:04:43.283 INFO:tasks.workunit.client.1.smithi152.stdout:5/573: dread d6/d8/f15 [0,4194304] 0 2024-03-04T16:04:43.285 INFO:tasks.workunit.client.1.smithi152.stdout:5/574: chown d6/d8/d1f/d3d/l6b 66348100 1 2024-03-04T16:04:43.286 INFO:tasks.workunit.client.1.smithi152.stdout:6/746: link d8/d42/cce d8/d23/c101 0 2024-03-04T16:04:43.291 INFO:tasks.workunit.client.1.smithi152.stdout:0/539: dwrite d9/f45 [0,4194304] 0 2024-03-04T16:04:43.297 INFO:tasks.workunit.client.1.smithi152.stdout:1/625: link d19/d20/f99 d19/d20/fcb 0 2024-03-04T16:04:43.298 INFO:tasks.workunit.client.1.smithi152.stdout:7/486: truncate d14/d21/d31/f57 1078919 0 2024-03-04T16:04:43.301 INFO:tasks.workunit.client.1.smithi152.stdout:9/553: creat d2/d13/d23/d66/d7d/da7/fb8 x:0 0 0 2024-03-04T16:04:43.307 INFO:tasks.workunit.client.1.smithi152.stdout:4/524: creat d8/d2f/d7c/fb1 x:0 0 0 2024-03-04T16:04:43.308 INFO:tasks.workunit.client.1.smithi152.stdout:4/525: stat d8/d2f/d6f/d84/f8e 0 2024-03-04T16:04:43.309 INFO:tasks.workunit.client.1.smithi152.stdout:1/626: readlink d19/d20/d2f/d6b/d38/l7b 0 2024-03-04T16:04:43.309 INFO:tasks.workunit.client.1.smithi152.stdout:7/487: rmdir d14 39 2024-03-04T16:04:43.311 INFO:tasks.workunit.client.1.smithi152.stdout:9/554: creat d2/d13/d23/d66/d7d/fb9 x:0 0 0 2024-03-04T16:04:43.314 INFO:tasks.workunit.client.1.smithi152.stdout:3/624: write d6/d10/f27 [9181125,59370] 0 2024-03-04T16:04:43.314 INFO:tasks.workunit.client.1.smithi152.stdout:6/747: write d8/d92/d59/d7d/d91/fcd [621566,22818] 0 2024-03-04T16:04:43.316 INFO:tasks.workunit.client.1.smithi152.stdout:8/732: write d2/d75/d89/f14 [505234,61434] 0 2024-03-04T16:04:43.317 INFO:tasks.workunit.client.1.smithi152.stdout:2/577: creat da/fc4 x:0 0 0 2024-03-04T16:04:43.317 INFO:tasks.workunit.client.1.smithi152.stdout:4/526: fsync f3 0 2024-03-04T16:04:43.321 INFO:tasks.workunit.client.1.smithi152.stdout:4/527: write d8/d9/d7a/f8f [184961,74594] 0 2024-03-04T16:04:43.322 INFO:tasks.workunit.client.1.smithi152.stdout:4/528: fdatasync d8/d9/f9e 0 2024-03-04T16:04:43.328 INFO:tasks.workunit.client.1.smithi152.stdout:3/625: fsync d6/d60/d8e/dc0/f72 0 2024-03-04T16:04:43.329 INFO:tasks.workunit.client.1.smithi152.stdout:8/733: mknod d2/d3/d21/d2d/d52/dcc/ce9 0 2024-03-04T16:04:43.331 INFO:tasks.workunit.client.1.smithi152.stdout:2/578: dread da/d2c/d3f/f72 [0,4194304] 0 2024-03-04T16:04:43.332 INFO:tasks.workunit.client.1.smithi152.stdout:2/579: read - da/df/d2e/d61/d8e/f93 zero size 2024-03-04T16:04:43.335 INFO:tasks.workunit.client.1.smithi152.stdout:6/748: truncate d8/d23/f33 1227041 0 2024-03-04T16:04:43.337 INFO:tasks.workunit.client.1.smithi152.stdout:7/488: dread - d14/d21/d47/d36/f74 zero size 2024-03-04T16:04:43.340 INFO:tasks.workunit.client.1.smithi152.stdout:5/575: getdents d6/d8/d1f/d3d/d6f/d70/d98 0 2024-03-04T16:04:43.341 INFO:tasks.workunit.client.1.smithi152.stdout:3/626: mknod d6/d10/cd5 0 2024-03-04T16:04:43.342 INFO:tasks.workunit.client.1.smithi152.stdout:3/627: write d6/d14/fc4 [544419,104828] 0 2024-03-04T16:04:43.343 INFO:tasks.workunit.client.1.smithi152.stdout:3/628: truncate d6/d60/fb8 918186 0 2024-03-04T16:04:43.344 INFO:tasks.workunit.client.1.smithi152.stdout:2/580: unlink da/d24/d47/c52 0 2024-03-04T16:04:43.345 INFO:tasks.workunit.client.1.smithi152.stdout:1/627: unlink d19/d4b/d53/d57/d85/lb7 0 2024-03-04T16:04:43.346 INFO:tasks.workunit.client.1.smithi152.stdout:6/749: symlink d8/d2a/l102 0 2024-03-04T16:04:43.347 INFO:tasks.workunit.client.1.smithi152.stdout:9/555: link d2/d13/d2e/c62 d2/d13/d96/cba 0 2024-03-04T16:04:43.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.386+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/2430672950 --> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1c0000bf0 con 0x7fe1c8038ab0 2024-03-04T16:04:43.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.387+0000 7fe1d2ffd700 1 -- 172.21.15.116:0/2430672950 <== mgr.14738 v2:172.21.15.116:6800/2966163948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7fe1c0000bf0 con 0x7fe1c8038ab0 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.389+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/2430672950 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fe1c8038ab0 msgr2=0x7fe1c803af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.389+0000 7fe1e3bdb700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fe1c8038ab0 0x7fe1c803af70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009c60 tx=0x7fe1cc009260 comp rx=0 tx=0).stop 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.389+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/2430672950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc0ffd30 msgr2=0x7fe1dc103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.389+0000 7fe1e3bdb700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc0ffd30 0x7fe1dc103ad0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8005b40 tx=0x7fe1d800fa00 comp rx=0 tx=0).stop 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.390+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/2430672950 shutdown_connections 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.390+0000 7fe1e3bdb700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc0ffd30 0x7fe1dc103ad0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.390+0000 7fe1e3bdb700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fe1c8038ab0 0x7fe1c803af70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.390+0000 7fe1e3bdb700 1 --2- 172.21.15.116:0/2430672950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc1010d0 0x7fe1dc102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.390+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/2430672950 >> 172.21.15.116:0/2430672950 conn(0x7fe1dc0fa7c0 msgr2=0x7fe1dc0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.390+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/2430672950 shutdown_connections 2024-03-04T16:04:43.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.390+0000 7fe1e3bdb700 1 -- 172.21.15.116:0/2430672950 wait complete. 2024-03-04T16:04:43.407 INFO:tasks.workunit.client.1.smithi152.stdout:7/489: unlink d14/d69/f6d 0 2024-03-04T16:04:43.408 INFO:tasks.workunit.client.1.smithi152.stdout:6/750: dwrite d8/f9 [0,4194304] 0 2024-03-04T16:04:43.408 INFO:tasks.workunit.client.1.smithi152.stdout:8/734: symlink d2/d3/d15/d1c/d84/dde/de4/lea 0 2024-03-04T16:04:43.410 INFO:tasks.workunit.client.1.smithi152.stdout:3/629: rmdir d6/d5c 39 2024-03-04T16:04:43.413 INFO:tasks.workunit.client.1.smithi152.stdout:5/576: dread d6/d50/f83 [0,4194304] 0 2024-03-04T16:04:43.414 INFO:tasks.workunit.client.1.smithi152.stdout:2/581: truncate da/d24/d80/d82/da8/f7d 1548749 0 2024-03-04T16:04:43.415 INFO:tasks.workunit.client.1.smithi152.stdout:5/577: chown d6/c2a 13466037 1 2024-03-04T16:04:43.419 INFO:tasks.workunit.client.1.smithi152.stdout:9/556: write d2/fd [280053,56303] 0 2024-03-04T16:04:43.422 INFO:tasks.workunit.client.1.smithi152.stdout:4/529: creat d8/d9/fb2 x:0 0 0 2024-03-04T16:04:43.426 INFO:tasks.workunit.client.1.smithi152.stdout:6/751: unlink d8/d10/c1a 0 2024-03-04T16:04:43.427 INFO:tasks.workunit.client.1.smithi152.stdout:9/557: dwrite d2/d63/f95 [0,4194304] 0 2024-03-04T16:04:43.433 INFO:tasks.workunit.client.1.smithi152.stdout:8/735: mkdir d2/d3/d21/d2d/d52/deb 0 2024-03-04T16:04:43.439 INFO:tasks.workunit.client.1.smithi152.stdout:5/578: creat d6/d8/d5a/d90/daf/fc6 x:0 0 0 2024-03-04T16:04:43.444 INFO:tasks.workunit.client.1.smithi152.stdout:7/490: dread d14/d2b/f4e [0,4194304] 0 2024-03-04T16:04:43.444 INFO:tasks.workunit.client.1.smithi152.stdout:4/530: rmdir d8/d2f/d67 39 2024-03-04T16:04:43.445 INFO:tasks.workunit.client.1.smithi152.stdout:1/628: dwrite d19/d20/f99 [0,4194304] 0 2024-03-04T16:04:43.447 INFO:tasks.workunit.client.1.smithi152.stdout:6/752: mknod d8/d23/d3a/d6d/c103 0 2024-03-04T16:04:43.451 INFO:tasks.workunit.client.1.smithi152.stdout:5/579: mkdir d6/d8/d5a/d64/d76/d9f/dc7 0 2024-03-04T16:04:43.452 INFO:tasks.workunit.client.1.smithi152.stdout:5/580: chown d6/d8/fc3 3195 1 2024-03-04T16:04:43.456 INFO:tasks.workunit.client.1.smithi152.stdout:4/531: rmdir d8/d9/d7a 39 2024-03-04T16:04:43.458 INFO:tasks.workunit.client.1.smithi152.stdout:7/491: fdatasync d14/d21/d47/f5e 0 2024-03-04T16:04:43.465 INFO:tasks.workunit.client.1.smithi152.stdout:6/753: mkdir d8/d92/d59/d7d/d104 0 2024-03-04T16:04:43.466 INFO:tasks.workunit.client.1.smithi152.stdout:6/754: write d8/d23/d3a/d71/d9d/fa8 [4699831,50576] 0 2024-03-04T16:04:43.469 INFO:tasks.workunit.client.1.smithi152.stdout:8/736: mknod d2/d75/d89/dd/d22/cec 0 2024-03-04T16:04:43.472 INFO:tasks.workunit.client.1.smithi152.stdout:3/630: write d6/d14/f90 [1794880,47667] 0 2024-03-04T16:04:43.473 INFO:tasks.workunit.client.1.smithi152.stdout:4/532: chown d8/f45 3 1 2024-03-04T16:04:43.480 INFO:tasks.workunit.client.1.smithi152.stdout:2/582: dwrite da/df/d2e/d61/f84 [0,4194304] 0 2024-03-04T16:04:43.480 INFO:tasks.workunit.client.1.smithi152.stdout:2/583: readlink da/df/d58/l59 0 2024-03-04T16:04:43.482 INFO:tasks.workunit.client.1.smithi152.stdout:9/558: link d2/d13/d35/f34 d2/d13/d23/d71/fbb 0 2024-03-04T16:04:43.487 INFO:tasks.workunit.client.1.smithi152.stdout:6/755: creat d8/d92/d75/f105 x:0 0 0 2024-03-04T16:04:43.496 INFO:tasks.workunit.client.1.smithi152.stdout:5/581: dwrite d6/d8/d5a/d64/d8a/fb9 [4194304,4194304] 0 2024-03-04T16:04:43.499 INFO:tasks.workunit.client.1.smithi152.stdout:3/631: dwrite d6/d5c/f86 [0,4194304] 0 2024-03-04T16:04:43.504 INFO:tasks.workunit.client.1.smithi152.stdout:3/632: truncate d6/d60/d8e/fbd 382280 0 2024-03-04T16:04:43.505 INFO:tasks.workunit.client.1.smithi152.stdout:3/633: fsync d6/d60/d8a/fc8 0 2024-03-04T16:04:43.508 INFO:tasks.workunit.client.1.smithi152.stdout:5/582: dread d6/d8/d1f/d3d/d6f/d70/fa8 [0,4194304] 0 2024-03-04T16:04:43.513 INFO:tasks.workunit.client.1.smithi152.stdout:7/492: truncate d14/f15 1204657 0 2024-03-04T16:04:43.516 INFO:tasks.workunit.client.1.smithi152.stdout:4/533: write d8/d3d/f44 [186140,103964] 0 2024-03-04T16:04:43.519 INFO:tasks.workunit.client.1.smithi152.stdout:9/559: rename d2/d6e/f9b to d2/d13/d23/d71/da3/fbc 0 2024-03-04T16:04:43.520 INFO:tasks.workunit.client.1.smithi152.stdout:9/560: dread - d2/d13/d35/f7f zero size 2024-03-04T16:04:43.521 INFO:tasks.workunit.client.1.smithi152.stdout:6/756: creat d8/d8f/db2/f106 x:0 0 0 2024-03-04T16:04:43.521 INFO:tasks.workunit.client.1.smithi152.stdout:9/561: fdatasync d2/d13/d2e/f76 0 2024-03-04T16:04:43.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.525+0000 7f9b980c0700 1 -- 172.21.15.116:0/1884740757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b900ffab0 msgr2=0x7f9b900ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:43.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.525+0000 7f9b980c0700 1 --2- 172.21.15.116:0/1884740757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b900ffab0 0x7f9b900ffed0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009910 tx=0x7f9b8c009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.525+0000 7f9b980c0700 1 -- 172.21.15.116:0/1884740757 shutdown_connections 2024-03-04T16:04:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.525+0000 7f9b980c0700 1 --2- 172.21.15.116:0/1884740757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b90100cb0 0x7f9b90101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.525+0000 7f9b980c0700 1 --2- 172.21.15.116:0/1884740757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b900ffab0 0x7f9b900ffed0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.525+0000 7f9b980c0700 1 -- 172.21.15.116:0/1884740757 >> 172.21.15.116:0/1884740757 conn(0x7f9b900fb030 msgr2=0x7f9b900fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.526+0000 7f9b980c0700 1 -- 172.21.15.116:0/1884740757 shutdown_connections 2024-03-04T16:04:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.526+0000 7f9b980c0700 1 -- 172.21.15.116:0/1884740757 wait complete. 2024-03-04T16:04:43.528 INFO:tasks.workunit.client.1.smithi152.stdout:3/634: mknod d6/d14/d41/d49/d84/cd6 0 2024-03-04T16:04:43.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.526+0000 7f9b980c0700 1 Processor -- start 2024-03-04T16:04:43.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.526+0000 7f9b980c0700 1 -- start start 2024-03-04T16:04:43.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b980c0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b900ffab0 0x7f9b90194120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:43.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b980c0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b90100cb0 0x7f9b90194660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b980c0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b90194c80 con 0x7f9b90100cb0 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b980c0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b90194dc0 con 0x7f9b900ffab0 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b9565b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b90100cb0 0x7f9b90194660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b95e5c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b900ffab0 0x7f9b90194120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b95e5c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b900ffab0 0x7f9b90194120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41514/0 (socket says 172.21.15.116:41514) 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b9565b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b90100cb0 0x7f9b90194660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53792/0 (socket says 172.21.15.116:53792) 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b95e5c700 1 -- 172.21.15.116:0/2987241247 learned_addr learned my addr 172.21.15.116:0/2987241247 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b95e5c700 1 -- 172.21.15.116:0/2987241247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b90100cb0 msgr2=0x7f9b90194660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b95e5c700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b90100cb0 0x7f9b90194660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b95e5c700 1 -- 172.21.15.116:0/2987241247 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c0095f0 con 0x7f9b900ffab0 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b9565b700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b90100cb0 0x7f9b90194660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.527+0000 7f9b95e5c700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b900ffab0 0x7f9b90194120 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c006010 tx=0x7f9b8c0053f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.528+0000 7f9b86ffd700 1 -- 172.21.15.116:0/2987241247 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b8c0041f0 con 0x7f9b900ffab0 2024-03-04T16:04:43.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.528+0000 7f9b86ffd700 1 -- 172.21.15.116:0/2987241247 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9b8c0058e0 con 0x7f9b900ffab0 2024-03-04T16:04:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.528+0000 7f9b86ffd700 1 -- 172.21.15.116:0/2987241247 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b8c003c80 con 0x7f9b900ffab0 2024-03-04T16:04:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.528+0000 7f9b980c0700 1 -- 172.21.15.116:0/2987241247 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b90199810 con 0x7f9b900ffab0 2024-03-04T16:04:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.528+0000 7f9b980c0700 1 -- 172.21.15.116:0/2987241247 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b90199d80 con 0x7f9b900ffab0 2024-03-04T16:04:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.529+0000 7f9b86ffd700 1 -- 172.21.15.116:0/2987241247 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 45716+0+0 (secure 0 0 0) 0x7f9b8c00d900 con 0x7f9b900ffab0 2024-03-04T16:04:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.529+0000 7f9b980c0700 1 -- 172.21.15.116:0/2987241247 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b90107500 con 0x7f9b900ffab0 2024-03-04T16:04:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.529+0000 7f9b86ffd700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f9b7c038af0 0x7f9b7c03afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.529+0000 7f9b86ffd700 1 -- 172.21.15.116:0/2987241247 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f9b8c01a040 con 0x7f9b900ffab0 2024-03-04T16:04:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.529+0000 7f9b9565b700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f9b7c038af0 0x7f9b7c03afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.530+0000 7f9b9565b700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f9b7c038af0 0x7f9b7c03afb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9b800096d0 tx=0x7f9b80009260 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:43.533 INFO:tasks.workunit.client.1.smithi152.stdout:2/584: mknod da/d5f/cc5 0 2024-03-04T16:04:43.533 INFO:tasks.workunit.client.1.smithi152.stdout:1/629: getdents d19/d20/d2f/d5d 0 2024-03-04T16:04:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.535+0000 7f9b86ffd700 1 -- 172.21.15.116:0/2987241247 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f9b8c003de0 con 0x7f9b900ffab0 2024-03-04T16:04:43.537 INFO:tasks.workunit.client.1.smithi152.stdout:6/757: creat d8/d23/d3a/d8d/f107 x:0 0 0 2024-03-04T16:04:43.543 INFO:tasks.workunit.client.1.smithi152.stdout:5/583: fdatasync d6/d8/f2c 0 2024-03-04T16:04:43.544 INFO:tasks.workunit.client.1.smithi152.stdout:6/758: dwrite d8/d8f/db2/fc7 [0,4194304] 0 2024-03-04T16:04:43.547 INFO:tasks.workunit.client.1.smithi152.stdout:7/493: fsync d14/f61 0 2024-03-04T16:04:43.549 INFO:tasks.workunit.client.1.smithi152.stdout:4/534: mkdir d8/d89/db3 0 2024-03-04T16:04:43.549 INFO:tasks.workunit.client.1.smithi152.stdout:4/535: chown d8/f25 81543568 1 2024-03-04T16:04:43.550 INFO:tasks.workunit.client.1.smithi152.stdout:9/562: dread d2/d13/d2e/f44 [0,4194304] 0 2024-03-04T16:04:43.550 INFO:tasks.workunit.client.1.smithi152.stdout:4/536: read - d8/d2f/d6f/f73 zero size 2024-03-04T16:04:43.552 INFO:tasks.workunit.client.1.smithi152.stdout:9/563: read d2/d13/f15 [2295644,13595] 0 2024-03-04T16:04:43.552 INFO:tasks.workunit.client.1.smithi152.stdout:1/630: creat d19/d20/d2f/fcc x:0 0 0 2024-03-04T16:04:43.553 INFO:tasks.workunit.client.1.smithi152.stdout:1/631: write d19/d75/f91 [981791,35773] 0 2024-03-04T16:04:43.554 INFO:tasks.workunit.client.1.smithi152.stdout:9/564: read d2/d13/d23/d71/f86 [66892,83802] 0 2024-03-04T16:04:43.558 INFO:tasks.workunit.client.1.smithi152.stdout:3/635: mkdir d6/d14/dd7 0 2024-03-04T16:04:43.559 INFO:tasks.workunit.client.1.smithi152.stdout:3/636: write d6/d10/f65 [1510109,18262] 0 2024-03-04T16:04:43.565 INFO:tasks.workunit.client.1.smithi152.stdout:2/585: dread da/d5f/f7a [0,4194304] 0 2024-03-04T16:04:43.570 INFO:tasks.workunit.client.1.smithi152.stdout:8/737: link d2/d3/d21/ce5 d2/d3/ced 0 2024-03-04T16:04:43.574 INFO:tasks.workunit.client.1.smithi152.stdout:1/632: creat d19/d20/d2f/d65/fcd x:0 0 0 2024-03-04T16:04:43.577 INFO:tasks.workunit.client.1.smithi152.stdout:6/759: link d8/d23/d3a/d6d/fd6 d8/d8f/da3/de3/f108 0 2024-03-04T16:04:43.578 INFO:tasks.workunit.client.1.smithi152.stdout:5/584: stat d6/d8/d1f/d3d/d61/lab 0 2024-03-04T16:04:43.584 INFO:tasks.workunit.client.1.smithi152.stdout:8/738: symlink d2/d75/d89/dd/d22/d83/d96/lee 0 2024-03-04T16:04:43.586 INFO:tasks.workunit.client.1.smithi152.stdout:9/565: dwrite d2/d13/f1a [0,4194304] 0 2024-03-04T16:04:43.587 INFO:tasks.workunit.client.1.smithi152.stdout:5/585: creat d6/d50/fc8 x:0 0 0 2024-03-04T16:04:43.589 INFO:tasks.workunit.client.1.smithi152.stdout:5/586: dread - d6/d8/d5a/d64/d76/d9f/f51 zero size 2024-03-04T16:04:43.592 INFO:tasks.workunit.client.1.smithi152.stdout:7/494: rename d14/d21/d31/c79 to d14/d21/d47/c89 0 2024-03-04T16:04:43.593 INFO:tasks.workunit.client.1.smithi152.stdout:7/495: chown d14/d21/d47/d40/l84 422 1 2024-03-04T16:04:43.593 INFO:tasks.workunit.client.1.smithi152.stdout:2/586: mkdir da/df/d2e/dc6 0 2024-03-04T16:04:43.594 INFO:tasks.workunit.client.1.smithi152.stdout:7/496: fdatasync d14/d2b/f3b 0 2024-03-04T16:04:43.595 INFO:tasks.workunit.client.1.smithi152.stdout:6/760: mknod d8/d23/d3a/c109 0 2024-03-04T16:04:43.597 INFO:tasks.workunit.client.1.smithi152.stdout:6/761: stat d8/d42/dde/dee 0 2024-03-04T16:04:43.598 INFO:tasks.workunit.client.1.smithi152.stdout:4/537: rename d8/c2d to d8/d4e/cb4 0 2024-03-04T16:04:43.603 INFO:tasks.workunit.client.1.smithi152.stdout:1/633: rename d19/d20/fcb to d19/d75/fce 0 2024-03-04T16:04:43.604 INFO:tasks.workunit.client.1.smithi152.stdout:7/497: getdents d14/d2b/d83 0 2024-03-04T16:04:43.605 INFO:tasks.workunit.client.1.smithi152.stdout:1/634: readlink d19/d20/d2f/d6b/l3f 0 2024-03-04T16:04:43.606 INFO:tasks.workunit.client.1.smithi152.stdout:1/635: chown d19/d20/d2f/d6b/d38/d63/c8c 11 1 2024-03-04T16:04:43.610 INFO:tasks.workunit.client.1.smithi152.stdout:5/587: rename d6/d8/d5a/d8b/dad to d6/d8/d1f/d3d/da4/dc9 0 2024-03-04T16:04:43.612 INFO:tasks.workunit.client.1.smithi152.stdout:5/588: truncate d6/d8/d5a/d90/daf/fc6 710446 0 2024-03-04T16:04:43.613 INFO:tasks.workunit.client.1.smithi152.stdout:5/589: chown d6/d8/l6d 3140617 1 2024-03-04T16:04:43.614 INFO:tasks.workunit.client.1.smithi152.stdout:8/739: link d2/d75/d89/dd/d22/d26/f38 d2/d3/d21/d2d/d52/fef 0 2024-03-04T16:04:43.615 INFO:tasks.workunit.client.1.smithi152.stdout:5/590: write d6/d8/d1f/d3d/f85 [3403502,58385] 0 2024-03-04T16:04:43.617 INFO:tasks.workunit.client.1.smithi152.stdout:7/498: dread d14/f15 [0,4194304] 0 2024-03-04T16:04:43.618 INFO:tasks.workunit.client.1.smithi152.stdout:8/740: write d2/d3/d15/d1c/d84/d87/f6f [3350239,16063] 0 2024-03-04T16:04:43.619 INFO:tasks.workunit.client.1.smithi152.stdout:2/587: rename da/d24/l78 to da/d2c/lc7 0 2024-03-04T16:04:43.619 INFO:tasks.workunit.client.1.smithi152.stdout:2/588: stat da/f1c 0 2024-03-04T16:04:43.626 INFO:tasks.workunit.client.1.smithi152.stdout:5/591: chown d6/d8/c55 705 1 2024-03-04T16:04:43.627 INFO:tasks.workunit.client.1.smithi152.stdout:7/499: dwrite d14/d21/d48/f4b [4194304,4194304] 0 2024-03-04T16:04:43.634 INFO:tasks.workunit.client.1.smithi152.stdout:1/636: dread d19/d3b/f49 [0,4194304] 0 2024-03-04T16:04:43.635 INFO:tasks.workunit.client.1.smithi152.stdout:1/637: stat d19/d75/l8b 0 2024-03-04T16:04:43.635 INFO:tasks.workunit.client.1.smithi152.stdout:9/566: dwrite d2/f2d [0,4194304] 0 2024-03-04T16:04:43.636 INFO:tasks.workunit.client.1.smithi152.stdout:1/638: stat d19/d3b/d6e/d97/cc9 0 2024-03-04T16:04:43.636 INFO:tasks.workunit.client.1.smithi152.stdout:9/567: fdatasync d2/d13/d23/d71/da3/fab 0 2024-03-04T16:04:43.637 INFO:tasks.workunit.client.1.smithi152.stdout:1/639: stat d19/d20/d2f/d6b/d38/f3a 0 2024-03-04T16:04:43.637 INFO:tasks.workunit.client.1.smithi152.stdout:6/762: rename d8/d23/d3a/d6d to d8/d23/d3a/d8d/d96/d10a 0 2024-03-04T16:04:43.640 INFO:tasks.workunit.client.1.smithi152.stdout:8/741: creat d2/d3/d21/d2d/d52/d68/d6e/ff0 x:0 0 0 2024-03-04T16:04:43.648 INFO:tasks.workunit.client.1.smithi152.stdout:8/742: dwrite d2/d7f/f98 [0,4194304] 0 2024-03-04T16:04:43.652 INFO:tasks.workunit.client.1.smithi152.stdout:2/589: write da/d24/d47/f6b [64889,34603] 0 2024-03-04T16:04:43.652 INFO:tasks.workunit.client.1.smithi152.stdout:6/763: mknod d8/d52/d58/c10b 0 2024-03-04T16:04:43.657 INFO:tasks.workunit.client.1.smithi152.stdout:5/592: getdents d6/d8/d5a/d64/d76/d9f/db0 0 2024-03-04T16:04:43.659 INFO:tasks.workunit.client.1.smithi152.stdout:9/568: mknod d2/cbd 0 2024-03-04T16:04:43.662 INFO:tasks.workunit.client.1.smithi152.stdout:1/640: link d19/d3b/d6e/d97/fad d19/d3b/fcf 0 2024-03-04T16:04:43.664 INFO:tasks.workunit.client.1.smithi152.stdout:6/764: rmdir d8/d10 39 2024-03-04T16:04:43.666 INFO:tasks.workunit.client.1.smithi152.stdout:7/500: dwrite d14/d21/d47/f5e [0,4194304] 0 2024-03-04T16:04:43.669 INFO:tasks.workunit.client.1.smithi152.stdout:8/743: dwrite d2/d75/d89/dd/d22/d83/d96/da6/fbd [0,4194304] 0 2024-03-04T16:04:43.672 INFO:tasks.workunit.client.1.smithi152.stdout:6/765: symlink d8/da9/l10c 0 2024-03-04T16:04:43.673 INFO:tasks.workunit.client.1.smithi152.stdout:5/593: truncate d6/d8/d5a/d64/d76/d9f/f53 972901 0 2024-03-04T16:04:43.673 INFO:tasks.workunit.client.1.smithi152.stdout:1/641: read d19/d20/d2f/d65/fae [73964,68393] 0 2024-03-04T16:04:43.674 INFO:tasks.workunit.client.1.smithi152.stdout:2/590: dwrite da/df/d35/f4a [4194304,4194304] 0 2024-03-04T16:04:43.679 INFO:tasks.workunit.client.1.smithi152.stdout:5/594: chown d6/d8/d5a/d64/d76/d9f 76760993 1 2024-03-04T16:04:43.682 INFO:tasks.workunit.client.1.smithi152.stdout:8/744: write d2/d3/d15/d1c/d84/d87/d5d/f7b [1375947,46880] 0 2024-03-04T16:04:43.683 INFO:tasks.workunit.client.1.smithi152.stdout:1/642: chown d19/d3b/d6e/f79 2269779 1 2024-03-04T16:04:43.693 INFO:tasks.workunit.client.1.smithi152.stdout:9/569: dwrite d2/d13/d96/fa4 [0,4194304] 0 2024-03-04T16:04:43.694 INFO:tasks.workunit.client.1.smithi152.stdout:6/766: dwrite d8/d8f/da3/fac [0,4194304] 0 2024-03-04T16:04:43.694 INFO:tasks.workunit.client.1.smithi152.stdout:5/595: rmdir d6/d8/d1f/d3d/d6f/d70/d98 39 2024-03-04T16:04:43.699 INFO:tasks.workunit.client.1.smithi152.stdout:5/596: read d6/d8/d5a/d64/d76/d9f/f43 [592988,71460] 0 2024-03-04T16:04:43.699 INFO:tasks.workunit.client.1.smithi152.stdout:8/745: read d2/d3/d15/f6a [980435,87587] 0 2024-03-04T16:04:43.706 INFO:tasks.workunit.client.1.smithi152.stdout:1/643: dwrite d19/d20/d2f/d65/fae [0,4194304] 0 2024-03-04T16:04:43.708 INFO:tasks.workunit.client.1.smithi152.stdout:9/570: symlink d2/d13/d23/d71/d80/lbe 0 2024-03-04T16:04:43.709 INFO:tasks.workunit.client.1.smithi152.stdout:8/746: truncate d2/d75/d89/dd/d22/d83/d96/da6/fcd 283845 0 2024-03-04T16:04:43.716 INFO:tasks.workunit.client.1.smithi152.stdout:8/747: dread - d2/d75/d89/dd/d22/d6c/f70 zero size 2024-03-04T16:04:43.716 INFO:tasks.workunit.client.1.smithi152.stdout:7/501: dread d14/f78 [0,4194304] 0 2024-03-04T16:04:43.717 INFO:tasks.workunit.client.1.smithi152.stdout:9/571: creat d2/d13/d23/d66/d7d/fbf x:0 0 0 2024-03-04T16:04:43.718 INFO:tasks.workunit.client.1.smithi152.stdout:8/748: rmdir d2/d75 39 2024-03-04T16:04:43.719 INFO:tasks.workunit.client.1.smithi152.stdout:2/591: truncate da/df/f13 4331031 0 2024-03-04T16:04:43.720 INFO:tasks.workunit.client.1.smithi152.stdout:2/592: chown da/f3c 5 1 2024-03-04T16:04:43.728 INFO:tasks.workunit.client.1.smithi152.stdout:2/593: dwrite da/df/d2e/d61/f84 [0,4194304] 0 2024-03-04T16:04:43.732 INFO:tasks.workunit.client.1.smithi152.stdout:7/502: rename d14/c29 to d14/d2b/c8a 0 2024-03-04T16:04:43.734 INFO:tasks.workunit.client.1.smithi152.stdout:8/749: creat d2/d75/d89/dd/d22/d83/ff1 x:0 0 0 2024-03-04T16:04:43.740 INFO:tasks.workunit.client.1.smithi152.stdout:2/594: unlink da/df/d35/f46 0 2024-03-04T16:04:43.740 INFO:tasks.workunit.client.1.smithi152.stdout:2/595: chown da/d24/d47/d4f/l55 69 1 2024-03-04T16:04:43.741 INFO:tasks.workunit.client.1.smithi152.stdout:1/644: dread d19/d4b/d53/d57/f88 [0,4194304] 0 2024-03-04T16:04:43.742 INFO:tasks.workunit.client.1.smithi152.stdout:2/596: dread - da/d5f/fbf zero size 2024-03-04T16:04:43.743 INFO:tasks.workunit.client.1.smithi152.stdout:1/645: fdatasync d19/d20/d2f/f59 0 2024-03-04T16:04:43.743 INFO:tasks.workunit.client.1.smithi152.stdout:5/597: dread d6/d8/ff [0,4194304] 0 2024-03-04T16:04:43.792 INFO:tasks.workunit.client.1.smithi152.stdout:9/572: dread d2/d13/d23/f5f [0,4194304] 0 2024-03-04T16:04:43.792 INFO:tasks.workunit.client.1.smithi152.stdout:1/646: dread d19/f5b [0,4194304] 0 2024-03-04T16:04:43.792 INFO:tasks.workunit.client.1.smithi152.stdout:5/598: chown d6/d8/d1f/l79 219844 1 2024-03-04T16:04:43.794 INFO:tasks.workunit.client.1.smithi152.stdout:8/750: mkdir d2/d3/d21/d2d/d52/dcc/de7/df2 0 2024-03-04T16:04:43.801 INFO:tasks.workunit.client.1.smithi152.stdout:9/573: creat d2/d63/fc0 x:0 0 0 2024-03-04T16:04:43.802 INFO:tasks.workunit.client.1.smithi152.stdout:7/503: dwrite d14/d21/d31/f57 [0,4194304] 0 2024-03-04T16:04:43.804 INFO:tasks.workunit.client.1.smithi152.stdout:9/574: fdatasync d2/d13/d23/d71/fa1 0 2024-03-04T16:04:43.805 INFO:tasks.workunit.client.1.smithi152.stdout:2/597: dwrite da/df/f37 [4194304,4194304] 0 2024-03-04T16:04:43.810 INFO:tasks.workunit.client.1.smithi152.stdout:9/575: dread d2/d13/f1a [0,4194304] 0 2024-03-04T16:04:43.811 INFO:tasks.workunit.client.1.smithi152.stdout:8/751: mkdir d2/d75/d89/dd/d22/ddf/de1/df3 0 2024-03-04T16:04:43.816 INFO:tasks.workunit.client.1.smithi152.stdout:7/504: readlink d14/d21/d48/l70 0 2024-03-04T16:04:43.819 INFO:tasks.workunit.client.1.smithi152.stdout:8/752: dwrite d2/d75/f8f [0,4194304] 0 2024-03-04T16:04:43.821 INFO:tasks.workunit.client.1.smithi152.stdout:5/599: link d6/d8/d1f/d3d/f5d d6/d50/fca 0 2024-03-04T16:04:43.823 INFO:tasks.workunit.client.1.smithi152.stdout:2/598: creat da/df/d58/d6e/fc8 x:0 0 0 2024-03-04T16:04:43.823 INFO:tasks.workunit.client.1.smithi152.stdout:2/599: dread - da/fc4 zero size 2024-03-04T16:04:43.828 INFO:tasks.workunit.client.1.smithi152.stdout:7/505: creat d14/d21/d47/d36/f8b x:0 0 0 2024-03-04T16:04:43.834 INFO:tasks.workunit.client.1.smithi152.stdout:2/600: rename da/f11 to da/d24/d47/d4f/fc9 0 2024-03-04T16:04:43.840 INFO:tasks.workunit.client.1.smithi152.stdout:5/600: mkdir d6/d8/d5a/d64/d76/d9f/dc7/dcb 0 2024-03-04T16:04:43.841 INFO:tasks.workunit.client.1.smithi152.stdout:2/601: dwrite da/f23 [4194304,4194304] 0 2024-03-04T16:04:43.844 INFO:tasks.workunit.client.1.smithi152.stdout:1/647: truncate d19/d4b/d53/d57/f81 140088 0 2024-03-04T16:04:43.846 INFO:tasks.workunit.client.1.smithi152.stdout:9/576: creat d2/d63/d69/fc1 x:0 0 0 2024-03-04T16:04:43.850 INFO:tasks.workunit.client.1.smithi152.stdout:2/602: mkdir da/d5f/d85/dca 0 2024-03-04T16:04:43.851 INFO:tasks.workunit.client.1.smithi152.stdout:7/506: creat d14/d21/d47/f8c x:0 0 0 2024-03-04T16:04:43.852 INFO:tasks.workunit.client.1.smithi152.stdout:8/753: getdents d2/d3/d21/dc0 0 2024-03-04T16:04:43.853 INFO:tasks.workunit.client.1.smithi152.stdout:5/601: dwrite d6/d8/d1f/d3d/d6f/f95 [4194304,4194304] 0 2024-03-04T16:04:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.853+0000 7f9b980c0700 1 -- 172.21.15.116:0/2987241247 --> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b90105600 con 0x7f9b7c038af0 2024-03-04T16:04:43.855 INFO:tasks.workunit.client.1.smithi152.stdout:9/577: symlink d2/d13/d23/d71/da3/lc2 0 2024-03-04T16:04:43.856 INFO:tasks.workunit.client.1.smithi152.stdout:5/602: chown d6/d8/d1f/d3d/f5d 676943316 1 2024-03-04T16:04:43.858 INFO:tasks.workunit.client.1.smithi152.stdout:8/754: symlink d2/d3/d21/d2d/d52/dcc/dd6/lf4 0 2024-03-04T16:04:43.859 INFO:tasks.workunit.client.1.smithi152.stdout:8/755: write d2/d7f/fc2 [906654,127689] 0 2024-03-04T16:04:43.860 INFO:tasks.workunit.client.1.smithi152.stdout:1/648: fdatasync d19/d20/f41 0 2024-03-04T16:04:43.862 INFO:tasks.workunit.client.1.smithi152.stdout:1/649: read - d19/d20/d2f/d65/fbd zero size 2024-03-04T16:04:43.863 INFO:tasks.workunit.client.1.smithi152.stdout:1/650: stat d19/d4b 0 2024-03-04T16:04:43.864 INFO:tasks.workunit.client.1.smithi152.stdout:9/578: chown d2/d63/d69/c7b 7482 1 2024-03-04T16:04:43.867 INFO:tasks.workunit.client.1.smithi152.stdout:8/756: unlink d2/d75/d89/dd/d22/fa4 0 2024-03-04T16:04:43.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.867+0000 7f9b86ffd700 1 -- 172.21.15.116:0/2987241247 <== mgr.14738 v2:172.21.15.116:6800/2966163948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9b90105600 con 0x7f9b7c038af0 2024-03-04T16:04:43.868 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:04:43.868 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (7m) 7s ago 9m 25.2M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:04:43.868 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (9m) 7s ago 9m 7646k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:04:43.868 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (8m) 51s ago 8m 7574k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:04:43.868 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (9m) 7s ago 9m 6665k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (8m) 51s ago 8m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (7m) 7s ago 8m 86.7M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (3m) 7s ago 3m 17.4M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (3m) 7s ago 3m 13.5M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (3m) 51s ago 3m 14.8M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (3m) 51s ago 3m 131M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (13s) 7s ago 10m 65.9M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (94s) 51s ago 8m 489M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6756dd312323 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (10m) 7s ago 10m 49.1M 2048M 18.2.1 5be31c24972a 60e0bf118c4d 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (8m) 51s ago 8m 41.4M 2048M 18.2.1 5be31c24972a 52450d235516 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (9m) 7s ago 9m 16.0M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (8m) 51s ago 8m 16.5M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (7m) 7s ago 7m 191M 1216M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:04:43.869 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (6m) 7s ago 6m 186M 1216M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:04:43.870 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (6m) 7s ago 6m 158M 1216M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:04:43.870 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (6m) 51s ago 6m 98.4M 2240M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:04:43.870 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (5m) 51s ago 5m 97.4M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:04:43.870 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (5m) 51s ago 5m 89.1M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:04:43.870 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (25s) 7s ago 8m 37.5M - 2.43.0 a07b618ecd1d 414f726d8ae2 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 -- 172.21.15.116:0/2987241247 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f9b7c038af0 msgr2=0x7f9b7c03afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f9b7c038af0 0x7f9b7c03afb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9b800096d0 tx=0x7f9b80009260 comp rx=0 tx=0).stop 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 -- 172.21.15.116:0/2987241247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b900ffab0 msgr2=0x7f9b90194120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b900ffab0 0x7f9b90194120 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c006010 tx=0x7f9b8c0053f0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 -- 172.21.15.116:0/2987241247 shutdown_connections 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f9b7c038af0 0x7f9b7c03afb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b900ffab0 0x7f9b90194120 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 --2- 172.21.15.116:0/2987241247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b90100cb0 0x7f9b90194660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 -- 172.21.15.116:0/2987241247 >> 172.21.15.116:0/2987241247 conn(0x7f9b900fb030 msgr2=0x7f9b90103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 -- 172.21.15.116:0/2987241247 shutdown_connections 2024-03-04T16:04:43.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.869+0000 7f9b980c0700 1 -- 172.21.15.116:0/2987241247 wait complete. 2024-03-04T16:04:43.873 INFO:tasks.workunit.client.1.smithi152.stdout:7/507: dread d14/d2b/f60 [0,4194304] 0 2024-03-04T16:04:43.879 INFO:tasks.workunit.client.1.smithi152.stdout:9/579: rename d2/d13/d38/l5e to d2/d13/d38/lc3 0 2024-03-04T16:04:43.881 INFO:tasks.workunit.client.1.smithi152.stdout:8/757: fdatasync d2/d75/d89/f90 0 2024-03-04T16:04:43.883 INFO:tasks.workunit.client.1.smithi152.stdout:9/580: creat d2/d13/d23/fc4 x:0 0 0 2024-03-04T16:04:43.884 INFO:tasks.workunit.client.1.smithi152.stdout:7/508: fdatasync d14/f16 0 2024-03-04T16:04:43.885 INFO:tasks.workunit.client.1.smithi152.stdout:7/509: stat d14/d21/f6b 0 2024-03-04T16:04:43.887 INFO:tasks.workunit.client.1.smithi152.stdout:8/758: chown d2/d3/d15/cad 59 1 2024-03-04T16:04:43.887 INFO:tasks.workunit.client.1.smithi152.stdout:7/510: chown d14/d21/d48/l70 12507 1 2024-03-04T16:04:43.890 INFO:tasks.workunit.client.1.smithi152.stdout:5/603: dwrite d6/f39 [4194304,4194304] 0 2024-03-04T16:04:43.896 INFO:tasks.workunit.client.1.smithi152.stdout:1/651: truncate d19/d3b/f92 820038 0 2024-03-04T16:04:43.899 INFO:tasks.workunit.client.1.smithi152.stdout:9/581: dwrite d2/d63/d69/f90 [0,4194304] 0 2024-03-04T16:04:43.900 INFO:tasks.workunit.client.1.smithi152.stdout:9/582: truncate d2/d63/f94 4719831 0 2024-03-04T16:04:43.901 INFO:tasks.workunit.client.1.smithi152.stdout:9/583: chown d2/d13/d23/f33 1412533 1 2024-03-04T16:04:43.905 INFO:tasks.workunit.client.1.smithi152.stdout:5/604: read - d6/f68 zero size 2024-03-04T16:04:43.911 INFO:tasks.workunit.client.1.smithi152.stdout:7/511: dwrite d14/d21/d47/f35 [0,4194304] 0 2024-03-04T16:04:43.917 INFO:tasks.workunit.client.1.smithi152.stdout:1/652: rename d19/d20/d2f/d65/fcd to d19/d20/d2f/d6b/d38/d63/fd0 0 2024-03-04T16:04:43.918 INFO:tasks.workunit.client.1.smithi152.stdout:1/653: chown d19/d20/d2f/d65 194471911 1 2024-03-04T16:04:43.919 INFO:tasks.workunit.client.1.smithi152.stdout:1/654: chown d19/d4b/d53/d57/d62/d82/fc2 112 1 2024-03-04T16:04:43.920 INFO:tasks.workunit.client.1.smithi152.stdout:1/655: readlink d19/d20/d2f/d6b/l77 0 2024-03-04T16:04:43.922 INFO:tasks.workunit.client.1.smithi152.stdout:8/759: creat d2/d3/d15/ff5 x:0 0 0 2024-03-04T16:04:43.925 INFO:tasks.workunit.client.1.smithi152.stdout:5/605: unlink d6/d8/d5a/d64/d76/d9f/c69 0 2024-03-04T16:04:43.927 INFO:tasks.workunit.client.1.smithi152.stdout:7/512: creat d14/d21/f8d x:0 0 0 2024-03-04T16:04:43.930 INFO:tasks.workunit.client.1.smithi152.stdout:1/656: rename d19/d20/d2f/d6b/d38/da6/cbf to d19/d20/d2f/d6b/d38/da6/cd1 0 2024-03-04T16:04:43.939 INFO:tasks.workunit.client.1.smithi152.stdout:8/760: rename d2/d75/d89/dd/d22/d26/f61 to d2/d3/d21/dce/ff6 0 2024-03-04T16:04:43.941 INFO:tasks.workunit.client.1.smithi152.stdout:5/606: unlink d6/d8/d1f/l79 0 2024-03-04T16:04:43.949 INFO:tasks.workunit.client.1.smithi152.stdout:1/657: dread d19/d20/d2f/f94 [0,4194304] 0 2024-03-04T16:04:43.951 INFO:tasks.workunit.client.1.smithi152.stdout:5/607: write d6/f38 [1246292,63202] 0 2024-03-04T16:04:43.951 INFO:tasks.workunit.client.1.smithi152.stdout:9/584: dwrite d2/f61 [0,4194304] 0 2024-03-04T16:04:43.960 INFO:tasks.workunit.client.1.smithi152.stdout:7/513: dread d14/d21/d31/f64 [0,4194304] 0 2024-03-04T16:04:43.962 INFO:tasks.workunit.client.1.smithi152.stdout:9/585: mknod d2/d13/d35/cc5 0 2024-03-04T16:04:43.963 INFO:tasks.workunit.client.1.smithi152.stdout:1/658: dread fa [0,4194304] 0 2024-03-04T16:04:43.964 INFO:tasks.workunit.client.1.smithi152.stdout:1/659: truncate d19/d75/fce 4485380 0 2024-03-04T16:04:43.966 INFO:tasks.workunit.client.1.smithi152.stdout:7/514: dread - d14/d21/d47/d36/f74 zero size 2024-03-04T16:04:43.969 INFO:tasks.workunit.client.1.smithi152.stdout:9/586: rmdir d2/d13/d23/d42 39 2024-03-04T16:04:43.978 INFO:tasks.workunit.client.1.smithi152.stdout:7/515: dread d14/d21/f49 [0,4194304] 0 2024-03-04T16:04:43.978 INFO:tasks.workunit.client.1.smithi152.stdout:5/608: getdents d6/d8/d1f 0 2024-03-04T16:04:43.978 INFO:tasks.workunit.client.1.smithi152.stdout:1/660: dwrite d19/d20/fb2 [0,4194304] 0 2024-03-04T16:04:43.980 INFO:tasks.workunit.client.1.smithi152.stdout:1/661: chown d19/d4b/d53/d57/d62/d82/fbe 193283047 1 2024-03-04T16:04:43.987 INFO:tasks.workunit.client.1.smithi152.stdout:7/516: dwrite d14/d21/d47/f8c [0,4194304] 0 2024-03-04T16:04:43.990 INFO:tasks.workunit.client.1.smithi152.stdout:1/662: creat d19/d75/da8/db8/fd2 x:0 0 0 2024-03-04T16:04:43.999 INFO:tasks.workunit.client.1.smithi152.stdout:1/663: mknod d19/d20/d2f/cd3 0 2024-03-04T16:04:44.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:43.999+0000 7f0f46ffd700 1 -- 172.21.15.116:0/2554899029 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f3c004a00 con 0x7f0f48101e60 2024-03-04T16:04:44.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.000+0000 7f0f4e851700 1 -- 172.21.15.116:0/2554899029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f48101e60 msgr2=0x7f0f481022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:44.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.000+0000 7f0f4e851700 1 --2- 172.21.15.116:0/2554899029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f48101e60 0x7f0f481022c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c009910 tx=0x7f0f3c009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:44.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.000+0000 7f0f4e851700 1 -- 172.21.15.116:0/2554899029 shutdown_connections 2024-03-04T16:04:44.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.000+0000 7f0f4e851700 1 --2- 172.21.15.116:0/2554899029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f48101e60 0x7f0f481022c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.000+0000 7f0f4e851700 1 --2- 172.21.15.116:0/2554899029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f480fee80 0x7f0f480ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.000+0000 7f0f4e851700 1 -- 172.21.15.116:0/2554899029 >> 172.21.15.116:0/2554899029 conn(0x7f0f480fa7c0 msgr2=0x7f0f480fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:44.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.000+0000 7f0f4e851700 1 -- 172.21.15.116:0/2554899029 shutdown_connections 2024-03-04T16:04:44.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.000+0000 7f0f4e851700 1 -- 172.21.15.116:0/2554899029 wait complete. 2024-03-04T16:04:44.003 INFO:tasks.workunit.client.1.smithi152.stdout:5/609: stat d6/d8/d5a/d64/d76/d9f/f7c 0 2024-03-04T16:04:44.003 INFO:tasks.workunit.client.1.smithi152.stdout:0/540: sync 2024-03-04T16:04:44.003 INFO:tasks.workunit.client.1.smithi152.stdout:5/610: chown d6/d8/fa3 2672235 1 2024-03-04T16:04:44.003 INFO:tasks.workunit.client.1.smithi152.stdout:0/541: chown d9/d1b/d3c/fa9 659 1 2024-03-04T16:04:44.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.000+0000 7f0f4e851700 1 Processor -- start 2024-03-04T16:04:44.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.000+0000 7f0f4e851700 1 -- start start 2024-03-04T16:04:44.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f4e851700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f480fee80 0x7f0f481017c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f4e851700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f48101e60 0x7f0f480ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f4e851700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f48100350 con 0x7f0f48101e60 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f4e851700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f48100490 con 0x7f0f480fee80 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f477fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f48101e60 0x7f0f480ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f477fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f48101e60 0x7f0f480ffe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53816/0 (socket says 172.21.15.116:53816) 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f477fe700 1 -- 172.21.15.116:0/869404526 learned_addr learned my addr 172.21.15.116:0/869404526 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f47fff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f480fee80 0x7f0f481017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f477fe700 1 -- 172.21.15.116:0/869404526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f480fee80 msgr2=0x7f0f481017c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f477fe700 1 --2- 172.21.15.116:0/869404526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f480fee80 0x7f0f481017c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f477fe700 1 -- 172.21.15.116:0/869404526 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c0095f0 con 0x7f0f48101e60 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f477fe700 1 --2- 172.21.15.116:0/869404526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f48101e60 0x7f0f480ffe10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00f690 tx=0x7f0f3c00f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.001+0000 7f0f457fa700 1 -- 172.21.15.116:0/869404526 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f3c01d070 con 0x7f0f48101e60 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.002+0000 7f0f4e851700 1 -- 172.21.15.116:0/869404526 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f48100710 con 0x7f0f48101e60 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.002+0000 7f0f457fa700 1 -- 172.21.15.116:0/869404526 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0f3c005dc0 con 0x7f0f48101e60 2024-03-04T16:04:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.002+0000 7f0f457fa700 1 -- 172.21.15.116:0/869404526 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f3c0226a0 con 0x7f0f48101e60 2024-03-04T16:04:44.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.002+0000 7f0f4e851700 1 -- 172.21.15.116:0/869404526 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f48100c00 con 0x7f0f48101e60 2024-03-04T16:04:44.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.003+0000 7f0f4e851700 1 -- 172.21.15.116:0/869404526 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f48109720 con 0x7f0f48101e60 2024-03-04T16:04:44.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.003+0000 7f0f457fa700 1 -- 172.21.15.116:0/869404526 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 28) v1 ==== 46012+0+0 (secure 0 0 0) 0x7f0f3c022800 con 0x7f0f48101e60 2024-03-04T16:04:44.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.004+0000 7f0f457fa700 1 --2- 172.21.15.116:0/869404526 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f0f30038e80 0x7f0f3003b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:44.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.004+0000 7f0f457fa700 1 -- 172.21.15.116:0/869404526 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f0f3c04e820 con 0x7f0f48101e60 2024-03-04T16:04:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.004+0000 7f0f47fff700 1 --2- 172.21.15.116:0/869404526 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f0f30038e80 0x7f0f3003b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.005+0000 7f0f47fff700 1 --2- 172.21.15.116:0/869404526 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f0f30038e80 0x7f0f3003b340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0f48068210 tx=0x7f0f38009260 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:44.007 INFO:tasks.workunit.client.1.smithi152.stdout:0/542: read d9/d1a/d7c/f7e [921340,104562] 0 2024-03-04T16:04:44.008 INFO:tasks.workunit.client.1.smithi152.stdout:5/611: getdents d6/d8/d1f/d3d/da4 0 2024-03-04T16:04:44.009 INFO:tasks.workunit.client.1.smithi152.stdout:9/587: truncate d2/d6e/f7a 764691 0 2024-03-04T16:04:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.012+0000 7f0f457fa700 1 -- 172.21.15.116:0/869404526 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f0f3c020de0 con 0x7f0f48101e60 2024-03-04T16:04:44.036 INFO:tasks.workunit.client.1.smithi152.stdout:5/612: stat d6/d8/d1f/d3d/d6f/d70/d98 0 2024-03-04T16:04:44.037 INFO:tasks.workunit.client.1.smithi152.stdout:5/613: chown d6/d8/d1f/d3d/d61/c63 332845779 1 2024-03-04T16:04:44.041 INFO:tasks.workunit.client.1.smithi152.stdout:9/588: mkdir d2/d13/d23/d71/da3/dc6 0 2024-03-04T16:04:44.052 INFO:tasks.workunit.client.1.smithi152.stdout:0/543: getdents d9/d1b/d54/d69 0 2024-03-04T16:04:44.056 INFO:tasks.workunit.client.1.smithi152.stdout:0/544: creat d9/d1a/d7c/faa x:0 0 0 2024-03-04T16:04:44.060 INFO:tasks.workunit.client.1.smithi152.stdout:9/589: link d2/d13/d23/c5a d2/d13/d2e/cc7 0 2024-03-04T16:04:44.065 INFO:tasks.workunit.client.1.smithi152.stdout:0/545: creat d9/d2c/d2e/d7f/d9d/da3/fab x:0 0 0 2024-03-04T16:04:44.065 INFO:tasks.workunit.client.1.smithi152.stdout:7/517: dwrite d14/d21/d47/d40/f6c [0,4194304] 0 2024-03-04T16:04:44.065 INFO:tasks.workunit.client.1.smithi152.stdout:9/590: symlink d2/d13/d23/d71/da3/lc8 0 2024-03-04T16:04:44.068 INFO:tasks.workunit.client.1.smithi152.stdout:9/591: write d2/d13/d23/d66/d7d/da7/fb8 [119314,25091] 0 2024-03-04T16:04:44.073 INFO:tasks.workunit.client.1.smithi152.stdout:7/518: rmdir d14/d21/d48 39 2024-03-04T16:04:44.075 INFO:tasks.workunit.client.1.smithi152.stdout:0/546: mkdir d9/d2c/d2e/d94/dac 0 2024-03-04T16:04:44.077 INFO:tasks.workunit.client.1.smithi152.stdout:5/614: write d6/d8/d1f/d3d/f5d [3390044,61148] 0 2024-03-04T16:04:44.077 INFO:tasks.workunit.client.1.smithi152.stdout:0/547: write d9/d1b/d3c/d84/d56/f63 [252096,50321] 0 2024-03-04T16:04:44.078 INFO:tasks.workunit.client.1.smithi152.stdout:5/615: stat d6/d50/fca 0 2024-03-04T16:04:44.079 INFO:tasks.workunit.client.1.smithi152.stdout:7/519: mkdir d14/d69/d8e 0 2024-03-04T16:04:44.085 INFO:tasks.workunit.client.1.smithi152.stdout:5/616: chown d6/c8c 3811237 1 2024-03-04T16:04:44.086 INFO:tasks.workunit.client.1.smithi152.stdout:0/548: dwrite d9/d1b/f6e [0,4194304] 0 2024-03-04T16:04:44.095 INFO:tasks.workunit.client.1.smithi152.stdout:5/617: truncate d6/d8/d1f/f3b 287718 0 2024-03-04T16:04:44.097 INFO:tasks.workunit.client.1.smithi152.stdout:5/618: dread - d6/d8/fa3 zero size 2024-03-04T16:04:44.097 INFO:tasks.workunit.client.1.smithi152.stdout:9/592: dwrite d2/d13/d2e/f3c [0,4194304] 0 2024-03-04T16:04:44.098 INFO:tasks.workunit.client.1.smithi152.stdout:7/520: symlink d14/d21/d47/d36/l8f 0 2024-03-04T16:04:44.099 INFO:tasks.workunit.client.1.smithi152.stdout:7/521: readlink d14/d21/d47/d40/l87 0 2024-03-04T16:04:44.109 INFO:tasks.workunit.client.1.smithi152.stdout:5/619: chown d6/d8/d5a/d64/d76/d9f/c40 62936142 1 2024-03-04T16:04:44.110 INFO:tasks.workunit.client.1.smithi152.stdout:9/593: dwrite d2/d13/d96/fa4 [0,4194304] 0 2024-03-04T16:04:44.110 INFO:tasks.workunit.client.1.smithi152.stdout:5/620: chown d6/d8/f9a 907827 1 2024-03-04T16:04:44.110 INFO:tasks.workunit.client.1.smithi152.stdout:0/549: dread d9/d1b/f4f [0,4194304] 0 2024-03-04T16:04:44.113 INFO:tasks.workunit.client.1.smithi152.stdout:0/550: truncate d9/d1b/d54/d69/d95/fa5 950428 0 2024-03-04T16:04:44.117 INFO:tasks.workunit.client.1.smithi152.stdout:7/522: creat d14/d21/f90 x:0 0 0 2024-03-04T16:04:44.117 INFO:tasks.workunit.client.1.smithi152.stdout:0/551: truncate d9/d2c/d2e/d7f/d9d/da3/fab 805450 0 2024-03-04T16:04:44.118 INFO:tasks.workunit.client.1.smithi152.stdout:9/594: mknod d2/d13/d38/cc9 0 2024-03-04T16:04:44.120 INFO:tasks.workunit.client.1.smithi152.stdout:5/621: unlink d6/d8/d1f/l94 0 2024-03-04T16:04:44.121 INFO:tasks.workunit.client.1.smithi152.stdout:0/552: symlink d9/d92/lad 0 2024-03-04T16:04:44.122 INFO:tasks.workunit.client.1.smithi152.stdout:0/553: dread - d9/d1b/d54/d80/f8e zero size 2024-03-04T16:04:44.123 INFO:tasks.workunit.client.1.smithi152.stdout:0/554: readlink d9/d1b/d3c/l67 0 2024-03-04T16:04:44.207 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:43 smithi116 ceph-mon[29120]: [04/Mar/2024:16:04:42] ENGINE Serving on http://172.21.15.116:8765 2024-03-04T16:04:44.207 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:43 smithi116 ceph-mon[29120]: [04/Mar/2024:16:04:42] ENGINE Serving on https://172.21.15.116:7150 2024-03-04T16:04:44.207 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:43 smithi116 ceph-mon[29120]: [04/Mar/2024:16:04:42] ENGINE Bus STARTED 2024-03-04T16:04:44.207 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:43 smithi116 ceph-mon[29120]: mgrmap e27: smithi116.jwdhop(active, since 1.01189s) 2024-03-04T16:04:44.207 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:43 smithi116 ceph-mon[29120]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:44.208 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:43 smithi116 ceph-mon[29120]: pgmap v3: 65 pgs: 65 active+clean; 2.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail 2024-03-04T16:04:44.208 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:43 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:44.208 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:43 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:44.208 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:43 smithi116 ceph-mon[29120]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:44.214 INFO:tasks.workunit.client.1.smithi152.stdout:3/637: sync 2024-03-04T16:04:44.214 INFO:tasks.workunit.client.1.smithi152.stdout:4/538: sync 2024-03-04T16:04:44.214 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:43 smithi152 ceph-mon[36876]: [04/Mar/2024:16:04:42] ENGINE Serving on http://172.21.15.116:8765 2024-03-04T16:04:44.214 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:43 smithi152 ceph-mon[36876]: [04/Mar/2024:16:04:42] ENGINE Serving on https://172.21.15.116:7150 2024-03-04T16:04:44.214 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:43 smithi152 ceph-mon[36876]: [04/Mar/2024:16:04:42] ENGINE Bus STARTED 2024-03-04T16:04:44.215 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:43 smithi152 ceph-mon[36876]: mgrmap e27: smithi116.jwdhop(active, since 1.01189s) 2024-03-04T16:04:44.215 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:43 smithi152 ceph-mon[36876]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:44.215 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:43 smithi152 ceph-mon[36876]: pgmap v3: 65 pgs: 65 active+clean; 2.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail 2024-03-04T16:04:44.215 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:43 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:44.215 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:43 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:44.215 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:43 smithi152 ceph-mon[36876]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:44.233 INFO:tasks.workunit.client.1.smithi152.stdout:9/595: chown d2/l74 248432263 1 2024-03-04T16:04:44.235 INFO:tasks.workunit.client.1.smithi152.stdout:0/555: chown d9/l12 73 1 2024-03-04T16:04:44.267 INFO:tasks.workunit.client.1.smithi152.stdout:3/638: creat d6/d60/d8e/fd8 x:0 0 0 2024-03-04T16:04:44.267 INFO:tasks.workunit.client.1.smithi152.stdout:3/639: stat d6/f1c 0 2024-03-04T16:04:44.269 INFO:tasks.workunit.client.1.smithi152.stdout:5/622: creat d6/d8/d5a/d82/fcc x:0 0 0 2024-03-04T16:04:44.270 INFO:tasks.workunit.client.1.smithi152.stdout:0/556: rename d9/d1b/d3c/c78 to d9/d1b/d3c/d84/d56/d5b/cae 0 2024-03-04T16:04:44.277 INFO:tasks.workunit.client.1.smithi152.stdout:0/557: creat d9/d2c/d2e/d7f/d9d/da3/faf x:0 0 0 2024-03-04T16:04:44.280 INFO:tasks.workunit.client.1.smithi152.stdout:3/640: mkdir d6/dd9 0 2024-03-04T16:04:44.283 INFO:tasks.workunit.client.1.smithi152.stdout:0/558: dwrite d9/d1a/d7c/faa [0,4194304] 0 2024-03-04T16:04:44.288 INFO:tasks.workunit.client.1.smithi152.stdout:3/641: rename d6/d14/d41/l48 to d6/d14/d41/d49/d84/lda 0 2024-03-04T16:04:44.293 INFO:tasks.workunit.client.1.smithi152.stdout:3/642: rename d6/d14/d41/d49/f95 to d6/d14/dd7/fdb 0 2024-03-04T16:04:44.295 INFO:tasks.workunit.client.1.smithi152.stdout:0/559: dread d9/f26 [0,4194304] 0 2024-03-04T16:04:44.296 INFO:tasks.workunit.client.1.smithi152.stdout:0/560: readlink d9/l12 0 2024-03-04T16:04:44.302 INFO:tasks.workunit.client.1.smithi152.stdout:0/561: dwrite d9/d2c/f6f [0,4194304] 0 2024-03-04T16:04:44.306 INFO:tasks.workunit.client.1.smithi152.stdout:0/562: rmdir d9/d1b/d3c/d84/d56 39 2024-03-04T16:04:44.307 INFO:tasks.workunit.client.1.smithi152.stdout:0/563: chown d9/d92/lad 4 1 2024-03-04T16:04:44.312 INFO:tasks.workunit.client.1.smithi152.stdout:3/643: dread d6/f9 [0,4194304] 0 2024-03-04T16:04:44.313 INFO:tasks.workunit.client.1.smithi152.stdout:0/564: creat d9/d2c/d2e/d7f/d9d/fb0 x:0 0 0 2024-03-04T16:04:44.315 INFO:tasks.workunit.client.1.smithi152.stdout:3/644: creat d6/d60/d8e/dc0/da1/fdc x:0 0 0 2024-03-04T16:04:44.316 INFO:tasks.workunit.client.1.smithi152.stdout:3/645: fsync d6/d5c/f86 0 2024-03-04T16:04:44.317 INFO:tasks.workunit.client.1.smithi152.stdout:0/565: symlink d9/d1b/d54/d80/lb1 0 2024-03-04T16:04:44.322 INFO:tasks.workunit.client.1.smithi152.stdout:0/566: mkdir d9/d1b/d54/d80/db2 0 2024-03-04T16:04:44.324 INFO:tasks.workunit.client.1.smithi152.stdout:0/567: chown d9/c11 418129 1 2024-03-04T16:04:44.333 INFO:tasks.workunit.client.1.smithi152.stdout:5/623: truncate d6/d8/d5a/d64/d76/d9f/f12 2993419 0 2024-03-04T16:04:44.335 INFO:tasks.workunit.client.1.smithi152.stdout:0/568: dwrite d9/d2c/f2f [0,4194304] 0 2024-03-04T16:04:44.342 INFO:tasks.workunit.client.1.smithi152.stdout:4/539: dwrite d8/d2f/f47 [0,4194304] 0 2024-03-04T16:04:44.342 INFO:tasks.workunit.client.1.smithi152.stdout:0/569: read d9/d2c/d2e/f31 [903428,87315] 0 2024-03-04T16:04:44.343 INFO:tasks.workunit.client.1.smithi152.stdout:9/596: dwrite d2/d13/d23/d71/fbb [0,4194304] 0 2024-03-04T16:04:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.344+0000 7f0f4e851700 1 -- 172.21.15.116:0/869404526 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0f4819e7d0 con 0x7f0f48101e60 2024-03-04T16:04:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.345+0000 7f0f457fa700 1 -- 172.21.15.116:0/869404526 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f0f3c020c00 con 0x7f0f48101e60 2024-03-04T16:04:44.346 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:04:44.346 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:04:44.346 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-03-04T16:04:44.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:04:44.346 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:04:44.346 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:04:44.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:04:44.347 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:04:44.348 INFO:tasks.workunit.client.1.smithi152.stdout:5/624: truncate d6/d50/f7e 4468544 0 2024-03-04T16:04:44.348 INFO:tasks.workunit.client.1.smithi152.stdout:5/625: read d6/d8/f1d [1165952,57830] 0 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 -- 172.21.15.116:0/869404526 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f0f30038e80 msgr2=0x7f0f3003b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 --2- 172.21.15.116:0/869404526 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f0f30038e80 0x7f0f3003b340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0f48068210 tx=0x7f0f38009260 comp rx=0 tx=0).stop 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 -- 172.21.15.116:0/869404526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f48101e60 msgr2=0x7f0f480ffe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 --2- 172.21.15.116:0/869404526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f48101e60 0x7f0f480ffe10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00f690 tx=0x7f0f3c00f6c0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 -- 172.21.15.116:0/869404526 shutdown_connections 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 --2- 172.21.15.116:0/869404526 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f0f30038e80 0x7f0f3003b340 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 --2- 172.21.15.116:0/869404526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f480fee80 0x7f0f481017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 --2- 172.21.15.116:0/869404526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f48101e60 0x7f0f480ffe10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 -- 172.21.15.116:0/869404526 >> 172.21.15.116:0/869404526 conn(0x7f0f480fa7c0 msgr2=0x7f0f48105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 -- 172.21.15.116:0/869404526 shutdown_connections 2024-03-04T16:04:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.347+0000 7f0f4e851700 1 -- 172.21.15.116:0/869404526 wait complete. 2024-03-04T16:04:44.350 INFO:tasks.workunit.client.1.smithi152.stdout:5/626: fsync d6/d8/d1f/d3d/f85 0 2024-03-04T16:04:44.353 INFO:tasks.workunit.client.1.smithi152.stdout:0/570: unlink d9/d1b/d54/d80/f8e 0 2024-03-04T16:04:44.358 INFO:tasks.workunit.client.1.smithi152.stdout:0/571: dread d9/f36 [0,4194304] 0 2024-03-04T16:04:44.370 INFO:tasks.workunit.client.1.smithi152.stdout:5/627: write d6/d8/d5a/d64/f96 [1449574,72769] 0 2024-03-04T16:04:44.371 INFO:tasks.workunit.client.1.smithi152.stdout:4/540: creat d8/d9/d24/fb5 x:0 0 0 2024-03-04T16:04:44.372 INFO:tasks.workunit.client.1.smithi152.stdout:4/541: write d8/f10 [3844797,36307] 0 2024-03-04T16:04:44.372 INFO:tasks.workunit.client.1.smithi152.stdout:0/572: symlink d9/d1b/d54/d69/d98/lb3 0 2024-03-04T16:04:44.373 INFO:tasks.workunit.client.1.smithi152.stdout:5/628: write d6/d8/d5a/d64/d76/d9f/f2d [2091835,17275] 0 2024-03-04T16:04:44.374 INFO:tasks.workunit.client.1.smithi152.stdout:5/629: stat d6/d50/l60 0 2024-03-04T16:04:44.382 INFO:tasks.workunit.client.1.smithi152.stdout:4/542: dwrite d8/d9/f50 [0,4194304] 0 2024-03-04T16:04:44.384 INFO:tasks.workunit.client.1.smithi152.stdout:0/573: unlink d9/f37 0 2024-03-04T16:04:44.391 INFO:tasks.workunit.client.1.smithi152.stdout:9/597: getdents d2/d13/d23 0 2024-03-04T16:04:44.394 INFO:tasks.workunit.client.1.smithi152.stdout:2/603: sync 2024-03-04T16:04:44.394 INFO:tasks.workunit.client.1.smithi152.stdout:6/767: sync 2024-03-04T16:04:44.396 INFO:tasks.workunit.client.1.smithi152.stdout:0/574: mkdir d9/d1a/db4 0 2024-03-04T16:04:44.400 INFO:tasks.workunit.client.1.smithi152.stdout:5/630: creat d6/d8/d5a/fcd x:0 0 0 2024-03-04T16:04:44.401 INFO:tasks.workunit.client.1.smithi152.stdout:2/604: chown da/df/d2e/f3a 63925337 1 2024-03-04T16:04:44.403 INFO:tasks.workunit.client.1.smithi152.stdout:2/605: chown da/df/d2e/d51/d8d/d9a/cc0 3967 1 2024-03-04T16:04:44.404 INFO:tasks.workunit.client.1.smithi152.stdout:9/598: rename d2/l59 to d2/d13/d35/lca 0 2024-03-04T16:04:44.405 INFO:tasks.workunit.client.1.smithi152.stdout:9/599: readlink d2/d13/d2e/l57 0 2024-03-04T16:04:44.408 INFO:tasks.workunit.client.1.smithi152.stdout:0/575: rename f4 to d9/d1b/d54/d69/d95/fb5 0 2024-03-04T16:04:44.411 INFO:tasks.workunit.client.1.smithi152.stdout:5/631: symlink d6/d8/dbe/lce 0 2024-03-04T16:04:44.411 INFO:tasks.workunit.client.1.smithi152.stdout:0/576: fdatasync d9/d2c/d2e/d7f/d9d/fa7 0 2024-03-04T16:04:44.411 INFO:tasks.workunit.client.1.smithi152.stdout:4/543: dwrite d8/d9/d24/d26/d42/f81 [0,4194304] 0 2024-03-04T16:04:44.413 INFO:tasks.workunit.client.1.smithi152.stdout:5/632: readlink d6/l9 0 2024-03-04T16:04:44.415 INFO:tasks.workunit.client.1.smithi152.stdout:4/544: chown d8/d2f/l35 5 1 2024-03-04T16:04:44.416 INFO:tasks.workunit.client.1.smithi152.stdout:9/600: rename d2/d13/d38/c3f to d2/d13/dad/ccb 0 2024-03-04T16:04:44.418 INFO:tasks.workunit.client.1.smithi152.stdout:9/601: write d2/d13/d96/fa4 [4960670,86071] 0 2024-03-04T16:04:44.422 INFO:tasks.workunit.client.1.smithi152.stdout:4/545: mknod d8/d2f/d6f/d90/cb6 0 2024-03-04T16:04:44.424 INFO:tasks.workunit.client.1.smithi152.stdout:5/633: dwrite d6/d50/f7b [0,4194304] 0 2024-03-04T16:04:44.424 INFO:tasks.workunit.client.1.smithi152.stdout:4/546: chown d8/d4e/f57 2446865 1 2024-03-04T16:04:44.427 INFO:tasks.workunit.client.1.smithi152.stdout:6/768: dread d8/d10/f44 [0,4194304] 0 2024-03-04T16:04:44.431 INFO:tasks.workunit.client.1.smithi152.stdout:9/602: dwrite d2/d13/d23/fa0 [0,4194304] 0 2024-03-04T16:04:44.437 INFO:tasks.workunit.client.1.smithi152.stdout:6/769: unlink d8/d92/d75/la4 0 2024-03-04T16:04:44.437 INFO:tasks.workunit.client.1.smithi152.stdout:0/577: write d9/d2c/d2e/f32 [953422,82783] 0 2024-03-04T16:04:44.438 INFO:tasks.workunit.client.1.smithi152.stdout:0/578: dread - d9/d2c/d2e/fa0 zero size 2024-03-04T16:04:44.439 INFO:tasks.workunit.client.1.smithi152.stdout:6/770: dread d8/d2a/f89 [0,4194304] 0 2024-03-04T16:04:44.441 INFO:tasks.workunit.client.1.smithi152.stdout:9/603: mkdir d2/d13/d23/d71/d80/dcc 0 2024-03-04T16:04:44.442 INFO:tasks.workunit.client.1.smithi152.stdout:9/604: write d2/f8d [3570701,66380] 0 2024-03-04T16:04:44.446 INFO:tasks.workunit.client.1.smithi152.stdout:9/605: write d2/d13/d38/f58 [620989,125266] 0 2024-03-04T16:04:44.446 INFO:tasks.workunit.client.1.smithi152.stdout:4/547: dread d8/d3d/f44 [0,4194304] 0 2024-03-04T16:04:44.449 INFO:tasks.workunit.client.1.smithi152.stdout:9/606: dread - d2/d13/d23/d66/d7d/f9a zero size 2024-03-04T16:04:44.450 INFO:tasks.workunit.client.1.smithi152.stdout:5/634: write d6/d8/d5a/d64/d76/d9f/f51 [963477,120856] 0 2024-03-04T16:04:44.453 INFO:tasks.workunit.client.1.smithi152.stdout:6/771: mknod d8/c10d 0 2024-03-04T16:04:44.456 INFO:tasks.workunit.client.1.smithi152.stdout:2/606: dread da/df/d2e/f9c [0,4194304] 0 2024-03-04T16:04:44.456 INFO:tasks.workunit.client.1.smithi152.stdout:0/579: rename d9/l89 to d9/d1b/d3c/d84/d56/d5b/d75/lb6 0 2024-03-04T16:04:44.458 INFO:tasks.workunit.client.1.smithi152.stdout:4/548: rename d8/d78/f95 to d8/d78/fb7 0 2024-03-04T16:04:44.459 INFO:tasks.workunit.client.1.smithi152.stdout:2/607: truncate da/df/fb7 287819 0 2024-03-04T16:04:44.460 INFO:tasks.workunit.client.1.smithi152.stdout:4/549: chown d8/d2f 121181 1 2024-03-04T16:04:44.461 INFO:tasks.workunit.client.1.smithi152.stdout:2/608: stat da/d5f/d85/d8c/lbe 0 2024-03-04T16:04:44.461 INFO:tasks.workunit.client.1.smithi152.stdout:5/635: rename d6/d8/f32 to d6/d8/d1f/d88/fcf 0 2024-03-04T16:04:44.463 INFO:tasks.workunit.client.1.smithi152.stdout:6/772: creat d8/d42/dde/dee/f10e x:0 0 0 2024-03-04T16:04:44.465 INFO:tasks.workunit.client.1.smithi152.stdout:0/580: dwrite d9/d1b/f35 [0,4194304] 0 2024-03-04T16:04:44.465 INFO:tasks.workunit.client.1.smithi152.stdout:9/607: dread d2/d13/d23/f4c [0,4194304] 0 2024-03-04T16:04:44.465 INFO:tasks.workunit.client.1.smithi152.stdout:4/550: write d8/d3d/f44 [1610518,36870] 0 2024-03-04T16:04:44.468 INFO:tasks.workunit.client.1.smithi152.stdout:0/581: dread - d9/d2c/d2e/d7f/d9d/fb0 zero size 2024-03-04T16:04:44.468 INFO:tasks.workunit.client.1.smithi152.stdout:0/582: chown d9/l2b 321438583 1 2024-03-04T16:04:44.471 INFO:tasks.workunit.client.1.smithi152.stdout:6/773: mknod d8/d92/d75/c10f 0 2024-03-04T16:04:44.472 INFO:tasks.workunit.client.1.smithi152.stdout:6/774: chown d8/d8f/da3/fba 919675 1 2024-03-04T16:04:44.473 INFO:tasks.workunit.client.1.smithi152.stdout:2/609: unlink da/c12 0 2024-03-04T16:04:44.477 INFO:tasks.workunit.client.1.smithi152.stdout:4/551: mknod d8/d4e/d8b/cb8 0 2024-03-04T16:04:44.478 INFO:tasks.workunit.client.1.smithi152.stdout:5/636: truncate d6/d8/f86 68253 0 2024-03-04T16:04:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.480+0000 7f615dbe4700 1 -- 172.21.15.116:0/2819684303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6158100cb0 msgr2=0x7f6158101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.480+0000 7f615dbe4700 1 --2- 172.21.15.116:0/2819684303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6158100cb0 0x7f6158101110 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f614c009910 tx=0x7f614c009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.480+0000 7f615dbe4700 1 -- 172.21.15.116:0/2819684303 shutdown_connections 2024-03-04T16:04:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.480+0000 7f615dbe4700 1 --2- 172.21.15.116:0/2819684303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6158100cb0 0x7f6158101110 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.480+0000 7f615dbe4700 1 --2- 172.21.15.116:0/2819684303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61580ffab0 0x7f61580ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.480+0000 7f615dbe4700 1 -- 172.21.15.116:0/2819684303 >> 172.21.15.116:0/2819684303 conn(0x7f61580fb030 msgr2=0x7f61580fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.480+0000 7f615dbe4700 1 -- 172.21.15.116:0/2819684303 shutdown_connections 2024-03-04T16:04:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.480+0000 7f615dbe4700 1 -- 172.21.15.116:0/2819684303 wait complete. 2024-03-04T16:04:44.483 INFO:tasks.workunit.client.1.smithi152.stdout:2/610: stat da/f77 0 2024-03-04T16:04:44.483 INFO:tasks.workunit.client.1.smithi152.stdout:0/583: dread d9/d2c/d2e/f31 [0,4194304] 0 2024-03-04T16:04:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.481+0000 7f615dbe4700 1 Processor -- start 2024-03-04T16:04:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.481+0000 7f615dbe4700 1 -- start start 2024-03-04T16:04:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.481+0000 7f615dbe4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61580ffab0 0x7f6158198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.481+0000 7f615dbe4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6158100cb0 0x7f6158198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.481+0000 7f615dbe4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6158199090 con 0x7f6158100cb0 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.481+0000 7f615dbe4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61581991d0 con 0x7f61580ffab0 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.481+0000 7f6156ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6158100cb0 0x7f6158198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.481+0000 7f6156ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6158100cb0 0x7f6158198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53848/0 (socket says 172.21.15.116:53848) 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f6156ffd700 1 -- 172.21.15.116:0/1057118869 learned_addr learned my addr 172.21.15.116:0/1057118869 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f61577fe700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61580ffab0 0x7f6158198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f6156ffd700 1 -- 172.21.15.116:0/1057118869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61580ffab0 msgr2=0x7f6158198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f6156ffd700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61580ffab0 0x7f6158198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f6156ffd700 1 -- 172.21.15.116:0/1057118869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c0095f0 con 0x7f6158100cb0 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f6156ffd700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6158100cb0 0x7f6158198a70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f614c000c00 tx=0x7f614c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f6154ff9700 1 -- 172.21.15.116:0/1057118869 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f614c01d070 con 0x7f6158100cb0 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f615dbe4700 1 -- 172.21.15.116:0/1057118869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615819dc20 con 0x7f6158100cb0 2024-03-04T16:04:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f6154ff9700 1 -- 172.21.15.116:0/1057118869 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f614c005dc0 con 0x7f6158100cb0 2024-03-04T16:04:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f6154ff9700 1 -- 172.21.15.116:0/1057118869 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f614c0226a0 con 0x7f6158100cb0 2024-03-04T16:04:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.482+0000 7f615dbe4700 1 -- 172.21.15.116:0/1057118869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615819e110 con 0x7f6158100cb0 2024-03-04T16:04:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.483+0000 7f6154ff9700 1 -- 172.21.15.116:0/1057118869 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 28) v1 ==== 46012+0+0 (secure 0 0 0) 0x7f614c022800 con 0x7f6158100cb0 2024-03-04T16:04:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.483+0000 7f615dbe4700 1 -- 172.21.15.116:0/1057118869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6158107500 con 0x7f6158100cb0 2024-03-04T16:04:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.483+0000 7f6154ff9700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6140038e10 0x7f614003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.484+0000 7f6154ff9700 1 -- 172.21.15.116:0/1057118869 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f614c04e510 con 0x7f6158100cb0 2024-03-04T16:04:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.484+0000 7f61577fe700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6140038e10 0x7f614003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.484+0000 7f61577fe700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6140038e10 0x7f614003b2d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6158100b10 tx=0x7f6148009260 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:44.487 INFO:tasks.workunit.client.1.smithi152.stdout:4/552: symlink d8/d4e/d86/lb9 0 2024-03-04T16:04:44.487 INFO:tasks.workunit.client.1.smithi152.stdout:4/553: write d8/d2f/d7c/fb1 [300259,95561] 0 2024-03-04T16:04:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.489+0000 7f6154ff9700 1 -- 172.21.15.116:0/1057118869 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f614c017920 con 0x7f6158100cb0 2024-03-04T16:04:44.490 INFO:tasks.workunit.client.1.smithi152.stdout:2/611: rename da/df/c18 to da/d5f/dbc/ccb 0 2024-03-04T16:04:44.492 INFO:tasks.workunit.client.1.smithi152.stdout:2/612: fsync da/d24/d80/f9b 0 2024-03-04T16:04:44.494 INFO:tasks.workunit.client.1.smithi152.stdout:2/613: readlink da/d24/d47/d73/lc3 0 2024-03-04T16:04:44.498 INFO:tasks.workunit.client.1.smithi152.stdout:9/608: dwrite d2/d13/d38/d79/f89 [0,4194304] 0 2024-03-04T16:04:44.500 INFO:tasks.workunit.client.1.smithi152.stdout:5/637: write d6/d50/f7e [925017,108640] 0 2024-03-04T16:04:44.500 INFO:tasks.workunit.client.1.smithi152.stdout:9/609: write d2/d13/d23/d71/da3/fab [3591774,10430] 0 2024-03-04T16:04:44.501 INFO:tasks.workunit.client.1.smithi152.stdout:6/775: dwrite d8/d10/d53/f5b [0,4194304] 0 2024-03-04T16:04:44.503 INFO:tasks.workunit.client.1.smithi152.stdout:6/776: chown d8/d23/c101 146722 1 2024-03-04T16:04:44.508 INFO:tasks.workunit.client.1.smithi152.stdout:2/614: creat da/d5f/dbc/fcc x:0 0 0 2024-03-04T16:04:44.510 INFO:tasks.workunit.client.1.smithi152.stdout:5/638: chown d6/d8/f86 50126363 1 2024-03-04T16:04:44.510 INFO:tasks.workunit.client.1.smithi152.stdout:9/610: mkdir d2/d13/d2e/dcd 0 2024-03-04T16:04:44.515 INFO:tasks.workunit.client.1.smithi152.stdout:4/554: getdents d8/d4e 0 2024-03-04T16:04:44.516 INFO:tasks.workunit.client.1.smithi152.stdout:4/555: write d8/d2f/d6f/d84/f8e [619484,48908] 0 2024-03-04T16:04:44.526 INFO:tasks.workunit.client.1.smithi152.stdout:4/556: write d8/d9/ff [2918152,71461] 0 2024-03-04T16:04:44.527 INFO:tasks.workunit.client.1.smithi152.stdout:6/777: write d8/d23/d3a/f8b [517977,44595] 0 2024-03-04T16:04:44.532 INFO:tasks.workunit.client.1.smithi152.stdout:9/611: write d2/d13/f15 [3540703,106888] 0 2024-03-04T16:04:44.532 INFO:tasks.workunit.client.1.smithi152.stdout:5/639: creat d6/d8/d5a/d64/fd0 x:0 0 0 2024-03-04T16:04:44.534 INFO:tasks.workunit.client.1.smithi152.stdout:5/640: chown d6/d8/f2f 843 1 2024-03-04T16:04:44.538 INFO:tasks.workunit.client.1.smithi152.stdout:6/778: dwrite d8/d92/db1/fd5 [0,4194304] 0 2024-03-04T16:04:44.538 INFO:tasks.workunit.client.1.smithi152.stdout:2/615: dread da/d5f/d85/f99 [0,4194304] 0 2024-03-04T16:04:44.540 INFO:tasks.workunit.client.1.smithi152.stdout:2/616: fsync da/d2c/f75 0 2024-03-04T16:04:44.547 INFO:tasks.workunit.client.1.smithi152.stdout:9/612: fsync d2/f28 0 2024-03-04T16:04:44.550 INFO:tasks.workunit.client.1.smithi152.stdout:2/617: dwrite da/df/d2e/d51/fb4 [0,4194304] 0 2024-03-04T16:04:44.550 INFO:tasks.workunit.client.1.smithi152.stdout:6/779: write d8/d92/d59/d61/d57/feb [2853920,75438] 0 2024-03-04T16:04:44.551 INFO:tasks.workunit.client.1.smithi152.stdout:4/557: dread f7 [0,4194304] 0 2024-03-04T16:04:44.555 INFO:tasks.workunit.client.1.smithi152.stdout:5/641: unlink d6/d8/d5a/d8b/fa5 0 2024-03-04T16:04:44.564 INFO:tasks.workunit.client.1.smithi152.stdout:9/613: creat d2/d13/d23/d66/d7d/da7/fce x:0 0 0 2024-03-04T16:04:44.565 INFO:tasks.workunit.client.1.smithi152.stdout:2/618: mknod da/df/d58/da2/ccd 0 2024-03-04T16:04:44.565 INFO:tasks.workunit.client.1.smithi152.stdout:9/614: dread - d2/d13/d23/d66/d7d/da7/fce zero size 2024-03-04T16:04:44.565 INFO:tasks.workunit.client.1.smithi152.stdout:4/558: mkdir d8/d4e/dba 0 2024-03-04T16:04:44.566 INFO:tasks.workunit.client.1.smithi152.stdout:2/619: dread - da/d2c/d5e/f69 zero size 2024-03-04T16:04:44.566 INFO:tasks.workunit.client.1.smithi152.stdout:6/780: mkdir d8/d92/d59/d61/d57/d110 0 2024-03-04T16:04:44.567 INFO:tasks.workunit.client.1.smithi152.stdout:9/615: dread - d2/d13/d23/d71/da3/fbc zero size 2024-03-04T16:04:44.572 INFO:tasks.workunit.client.1.smithi152.stdout:4/559: rmdir d8/d9/d79 39 2024-03-04T16:04:44.572 INFO:tasks.workunit.client.1.smithi152.stdout:6/781: mkdir d8/da9/d111 0 2024-03-04T16:04:44.580 INFO:tasks.workunit.client.1.smithi152.stdout:2/620: creat da/d24/d81/fce x:0 0 0 2024-03-04T16:04:44.580 INFO:tasks.workunit.client.1.smithi152.stdout:2/621: readlink da/d24/d47/d4f/laf 0 2024-03-04T16:04:44.580 INFO:tasks.workunit.client.1.smithi152.stdout:9/616: rename d2/l1d to d2/d6e/lcf 0 2024-03-04T16:04:44.580 INFO:tasks.workunit.client.1.smithi152.stdout:2/622: truncate da/d24/d81/fce 731232 0 2024-03-04T16:04:44.581 INFO:tasks.workunit.client.1.smithi152.stdout:9/617: stat d2/d13/d23/f5c 0 2024-03-04T16:04:44.583 INFO:tasks.workunit.client.1.smithi152.stdout:6/782: creat d8/d23/d3a/d8d/d96/d10a/f112 x:0 0 0 2024-03-04T16:04:44.583 INFO:tasks.workunit.client.1.smithi152.stdout:4/560: mknod d8/cbb 0 2024-03-04T16:04:44.585 INFO:tasks.workunit.client.1.smithi152.stdout:6/783: truncate d8/d23/d3a/f8b 575338 0 2024-03-04T16:04:44.585 INFO:tasks.workunit.client.1.smithi152.stdout:4/561: readlink d8/d9/d24/l6a 0 2024-03-04T16:04:44.585 INFO:tasks.workunit.client.1.smithi152.stdout:2/623: mkdir da/d24/d47/dcf 0 2024-03-04T16:04:44.586 INFO:tasks.workunit.client.1.smithi152.stdout:9/618: symlink d2/d13/d2e/ld0 0 2024-03-04T16:04:44.586 INFO:tasks.workunit.client.1.smithi152.stdout:9/619: chown d2/d13/d23/d71/c85 0 1 2024-03-04T16:04:44.588 INFO:tasks.workunit.client.1.smithi152.stdout:2/624: rmdir da/df/d2e/d51/d8d 39 2024-03-04T16:04:44.589 INFO:tasks.workunit.client.1.smithi152.stdout:5/642: dwrite d6/d8/f86 [0,4194304] 0 2024-03-04T16:04:44.589 INFO:tasks.workunit.client.1.smithi152.stdout:6/784: creat d8/d92/d59/d61/d57/d110/f113 x:0 0 0 2024-03-04T16:04:44.590 INFO:tasks.workunit.client.1.smithi152.stdout:2/625: chown da/d24/d47/f5a 165089572 1 2024-03-04T16:04:44.591 INFO:tasks.workunit.client.1.smithi152.stdout:2/626: chown da/df/d2e/l3b 10091 1 2024-03-04T16:04:44.592 INFO:tasks.workunit.client.1.smithi152.stdout:6/785: write d8/d92/d75/ffc [110477,119341] 0 2024-03-04T16:04:44.594 INFO:tasks.workunit.client.1.smithi152.stdout:2/627: fsync da/d2c/d3f/f9d 0 2024-03-04T16:04:44.599 INFO:tasks.workunit.client.1.smithi152.stdout:6/786: unlink d8/d23/d3a/f50 0 2024-03-04T16:04:44.601 INFO:tasks.workunit.client.1.smithi152.stdout:2/628: mkdir da/d24/d80/dd0 0 2024-03-04T16:04:44.604 INFO:tasks.workunit.client.1.smithi152.stdout:5/643: mkdir d6/d8/d5a/d64/d76/d9f/dc7/dcb/dd1 0 2024-03-04T16:04:44.606 INFO:tasks.workunit.client.1.smithi152.stdout:4/562: dwrite d8/d2f/f4a [0,4194304] 0 2024-03-04T16:04:44.607 INFO:tasks.workunit.client.1.smithi152.stdout:5/644: write d6/d50/f83 [1170797,105659] 0 2024-03-04T16:04:44.610 INFO:tasks.workunit.client.1.smithi152.stdout:4/563: read d8/d9/d24/d26/f32 [3466009,14408] 0 2024-03-04T16:04:44.611 INFO:tasks.workunit.client.1.smithi152.stdout:2/629: symlink da/d24/d47/dcf/ld1 0 2024-03-04T16:04:44.612 INFO:tasks.workunit.client.1.smithi152.stdout:2/630: chown da/d24/d80/d82/da8/f5c 4753995 1 2024-03-04T16:04:44.616 INFO:tasks.workunit.client.1.smithi152.stdout:5/645: unlink d6/f4e 0 2024-03-04T16:04:44.617 INFO:tasks.workunit.client.1.smithi152.stdout:6/787: link d8/d92/d59/d7d/d91/lb0 d8/d42/l114 0 2024-03-04T16:04:44.622 INFO:tasks.workunit.client.1.smithi152.stdout:6/788: mknod d8/d23/d98/c115 0 2024-03-04T16:04:44.624 INFO:tasks.workunit.client.1.smithi152.stdout:5/646: truncate d6/d8/d5a/d64/d76/d9f/f7c 343202 0 2024-03-04T16:04:44.625 INFO:tasks.workunit.client.1.smithi152.stdout:2/631: creat da/df/d58/da2/db5/fd2 x:0 0 0 2024-03-04T16:04:44.629 INFO:tasks.workunit.client.1.smithi152.stdout:5/647: creat d6/d8/d5a/d64/d76/d9f/dc7/dcb/fd2 x:0 0 0 2024-03-04T16:04:44.630 INFO:tasks.workunit.client.1.smithi152.stdout:2/632: creat da/d24/d80/fd3 x:0 0 0 2024-03-04T16:04:44.633 INFO:tasks.workunit.client.1.smithi152.stdout:6/789: symlink d8/d92/db1/d87/l116 0 2024-03-04T16:04:44.633 INFO:tasks.workunit.client.1.smithi152.stdout:5/648: symlink d6/d8/d5a/d64/d76/d9f/dc7/dcb/ld3 0 2024-03-04T16:04:44.638 INFO:tasks.workunit.client.1.smithi152.stdout:5/649: mknod d6/d8/d5a/d64/d76/d9f/cd4 0 2024-03-04T16:04:44.644 INFO:tasks.workunit.client.1.smithi152.stdout:5/650: rename d6/d50/l60 to d6/d8/d1f/d3d/da4/dc9/ld5 0 2024-03-04T16:04:44.647 INFO:tasks.workunit.client.1.smithi152.stdout:2/633: dread da/f1a [0,4194304] 0 2024-03-04T16:04:44.647 INFO:tasks.workunit.client.1.smithi152.stdout:2/634: chown f6 2 1 2024-03-04T16:04:44.652 INFO:tasks.workunit.client.1.smithi152.stdout:8/761: sync 2024-03-04T16:04:44.652 INFO:tasks.workunit.client.1.smithi152.stdout:1/664: sync 2024-03-04T16:04:44.652 INFO:tasks.workunit.client.1.smithi152.stdout:7/523: sync 2024-03-04T16:04:44.652 INFO:tasks.workunit.client.1.smithi152.stdout:7/524: write d14/d21/d47/f5e [4467296,10726] 0 2024-03-04T16:04:44.658 INFO:tasks.workunit.client.1.smithi152.stdout:7/525: truncate d14/d21/d47/d36/f68 2084343 0 2024-03-04T16:04:44.659 INFO:tasks.workunit.client.1.smithi152.stdout:8/762: dwrite d2/d3/d21/d2d/d52/d68/fae [0,4194304] 0 2024-03-04T16:04:44.660 INFO:tasks.workunit.client.1.smithi152.stdout:7/526: write d14/f78 [1984377,83952] 0 2024-03-04T16:04:44.671 INFO:tasks.workunit.client.1.smithi152.stdout:8/763: symlink d2/d75/d89/dd/d22/lf7 0 2024-03-04T16:04:44.675 INFO:tasks.workunit.client.1.smithi152.stdout:1/665: link d19/d20/d2f/d6b/d38/d63/c9e d19/d20/d2f/d6b/d38/d63/d9a/cd4 0 2024-03-04T16:04:44.676 INFO:tasks.workunit.client.1.smithi152.stdout:5/651: rmdir d6 39 2024-03-04T16:04:44.679 INFO:tasks.workunit.client.1.smithi152.stdout:6/790: dwrite d8/d23/f2c [0,4194304] 0 2024-03-04T16:04:44.681 INFO:tasks.workunit.client.1.smithi152.stdout:6/791: readlink d8/d42/l62 0 2024-03-04T16:04:44.681 INFO:tasks.workunit.client.1.smithi152.stdout:1/666: symlink d19/d20/d2f/d6b/d38/da6/ld5 0 2024-03-04T16:04:44.683 INFO:tasks.workunit.client.1.smithi152.stdout:8/764: mknod d2/d3/d21/d2d/d52/dcc/de7/df2/cf8 0 2024-03-04T16:04:44.688 INFO:tasks.workunit.client.1.smithi152.stdout:2/635: dwrite da/f48 [0,4194304] 0 2024-03-04T16:04:44.689 INFO:tasks.workunit.client.1.smithi152.stdout:6/792: symlink d8/d92/l117 0 2024-03-04T16:04:44.696 INFO:tasks.workunit.client.1.smithi152.stdout:1/667: dwrite d19/d4b/d53/d57/d62/d82/fc2 [0,4194304] 0 2024-03-04T16:04:44.699 INFO:tasks.workunit.client.1.smithi152.stdout:8/765: mkdir d2/d3/d21/d2d/d52/d68/d6e/d7d/df9 0 2024-03-04T16:04:44.700 INFO:tasks.workunit.client.1.smithi152.stdout:2/636: rmdir da/df/d2e/d61 39 2024-03-04T16:04:44.700 INFO:tasks.workunit.client.1.smithi152.stdout:1/668: stat l11 0 2024-03-04T16:04:44.704 INFO:tasks.workunit.client.1.smithi152.stdout:6/793: dwrite d8/d42/f4a [4194304,4194304] 0 2024-03-04T16:04:44.708 INFO:tasks.workunit.client.1.smithi152.stdout:5/652: mkdir d6/d8/d5a/d90/dd6 0 2024-03-04T16:04:44.710 INFO:tasks.workunit.client.1.smithi152.stdout:8/766: mkdir d2/d3/d21/d2d/d52/dfa 0 2024-03-04T16:04:44.715 INFO:tasks.workunit.client.1.smithi152.stdout:6/794: unlink d8/d42/cce 0 2024-03-04T16:04:44.715 INFO:tasks.workunit.client.1.smithi152.stdout:8/767: dread d2/d7f/fc2 [0,4194304] 0 2024-03-04T16:04:44.717 INFO:tasks.workunit.client.1.smithi152.stdout:2/637: mknod da/df/d2e/cd4 0 2024-03-04T16:04:44.717 INFO:tasks.workunit.client.1.smithi152.stdout:2/638: stat da/d2c/c4c 0 2024-03-04T16:04:44.718 INFO:tasks.workunit.client.1.smithi152.stdout:8/768: stat d2/d75/d89/dd/d22/d83/d96/da6/fbd 0 2024-03-04T16:04:44.718 INFO:tasks.workunit.client.1.smithi152.stdout:1/669: mkdir d19/d4b/d53/d57/dd6 0 2024-03-04T16:04:44.721 INFO:tasks.workunit.client.1.smithi152.stdout:5/653: mknod d6/d8/cd7 0 2024-03-04T16:04:44.722 INFO:tasks.workunit.client.1.smithi152.stdout:8/769: mknod d2/d75/d89/dd/d22/d26/cfb 0 2024-03-04T16:04:44.723 INFO:tasks.workunit.client.1.smithi152.stdout:2/639: dread da/d2c/d3f/f72 [0,4194304] 0 2024-03-04T16:04:44.726 INFO:tasks.workunit.client.1.smithi152.stdout:6/795: rename d8/d92/d59/c6c to d8/d10/d53/c118 0 2024-03-04T16:04:44.730 INFO:tasks.workunit.client.1.smithi152.stdout:8/770: fdatasync d2/d3/d15/d1c/d84/d87/f6f 0 2024-03-04T16:04:44.731 INFO:tasks.workunit.client.1.smithi152.stdout:2/640: mkdir da/d5f/dbc/dd5 0 2024-03-04T16:04:44.732 INFO:tasks.workunit.client.1.smithi152.stdout:2/641: chown da/df/d2e/l98 25870613 1 2024-03-04T16:04:44.734 INFO:tasks.workunit.client.1.smithi152.stdout:1/670: rename d19/d20/d2f/d6b/d38 to d19/d3b/d6e/d97/dd7 0 2024-03-04T16:04:44.745 INFO:tasks.workunit.client.1.smithi152.stdout:6/796: dread d8/d23/d3a/d71/d9d/fa8 [0,4194304] 0 2024-03-04T16:04:44.746 INFO:tasks.workunit.client.1.smithi152.stdout:2/642: creat da/df/d2e/dc6/fd6 x:0 0 0 2024-03-04T16:04:44.748 INFO:tasks.workunit.client.1.smithi152.stdout:1/671: dread d19/d75/fa0 [4194304,4194304] 0 2024-03-04T16:04:44.750 INFO:tasks.workunit.client.1.smithi152.stdout:1/672: chown d19/d4b/d53/d57/d62/d82/fbe 72 1 2024-03-04T16:04:44.755 INFO:tasks.workunit.client.1.smithi152.stdout:1/673: dwrite f16 [0,4194304] 0 2024-03-04T16:04:44.758 INFO:tasks.workunit.client.1.smithi152.stdout:5/654: getdents d6/d50 0 2024-03-04T16:04:44.760 INFO:tasks.workunit.client.1.smithi152.stdout:6/797: symlink d8/d42/dde/dee/l119 0 2024-03-04T16:04:44.765 INFO:tasks.workunit.client.1.smithi152.stdout:8/771: rename d2/d3/d21/dce to d2/d75/d89/dd/dfc 0 2024-03-04T16:04:44.767 INFO:tasks.workunit.client.1.smithi152.stdout:1/674: mkdir d19/d4b/d53/d57/d62/dd8 0 2024-03-04T16:04:44.767 INFO:tasks.workunit.client.1.smithi152.stdout:8/772: write d2/d3/d21/dac/fdb [212560,76213] 0 2024-03-04T16:04:44.768 INFO:tasks.workunit.client.1.smithi152.stdout:1/675: chown d19/d4b/d53/d57/d62/d82/faa 0 1 2024-03-04T16:04:44.770 INFO:tasks.workunit.client.1.smithi152.stdout:6/798: mkdir d8/d10/d53/d9a/d11a 0 2024-03-04T16:04:44.772 INFO:tasks.workunit.client.1.smithi152.stdout:2/643: creat da/d24/d80/dd0/fd7 x:0 0 0 2024-03-04T16:04:44.773 INFO:tasks.workunit.client.1.smithi152.stdout:2/644: write da/f3c [179738,50181] 0 2024-03-04T16:04:44.774 INFO:tasks.workunit.client.1.smithi152.stdout:1/676: mkdir d19/d75/dd9 0 2024-03-04T16:04:44.774 INFO:tasks.workunit.client.1.smithi152.stdout:2/645: chown da/df/d2e/dc6 5368197 1 2024-03-04T16:04:44.776 INFO:tasks.workunit.client.1.smithi152.stdout:2/646: readlink da/d24/d47/d4f/l6c 0 2024-03-04T16:04:44.781 INFO:tasks.workunit.client.1.smithi152.stdout:6/799: creat d8/d8f/f11b x:0 0 0 2024-03-04T16:04:44.784 INFO:tasks.workunit.client.1.smithi152.stdout:1/677: symlink d19/d3b/d6e/d97/lda 0 2024-03-04T16:04:44.784 INFO:tasks.workunit.client.1.smithi152.stdout:5/655: write d6/d8/d5a/d64/f6e [931898,127330] 0 2024-03-04T16:04:44.785 INFO:tasks.workunit.client.1.smithi152.stdout:5/656: read d6/f38 [3029796,34879] 0 2024-03-04T16:04:44.785 INFO:tasks.workunit.client.1.smithi152.stdout:6/800: truncate d8/d92/d59/d61/f78 346740 0 2024-03-04T16:04:44.787 INFO:tasks.workunit.client.1.smithi152.stdout:1/678: symlink d19/d75/ldb 0 2024-03-04T16:04:44.788 INFO:tasks.workunit.client.1.smithi152.stdout:2/647: dwrite da/df/d2e/d61/f84 [0,4194304] 0 2024-03-04T16:04:44.790 INFO:tasks.workunit.client.1.smithi152.stdout:8/773: getdents d2/d75/d89/dd/dfc 0 2024-03-04T16:04:44.798 INFO:tasks.workunit.client.1.smithi152.stdout:1/679: creat d19/d4b/d53/d57/d62/fdc x:0 0 0 2024-03-04T16:04:44.798 INFO:tasks.workunit.client.1.smithi152.stdout:5/657: dread d6/d8/d1f/d3d/d6f/d70/fa8 [0,4194304] 0 2024-03-04T16:04:44.799 INFO:tasks.workunit.client.1.smithi152.stdout:1/680: chown d19/f5b 1035 1 2024-03-04T16:04:44.799 INFO:tasks.workunit.client.1.smithi152.stdout:2/648: unlink da/d5f/f9e 0 2024-03-04T16:04:44.801 INFO:tasks.workunit.client.1.smithi152.stdout:2/649: write da/df/d58/d6e/fc8 [109052,12514] 0 2024-03-04T16:04:44.802 INFO:tasks.workunit.client.1.smithi152.stdout:8/774: creat d2/d75/d89/dd/dfc/ffd x:0 0 0 2024-03-04T16:04:44.804 INFO:tasks.workunit.client.1.smithi152.stdout:6/801: dread - d8/d23/d3a/d8d/d96/d10a/f112 zero size 2024-03-04T16:04:44.805 INFO:tasks.workunit.client.1.smithi152.stdout:8/775: write d2/d3/d15/d1c/d84/d87/d5d/f8b [725542,45705] 0 2024-03-04T16:04:44.806 INFO:tasks.workunit.client.1.smithi152.stdout:2/650: creat da/d24/d81/fd8 x:0 0 0 2024-03-04T16:04:44.810 INFO:tasks.workunit.client.1.smithi152.stdout:1/681: getdents d19/d4b 0 2024-03-04T16:04:44.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.813+0000 7f615dbe4700 1 -- 172.21.15.116:0/1057118869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f615819e720 con 0x7f6158100cb0 2024-03-04T16:04:44.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.814+0000 7f6154ff9700 1 -- 172.21.15.116:0/1057118869 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1831 (secure 0 0 0) 0x7f614c020c00 con 0x7f6158100cb0 2024-03-04T16:04:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:04:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:04:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:04:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:04:44.818 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:04:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:04:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:04:44.820 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:04:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:04:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:04:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:04:44.822 INFO:tasks.workunit.client.1.smithi152.stdout:2/651: dwrite da/d2c/d5e/f69 [0,4194304] 0 2024-03-04T16:04:44.822 INFO:tasks.workunit.client.1.smithi152.stdout:8/776: dwrite d2/d3/d21/d2d/d52/d68/fae [0,4194304] 0 2024-03-04T16:04:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 -- 172.21.15.116:0/1057118869 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6140038e10 msgr2=0x7f614003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6140038e10 0x7f614003b2d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6158100b10 tx=0x7f6148009260 comp rx=0 tx=0).stop 2024-03-04T16:04:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 -- 172.21.15.116:0/1057118869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6158100cb0 msgr2=0x7f6158198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6158100cb0 0x7f6158198a70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f614c000c00 tx=0x7f614c00f690 comp rx=0 tx=0).stop 2024-03-04T16:04:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 -- 172.21.15.116:0/1057118869 shutdown_connections 2024-03-04T16:04:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6140038e10 0x7f614003b2d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61580ffab0 0x7f6158198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 --2- 172.21.15.116:0/1057118869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6158100cb0 0x7f6158198a70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 -- 172.21.15.116:0/1057118869 >> 172.21.15.116:0/1057118869 conn(0x7f61580fb030 msgr2=0x7f6158103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:44.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 -- 172.21.15.116:0/1057118869 shutdown_connections 2024-03-04T16:04:44.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.816+0000 7f615dbe4700 1 -- 172.21.15.116:0/1057118869 wait complete. 2024-03-04T16:04:44.823 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:04:44.823 INFO:tasks.workunit.client.1.smithi152.stdout:1/682: rename d19/f28 to d19/d75/dc3/fdd 0 2024-03-04T16:04:44.826 INFO:tasks.workunit.client.1.smithi152.stdout:5/658: dread d6/d8/f15 [0,4194304] 0 2024-03-04T16:04:44.830 INFO:tasks.workunit.client.1.smithi152.stdout:5/659: read d6/d8/f30 [1384063,98266] 0 2024-03-04T16:04:44.832 INFO:tasks.workunit.client.1.smithi152.stdout:8/777: dread d2/d3/d21/d2d/d31/f7a [0,4194304] 0 2024-03-04T16:04:44.832 INFO:tasks.workunit.client.1.smithi152.stdout:2/652: rename da/d5f/c7e to da/df/d58/da2/cd9 0 2024-03-04T16:04:44.833 INFO:tasks.workunit.client.1.smithi152.stdout:2/653: fdatasync da/d2c/f75 0 2024-03-04T16:04:44.833 INFO:tasks.workunit.client.1.smithi152.stdout:8/778: dread - d2/d3/d15/d1c/d84/fbb zero size 2024-03-04T16:04:44.835 INFO:tasks.workunit.client.1.smithi152.stdout:2/654: fdatasync da/d24/d47/f5a 0 2024-03-04T16:04:44.835 INFO:tasks.workunit.client.1.smithi152.stdout:8/779: stat d2/d3/d21/d2d/d52/d68/d6e/d7d/df9 0 2024-03-04T16:04:44.837 INFO:tasks.workunit.client.1.smithi152.stdout:8/780: write d2/d3/d15/d1c/d84/d87/fd1 [661916,63502] 0 2024-03-04T16:04:44.843 INFO:tasks.workunit.client.1.smithi152.stdout:5/660: mkdir d6/d8/d5a/d64/d76/d9f/db0/dd8 0 2024-03-04T16:04:44.844 INFO:tasks.workunit.client.1.smithi152.stdout:8/781: creat d2/d3/d21/d2d/d52/d68/ffe x:0 0 0 2024-03-04T16:04:44.846 INFO:tasks.workunit.client.1.smithi152.stdout:5/661: dread - d6/d8/d5a/d64/d76/d9f/dc7/dcb/fd2 zero size 2024-03-04T16:04:44.846 INFO:tasks.workunit.client.1.smithi152.stdout:8/782: write d2/d75/d89/dd/f93 [1817568,111949] 0 2024-03-04T16:04:44.848 INFO:tasks.workunit.client.1.smithi152.stdout:8/783: truncate d2/d3/fcf 665269 0 2024-03-04T16:04:44.854 INFO:tasks.workunit.client.1.smithi152.stdout:2/655: dread da/d24/d80/d82/da8/f5c [0,4194304] 0 2024-03-04T16:04:44.855 INFO:tasks.workunit.client.1.smithi152.stdout:2/656: readlink da/d24/d47/dcf/ld1 0 2024-03-04T16:04:44.877 INFO:tasks.workunit.client.1.smithi152.stdout:0/584: sync 2024-03-04T16:04:44.877 INFO:tasks.workunit.client.1.smithi152.stdout:3/646: sync 2024-03-04T16:04:44.878 INFO:tasks.workunit.client.1.smithi152.stdout:0/585: write d9/d1b/f35 [4572981,28676] 0 2024-03-04T16:04:44.879 INFO:tasks.workunit.client.1.smithi152.stdout:3/647: readlink d6/d60/d8e/dc0/da1/lb4 0 2024-03-04T16:04:44.880 INFO:tasks.workunit.client.1.smithi152.stdout:1/683: write d19/d4b/d53/d57/f88 [1412598,4263] 0 2024-03-04T16:04:44.881 INFO:tasks.workunit.client.1.smithi152.stdout:3/648: read - d6/d60/d8e/dc0/faf zero size 2024-03-04T16:04:44.882 INFO:tasks.workunit.client.1.smithi152.stdout:3/649: fdatasync d6/d14/d58/f7e 0 2024-03-04T16:04:44.884 INFO:tasks.workunit.client.1.smithi152.stdout:8/784: mkdir d2/d3/d21/d2d/d31/dff 0 2024-03-04T16:04:44.884 INFO:tasks.workunit.client.1.smithi152.stdout:5/662: creat d6/d8/d5a/d90/dd6/fd9 x:0 0 0 2024-03-04T16:04:44.886 INFO:tasks.workunit.client.1.smithi152.stdout:8/785: dread d2/d3/fcf [0,4194304] 0 2024-03-04T16:04:44.888 INFO:tasks.workunit.client.1.smithi152.stdout:2/657: write da/f1a [2205906,122994] 0 2024-03-04T16:04:44.899 INFO:tasks.workunit.client.1.smithi152.stdout:8/786: mkdir d2/d3/d21/d2d/d52/dcc/dd6/d100 0 2024-03-04T16:04:44.901 INFO:tasks.workunit.client.1.smithi152.stdout:0/586: write d9/d2c/d2e/d7f/f9f [786322,97706] 0 2024-03-04T16:04:44.901 INFO:tasks.workunit.client.1.smithi152.stdout:2/658: dread da/df/f13 [4194304,4194304] 0 2024-03-04T16:04:44.904 INFO:tasks.workunit.client.1.smithi152.stdout:1/684: rename d19/d3b/d6e/d97/dd7/cc8 to d19/d4b/d53/d57/d62/dac/cde 0 2024-03-04T16:04:44.906 INFO:tasks.workunit.client.1.smithi152.stdout:3/650: creat d6/dd9/fdd x:0 0 0 2024-03-04T16:04:44.907 INFO:tasks.workunit.client.1.smithi152.stdout:3/651: stat d6/d14/d41/d49/d84 0 2024-03-04T16:04:44.909 INFO:tasks.workunit.client.1.smithi152.stdout:5/663: creat d6/d8/d5a/d64/d76/d9f/dc7/dcb/dd1/fda x:0 0 0 2024-03-04T16:04:44.913 INFO:tasks.workunit.client.1.smithi152.stdout:0/587: rmdir d9/d1a/d7c 39 2024-03-04T16:04:44.918 INFO:tasks.workunit.client.1.smithi152.stdout:8/787: dwrite d2/d3/d21/d2d/d52/d68/f72 [0,4194304] 0 2024-03-04T16:04:44.918 INFO:tasks.workunit.client.1.smithi152.stdout:0/588: dread d9/d1b/d54/d69/d95/fa5 [0,4194304] 0 2024-03-04T16:04:44.918 INFO:tasks.workunit.client.1.smithi152.stdout:2/659: dread f6 [0,4194304] 0 2024-03-04T16:04:44.920 INFO:tasks.workunit.client.1.smithi152.stdout:0/589: dread - d9/d2c/d2e/fa0 zero size 2024-03-04T16:04:44.921 INFO:tasks.workunit.client.1.smithi152.stdout:0/590: chown d9/d2c/d2e 60797134 1 2024-03-04T16:04:44.925 INFO:tasks.workunit.client.1.smithi152.stdout:0/591: dread d9/d2c/d2e/d7f/d9d/da3/fab [0,4194304] 0 2024-03-04T16:04:44.934 INFO:tasks.workunit.client.1.smithi152.stdout:5/664: write d6/d8/f30 [3123401,28710] 0 2024-03-04T16:04:44.935 INFO:tasks.workunit.client.1.smithi152.stdout:3/652: creat d6/d14/d58/dad/fde x:0 0 0 2024-03-04T16:04:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.940+0000 7f8ce50ec700 1 -- 172.21.15.116:0/2398549166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0100cb0 msgr2=0x7f8ce0101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.940+0000 7f8ce50ec700 1 --2- 172.21.15.116:0/2398549166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0100cb0 0x7f8ce0101110 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4009910 tx=0x7f8cd4009c20 comp rx=0 tx=0).stop 2024-03-04T16:04:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.940+0000 7f8ce50ec700 1 -- 172.21.15.116:0/2398549166 shutdown_connections 2024-03-04T16:04:44.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.940+0000 7f8ce50ec700 1 --2- 172.21.15.116:0/2398549166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0100cb0 0x7f8ce0101110 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.940+0000 7f8ce50ec700 1 --2- 172.21.15.116:0/2398549166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ce00ffab0 0x7f8ce00ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.940+0000 7f8ce50ec700 1 -- 172.21.15.116:0/2398549166 >> 172.21.15.116:0/2398549166 conn(0x7f8ce00fb050 msgr2=0x7f8ce00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:44.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.941+0000 7f8ce50ec700 1 -- 172.21.15.116:0/2398549166 shutdown_connections 2024-03-04T16:04:44.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.941+0000 7f8ce50ec700 1 -- 172.21.15.116:0/2398549166 wait complete. 2024-03-04T16:04:44.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.942+0000 7f8ce50ec700 1 Processor -- start 2024-03-04T16:04:44.943 INFO:tasks.workunit.client.1.smithi152.stdout:8/788: rename d2/d3/d21/d2d/d52/d68/d6e/l7c to d2/d75/d89/dd/d22/ddf/l101 0 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.942+0000 7f8ce50ec700 1 -- start start 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.942+0000 7f8ce50ec700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ce00ffab0 0x7f8ce019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.942+0000 7f8ce50ec700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0100cb0 0x7f8ce019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.942+0000 7f8ce50ec700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce019d4b0 con 0x7f8ce0100cb0 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.942+0000 7f8ce50ec700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce019d5f0 con 0x7f8ce00ffab0 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.942+0000 7f8cde59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0100cb0 0x7f8ce019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.942+0000 7f8cded9d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ce00ffab0 0x7f8ce019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.942+0000 7f8cde59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0100cb0 0x7f8ce019ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46102/0 (socket says 172.21.15.116:46102) 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.942+0000 7f8cde59c700 1 -- 172.21.15.116:0/186446646 learned_addr learned my addr 172.21.15.116:0/186446646 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.943+0000 7f8cde59c700 1 -- 172.21.15.116:0/186446646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ce00ffab0 msgr2=0x7f8ce019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.943+0000 7f8cde59c700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ce00ffab0 0x7f8ce019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.943+0000 7f8cde59c700 1 -- 172.21.15.116:0/186446646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd40095f0 con 0x7f8ce0100cb0 2024-03-04T16:04:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.943+0000 7f8cded9d700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ce00ffab0 0x7f8ce019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T16:04:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.943+0000 7f8cde59c700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0100cb0 0x7f8ce019ce90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400b5c0 tx=0x7f8cd400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.946+0000 7f8cc7fff700 1 -- 172.21.15.116:0/186446646 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cd401d070 con 0x7f8ce0100cb0 2024-03-04T16:04:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.946+0000 7f8cc7fff700 1 -- 172.21.15.116:0/186446646 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8cd400f460 con 0x7f8ce0100cb0 2024-03-04T16:04:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.946+0000 7f8ce50ec700 1 -- 172.21.15.116:0/186446646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce01a2040 con 0x7f8ce0100cb0 2024-03-04T16:04:44.947 INFO:tasks.workunit.client.1.smithi152.stdout:1/685: fdatasync d19/d20/d2f/d6b/f37 0 2024-03-04T16:04:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.946+0000 7f8ce50ec700 1 -- 172.21.15.116:0/186446646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce01a2530 con 0x7f8ce0100cb0 2024-03-04T16:04:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.946+0000 7f8cc7fff700 1 -- 172.21.15.116:0/186446646 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cd4021640 con 0x7f8ce0100cb0 2024-03-04T16:04:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.947+0000 7f8cc7fff700 1 -- 172.21.15.116:0/186446646 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 28) v1 ==== 46012+0+0 (secure 0 0 0) 0x7f8cd402b440 con 0x7f8ce0100cb0 2024-03-04T16:04:44.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.947+0000 7f8ce50ec700 1 -- 172.21.15.116:0/186446646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce01074b0 con 0x7f8ce0100cb0 2024-03-04T16:04:44.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.947+0000 7f8cc7fff700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8cc8038e80 0x7f8cc803b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:44.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.947+0000 7f8cc7fff700 1 -- 172.21.15.116:0/186446646 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f8cd404e2b0 con 0x7f8ce0100cb0 2024-03-04T16:04:44.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.948+0000 7f8cded9d700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8cc8038e80 0x7f8cc803b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:44.949 INFO:tasks.workunit.client.1.smithi152.stdout:2/660: dwrite da/d24/d80/d82/da8/f5c [4194304,4194304] 0 2024-03-04T16:04:44.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.948+0000 7f8cded9d700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8cc8038e80 0x7f8cc803b340 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0100b10 tx=0x7f8cd000b410 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:44.955+0000 7f8cc7fff700 1 -- 172.21.15.116:0/186446646 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f8cd40298f0 con 0x7f8ce0100cb0 2024-03-04T16:04:44.959 INFO:tasks.workunit.client.1.smithi152.stdout:8/789: write d2/d3/d15/d1c/d84/d87/d5d/f7b [570386,24479] 0 2024-03-04T16:04:44.964 INFO:tasks.workunit.client.1.smithi152.stdout:2/661: mknod da/df/d58/cda 0 2024-03-04T16:04:44.965 INFO:tasks.workunit.client.1.smithi152.stdout:2/662: stat da/df/d35/la9 0 2024-03-04T16:04:44.966 INFO:tasks.workunit.client.1.smithi152.stdout:3/653: write d6/d10/d2d/f3b [113482,37389] 0 2024-03-04T16:04:44.968 INFO:tasks.workunit.client.1.smithi152.stdout:5/665: mkdir d6/d8/d1f/d3d/d6f/d70/ddb 0 2024-03-04T16:04:44.971 INFO:tasks.workunit.client.1.smithi152.stdout:0/592: link d9/d2c/d2e/fa1 d9/d1b/d54/fb7 0 2024-03-04T16:04:44.971 INFO:tasks.workunit.client.1.smithi152.stdout:8/790: chown d2/d3/d21/d2d/d52/le3 7897847 1 2024-03-04T16:04:44.974 INFO:tasks.workunit.client.1.smithi152.stdout:3/654: mkdir d6/d14/d41/ddf 0 2024-03-04T16:04:44.975 INFO:tasks.workunit.client.1.smithi152.stdout:3/655: dread - d6/d60/d8e/dc0/fc1 zero size 2024-03-04T16:04:44.976 INFO:tasks.workunit.client.1.smithi152.stdout:3/656: chown d6/d14/d41/d49/f5a 41 1 2024-03-04T16:04:44.976 INFO:tasks.workunit.client.1.smithi152.stdout:3/657: chown d6/d5c 3 1 2024-03-04T16:04:44.977 INFO:tasks.workunit.client.1.smithi152.stdout:5/666: rmdir d6/d8 39 2024-03-04T16:04:44.980 INFO:tasks.workunit.client.1.smithi152.stdout:2/663: symlink da/df/d2e/ldb 0 2024-03-04T16:04:44.984 INFO:tasks.workunit.client.1.smithi152.stdout:3/658: unlink d6/d60/d8e/fbd 0 2024-03-04T16:04:44.988 INFO:tasks.workunit.client.1.smithi152.stdout:3/659: chown d6/d5c/f97 214 1 2024-03-04T16:04:44.990 INFO:tasks.workunit.client.1.smithi152.stdout:3/660: write d6/d14/d41/d49/d84/fba [525003,120404] 0 2024-03-04T16:04:44.990 INFO:tasks.workunit.client.1.smithi152.stdout:5/667: unlink d6/d8/d1f/d3d/f5d 0 2024-03-04T16:04:44.992 INFO:tasks.workunit.client.1.smithi152.stdout:3/661: truncate d6/d14/d58/dad/fde 631524 0 2024-03-04T16:04:44.994 INFO:tasks.workunit.client.1.smithi152.stdout:0/593: getdents d9/d1b/d3c 0 2024-03-04T16:04:44.996 INFO:tasks.workunit.client.1.smithi152.stdout:5/668: fsync d6/d8/d1f/f3a 0 2024-03-04T16:04:45.001 INFO:tasks.workunit.client.1.smithi152.stdout:0/594: dwrite d9/d1b/d54/d69/f9a [0,4194304] 0 2024-03-04T16:04:45.004 INFO:tasks.workunit.client.1.smithi152.stdout:0/595: chown d9/d1b/d3c/d84/d56/f63 1 1 2024-03-04T16:04:45.005 INFO:tasks.workunit.client.1.smithi152.stdout:5/669: dwrite d6/d50/fc8 [0,4194304] 0 2024-03-04T16:04:45.006 INFO:tasks.workunit.client.1.smithi152.stdout:0/596: dread - d9/d2c/d2e/d7f/d9d/fa7 zero size 2024-03-04T16:04:45.012 INFO:tasks.workunit.client.1.smithi152.stdout:0/597: dread d9/d1b/d54/d69/f9a [0,4194304] 0 2024-03-04T16:04:45.018 INFO:tasks.workunit.client.1.smithi152.stdout:0/598: link d9/d2c/d2e/d7f/d9d/da3/faf d9/d1b/d54/d80/fb8 0 2024-03-04T16:04:45.019 INFO:tasks.workunit.client.1.smithi152.stdout:0/599: write d9/d1b/d3c/fa9 [53968,5655] 0 2024-03-04T16:04:45.026 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:44 smithi116 ceph-mon[29120]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:45.026 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:44 smithi116 ceph-mon[29120]: pgmap v4: 65 pgs: 65 active+clean; 2.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail 2024-03-04T16:04:45.026 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:44 smithi116 ceph-mon[29120]: mgrmap e28: smithi116.jwdhop(active, since 2s) 2024-03-04T16:04:45.026 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:44 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/869404526' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:04:45.026 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:44 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1057118869' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:04:45.027 INFO:tasks.workunit.client.1.smithi152.stdout:0/600: fsync d9/f49 0 2024-03-04T16:04:45.027 INFO:tasks.workunit.client.1.smithi152.stdout:8/791: dwrite d2/d3/d21/d2d/fc [8388608,4194304] 0 2024-03-04T16:04:45.032 INFO:tasks.workunit.client.1.smithi152.stdout:8/792: creat d2/d75/d89/dd/d22/d26/f102 x:0 0 0 2024-03-04T16:04:45.038 INFO:tasks.workunit.client.1.smithi152.stdout:8/793: creat d2/d3/d21/d2d/d48/f103 x:0 0 0 2024-03-04T16:04:45.040 INFO:tasks.workunit.client.1.smithi152.stdout:2/664: dwrite da/df/d2e/f38 [0,4194304] 0 2024-03-04T16:04:45.049 INFO:tasks.workunit.client.1.smithi152.stdout:8/794: link d2/d3/d9e/fda d2/d3/d21/d2d/d48/db8/f104 0 2024-03-04T16:04:45.049 INFO:tasks.workunit.client.1.smithi152.stdout:8/795: stat d2/d3/d9e 0 2024-03-04T16:04:45.052 INFO:tasks.workunit.client.1.smithi152.stdout:8/796: fsync d2/d3/d21/f32 0 2024-03-04T16:04:45.056 INFO:tasks.workunit.client.1.smithi152.stdout:3/662: dwrite d6/d14/d93/f89 [0,4194304] 0 2024-03-04T16:04:45.056 INFO:tasks.workunit.client.1.smithi152.stdout:8/797: fdatasync d2/d75/d89/fd4 0 2024-03-04T16:04:45.059 INFO:tasks.workunit.client.1.smithi152.stdout:9/620: sync 2024-03-04T16:04:45.060 INFO:tasks.workunit.client.1.smithi152.stdout:5/670: write d6/d8/d1f/d3d/f78 [1335592,840] 0 2024-03-04T16:04:45.063 INFO:tasks.workunit.client.1.smithi152.stdout:9/621: dread - d2/d13/d23/fc4 zero size 2024-03-04T16:04:45.063 INFO:tasks.workunit.client.1.smithi152.stdout:8/798: read d2/d75/d89/f45 [6251705,64443] 0 2024-03-04T16:04:45.069 INFO:tasks.workunit.client.1.smithi152.stdout:9/622: read d2/d13/d23/f5b [2939795,107946] 0 2024-03-04T16:04:45.074 INFO:tasks.workunit.client.1.smithi152.stdout:8/799: rename d2/d3/d15/d1c/d84/d87/f6f to d2/d3/d21/d2d/d52/d68/d6e/d7d/f105 0 2024-03-04T16:04:45.079 INFO:tasks.workunit.client.1.smithi152.stdout:9/623: dwrite d2/d13/d23/d66/d7d/fbf [0,4194304] 0 2024-03-04T16:04:45.086 INFO:tasks.workunit.client.1.smithi152.stdout:8/800: dwrite d2/d3/d21/d2d/f79 [4194304,4194304] 0 2024-03-04T16:04:45.093 INFO:tasks.workunit.client.1.smithi152.stdout:8/801: creat d2/d3/d21/dac/f106 x:0 0 0 2024-03-04T16:04:45.095 INFO:tasks.workunit.client.1.smithi152.stdout:3/663: dread d6/d14/f15 [0,4194304] 0 2024-03-04T16:04:45.097 INFO:tasks.workunit.client.1.smithi152.stdout:9/624: mknod d2/d13/d2e/dcd/cd1 0 2024-03-04T16:04:45.098 INFO:tasks.workunit.client.1.smithi152.stdout:0/601: truncate d9/f26 410103 0 2024-03-04T16:04:45.102 INFO:tasks.workunit.client.1.smithi152.stdout:9/625: mkdir d2/d13/dad/dd2 0 2024-03-04T16:04:45.103 INFO:tasks.workunit.client.1.smithi152.stdout:3/664: mkdir d6/d60/d8e/dc0/da1/de0 0 2024-03-04T16:04:45.104 INFO:tasks.workunit.client.1.smithi152.stdout:9/626: dread - d2/d13/d23/d42/db1/fb2 zero size 2024-03-04T16:04:45.109 INFO:tasks.workunit.client.1.smithi152.stdout:0/602: unlink d9/d1b/c48 0 2024-03-04T16:04:45.110 INFO:tasks.workunit.client.1.smithi152.stdout:3/665: symlink d6/d14/d93/db7/le1 0 2024-03-04T16:04:45.115 INFO:tasks.workunit.client.1.smithi152.stdout:9/627: dwrite d2/d13/d23/d66/d7d/f9a [0,4194304] 0 2024-03-04T16:04:45.121 INFO:tasks.workunit.client.1.smithi152.stdout:2/665: dwrite da/df/d2e/f9c [4194304,4194304] 0 2024-03-04T16:04:45.129 INFO:tasks.workunit.client.1.smithi152.stdout:8/802: dread d2/d75/d89/f14 [0,4194304] 0 2024-03-04T16:04:45.129 INFO:tasks.workunit.client.1.smithi152.stdout:3/666: mkdir d6/d14/d41/d49/d84/de2 0 2024-03-04T16:04:45.130 INFO:tasks.workunit.client.1.smithi152.stdout:8/803: write d2/d75/d89/dd/d22/d26/fcb [56089,85563] 0 2024-03-04T16:04:45.140 INFO:tasks.workunit.client.1.smithi152.stdout:8/804: readlink d2/d3/d15/d1c/l97 0 2024-03-04T16:04:45.140 INFO:tasks.workunit.client.1.smithi152.stdout:9/628: creat d2/d13/d23/d71/da3/fd3 x:0 0 0 2024-03-04T16:04:45.140 INFO:tasks.workunit.client.1.smithi152.stdout:2/666: dwrite da/d2c/f96 [0,4194304] 0 2024-03-04T16:04:45.140 INFO:tasks.workunit.client.1.smithi152.stdout:9/629: dread - d2/d13/d23/d66/d7d/da7/fce zero size 2024-03-04T16:04:45.141 INFO:tasks.workunit.client.1.smithi152.stdout:8/805: fdatasync d2/d75/d89/f73 0 2024-03-04T16:04:45.146 INFO:tasks.workunit.client.1.smithi152.stdout:0/603: creat d9/d1b/d54/d80/db2/fb9 x:0 0 0 2024-03-04T16:04:45.151 INFO:tasks.workunit.client.1.smithi152.stdout:0/604: read - d9/d1b/d54/d80/f96 zero size 2024-03-04T16:04:45.152 INFO:tasks.workunit.client.1.smithi152.stdout:3/667: dread - d6/d10/f39 zero size 2024-03-04T16:04:45.154 INFO:tasks.workunit.client.1.smithi152.stdout:3/668: chown d6/d14/d41 133741117 1 2024-03-04T16:04:45.155 INFO:tasks.workunit.client.1.smithi152.stdout:0/605: write d9/d1b/d3c/fa9 [692708,49939] 0 2024-03-04T16:04:45.159 INFO:tasks.workunit.client.1.smithi152.stdout:3/669: write d6/d14/d93/f77 [4039571,29983] 0 2024-03-04T16:04:45.161 INFO:tasks.workunit.client.1.smithi152.stdout:5/671: dwrite d6/f7f [0,4194304] 0 2024-03-04T16:04:45.163 INFO:tasks.workunit.client.1.smithi152.stdout:0/606: chown d9/d1b/d3c/fa9 1550481671 1 2024-03-04T16:04:45.174 INFO:tasks.workunit.client.1.smithi152.stdout:2/667: creat da/d2c/fdc x:0 0 0 2024-03-04T16:04:45.180 INFO:tasks.workunit.client.1.smithi152.stdout:5/672: symlink d6/d8/d5a/d64/d76/d9f/dc7/dcb/dd1/ldc 0 2024-03-04T16:04:45.184 INFO:tasks.workunit.client.1.smithi152.stdout:3/670: truncate d6/d14/d58/f71 899559 0 2024-03-04T16:04:45.185 INFO:tasks.workunit.client.1.smithi152.stdout:2/668: mkdir da/d2c/ddd 0 2024-03-04T16:04:45.187 INFO:tasks.workunit.client.1.smithi152.stdout:0/607: symlink d9/lba 0 2024-03-04T16:04:45.188 INFO:tasks.workunit.client.1.smithi152.stdout:5/673: dwrite d6/d8/d1f/d3d/d6f/d70/fa2 [0,4194304] 0 2024-03-04T16:04:45.190 INFO:tasks.workunit.client.1.smithi152.stdout:5/674: fsync d6/d8/d5a/d90/dd6/fd9 0 2024-03-04T16:04:45.194 INFO:tasks.workunit.client.1.smithi152.stdout:5/675: fsync d6/d8/d1f/d3d/d6f/d70/fa8 0 2024-03-04T16:04:45.195 INFO:tasks.workunit.client.1.smithi152.stdout:2/669: symlink da/d5f/d85/dca/lde 0 2024-03-04T16:04:45.196 INFO:tasks.workunit.client.1.smithi152.stdout:5/676: truncate d6/d8/fc3 839870 0 2024-03-04T16:04:45.198 INFO:tasks.workunit.client.1.smithi152.stdout:0/608: rename d9/d1b/d3c/d84/d56/d5b/d75/lb6 to d9/d1b/d3c/lbb 0 2024-03-04T16:04:45.202 INFO:tasks.workunit.client.1.smithi152.stdout:5/677: rename d6/d8/d5a/d64/f96 to d6/d8/d1f/d3d/d6f/d70/d98/fdd 0 2024-03-04T16:04:45.205 INFO:tasks.workunit.client.1.smithi152.stdout:0/609: mknod d9/cbc 0 2024-03-04T16:04:45.208 INFO:tasks.workunit.client.1.smithi152.stdout:2/670: link da/d24/d80/d82/da8/l76 da/d5f/ldf 0 2024-03-04T16:04:45.209 INFO:tasks.workunit.client.1.smithi152.stdout:5/678: mkdir d6/d8/d1f/d3d/d6f/d70/ddb/dde 0 2024-03-04T16:04:45.209 INFO:tasks.workunit.client.1.smithi152.stdout:2/671: dread - da/df/d2e/d61/d8e/f93 zero size 2024-03-04T16:04:45.210 INFO:tasks.workunit.client.1.smithi152.stdout:5/679: chown d6/d8/dbe 18002694 1 2024-03-04T16:04:45.214 INFO:tasks.workunit.client.1.smithi152.stdout:5/680: truncate d6/d8/d5a/d64/d76/d9f/f49 2428990 0 2024-03-04T16:04:45.216 INFO:tasks.workunit.client.1.smithi152.stdout:2/672: getdents da/df/d58/d6e 0 2024-03-04T16:04:45.217 INFO:tasks.workunit.client.1.smithi152.stdout:5/681: creat d6/d8/d1f/d3d/d6f/fdf x:0 0 0 2024-03-04T16:04:45.218 INFO:tasks.workunit.client.1.smithi152.stdout:2/673: rename da/fc4 to da/d24/d47/fe0 0 2024-03-04T16:04:45.230 INFO:tasks.workunit.client.1.smithi152.stdout:8/806: truncate d2/d75/d89/dd/d22/d83/d96/da6/fbd 1605664 0 2024-03-04T16:04:45.232 INFO:tasks.workunit.client.1.smithi152.stdout:3/671: write d6/f2c [187195,102624] 0 2024-03-04T16:04:45.235 INFO:tasks.workunit.client.1.smithi152.stdout:3/672: unlink d6/d10/d2d/c76 0 2024-03-04T16:04:45.237 INFO:tasks.workunit.client.1.smithi152.stdout:3/673: dread - d6/d60/d8e/dc0/fc1 zero size 2024-03-04T16:04:45.239 INFO:tasks.workunit.client.1.smithi152.stdout:0/610: getdents d9 0 2024-03-04T16:04:45.239 INFO:tasks.workunit.client.1.smithi152.stdout:8/807: dwrite d2/d75/d89/dd/dfc/ffd [0,4194304] 0 2024-03-04T16:04:45.240 INFO:tasks.workunit.client.1.smithi152.stdout:3/674: chown d6/d10/d2d/l33 1047768 1 2024-03-04T16:04:45.242 INFO:tasks.workunit.client.1.smithi152.stdout:5/682: write d6/f68 [293207,122012] 0 2024-03-04T16:04:45.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.244+0000 7f8ce50ec700 1 -- 172.21.15.116:0/186446646 --> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ce0105290 con 0x7f8cc8038e80 2024-03-04T16:04:45.245 INFO:tasks.workunit.client.1.smithi152.stdout:3/675: read d6/d14/dd7/fdb [2661968,55510] 0 2024-03-04T16:04:45.246 INFO:tasks.workunit.client.1.smithi152.stdout:0/611: chown d9/l5a 32 1 2024-03-04T16:04:45.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.245+0000 7f8cc7fff700 1 -- 172.21.15.116:0/186446646 <== mgr.14738 v2:172.21.15.116:6800/2966163948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f8ce0105290 con 0x7f8cc8038e80 2024-03-04T16:04:45.246 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:04:45.246 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:04:45.247 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:04:45.247 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:04:45.247 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:04:45.247 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:04:45.247 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:04:45.247 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "2/23 daemons upgraded", 2024-03-04T16:04:45.247 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2024-03-04T16:04:45.247 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:04:45.247 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 -- 172.21.15.116:0/186446646 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8cc8038e80 msgr2=0x7f8cc803b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8cc8038e80 0x7f8cc803b340 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0100b10 tx=0x7f8cd000b410 comp rx=0 tx=0).stop 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 -- 172.21.15.116:0/186446646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0100cb0 msgr2=0x7f8ce019ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0100cb0 0x7f8ce019ce90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400b5c0 tx=0x7f8cd400bd70 comp rx=0 tx=0).stop 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 -- 172.21.15.116:0/186446646 shutdown_connections 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8cc8038e80 0x7f8cc803b340 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ce00ffab0 0x7f8ce019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 --2- 172.21.15.116:0/186446646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0100cb0 0x7f8ce019ce90 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 -- 172.21.15.116:0/186446646 >> 172.21.15.116:0/186446646 conn(0x7f8ce00fb050 msgr2=0x7f8ce00fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 -- 172.21.15.116:0/186446646 shutdown_connections 2024-03-04T16:04:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.248+0000 7f8ce50ec700 1 -- 172.21.15.116:0/186446646 wait complete. 2024-03-04T16:04:45.250 INFO:tasks.workunit.client.1.smithi152.stdout:2/674: dwrite da/fc [0,4194304] 0 2024-03-04T16:04:45.253 INFO:tasks.workunit.client.1.smithi152.stdout:5/683: creat d6/d8/d1f/d3d/d6f/d70/dc4/fe0 x:0 0 0 2024-03-04T16:04:45.254 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:44 smithi152 ceph-mon[36876]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:45.254 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:44 smithi152 ceph-mon[36876]: pgmap v4: 65 pgs: 65 active+clean; 2.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail 2024-03-04T16:04:45.254 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:44 smithi152 ceph-mon[36876]: mgrmap e28: smithi116.jwdhop(active, since 2s) 2024-03-04T16:04:45.254 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:44 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/869404526' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:04:45.255 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:44 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/1057118869' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:04:45.255 INFO:tasks.workunit.client.1.smithi152.stdout:8/808: dread d2/f42 [0,4194304] 0 2024-03-04T16:04:45.256 INFO:tasks.workunit.client.1.smithi152.stdout:2/675: dread da/d5f/f68 [0,4194304] 0 2024-03-04T16:04:45.257 INFO:tasks.workunit.client.1.smithi152.stdout:3/676: mkdir d6/d14/d41/d49/d84/de3 0 2024-03-04T16:04:45.258 INFO:tasks.workunit.client.1.smithi152.stdout:0/612: creat d9/d2c/d2e/d7f/d9d/fbd x:0 0 0 2024-03-04T16:04:45.259 INFO:tasks.workunit.client.1.smithi152.stdout:2/676: truncate da/d2c/d5e/f8b 349661 0 2024-03-04T16:04:45.260 INFO:tasks.workunit.client.1.smithi152.stdout:0/613: dread - d9/d1b/d3c/d84/d56/d5b/f93 zero size 2024-03-04T16:04:45.262 INFO:tasks.workunit.client.1.smithi152.stdout:5/684: unlink d6/d8/dbe/lce 0 2024-03-04T16:04:45.268 INFO:tasks.workunit.client.1.smithi152.stdout:3/677: mkdir d6/d14/d58/de4 0 2024-03-04T16:04:45.268 INFO:tasks.workunit.client.1.smithi152.stdout:5/685: write d6/d8/fa3 [291738,102940] 0 2024-03-04T16:04:45.269 INFO:tasks.workunit.client.1.smithi152.stdout:3/678: chown d6/d5c/la5 2073822 1 2024-03-04T16:04:45.269 INFO:tasks.workunit.client.1.smithi152.stdout:2/677: mkdir da/d2c/de1 0 2024-03-04T16:04:45.270 INFO:tasks.workunit.client.1.smithi152.stdout:0/614: dwrite d9/d2c/d2e/d7f/d9d/fa7 [0,4194304] 0 2024-03-04T16:04:45.272 INFO:tasks.workunit.client.1.smithi152.stdout:0/615: write d9/d2c/d2e/d7f/d9d/fb0 [705562,45821] 0 2024-03-04T16:04:45.273 INFO:tasks.workunit.client.1.smithi152.stdout:3/679: creat d6/db1/fe5 x:0 0 0 2024-03-04T16:04:45.277 INFO:tasks.workunit.client.1.smithi152.stdout:5/686: mknod d6/d8/d1f/d3d/dc0/ce1 0 2024-03-04T16:04:45.280 INFO:tasks.workunit.client.1.smithi152.stdout:3/680: chown d6/f4a 106 1 2024-03-04T16:04:45.282 INFO:tasks.workunit.client.1.smithi152.stdout:0/616: rename d9/d2c/l3b to d9/d2c/d2e/d7f/lbe 0 2024-03-04T16:04:45.283 INFO:tasks.workunit.client.1.smithi152.stdout:2/678: mknod da/d2c/d3f/ce2 0 2024-03-04T16:04:45.284 INFO:tasks.workunit.client.1.smithi152.stdout:2/679: stat da/d2c/fa3 0 2024-03-04T16:04:45.285 INFO:tasks.workunit.client.1.smithi152.stdout:8/809: dwrite d2/d3/d15/d1c/d84/d87/fd2 [0,4194304] 0 2024-03-04T16:04:45.289 INFO:tasks.workunit.client.1.smithi152.stdout:8/810: truncate d2/d75/d89/dd/d22/d26/fcb 1028866 0 2024-03-04T16:04:45.291 INFO:tasks.workunit.client.1.smithi152.stdout:8/811: chown d2/d75/d89/l12 3 1 2024-03-04T16:04:45.292 INFO:tasks.workunit.client.1.smithi152.stdout:4/564: sync 2024-03-04T16:04:45.292 INFO:tasks.workunit.client.1.smithi152.stdout:7/527: sync 2024-03-04T16:04:45.294 INFO:tasks.workunit.client.1.smithi152.stdout:8/812: truncate d2/d3/d15/ff5 1017361 0 2024-03-04T16:04:45.295 INFO:tasks.workunit.client.1.smithi152.stdout:5/687: write d6/d8/d1f/f2e [695562,4757] 0 2024-03-04T16:04:45.298 INFO:tasks.workunit.client.1.smithi152.stdout:7/528: rmdir d14 39 2024-03-04T16:04:45.299 INFO:tasks.workunit.client.1.smithi152.stdout:2/680: dwrite da/df/d58/d6e/f89 [0,4194304] 0 2024-03-04T16:04:45.302 INFO:tasks.workunit.client.1.smithi152.stdout:2/681: chown da/d24/d80/d82 1 1 2024-03-04T16:04:45.303 INFO:tasks.workunit.client.1.smithi152.stdout:0/617: dread d9/d1b/f6e [0,4194304] 0 2024-03-04T16:04:45.305 INFO:tasks.workunit.client.1.smithi152.stdout:4/565: write d8/d9/fae [4422254,6428] 0 2024-03-04T16:04:45.307 INFO:tasks.workunit.client.1.smithi152.stdout:8/813: creat d2/d75/d89/dd/d22/d6c/d86/f107 x:0 0 0 2024-03-04T16:04:45.308 INFO:tasks.workunit.client.1.smithi152.stdout:0/618: read d9/d1b/d54/d80/f9e [315142,117069] 0 2024-03-04T16:04:45.309 INFO:tasks.workunit.client.1.smithi152.stdout:3/681: dwrite d6/d10/d2d/f4c [4194304,4194304] 0 2024-03-04T16:04:45.312 INFO:tasks.workunit.client.1.smithi152.stdout:8/814: fdatasync d2/d3/d21/dac/fdb 0 2024-03-04T16:04:45.314 INFO:tasks.workunit.client.1.smithi152.stdout:2/682: rename da/d5f/fbf to da/d2c/d5e/fe3 0 2024-03-04T16:04:45.317 INFO:tasks.workunit.client.1.smithi152.stdout:4/566: write d8/d2f/d6f/d90/fa5 [404019,119181] 0 2024-03-04T16:04:45.319 INFO:tasks.workunit.client.1.smithi152.stdout:0/619: chown d9/d1a/d7c 24793453 1 2024-03-04T16:04:45.320 INFO:tasks.workunit.client.1.smithi152.stdout:8/815: readlink d2/d3/d21/d2d/d48/l5c 0 2024-03-04T16:04:45.321 INFO:tasks.workunit.client.1.smithi152.stdout:5/688: dwrite d6/d8/f2f [0,4194304] 0 2024-03-04T16:04:45.321 INFO:tasks.workunit.client.1.smithi152.stdout:3/682: mkdir d6/d60/d8e/dc0/db5/de6 0 2024-03-04T16:04:45.321 INFO:tasks.workunit.client.1.smithi152.stdout:0/620: fsync d9/d1b/f2a 0 2024-03-04T16:04:45.326 INFO:tasks.workunit.client.1.smithi152.stdout:2/683: creat da/d24/d80/d82/da6/fe4 x:0 0 0 2024-03-04T16:04:45.328 INFO:tasks.workunit.client.1.smithi152.stdout:7/529: dwrite d14/d21/d47/d36/f82 [0,4194304] 0 2024-03-04T16:04:45.328 INFO:tasks.workunit.client.1.smithi152.stdout:2/684: write da/f48 [1333394,7568] 0 2024-03-04T16:04:45.329 INFO:tasks.workunit.client.1.smithi152.stdout:3/683: dread - d6/d14/d41/fa3 zero size 2024-03-04T16:04:45.330 INFO:tasks.workunit.client.1.smithi152.stdout:8/816: creat d2/d3/d15/d1c/d84/d87/f108 x:0 0 0 2024-03-04T16:04:45.331 INFO:tasks.workunit.client.1.smithi152.stdout:8/817: readlink d2/d75/d89/l12 0 2024-03-04T16:04:45.334 INFO:tasks.workunit.client.1.smithi152.stdout:4/567: creat d8/d2f/d67/fbc x:0 0 0 2024-03-04T16:04:45.334 INFO:tasks.workunit.client.1.smithi152.stdout:0/621: mkdir d9/d1b/d54/d69/d6c/dbf 0 2024-03-04T16:04:45.338 INFO:tasks.workunit.client.1.smithi152.stdout:3/684: mkdir d6/d60/d8e/dc0/db5/de7 0 2024-03-04T16:04:45.339 INFO:tasks.workunit.client.1.smithi152.stdout:8/818: creat d2/d3/d21/d2d/d52/d68/d6e/d7d/f109 x:0 0 0 2024-03-04T16:04:45.343 INFO:tasks.workunit.client.1.smithi152.stdout:7/530: read d14/d21/d31/f33 [31129,58619] 0 2024-03-04T16:04:45.344 INFO:tasks.workunit.client.1.smithi152.stdout:4/568: rename d8/d4e/f9b to d8/d2f/d6f/d90/fbd 0 2024-03-04T16:04:45.347 INFO:tasks.workunit.client.1.smithi152.stdout:0/622: dwrite d9/d2c/d2e/f4a [0,4194304] 0 2024-03-04T16:04:45.348 INFO:tasks.workunit.client.1.smithi152.stdout:4/569: dread - d8/d2f/d7c/d87/f97 zero size 2024-03-04T16:04:45.355 INFO:tasks.workunit.client.1.smithi152.stdout:3/685: mknod d6/d60/d8a/ce8 0 2024-03-04T16:04:45.356 INFO:tasks.workunit.client.1.smithi152.stdout:3/686: dread - d6/d14/f51 zero size 2024-03-04T16:04:45.359 INFO:tasks.workunit.client.1.smithi152.stdout:5/689: dread d6/d8/d1f/f44 [0,4194304] 0 2024-03-04T16:04:45.365 INFO:tasks.workunit.client.1.smithi152.stdout:0/623: creat d9/d1b/d3c/fc0 x:0 0 0 2024-03-04T16:04:45.366 INFO:tasks.workunit.client.1.smithi152.stdout:0/624: chown d9/lba 476808010 1 2024-03-04T16:04:45.367 INFO:tasks.workunit.client.1.smithi152.stdout:7/531: dread d14/d21/d47/d36/f73 [4194304,4194304] 0 2024-03-04T16:04:45.372 INFO:tasks.workunit.client.1.smithi152.stdout:5/690: creat d6/d8/d5a/d64/d76/d9f/fe2 x:0 0 0 2024-03-04T16:04:45.373 INFO:tasks.workunit.client.1.smithi152.stdout:0/625: symlink d9/d92/lc1 0 2024-03-04T16:04:45.375 INFO:tasks.workunit.client.1.smithi152.stdout:7/532: dwrite d14/d21/f8d [0,4194304] 0 2024-03-04T16:04:45.380 INFO:tasks.workunit.client.1.smithi152.stdout:5/691: write d6/d8/d5a/d64/fd0 [974459,41956] 0 2024-03-04T16:04:45.380 INFO:tasks.workunit.client.1.smithi152.stdout:8/819: creat d2/d3/d21/f10a x:0 0 0 2024-03-04T16:04:45.384 INFO:tasks.workunit.client.1.smithi152.stdout:0/626: write d9/d1b/d3c/d84/d56/f63 [1593973,21171] 0 2024-03-04T16:04:45.384 INFO:tasks.workunit.client.1.smithi152.stdout:2/685: dwrite da/d24/f42 [0,4194304] 0 2024-03-04T16:04:45.389 INFO:tasks.workunit.client.1.smithi152.stdout:0/627: fsync d9/d1b/f4f 0 2024-03-04T16:04:45.392 INFO:tasks.workunit.client.1.smithi152.stdout:0/628: write d9/d1b/f4f [4066295,125879] 0 2024-03-04T16:04:45.394 INFO:tasks.workunit.client.1.smithi152.stdout:3/687: dwrite d6/d5c/fa0 [0,4194304] 0 2024-03-04T16:04:45.397 INFO:tasks.workunit.client.1.smithi152.stdout:2/686: creat da/d24/d80/d82/da6/fe5 x:0 0 0 2024-03-04T16:04:45.398 INFO:tasks.workunit.client.1.smithi152.stdout:7/533: creat d14/d21/d47/f91 x:0 0 0 2024-03-04T16:04:45.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.397+0000 7f809dbc7700 1 -- 172.21.15.116:0/1725080344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8098100440 msgr2=0x7f80981008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:45.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.397+0000 7f809dbc7700 1 --2- 172.21.15.116:0/1725080344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8098100440 0x7f80981008a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f808c00aa20 tx=0x7f808c00ad30 comp rx=0 tx=0).stop 2024-03-04T16:04:45.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.397+0000 7f809dbc7700 1 -- 172.21.15.116:0/1725080344 shutdown_connections 2024-03-04T16:04:45.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.397+0000 7f809dbc7700 1 --2- 172.21.15.116:0/1725080344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8098100440 0x7f80981008a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:45.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.397+0000 7f809dbc7700 1 --2- 172.21.15.116:0/1725080344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80980ff240 0x7f80980ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:45.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.397+0000 7f809dbc7700 1 -- 172.21.15.116:0/1725080344 >> 172.21.15.116:0/1725080344 conn(0x7f80980fa7c0 msgr2=0x7f80980fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:45.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.397+0000 7f809dbc7700 1 -- 172.21.15.116:0/1725080344 shutdown_connections 2024-03-04T16:04:45.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.397+0000 7f809dbc7700 1 -- 172.21.15.116:0/1725080344 wait complete. 2024-03-04T16:04:45.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.398+0000 7f809dbc7700 1 Processor -- start 2024-03-04T16:04:45.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.399+0000 7f809dbc7700 1 -- start start 2024-03-04T16:04:45.400 INFO:tasks.workunit.client.1.smithi152.stdout:2/687: chown da/df/d2e/f9c 42 1 2024-03-04T16:04:45.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.399+0000 7f809dbc7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80980ff240 0x7f8098196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:45.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.399+0000 7f809dbc7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8098100440 0x7f8098196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:45.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.399+0000 7f809dbc7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8098196f60 con 0x7f8098100440 2024-03-04T16:04:45.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.399+0000 7f809dbc7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80981970a0 con 0x7f80980ff240 2024-03-04T16:04:45.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.399+0000 7f8096ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8098100440 0x7f8098196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:45.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.399+0000 7f8096ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8098100440 0x7f8098196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46116/0 (socket says 172.21.15.116:46116) 2024-03-04T16:04:45.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.399+0000 7f8096ffd700 1 -- 172.21.15.116:0/1254958385 learned_addr learned my addr 172.21.15.116:0/1254958385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:04:45.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.399+0000 7f80977fe700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80980ff240 0x7f8098196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:45.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.400+0000 7f8096ffd700 1 -- 172.21.15.116:0/1254958385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80980ff240 msgr2=0x7f8098196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:45.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.400+0000 7f8096ffd700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80980ff240 0x7f8098196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:45.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.400+0000 7f8096ffd700 1 -- 172.21.15.116:0/1254958385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808c00a700 con 0x7f8098100440 2024-03-04T16:04:45.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.400+0000 7f8096ffd700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8098100440 0x7f8098196940 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f808c00fd90 tx=0x7f808c00fdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:45.402 INFO:tasks.workunit.client.1.smithi152.stdout:0/629: dread d9/d1b/f6e [0,4194304] 0 2024-03-04T16:04:45.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.400+0000 7f8094ff9700 1 -- 172.21.15.116:0/1254958385 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f808c00ccc0 con 0x7f8098100440 2024-03-04T16:04:45.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.400+0000 7f8094ff9700 1 -- 172.21.15.116:0/1254958385 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f808c005070 con 0x7f8098100440 2024-03-04T16:04:45.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.401+0000 7f809dbc7700 1 -- 172.21.15.116:0/1254958385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809819baf0 con 0x7f8098100440 2024-03-04T16:04:45.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.402+0000 7f8094ff9700 1 -- 172.21.15.116:0/1254958385 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f808c0167c0 con 0x7f8098100440 2024-03-04T16:04:45.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.402+0000 7f809dbc7700 1 -- 172.21.15.116:0/1254958385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809819bfe0 con 0x7f8098100440 2024-03-04T16:04:45.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.403+0000 7f8094ff9700 1 -- 172.21.15.116:0/1254958385 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 28) v1 ==== 46012+0+0 (secure 0 0 0) 0x7f808c003680 con 0x7f8098100440 2024-03-04T16:04:45.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.403+0000 7f8094ff9700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8080038c70 0x7f808003b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:04:45.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.403+0000 7f80977fe700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8080038c70 0x7f808003b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:04:45.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.404+0000 7f809dbc7700 1 -- 172.21.15.116:0/1254958385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8098107510 con 0x7f8098100440 2024-03-04T16:04:45.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.404+0000 7f8094ff9700 1 -- 172.21.15.116:0/1254958385 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f808c012070 con 0x7f8098100440 2024-03-04T16:04:45.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.404+0000 7f80977fe700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8080038c70 0x7f808003b130 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f80981002a0 tx=0x7f8088009260 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:04:45.406 INFO:tasks.workunit.client.1.smithi152.stdout:3/688: mkdir d6/d14/d41/d49/d84/de2/de9 0 2024-03-04T16:04:45.409 INFO:tasks.workunit.client.1.smithi152.stdout:2/688: dwrite da/df/d2e/f30 [4194304,4194304] 0 2024-03-04T16:04:45.409 INFO:tasks.workunit.client.1.smithi152.stdout:7/534: mknod d14/d69/d8e/c92 0 2024-03-04T16:04:45.411 INFO:tasks.workunit.client.1.smithi152.stdout:0/630: symlink d9/d1b/d54/d69/d95/lc2 0 2024-03-04T16:04:45.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.410+0000 7f8094ff9700 1 -- 172.21.15.116:0/1254958385 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f808c029460 con 0x7f8098100440 2024-03-04T16:04:45.412 INFO:tasks.workunit.client.1.smithi152.stdout:0/631: chown d9/d1a/db4 1 1 2024-03-04T16:04:45.412 INFO:tasks.workunit.client.1.smithi152.stdout:2/689: write da/d2c/d3f/f9d [623148,90154] 0 2024-03-04T16:04:45.413 INFO:tasks.workunit.client.1.smithi152.stdout:3/689: mknod d6/d60/d8a/cea 0 2024-03-04T16:04:45.418 INFO:tasks.workunit.client.1.smithi152.stdout:2/690: rename da/df/d35/f4a to da/df/d58/da2/db5/fe6 0 2024-03-04T16:04:45.419 INFO:tasks.workunit.client.1.smithi152.stdout:3/690: mkdir d6/d14/d41/d49/d84/deb 0 2024-03-04T16:04:45.423 INFO:tasks.workunit.client.1.smithi152.stdout:0/632: mkdir d9/d1b/d54/d80/db2/dc3 0 2024-03-04T16:04:45.424 INFO:tasks.workunit.client.1.smithi152.stdout:5/692: dread d6/d8/d1f/f57 [0,4194304] 0 2024-03-04T16:04:45.425 INFO:tasks.workunit.client.1.smithi152.stdout:5/693: write d6/d8/d1f/d3d/da4/fba [998055,82015] 0 2024-03-04T16:04:45.427 INFO:tasks.workunit.client.1.smithi152.stdout:2/691: symlink da/d24/d47/le7 0 2024-03-04T16:04:45.429 INFO:tasks.workunit.client.1.smithi152.stdout:3/691: creat d6/d10/d2d/fec x:0 0 0 2024-03-04T16:04:45.430 INFO:tasks.workunit.client.1.smithi152.stdout:3/692: truncate d6/d5c/f97 532697 0 2024-03-04T16:04:45.431 INFO:tasks.workunit.client.1.smithi152.stdout:0/633: mknod d9/d1b/d54/d69/d6c/cc4 0 2024-03-04T16:04:45.432 INFO:tasks.workunit.client.1.smithi152.stdout:0/634: chown d9/d1b/d3c/d84/d56/d5b/d7a/f8f 63 1 2024-03-04T16:04:45.432 INFO:tasks.workunit.client.1.smithi152.stdout:0/635: stat d9/d2c/d2e/f4a 0 2024-03-04T16:04:45.434 INFO:tasks.workunit.client.1.smithi152.stdout:5/694: rename d6/d8/d5a/d64/d8a to d6/d8/d1f/d3d/d6f/dbb/de3 0 2024-03-04T16:04:45.437 INFO:tasks.workunit.client.1.smithi152.stdout:3/693: creat d6/d14/d41/d49/d84/fed x:0 0 0 2024-03-04T16:04:45.440 INFO:tasks.workunit.client.1.smithi152.stdout:7/535: getdents d14/d21/d47 0 2024-03-04T16:04:45.445 INFO:tasks.workunit.client.1.smithi152.stdout:0/636: mknod d9/d1b/d54/d69/cc5 0 2024-03-04T16:04:45.446 INFO:tasks.workunit.client.1.smithi152.stdout:5/695: dread - d6/d8/d1f/d3d/d6f/d70/d98/fa9 zero size 2024-03-04T16:04:45.447 INFO:tasks.workunit.client.1.smithi152.stdout:5/696: truncate d6/d8/f86 4468374 0 2024-03-04T16:04:45.456 INFO:tasks.workunit.client.1.smithi152.stdout:0/637: rmdir d9/d1b/d3c/d84 39 2024-03-04T16:04:45.461 INFO:tasks.workunit.client.1.smithi152.stdout:5/697: write d6/d8/d1f/d3d/d6f/d70/d98/fdd [2369202,73637] 0 2024-03-04T16:04:45.463 INFO:tasks.workunit.client.1.smithi152.stdout:0/638: dwrite d9/d1b/d54/d80/f96 [0,4194304] 0 2024-03-04T16:04:45.465 INFO:tasks.workunit.client.1.smithi152.stdout:2/692: getdents da/df/d2e/dc6 0 2024-03-04T16:04:45.466 INFO:tasks.workunit.client.1.smithi152.stdout:2/693: readlink da/d5f/d85/dca/lde 0 2024-03-04T16:04:45.468 INFO:tasks.workunit.client.1.smithi152.stdout:3/694: link d6/d14/d41/d49/l5e d6/d14/dd7/lee 0 2024-03-04T16:04:45.469 INFO:tasks.workunit.client.1.smithi152.stdout:3/695: read - d6/d60/d8e/dc0/da1/fdc zero size 2024-03-04T16:04:45.478 INFO:tasks.workunit.client.1.smithi152.stdout:0/639: dwrite d9/d1b/d54/d80/db2/fb9 [0,4194304] 0 2024-03-04T16:04:45.478 INFO:tasks.workunit.client.1.smithi152.stdout:7/536: getdents d14/d2b 0 2024-03-04T16:04:45.479 INFO:tasks.workunit.client.1.smithi152.stdout:3/696: dwrite d6/d60/d8e/fa8 [0,4194304] 0 2024-03-04T16:04:45.490 INFO:tasks.workunit.client.1.smithi152.stdout:2/694: mknod da/d5f/dbc/dd5/ce8 0 2024-03-04T16:04:45.490 INFO:tasks.workunit.client.1.smithi152.stdout:3/697: mkdir d6/d14/d93/def 0 2024-03-04T16:04:45.490 INFO:tasks.workunit.client.1.smithi152.stdout:3/698: chown d6/d10/d2d/d4d 88883 1 2024-03-04T16:04:45.492 INFO:tasks.workunit.client.1.smithi152.stdout:7/537: rename d14/d21/d47/f51 to d14/d2b/f93 0 2024-03-04T16:04:45.493 INFO:tasks.workunit.client.1.smithi152.stdout:0/640: getdents d9/d1b/d54/d80/db2/dc3 0 2024-03-04T16:04:45.494 INFO:tasks.workunit.client.1.smithi152.stdout:5/698: truncate d6/d8/f2f 4003807 0 2024-03-04T16:04:45.496 INFO:tasks.workunit.client.1.smithi152.stdout:3/699: mkdir d6/d60/d8e/df0 0 2024-03-04T16:04:45.499 INFO:tasks.workunit.client.1.smithi152.stdout:0/641: creat d9/d1a/d7c/fc6 x:0 0 0 2024-03-04T16:04:45.500 INFO:tasks.workunit.client.1.smithi152.stdout:5/699: creat d6/d8/d5a/d64/d76/d9f/dc7/dcb/dd1/fe4 x:0 0 0 2024-03-04T16:04:45.502 INFO:tasks.workunit.client.1.smithi152.stdout:2/695: rename da/df/l17 to da/d5f/d85/le9 0 2024-03-04T16:04:45.514 INFO:tasks.workunit.client.1.smithi152.stdout:2/696: symlink da/lea 0 2024-03-04T16:04:45.515 INFO:tasks.workunit.client.1.smithi152.stdout:0/642: creat d9/d1b/d54/d69/d98/fc7 x:0 0 0 2024-03-04T16:04:45.516 INFO:tasks.workunit.client.1.smithi152.stdout:6/802: sync 2024-03-04T16:04:45.517 INFO:tasks.workunit.client.1.smithi152.stdout:3/700: rename d6/db1/fe5 to d6/ff1 0 2024-03-04T16:04:45.522 INFO:tasks.workunit.client.1.smithi152.stdout:5/700: dwrite d6/d50/f59 [0,4194304] 0 2024-03-04T16:04:45.526 INFO:tasks.workunit.client.1.smithi152.stdout:5/701: fdatasync d6/d8/d5a/d82/fcc 0 2024-03-04T16:04:45.528 INFO:tasks.workunit.client.1.smithi152.stdout:6/803: dwrite d8/d10/d53/d9a/fe8 [0,4194304] 0 2024-03-04T16:04:45.531 INFO:tasks.workunit.client.1.smithi152.stdout:0/643: write d9/f49 [3589927,31858] 0 2024-03-04T16:04:45.532 INFO:tasks.workunit.client.1.smithi152.stdout:0/644: fdatasync d9/d2c/d2e/f4a 0 2024-03-04T16:04:45.536 INFO:tasks.workunit.client.1.smithi152.stdout:2/697: creat da/feb x:0 0 0 2024-03-04T16:04:45.538 INFO:tasks.workunit.client.1.smithi152.stdout:6/804: dread - d8/d42/dde/f73 zero size 2024-03-04T16:04:45.542 INFO:tasks.workunit.client.1.smithi152.stdout:7/538: rename d14/d21/d31/f4f to d14/d21/d47/f94 0 2024-03-04T16:04:45.543 INFO:tasks.workunit.client.1.smithi152.stdout:2/698: symlink da/df/d58/d6e/lec 0 2024-03-04T16:04:45.548 INFO:tasks.workunit.client.1.smithi152.stdout:2/699: dread da/df/d2e/d61/f84 [0,4194304] 0 2024-03-04T16:04:45.548 INFO:tasks.workunit.client.1.smithi152.stdout:5/702: rename d6/d50/f7b to d6/d8/d5a/d64/d76/d9f/db0/fe5 0 2024-03-04T16:04:45.549 INFO:tasks.workunit.client.1.smithi152.stdout:3/701: truncate d6/d10/d2d/d4d/f55 402088 0 2024-03-04T16:04:45.550 INFO:tasks.workunit.client.1.smithi152.stdout:5/703: write d6/d8/d5a/d82/f92 [13709,94428] 0 2024-03-04T16:04:45.556 INFO:tasks.workunit.client.1.smithi152.stdout:3/702: dwrite d6/fd4 [0,4194304] 0 2024-03-04T16:04:45.564 INFO:tasks.workunit.client.1.smithi152.stdout:7/539: write d14/f61 [1135930,25537] 0 2024-03-04T16:04:45.571 INFO:tasks.workunit.client.1.smithi152.stdout:0/645: rename d9/d1b/f23 to d9/d2c/d2e/d94/dac/fc8 0 2024-03-04T16:04:45.580 INFO:tasks.workunit.client.1.smithi152.stdout:6/805: dwrite d8/f12 [0,4194304] 0 2024-03-04T16:04:45.587 INFO:tasks.workunit.client.1.smithi152.stdout:3/703: dread d6/d14/f37 [0,4194304] 0 2024-03-04T16:04:45.591 INFO:tasks.workunit.client.1.smithi152.stdout:6/806: dwrite d8/d23/d3a/d8d/d96/d10a/f112 [0,4194304] 0 2024-03-04T16:04:45.591 INFO:tasks.workunit.client.1.smithi152.stdout:3/704: dread d6/d60/d8e/dc0/f88 [0,4194304] 0 2024-03-04T16:04:45.592 INFO:tasks.workunit.client.1.smithi152.stdout:6/807: read - d8/d23/d3a/d8d/f107 zero size 2024-03-04T16:04:45.595 INFO:tasks.workunit.client.1.smithi152.stdout:5/704: rename d6/d8/d1f/d3d/d6f/d70/dc4/fe0 to d6/d8/d1f/d3d/d61/fe6 0 2024-03-04T16:04:45.596 INFO:tasks.workunit.client.1.smithi152.stdout:0/646: write d9/d2c/d2e/d94/dac/fc8 [2411487,58116] 0 2024-03-04T16:04:45.602 INFO:tasks.workunit.client.1.smithi152.stdout:7/540: creat d14/d21/d47/d40/f95 x:0 0 0 2024-03-04T16:04:45.603 INFO:tasks.workunit.client.1.smithi152.stdout:5/705: dwrite d6/d8/d1f/d3d/da4/fba [0,4194304] 0 2024-03-04T16:04:45.611 INFO:tasks.workunit.client.1.smithi152.stdout:2/700: link da/d5f/dbc/dd5/ce8 da/df/d58/ced 0 2024-03-04T16:04:45.613 INFO:tasks.workunit.client.1.smithi152.stdout:3/705: mknod d6/d60/d8e/df0/cf2 0 2024-03-04T16:04:45.614 INFO:tasks.workunit.client.1.smithi152.stdout:6/808: symlink d8/d23/d93/d99/l11c 0 2024-03-04T16:04:45.614 INFO:tasks.workunit.client.1.smithi152.stdout:6/809: readlink d8/d92/d75/lf3 0 2024-03-04T16:04:45.616 INFO:tasks.workunit.client.1.smithi152.stdout:6/810: write d8/d23/d3a/ff1 [843442,84688] 0 2024-03-04T16:04:45.617 INFO:tasks.workunit.client.1.smithi152.stdout:7/541: truncate d14/d21/d31/f33 904963 0 2024-03-04T16:04:45.618 INFO:tasks.workunit.client.1.smithi152.stdout:0/647: fdatasync d9/d1b/d54/d69/d95/fa5 0 2024-03-04T16:04:45.619 INFO:tasks.workunit.client.1.smithi152.stdout:3/706: mknod d6/d60/cf3 0 2024-03-04T16:04:45.620 INFO:tasks.workunit.client.1.smithi152.stdout:0/648: read d9/d1b/d54/d69/d95/fb5 [1686107,24128] 0 2024-03-04T16:04:45.621 INFO:tasks.workunit.client.1.smithi152.stdout:3/707: write d6/d14/fc4 [844639,52102] 0 2024-03-04T16:04:45.633 INFO:tasks.workunit.client.1.smithi152.stdout:6/811: rename d8/d23/d3a/d8d/d96/d10a/f82 to d8/d52/d58/dd0/f11d 0 2024-03-04T16:04:45.633 INFO:tasks.workunit.client.1.smithi152.stdout:5/706: write d6/d8/d5a/d64/d76/d9f/f49 [1296432,15095] 0 2024-03-04T16:04:45.636 INFO:tasks.workunit.client.1.smithi152.stdout:0/649: symlink d9/d92/lc9 0 2024-03-04T16:04:45.636 INFO:tasks.workunit.client.1.smithi152.stdout:2/701: truncate da/f1a 2364689 0 2024-03-04T16:04:45.639 INFO:tasks.workunit.client.1.smithi152.stdout:3/708: creat d6/d10/d2d/d4d/ff4 x:0 0 0 2024-03-04T16:04:45.640 INFO:tasks.workunit.client.1.smithi152.stdout:9/630: sync 2024-03-04T16:04:45.640 INFO:tasks.workunit.client.1.smithi152.stdout:1/686: sync 2024-03-04T16:04:45.640 INFO:tasks.workunit.client.1.smithi152.stdout:7/542: truncate d14/d2b/f93 554756 0 2024-03-04T16:04:45.641 INFO:tasks.workunit.client.1.smithi152.stdout:1/687: chown d19/d3b/d6e/d97/fad 98 1 2024-03-04T16:04:45.644 INFO:tasks.workunit.client.1.smithi152.stdout:5/707: rmdir d6/d8/d1f/d3d/d6f/d70/d98 39 2024-03-04T16:04:45.647 INFO:tasks.workunit.client.1.smithi152.stdout:7/543: dwrite d14/d21/d47/d40/f6c [4194304,4194304] 0 2024-03-04T16:04:45.650 INFO:tasks.workunit.client.1.smithi152.stdout:5/708: dread d6/d50/fc8 [0,4194304] 0 2024-03-04T16:04:45.655 INFO:tasks.workunit.client.1.smithi152.stdout:9/631: mknod d2/d13/d23/d71/d80/cd4 0 2024-03-04T16:04:45.656 INFO:tasks.workunit.client.1.smithi152.stdout:1/688: unlink d19/d20/f41 0 2024-03-04T16:04:45.669 INFO:tasks.workunit.client.1.smithi152.stdout:3/709: rename d6/d14/c17 to d6/d14/d58/cf5 0 2024-03-04T16:04:45.671 INFO:tasks.workunit.client.1.smithi152.stdout:0/650: write d9/d2c/d2e/d7f/f9f [1216392,115163] 0 2024-03-04T16:04:45.672 INFO:tasks.workunit.client.1.smithi152.stdout:3/710: chown d6/d14/d41/d49/d84/deb 698662602 1 2024-03-04T16:04:45.673 INFO:tasks.workunit.client.1.smithi152.stdout:7/544: dwrite d14/d21/f6b [0,4194304] 0 2024-03-04T16:04:45.679 INFO:tasks.workunit.client.1.smithi152.stdout:1/689: dread d19/d20/d2f/d65/fae [0,4194304] 0 2024-03-04T16:04:45.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/632: dread d2/f8d [0,4194304] 0 2024-03-04T16:04:45.683 INFO:tasks.workunit.client.1.smithi152.stdout:5/709: mkdir d6/d8/d1f/d3d/d6f/d70/d98/de7 0 2024-03-04T16:04:45.690 INFO:tasks.workunit.client.1.smithi152.stdout:0/651: creat d9/d2c/d2e/d94/fca x:0 0 0 2024-03-04T16:04:45.691 INFO:tasks.workunit.client.1.smithi152.stdout:9/633: dwrite d2/d13/d23/d71/fa1 [0,4194304] 0 2024-03-04T16:04:45.692 INFO:tasks.workunit.client.1.smithi152.stdout:4/570: sync 2024-03-04T16:04:45.692 INFO:tasks.workunit.client.1.smithi152.stdout:8/820: sync 2024-03-04T16:04:45.693 INFO:tasks.workunit.client.1.smithi152.stdout:3/711: dread d6/d14/d41/d49/f62 [4194304,4194304] 0 2024-03-04T16:04:45.693 INFO:tasks.workunit.client.1.smithi152.stdout:7/545: symlink d14/d21/d31/l96 0 2024-03-04T16:04:45.696 INFO:tasks.workunit.client.1.smithi152.stdout:2/702: dwrite da/d2c/d3f/f86 [0,4194304] 0 2024-03-04T16:04:45.698 INFO:tasks.workunit.client.1.smithi152.stdout:4/571: write d8/d9/fb2 [45557,44736] 0 2024-03-04T16:04:45.701 INFO:tasks.workunit.client.1.smithi152.stdout:6/812: link d8/d92/d75/cf5 d8/d23/d98/c11e 0 2024-03-04T16:04:45.705 INFO:tasks.workunit.client.1.smithi152.stdout:3/712: dread d6/fd4 [0,4194304] 0 2024-03-04T16:04:45.708 INFO:tasks.workunit.client.1.smithi152.stdout:9/634: rmdir d2/d63/d69 39 2024-03-04T16:04:45.713 INFO:tasks.workunit.client.1.smithi152.stdout:8/821: fsync d2/d3/d15/fc1 0 2024-03-04T16:04:45.715 INFO:tasks.workunit.client.1.smithi152.stdout:2/703: creat da/d24/d47/dcf/fee x:0 0 0 2024-03-04T16:04:45.717 INFO:tasks.workunit.client.1.smithi152.stdout:7/546: fsync d14/d21/d47/f46 0 2024-03-04T16:04:45.719 INFO:tasks.workunit.client.1.smithi152.stdout:4/572: truncate d8/f25 4247504 0 2024-03-04T16:04:45.720 INFO:tasks.workunit.client.1.smithi152.stdout:1/690: write d19/d3b/d6e/d97/dd7/f3a [3243665,39787] 0 2024-03-04T16:04:45.721 INFO:tasks.workunit.client.1.smithi152.stdout:3/713: creat d6/d10/d2d/ff6 x:0 0 0 2024-03-04T16:04:45.722 INFO:tasks.workunit.client.1.smithi152.stdout:5/710: write d6/d8/d1f/f44 [2968482,77386] 0 2024-03-04T16:04:45.730 INFO:tasks.workunit.client.1.smithi152.stdout:7/547: unlink lc 0 2024-03-04T16:04:45.737 INFO:tasks.workunit.client.1.smithi152.stdout:4/573: dread d8/d2f/f47 [0,4194304] 0 2024-03-04T16:04:45.749 INFO:tasks.workunit.client.1.smithi152.stdout:8/822: write d2/d75/d89/dd/d22/d26/f38 [584553,8225] 0 2024-03-04T16:04:45.752 INFO:tasks.workunit.client.1.smithi152.stdout:0/652: rename d9/d2c/d2e/f4a to d9/d1b/d3c/d84/fcb 0 2024-03-04T16:04:45.756 INFO:tasks.workunit.client.1.smithi152.stdout:4/574: mkdir d8/d2f/d7c/d87/dbe 0 2024-03-04T16:04:45.757 INFO:tasks.workunit.client.1.smithi152.stdout:8/823: dwrite d2/d3/d21/d2d/d52/fef [0,4194304] 0 2024-03-04T16:04:45.759 INFO:tasks.workunit.client.1.smithi152.stdout:8/824: readlink d2/d3/d15/d1c/l97 0 2024-03-04T16:04:45.760 INFO:tasks.workunit.client.1.smithi152.stdout:3/714: write d6/d60/d8a/fca [816405,9963] 0 2024-03-04T16:04:45.762 INFO:tasks.workunit.client.1.smithi152.stdout:6/813: getdents d8/d8f 0 2024-03-04T16:04:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.762+0000 7f809dbc7700 1 -- 172.21.15.116:0/1254958385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f809819c5f0 con 0x7f8098100440 2024-03-04T16:04:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.763+0000 7f8094ff9700 1 -- 172.21.15.116:0/1254958385 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f808c015cb0 con 0x7f8098100440 2024-03-04T16:04:45.764 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:04:45.765 INFO:tasks.workunit.client.1.smithi152.stdout:7/548: fsync d14/d21/d47/f34 0 2024-03-04T16:04:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 -- 172.21.15.116:0/1254958385 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8080038c70 msgr2=0x7f808003b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8080038c70 0x7f808003b130 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f80981002a0 tx=0x7f8088009260 comp rx=0 tx=0).stop 2024-03-04T16:04:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 -- 172.21.15.116:0/1254958385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8098100440 msgr2=0x7f8098196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:04:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8098100440 0x7f8098196940 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f808c00fd90 tx=0x7f808c00fdc0 comp rx=0 tx=0).stop 2024-03-04T16:04:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 -- 172.21.15.116:0/1254958385 shutdown_connections 2024-03-04T16:04:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f8080038c70 0x7f808003b130 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80980ff240 0x7f8098196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 --2- 172.21.15.116:0/1254958385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8098100440 0x7f8098196940 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:04:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 -- 172.21.15.116:0/1254958385 >> 172.21.15.116:0/1254958385 conn(0x7f80980fa7c0 msgr2=0x7f80980fc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:04:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 -- 172.21.15.116:0/1254958385 shutdown_connections 2024-03-04T16:04:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:04:45.766+0000 7f809dbc7700 1 -- 172.21.15.116:0/1254958385 wait complete. 2024-03-04T16:04:45.768 INFO:tasks.workunit.client.1.smithi152.stdout:2/704: rename da/df/d2e/d51/cc2 to da/d24/d81/cef 0 2024-03-04T16:04:45.771 INFO:tasks.workunit.client.1.smithi152.stdout:9/635: getdents d2/d13/d96 0 2024-03-04T16:04:45.778 INFO:tasks.workunit.client.1.smithi152.stdout:1/691: link d19/d75/da8/db8/fd2 d19/d3b/d6e/fdf 0 2024-03-04T16:04:45.779 INFO:tasks.workunit.client.1.smithi152.stdout:2/705: mknod da/d24/d80/d82/da8/cf0 0 2024-03-04T16:04:45.781 INFO:tasks.workunit.client.1.smithi152.stdout:0/653: rename d9/d1b/d3c/d84/l9b to d9/d1b/d54/d69/d6c/dbf/lcc 0 2024-03-04T16:04:45.782 INFO:tasks.workunit.client.1.smithi152.stdout:0/654: chown d9/d2c/d2e/l4e 4 1 2024-03-04T16:04:45.786 INFO:tasks.workunit.client.1.smithi152.stdout:5/711: link d6/c3f d6/d8/d1f/d3d/d6f/d70/ce8 0 2024-03-04T16:04:45.788 INFO:tasks.workunit.client.1.smithi152.stdout:4/575: mknod d8/d2f/d6f/d90/daf/cbf 0 2024-03-04T16:04:45.789 INFO:tasks.workunit.client.1.smithi152.stdout:3/715: write d6/f46 [1380920,64592] 0 2024-03-04T16:04:45.791 INFO:tasks.workunit.client.1.smithi152.stdout:3/716: write d6/d14/d58/f9a [27403,22399] 0 2024-03-04T16:04:45.795 INFO:tasks.workunit.client.1.smithi152.stdout:0/655: creat d9/d2c/d2e/d7f/d9d/fcd x:0 0 0 2024-03-04T16:04:45.798 INFO:tasks.workunit.client.1.smithi152.stdout:3/717: write d6/d60/d8e/dc0/db5/fcc [397037,69646] 0 2024-03-04T16:04:45.799 INFO:tasks.workunit.client.1.smithi152.stdout:8/825: link d2/d3/fcf d2/d3/d21/d2d/d52/d68/d6e/d7d/f10b 0 2024-03-04T16:04:45.800 INFO:tasks.workunit.client.1.smithi152.stdout:7/549: creat d14/d21/d47/f97 x:0 0 0 2024-03-04T16:04:45.801 INFO:tasks.workunit.client.1.smithi152.stdout:7/550: dread - d14/d69/f88 zero size 2024-03-04T16:04:45.802 INFO:tasks.workunit.client.1.smithi152.stdout:1/692: creat d19/d4b/d53/d57/dd6/fe0 x:0 0 0 2024-03-04T16:04:45.811 INFO:tasks.workunit.client.1.smithi152.stdout:1/693: creat d19/d3b/d6e/d97/dd7/da6/fe1 x:0 0 0 2024-03-04T16:04:45.812 INFO:tasks.workunit.client.1.smithi152.stdout:9/636: link d2/d13/d23/l39 d2/d13/d23/d71/da3/ld5 0 2024-03-04T16:04:45.816 INFO:tasks.workunit.client.1.smithi152.stdout:5/712: creat d6/d8/d1f/d3d/d6f/d70/fe9 x:0 0 0 2024-03-04T16:04:45.819 INFO:tasks.workunit.client.1.smithi152.stdout:3/718: creat d6/d14/d41/d49/d84/de3/ff7 x:0 0 0 2024-03-04T16:04:45.820 INFO:tasks.workunit.client.1.smithi152.stdout:9/637: dwrite d2/d13/d23/d66/d7d/da7/fb8 [0,4194304] 0 2024-03-04T16:04:45.822 INFO:tasks.workunit.client.1.smithi152.stdout:8/826: mknod d2/d3/d21/d2d/d31/dff/c10c 0 2024-03-04T16:04:45.825 INFO:tasks.workunit.client.1.smithi152.stdout:7/551: unlink d14/d21/d47/f5e 0 2024-03-04T16:04:45.829 INFO:tasks.workunit.client.1.smithi152.stdout:4/576: write d8/f25 [133558,81640] 0 2024-03-04T16:04:45.830 INFO:tasks.workunit.client.1.smithi152.stdout:4/577: truncate d8/d9/f99 172052 0 2024-03-04T16:04:45.833 INFO:tasks.workunit.client.1.smithi152.stdout:0/656: creat d9/d1b/d3c/d84/fce x:0 0 0 2024-03-04T16:04:45.834 INFO:tasks.workunit.client.1.smithi152.stdout:3/719: mknod d6/d10/d2d/d4d/cf8 0 2024-03-04T16:04:45.835 INFO:tasks.workunit.client.1.smithi152.stdout:3/720: stat d6/d14/d58/dad 0 2024-03-04T16:04:45.840 INFO:tasks.workunit.client.1.smithi152.stdout:1/694: mknod d19/d20/ce2 0 2024-03-04T16:04:45.846 INFO:tasks.workunit.client.1.smithi152.stdout:9/638: creat d2/d13/dad/dd2/fd6 x:0 0 0 2024-03-04T16:04:45.848 INFO:tasks.workunit.client.1.smithi152.stdout:7/552: creat d14/d71/f98 x:0 0 0 2024-03-04T16:04:45.848 INFO:tasks.workunit.client.1.smithi152.stdout:9/639: read d2/d13/d96/fa4 [890462,99123] 0 2024-03-04T16:04:45.849 INFO:tasks.workunit.client.1.smithi152.stdout:9/640: stat d2/d13/d35/c3d 0 2024-03-04T16:04:45.854 INFO:tasks.workunit.client.1.smithi152.stdout:0/657: link d9/d1b/d3c/d84/d56/d5b/f6b d9/d1b/d54/d69/d6c/dbf/fcf 0 2024-03-04T16:04:45.856 INFO:tasks.workunit.client.1.smithi152.stdout:4/578: write d8/d78/f82 [469166,111752] 0 2024-03-04T16:04:45.859 INFO:tasks.workunit.client.1.smithi152.stdout:1/695: unlink d19/d4b/d53/d57/f81 0 2024-03-04T16:04:45.862 INFO:tasks.workunit.client.1.smithi152.stdout:0/658: creat d9/fd0 x:0 0 0 2024-03-04T16:04:45.865 INFO:tasks.workunit.client.1.smithi152.stdout:7/553: unlink d14/d21/d47/d36/c4d 0 2024-03-04T16:04:45.870 INFO:tasks.workunit.client.1.smithi152.stdout:9/641: write d2/d13/d23/f40 [1108231,54197] 0 2024-03-04T16:04:45.872 INFO:tasks.workunit.client.1.smithi152.stdout:9/642: chown d2/d13 1 1 2024-03-04T16:04:45.874 INFO:tasks.workunit.client.1.smithi152.stdout:7/554: rename d14/d21/d47/d36/l8f to d14/d21/d47/d40/l99 0 2024-03-04T16:04:45.876 INFO:tasks.workunit.client.1.smithi152.stdout:7/555: stat d14/d2b/f56 0 2024-03-04T16:04:45.878 INFO:tasks.workunit.client.1.smithi152.stdout:4/579: dwrite d8/d78/fb7 [0,4194304] 0 2024-03-04T16:04:45.878 INFO:tasks.workunit.client.1.smithi152.stdout:9/643: read d2/f67 [111309,11846] 0 2024-03-04T16:04:45.880 INFO:tasks.workunit.client.1.smithi152.stdout:9/644: fsync d2/f2d 0 2024-03-04T16:04:45.881 INFO:tasks.workunit.client.1.smithi152.stdout:1/696: write d19/d3b/fcf [875635,100544] 0 2024-03-04T16:04:45.881 INFO:tasks.workunit.client.1.smithi152.stdout:0/659: mknod d9/d1b/d54/d69/d6c/dbf/cd1 0 2024-03-04T16:04:45.885 INFO:tasks.workunit.client.1.smithi152.stdout:7/556: rmdir d14/d21/d47/d36 39 2024-03-04T16:04:45.887 INFO:tasks.workunit.client.1.smithi152.stdout:7/557: read - d14/d71/f98 zero size 2024-03-04T16:04:45.889 INFO:tasks.workunit.client.1.smithi152.stdout:1/697: dwrite d19/d75/f91 [0,4194304] 0 2024-03-04T16:04:45.891 INFO:tasks.workunit.client.1.smithi152.stdout:1/698: dread - d19/d20/d2f/f8a zero size 2024-03-04T16:04:45.891 INFO:tasks.workunit.client.1.smithi152.stdout:9/645: fdatasync d2/d13/d23/f5f 0 2024-03-04T16:04:45.893 INFO:tasks.workunit.client.1.smithi152.stdout:1/699: stat d19/d3b/d6e/d97/dd7/f3e 0 2024-03-04T16:04:45.893 INFO:tasks.workunit.client.1.smithi152.stdout:1/700: stat d19/d4b/d53/d5c/f6d 0 2024-03-04T16:04:45.894 INFO:tasks.workunit.client.1.smithi152.stdout:1/701: write f10 [2923235,87233] 0 2024-03-04T16:04:45.901 INFO:tasks.workunit.client.1.smithi152.stdout:4/580: mknod d8/d2f/cc0 0 2024-03-04T16:04:45.908 INFO:tasks.workunit.client.1.smithi152.stdout:4/581: stat d8/d9/d7a/la8 0 2024-03-04T16:04:45.912 INFO:tasks.workunit.client.1.smithi152.stdout:4/582: unlink d8/f23 0 2024-03-04T16:04:45.915 INFO:tasks.workunit.client.1.smithi152.stdout:7/558: link c5 d14/d21/d31/c9a 0 2024-03-04T16:04:45.917 INFO:tasks.workunit.client.1.smithi152.stdout:9/646: link d2/f61 d2/d13/fd7 0 2024-03-04T16:04:45.920 INFO:tasks.workunit.client.1.smithi152.stdout:9/647: write d2/d13/d23/d42/db1/fb2 [122897,11588] 0 2024-03-04T16:04:45.920 INFO:tasks.workunit.client.1.smithi152.stdout:4/583: dwrite d8/d2f/d7c/fb1 [0,4194304] 0 2024-03-04T16:04:45.921 INFO:tasks.workunit.client.1.smithi152.stdout:4/584: dread - d8/d2f/d67/fbc zero size 2024-03-04T16:04:45.931 INFO:tasks.workunit.client.1.smithi152.stdout:1/702: link d19/d3b/d6e/d97/lbb d19/le3 0 2024-03-04T16:04:45.931 INFO:tasks.workunit.client.1.smithi152.stdout:7/559: dread - d14/d21/d47/d36/f8b zero size 2024-03-04T16:04:45.933 INFO:tasks.workunit.client.1.smithi152.stdout:4/585: dwrite d8/d78/f82 [0,4194304] 0 2024-03-04T16:04:45.934 INFO:tasks.workunit.client.1.smithi152.stdout:4/586: stat d8/d3d/f44 0 2024-03-04T16:04:45.935 INFO:tasks.workunit.client.1.smithi152.stdout:9/648: creat d2/fd8 x:0 0 0 2024-03-04T16:04:45.939 INFO:tasks.workunit.client.1.smithi152.stdout:1/703: fsync d19/d20/d2f/fcc 0 2024-03-04T16:04:45.942 INFO:tasks.workunit.client.1.smithi152.stdout:9/649: fdatasync d2/d13/d23/d42/fa2 0 2024-03-04T16:04:45.942 INFO:tasks.workunit.client.1.smithi152.stdout:0/660: dwrite d9/d1b/d54/d80/fb8 [0,4194304] 0 2024-03-04T16:04:45.951 INFO:tasks.workunit.client.1.smithi152.stdout:0/661: symlink d9/d1b/d54/d69/d6c/dbf/ld2 0 2024-03-04T16:04:45.964 INFO:tasks.workunit.client.1.smithi152.stdout:9/650: creat d2/d63/fd9 x:0 0 0 2024-03-04T16:04:45.964 INFO:tasks.workunit.client.1.smithi152.stdout:9/651: stat d2/d13/d23/f5f 0 2024-03-04T16:04:45.969 INFO:tasks.workunit.client.1.smithi152.stdout:4/587: dread d8/d2f/f9c [0,4194304] 0 2024-03-04T16:04:45.971 INFO:tasks.workunit.client.1.smithi152.stdout:7/560: link d14/d21/f49 d14/f9b 0 2024-03-04T16:04:45.975 INFO:tasks.workunit.client.1.smithi152.stdout:0/662: write d9/d2c/d2e/f6a [651322,89704] 0 2024-03-04T16:04:45.976 INFO:tasks.workunit.client.1.smithi152.stdout:4/588: dread d8/d9/d7a/f8f [0,4194304] 0 2024-03-04T16:04:45.977 INFO:tasks.workunit.client.1.smithi152.stdout:9/652: chown d2/d13/d23/c45 2355 1 2024-03-04T16:04:45.977 INFO:tasks.workunit.client.1.smithi152.stdout:1/704: link d19/d3b/l50 d19/d3b/d6e/dc6/le4 0 2024-03-04T16:04:45.981 INFO:tasks.workunit.client.1.smithi152.stdout:7/561: dwrite d14/d71/f7d [0,4194304] 0 2024-03-04T16:04:45.981 INFO:tasks.workunit.client.1.smithi152.stdout:0/663: mknod d9/d1b/d54/d69/d6c/cd3 0 2024-03-04T16:04:45.983 INFO:tasks.workunit.client.1.smithi152.stdout:0/664: chown d9/d1b/d3c/d84/d56 4337214 1 2024-03-04T16:04:45.984 INFO:tasks.workunit.client.1.smithi152.stdout:0/665: chown d9/d1b/d54/fb7 6885 1 2024-03-04T16:04:45.988 INFO:tasks.workunit.client.1.smithi152.stdout:1/705: chown d19/d75/fa7 825 1 2024-03-04T16:04:45.990 INFO:tasks.workunit.client.1.smithi152.stdout:0/666: unlink d9/d2c/d2e/f47 0 2024-03-04T16:04:45.992 INFO:tasks.workunit.client.1.smithi152.stdout:9/653: stat d2/d63/d69/l87 0 2024-03-04T16:04:45.994 INFO:tasks.workunit.client.1.smithi152.stdout:9/654: chown d2/d13/d23/d71/d80/l92 6 1 2024-03-04T16:04:45.999 INFO:tasks.workunit.client.1.smithi152.stdout:9/655: unlink d2/d13/d35/cb0 0 2024-03-04T16:04:46.000 INFO:tasks.workunit.client.1.smithi152.stdout:7/562: link d14/f16 d14/d2b/d83/f9c 0 2024-03-04T16:04:46.000 INFO:tasks.workunit.client.1.smithi152.stdout:9/656: chown d2/d13/d96/fa4 222940913 1 2024-03-04T16:04:46.005 INFO:tasks.workunit.client.1.smithi152.stdout:4/589: getdents d8/d4e 0 2024-03-04T16:04:46.006 INFO:tasks.workunit.client.1.smithi152.stdout:9/657: mknod d2/d6e/cda 0 2024-03-04T16:04:46.009 INFO:tasks.workunit.client.1.smithi152.stdout:7/563: dwrite d14/d21/d47/d36/f8b [0,4194304] 0 2024-03-04T16:04:46.024 INFO:tasks.workunit.client.1.smithi152.stdout:4/590: mkdir d8/d9/dc1 0 2024-03-04T16:04:46.025 INFO:tasks.workunit.client.1.smithi152.stdout:7/564: dwrite f7 [0,4194304] 0 2024-03-04T16:04:46.028 INFO:tasks.workunit.client.1.smithi152.stdout:9/658: creat d2/d63/d69/fdb x:0 0 0 2024-03-04T16:04:46.030 INFO:tasks.workunit.client.1.smithi152.stdout:7/565: rmdir d14/d69/d8e 39 2024-03-04T16:04:46.035 INFO:tasks.workunit.client.1.smithi152.stdout:9/659: creat d2/d13/d23/d66/d7d/da7/fdc x:0 0 0 2024-03-04T16:04:46.036 INFO:tasks.workunit.client.1.smithi152.stdout:9/660: chown d2/d13/d23/d71/d80/l92 26685777 1 2024-03-04T16:04:46.040 INFO:tasks.workunit.client.1.smithi152.stdout:9/661: symlink d2/d13/d23/d71/da3/ldd 0 2024-03-04T16:04:46.048 INFO:tasks.workunit.client.1.smithi152.stdout:9/662: dread d2/d13/d35/f8c [0,4194304] 0 2024-03-04T16:04:46.049 INFO:tasks.workunit.client.1.smithi152.stdout:1/706: truncate d19/d20/d2f/d6b/f46 1545973 0 2024-03-04T16:04:46.051 INFO:tasks.workunit.client.1.smithi152.stdout:7/566: getdents d14/d69 0 2024-03-04T16:04:46.051 INFO:tasks.workunit.client.1.smithi152.stdout:0/667: truncate d9/d1b/d3c/d84/fcb 2514374 0 2024-03-04T16:04:46.053 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:45 smithi116 ceph-mon[29120]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:46.053 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:45 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/1254958385' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:04:46.053 INFO:tasks.workunit.client.1.smithi152.stdout:7/567: dread - d14/d21/d47/f97 zero size 2024-03-04T16:04:46.054 INFO:tasks.workunit.client.1.smithi152.stdout:1/707: chown d19/d4b/f51 1193513 1 2024-03-04T16:04:46.057 INFO:tasks.workunit.client.1.smithi152.stdout:9/663: rename d2/d13/d23/d42/lb3 to d2/d13/d38/d79/lde 0 2024-03-04T16:04:46.065 INFO:tasks.workunit.client.1.smithi152.stdout:0/668: symlink d9/d1a/db4/ld4 0 2024-03-04T16:04:46.070 INFO:tasks.workunit.client.1.smithi152.stdout:0/669: dread d9/d1b/d54/d80/db2/fb9 [0,4194304] 0 2024-03-04T16:04:46.072 INFO:tasks.workunit.client.1.smithi152.stdout:9/664: getdents d2/d13/dad 0 2024-03-04T16:04:46.076 INFO:tasks.workunit.client.1.smithi152.stdout:9/665: fsync d2/f8d 0 2024-03-04T16:04:46.077 INFO:tasks.workunit.client.1.smithi152.stdout:7/568: truncate d14/d21/d48/f4b 4128560 0 2024-03-04T16:04:46.082 INFO:tasks.workunit.client.1.smithi152.stdout:0/670: dwrite d9/d1b/d54/d69/d98/fc7 [0,4194304] 0 2024-03-04T16:04:46.088 INFO:tasks.workunit.client.1.smithi152.stdout:0/671: readlink d9/l2b 0 2024-03-04T16:04:46.093 INFO:tasks.workunit.client.1.smithi152.stdout:9/666: dwrite d2/d13/d23/d66/d7d/da7/fb8 [0,4194304] 0 2024-03-04T16:04:46.093 INFO:tasks.workunit.client.1.smithi152.stdout:9/667: dread - d2/d63/fd9 zero size 2024-03-04T16:04:46.096 INFO:tasks.workunit.client.1.smithi152.stdout:9/668: readlink d2/l74 0 2024-03-04T16:04:46.099 INFO:tasks.workunit.client.1.smithi152.stdout:7/569: dwrite d14/d21/f30 [0,4194304] 0 2024-03-04T16:04:46.102 INFO:tasks.workunit.client.1.smithi152.stdout:0/672: dwrite d9/d2c/f6f [0,4194304] 0 2024-03-04T16:04:46.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:45 smithi152 ceph-mon[36876]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:04:46.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:45 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/1254958385' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:04:46.110 INFO:tasks.workunit.client.1.smithi152.stdout:7/570: mkdir d14/d9d 0 2024-03-04T16:04:46.115 INFO:tasks.workunit.client.1.smithi152.stdout:7/571: symlink d14/d21/d47/d40/d6e/l9e 0 2024-03-04T16:04:46.116 INFO:tasks.workunit.client.1.smithi152.stdout:0/673: getdents d9/d1b/d54/d69/d6c 0 2024-03-04T16:04:46.116 INFO:tasks.workunit.client.1.smithi152.stdout:7/572: rmdir d14/d2b 39 2024-03-04T16:04:46.120 INFO:tasks.workunit.client.1.smithi152.stdout:7/573: unlink d14/d21/d47/l5a 0 2024-03-04T16:04:46.120 INFO:tasks.workunit.client.1.smithi152.stdout:0/674: truncate d9/d1b/d3c/d84/d56/d5b/f6b 3463187 0 2024-03-04T16:04:46.123 INFO:tasks.workunit.client.1.smithi152.stdout:0/675: chown d9/f36 2059338837 1 2024-03-04T16:04:46.127 INFO:tasks.workunit.client.1.smithi152.stdout:0/676: symlink d9/d1a/db4/ld5 0 2024-03-04T16:04:46.128 INFO:tasks.workunit.client.1.smithi152.stdout:7/574: dwrite d14/d2b/f58 [0,4194304] 0 2024-03-04T16:04:46.132 INFO:tasks.workunit.client.1.smithi152.stdout:0/677: mknod d9/d92/cd6 0 2024-03-04T16:04:46.157 INFO:tasks.workunit.client.1.smithi152.stdout:7/575: dwrite d14/d2b/f4e [0,4194304] 0 2024-03-04T16:04:46.159 INFO:tasks.workunit.client.1.smithi152.stdout:7/576: read d14/f15 [806706,121279] 0 2024-03-04T16:04:46.159 INFO:tasks.workunit.client.1.smithi152.stdout:7/577: readlink d14/d21/d47/d40/l77 0 2024-03-04T16:04:46.174 INFO:tasks.workunit.client.1.smithi152.stdout:7/578: dread d14/f6a [0,4194304] 0 2024-03-04T16:04:46.203 INFO:tasks.workunit.client.1.smithi152.stdout:7/579: fdatasync d14/d2b/f7e 0 2024-03-04T16:04:46.205 INFO:tasks.workunit.client.1.smithi152.stdout:0/678: rename d9/d1b/d3c/lbb to d9/d1b/d3c/d84/d56/ld7 0 2024-03-04T16:04:46.207 INFO:tasks.workunit.client.1.smithi152.stdout:7/580: creat d14/d69/d8e/f9f x:0 0 0 2024-03-04T16:04:46.210 INFO:tasks.workunit.client.1.smithi152.stdout:7/581: mkdir d14/d21/d47/d36/da0 0 2024-03-04T16:04:46.212 INFO:tasks.workunit.client.1.smithi152.stdout:0/679: dread d9/d1a/d7c/f7e [0,4194304] 0 2024-03-04T16:04:46.215 INFO:tasks.workunit.client.1.smithi152.stdout:0/680: fsync d9/d1b/f71 0 2024-03-04T16:04:46.216 INFO:tasks.workunit.client.1.smithi152.stdout:7/582: fdatasync d14/d21/d47/f76 0 2024-03-04T16:04:46.217 INFO:tasks.workunit.client.1.smithi152.stdout:7/583: chown d14/d21/d47/d40/l59 136777 1 2024-03-04T16:04:46.224 INFO:tasks.workunit.client.1.smithi152.stdout:0/681: rmdir d9/d1b/d54/d80/db2/dc3 0 2024-03-04T16:04:46.226 INFO:tasks.workunit.client.1.smithi152.stdout:7/584: dwrite d14/d21/d47/f91 [0,4194304] 0 2024-03-04T16:04:46.227 INFO:tasks.workunit.client.1.smithi152.stdout:0/682: truncate d9/f36 1032113 0 2024-03-04T16:04:46.231 INFO:tasks.workunit.client.1.smithi152.stdout:7/585: symlink d14/d69/d8e/la1 0 2024-03-04T16:04:46.233 INFO:tasks.workunit.client.1.smithi152.stdout:0/683: mkdir d9/d2c/d2e/d94/dd8 0 2024-03-04T16:04:46.235 INFO:tasks.workunit.client.1.smithi152.stdout:0/684: write d9/d1b/d54/d80/fb8 [2920213,79921] 0 2024-03-04T16:04:46.248 INFO:tasks.workunit.client.1.smithi152.stdout:7/586: dread f4 [0,4194304] 0 2024-03-04T16:04:46.251 INFO:tasks.workunit.client.1.smithi152.stdout:7/587: unlink d14/d21/d47/d40/f95 0 2024-03-04T16:04:46.257 INFO:tasks.workunit.client.1.smithi152.stdout:0/685: dwrite d9/d1b/d54/d69/d6c/dbf/fcf [0,4194304] 0 2024-03-04T16:04:46.262 INFO:tasks.workunit.client.1.smithi152.stdout:7/588: write d14/d21/d31/f33 [1372018,33744] 0 2024-03-04T16:04:46.263 INFO:tasks.workunit.client.1.smithi152.stdout:0/686: creat d9/d1b/d54/d80/da8/fd9 x:0 0 0 2024-03-04T16:04:46.268 INFO:tasks.workunit.client.1.smithi152.stdout:0/687: symlink d9/d1b/d54/lda 0 2024-03-04T16:04:46.274 INFO:tasks.workunit.client.1.smithi152.stdout:7/589: write d14/d21/d47/d40/f43 [3088979,32328] 0 2024-03-04T16:04:46.278 INFO:tasks.workunit.client.1.smithi152.stdout:0/688: link d9/cf d9/d1b/cdb 0 2024-03-04T16:04:46.280 INFO:tasks.workunit.client.1.smithi152.stdout:7/590: unlink d14/d21/d47/f35 0 2024-03-04T16:04:46.281 INFO:tasks.workunit.client.1.smithi152.stdout:0/689: symlink d9/d1b/d54/d69/d98/ldc 0 2024-03-04T16:04:46.283 INFO:tasks.workunit.client.1.smithi152.stdout:7/591: symlink d14/d21/d47/d36/la2 0 2024-03-04T16:04:46.292 INFO:tasks.workunit.client.1.smithi152.stdout:7/592: dwrite d14/d2b/f60 [0,4194304] 0 2024-03-04T16:04:46.296 INFO:tasks.workunit.client.1.smithi152.stdout:0/690: dread d9/d1b/d54/d69/d95/fb5 [0,4194304] 0 2024-03-04T16:04:46.296 INFO:tasks.workunit.client.1.smithi152.stdout:7/593: creat d14/d69/fa3 x:0 0 0 2024-03-04T16:04:46.303 INFO:tasks.workunit.client.1.smithi152.stdout:7/594: link d14/f15 d14/d21/d47/d40/d6e/fa4 0 2024-03-04T16:04:46.326 INFO:tasks.workunit.client.1.smithi152.stdout:7/595: dread d14/d21/d48/f4b [0,4194304] 0 2024-03-04T16:04:46.328 INFO:tasks.workunit.client.1.smithi152.stdout:7/596: chown d14/d21/d31/c9a 7882 1 2024-03-04T16:04:46.330 INFO:tasks.workunit.client.1.smithi152.stdout:7/597: read d14/d21/d47/d40/f4c [154611,32256] 0 2024-03-04T16:04:46.332 INFO:tasks.workunit.client.1.smithi152.stdout:7/598: fsync d14/d21/d47/d40/f4c 0 2024-03-04T16:04:46.333 INFO:tasks.workunit.client.1.smithi152.stdout:7/599: chown d14/d21/d47/d36/da0 48 1 2024-03-04T16:04:46.339 INFO:tasks.workunit.client.1.smithi152.stdout:6/814: sync 2024-03-04T16:04:46.343 INFO:tasks.workunit.client.1.smithi152.stdout:6/815: dread d8/d92/d59/d61/f78 [0,4194304] 0 2024-03-04T16:04:46.344 INFO:tasks.workunit.client.1.smithi152.stdout:6/816: readlink d8/d10/le4 0 2024-03-04T16:04:46.348 INFO:tasks.workunit.client.1.smithi152.stdout:7/600: dread d14/d21/f25 [0,4194304] 0 2024-03-04T16:04:46.351 INFO:tasks.workunit.client.1.smithi152.stdout:6/817: getdents d8/da9 0 2024-03-04T16:04:46.354 INFO:tasks.workunit.client.1.smithi152.stdout:7/601: creat d14/d69/d8e/fa5 x:0 0 0 2024-03-04T16:04:46.356 INFO:tasks.workunit.client.1.smithi152.stdout:7/602: fsync f4 0 2024-03-04T16:04:46.361 INFO:tasks.workunit.client.1.smithi152.stdout:6/818: getdents d8/d92/d59 0 2024-03-04T16:04:46.361 INFO:tasks.workunit.client.1.smithi152.stdout:6/819: readlink d8/d2a/l40 0 2024-03-04T16:04:46.362 INFO:tasks.workunit.client.1.smithi152.stdout:7/603: mknod d14/ca6 0 2024-03-04T16:04:46.362 INFO:tasks.workunit.client.1.smithi152.stdout:7/604: readlink d14/d21/d47/d40/l59 0 2024-03-04T16:04:46.364 INFO:tasks.workunit.client.1.smithi152.stdout:6/820: symlink d8/da9/l11f 0 2024-03-04T16:04:46.366 INFO:tasks.workunit.client.1.smithi152.stdout:7/605: mkdir d14/d21/d47/d36/da7 0 2024-03-04T16:04:46.367 INFO:tasks.workunit.client.1.smithi152.stdout:6/821: stat d8/d23/lda 0 2024-03-04T16:04:46.370 INFO:tasks.workunit.client.1.smithi152.stdout:7/606: dread f7 [0,4194304] 0 2024-03-04T16:04:46.378 INFO:tasks.workunit.client.1.smithi152.stdout:7/607: dwrite d14/d21/d47/f8c [0,4194304] 0 2024-03-04T16:04:46.384 INFO:tasks.workunit.client.1.smithi152.stdout:7/608: creat d14/d2b/fa8 x:0 0 0 2024-03-04T16:04:46.388 INFO:tasks.workunit.client.1.smithi152.stdout:7/609: rename d14/d2b/fa8 to d14/d2b/d83/fa9 0 2024-03-04T16:04:46.396 INFO:tasks.workunit.client.1.smithi152.stdout:7/610: rmdir d14/d21/d47 39 2024-03-04T16:04:46.419 INFO:tasks.workunit.client.1.smithi152.stdout:7/611: dread d14/d21/d47/d40/f43 [0,4194304] 0 2024-03-04T16:04:46.434 INFO:tasks.workunit.client.1.smithi152.stdout:7/612: dread d14/d21/d31/f57 [0,4194304] 0 2024-03-04T16:04:46.442 INFO:tasks.workunit.client.1.smithi152.stdout:7/613: write d14/f42 [1384738,85457] 0 2024-03-04T16:04:46.443 INFO:tasks.workunit.client.1.smithi152.stdout:7/614: read d14/d2b/f7e [52568,21833] 0 2024-03-04T16:04:46.447 INFO:tasks.workunit.client.1.smithi152.stdout:7/615: mknod d14/d21/d47/d40/caa 0 2024-03-04T16:04:46.455 INFO:tasks.workunit.client.1.smithi152.stdout:7/616: write d14/d21/d47/d40/d6e/fa4 [1571069,109665] 0 2024-03-04T16:04:46.456 INFO:tasks.workunit.client.1.smithi152.stdout:7/617: chown l3 0 1 2024-03-04T16:04:46.460 INFO:tasks.workunit.client.1.smithi152.stdout:7/618: dread - d14/d69/f88 zero size 2024-03-04T16:04:46.463 INFO:tasks.workunit.client.1.smithi152.stdout:7/619: rename d14/d21/d47/d40/f6c to d14/d21/d48/fab 0 2024-03-04T16:04:46.469 INFO:tasks.workunit.client.1.smithi152.stdout:7/620: mknod d14/d71/cac 0 2024-03-04T16:04:46.472 INFO:tasks.workunit.client.1.smithi152.stdout:7/621: creat d14/d21/d47/d36/da7/fad x:0 0 0 2024-03-04T16:04:46.473 INFO:tasks.workunit.client.1.smithi152.stdout:7/622: write d14/d69/d8e/fa5 [398330,28266] 0 2024-03-04T16:04:46.474 INFO:tasks.workunit.client.1.smithi152.stdout:7/623: fsync d14/d21/d47/d36/f8b 0 2024-03-04T16:04:46.490 INFO:tasks.workunit.client.1.smithi152.stdout:7/624: dwrite d14/d21/d47/f65 [0,4194304] 0 2024-03-04T16:04:46.494 INFO:tasks.workunit.client.1.smithi152.stdout:7/625: mknod d14/d21/d47/d40/d6e/cae 0 2024-03-04T16:04:46.497 INFO:tasks.workunit.client.1.smithi152.stdout:7/626: creat d14/d71/faf x:0 0 0 2024-03-04T16:04:46.500 INFO:tasks.workunit.client.1.smithi152.stdout:7/627: creat d14/d21/d47/d32/fb0 x:0 0 0 2024-03-04T16:04:46.502 INFO:tasks.workunit.client.1.smithi152.stdout:7/628: creat d14/d21/d47/d36/da7/fb1 x:0 0 0 2024-03-04T16:04:46.508 INFO:tasks.workunit.client.1.smithi152.stdout:7/629: dwrite d14/d21/d31/f57 [0,4194304] 0 2024-03-04T16:04:46.513 INFO:tasks.workunit.client.1.smithi152.stdout:7/630: creat d14/d21/d47/fb2 x:0 0 0 2024-03-04T16:04:46.516 INFO:tasks.workunit.client.1.smithi152.stdout:7/631: mknod d14/d21/d48/cb3 0 2024-03-04T16:04:46.517 INFO:tasks.workunit.client.1.smithi152.stdout:7/632: chown d14/d21/d48/l70 1456513 1 2024-03-04T16:04:46.517 INFO:tasks.workunit.client.1.smithi152.stdout:7/633: chown d14/ca6 1068409887 1 2024-03-04T16:04:46.519 INFO:tasks.workunit.client.1.smithi152.stdout:7/634: truncate d14/d21/d47/f76 769982 0 2024-03-04T16:04:46.520 INFO:tasks.workunit.client.1.smithi152.stdout:7/635: creat d14/d71/fb4 x:0 0 0 2024-03-04T16:04:46.574 INFO:tasks.workunit.client.1.smithi152.stdout:7/636: truncate d14/f78 1641064 0 2024-03-04T16:04:46.854 INFO:tasks.workunit.client.1.smithi152.stdout:5/713: sync 2024-03-04T16:04:46.855 INFO:tasks.workunit.client.1.smithi152.stdout:2/706: sync 2024-03-04T16:04:46.859 INFO:tasks.workunit.client.1.smithi152.stdout:2/707: rename da/d24/d81/fd8 to da/d2c/ddd/ff1 0 2024-03-04T16:04:46.862 INFO:tasks.workunit.client.1.smithi152.stdout:5/714: dwrite d6/d8/d5a/d82/fcc [0,4194304] 0 2024-03-04T16:04:46.865 INFO:tasks.workunit.client.1.smithi152.stdout:2/708: unlink da/df/d58/cda 0 2024-03-04T16:04:46.869 INFO:tasks.workunit.client.1.smithi152.stdout:2/709: creat da/d24/d80/d82/ff2 x:0 0 0 2024-03-04T16:04:46.872 INFO:tasks.workunit.client.1.smithi152.stdout:5/715: unlink d6/d8/d5a/l9d 0 2024-03-04T16:04:46.879 INFO:tasks.workunit.client.1.smithi152.stdout:2/710: rename da/df/d35/d92/cb9 to da/d5f/d85/dca/cf3 0 2024-03-04T16:04:46.880 INFO:tasks.workunit.client.1.smithi152.stdout:2/711: truncate da/d24/d47/dcf/fee 740059 0 2024-03-04T16:04:46.884 INFO:tasks.workunit.client.1.smithi152.stdout:2/712: mknod da/df/d2e/d51/cf4 0 2024-03-04T16:04:46.885 INFO:tasks.workunit.client.1.smithi152.stdout:2/713: chown da/d2c/d3f/f72 10 1 2024-03-04T16:04:46.886 INFO:tasks.workunit.client.1.smithi152.stdout:2/714: write da/df/d58/d6e/f89 [2607615,87423] 0 2024-03-04T16:04:46.894 INFO:tasks.workunit.client.1.smithi152.stdout:2/715: dwrite da/d24/d80/d82/da6/fe4 [0,4194304] 0 2024-03-04T16:04:46.902 INFO:tasks.workunit.client.1.smithi152.stdout:2/716: dwrite da/d24/d80/d82/da6/fe4 [0,4194304] 0 2024-03-04T16:04:46.903 INFO:tasks.workunit.client.1.smithi152.stdout:5/716: dwrite d6/d8/d5a/d8b/fb1 [0,4194304] 0 2024-03-04T16:04:46.911 INFO:tasks.workunit.client.1.smithi152.stdout:2/717: dwrite f6 [0,4194304] 0 2024-03-04T16:04:46.911 INFO:tasks.workunit.client.1.smithi152.stdout:5/717: creat d6/d8/d5a/d90/dd6/fea x:0 0 0 2024-03-04T16:04:46.919 INFO:tasks.workunit.client.1.smithi152.stdout:8/827: sync 2024-03-04T16:04:46.919 INFO:tasks.workunit.client.1.smithi152.stdout:3/721: sync 2024-03-04T16:04:46.921 INFO:tasks.workunit.client.1.smithi152.stdout:5/718: creat d6/d8/d5a/d64/d76/d9f/dc7/dcb/dd1/feb x:0 0 0 2024-03-04T16:04:46.921 INFO:tasks.workunit.client.1.smithi152.stdout:2/718: dwrite da/df/d2e/f4e [8388608,4194304] 0 2024-03-04T16:04:46.923 INFO:tasks.workunit.client.1.smithi152.stdout:5/719: readlink d6/d8/d1f/d3d/l6b 0 2024-03-04T16:04:46.925 INFO:tasks.workunit.client.1.smithi152.stdout:5/720: chown d6/d8/d1f/f44 319109344 1 2024-03-04T16:04:46.927 INFO:tasks.workunit.client.1.smithi152.stdout:5/721: write d6/d8/d1f/d3d/d6f/d70/fe9 [417148,74459] 0 2024-03-04T16:04:46.927 INFO:tasks.workunit.client.1.smithi152.stdout:3/722: creat d6/d14/d41/d49/d84/de3/ff9 x:0 0 0 2024-03-04T16:04:46.931 INFO:tasks.workunit.client.1.smithi152.stdout:2/719: creat da/d5f/dbc/ff5 x:0 0 0 2024-03-04T16:04:46.932 INFO:tasks.workunit.client.1.smithi152.stdout:2/720: dread - da/d2c/fdc zero size 2024-03-04T16:04:46.934 INFO:tasks.workunit.client.1.smithi152.stdout:8/828: truncate d2/d3/d15/d1c/fb5 46866 0 2024-03-04T16:04:46.937 INFO:tasks.workunit.client.1.smithi152.stdout:3/723: unlink d6/d60/d8e/dc0/f72 0 2024-03-04T16:04:46.938 INFO:tasks.workunit.client.1.smithi152.stdout:5/722: rmdir d6/d8/d1f/d3d/da4/dc9 39 2024-03-04T16:04:46.939 INFO:tasks.workunit.client.1.smithi152.stdout:5/723: chown d6/d8/d1f/c62 115806 1 2024-03-04T16:04:46.941 INFO:tasks.workunit.client.1.smithi152.stdout:2/721: creat da/d24/d80/d82/da8/ff6 x:0 0 0 2024-03-04T16:04:46.943 INFO:tasks.workunit.client.1.smithi152.stdout:8/829: rename d2/c20 to d2/d3/d21/d2d/d31/c10d 0 2024-03-04T16:04:46.949 INFO:tasks.workunit.client.1.smithi152.stdout:3/724: write d6/d10/d2d/f9e [949029,24222] 0 2024-03-04T16:04:46.952 INFO:tasks.workunit.client.1.smithi152.stdout:3/725: fdatasync d6/d14/d41/d49/f62 0 2024-03-04T16:04:46.954 INFO:tasks.workunit.client.1.smithi152.stdout:2/722: mkdir da/df/d2e/d51/d8d/d9a/df7 0 2024-03-04T16:04:46.955 INFO:tasks.workunit.client.1.smithi152.stdout:8/830: creat d2/d3/d21/d2d/d52/deb/f10e x:0 0 0 2024-03-04T16:04:46.958 INFO:tasks.workunit.client.1.smithi152.stdout:5/724: getdents d6/d8/d1f 0 2024-03-04T16:04:46.960 INFO:tasks.workunit.client.1.smithi152.stdout:8/831: rmdir d2/d75/d89/dd/d22/d83 39 2024-03-04T16:04:46.962 INFO:tasks.workunit.client.1.smithi152.stdout:5/725: dread d6/d8/d5a/d82/f92 [0,4194304] 0 2024-03-04T16:04:46.964 INFO:tasks.workunit.client.1.smithi152.stdout:8/832: write d2/d3/d21/d2d/d48/f81 [1746355,2244] 0 2024-03-04T16:04:46.965 INFO:tasks.workunit.client.1.smithi152.stdout:2/723: rmdir da/d5f/dbc 39 2024-03-04T16:04:46.967 INFO:tasks.workunit.client.1.smithi152.stdout:3/726: creat d6/d14/ffa x:0 0 0 2024-03-04T16:04:46.973 INFO:tasks.workunit.client.1.smithi152.stdout:3/727: creat d6/d14/d58/dad/ffb x:0 0 0 2024-03-04T16:04:46.975 INFO:tasks.workunit.client.1.smithi152.stdout:5/726: symlink d6/d8/d1f/d3d/d6f/d70/lec 0 2024-03-04T16:04:46.976 INFO:tasks.workunit.client.1.smithi152.stdout:5/727: write d6/d8/f74 [3039364,124621] 0 2024-03-04T16:04:46.977 INFO:tasks.workunit.client.1.smithi152.stdout:8/833: unlink d2/d3/lb6 0 2024-03-04T16:04:46.977 INFO:tasks.workunit.client.1.smithi152.stdout:8/834: chown d2/d75 459 1 2024-03-04T16:04:46.978 INFO:tasks.workunit.client.1.smithi152.stdout:5/728: read d6/d8/d1f/f48 [436159,72186] 0 2024-03-04T16:04:46.983 INFO:tasks.workunit.client.1.smithi152.stdout:7/637: sync 2024-03-04T16:04:46.983 INFO:tasks.workunit.client.1.smithi152.stdout:0/691: sync 2024-03-04T16:04:46.984 INFO:tasks.workunit.client.1.smithi152.stdout:4/591: sync 2024-03-04T16:04:46.984 INFO:tasks.workunit.client.1.smithi152.stdout:9/669: sync 2024-03-04T16:04:46.984 INFO:tasks.workunit.client.1.smithi152.stdout:6/822: sync 2024-03-04T16:04:46.984 INFO:tasks.workunit.client.1.smithi152.stdout:1/708: sync 2024-03-04T16:04:46.987 INFO:tasks.workunit.client.1.smithi152.stdout:8/835: mkdir d2/d3/d21/dac/d10f 0 2024-03-04T16:04:46.988 INFO:tasks.workunit.client.1.smithi152.stdout:2/724: creat da/ff8 x:0 0 0 2024-03-04T16:04:46.988 INFO:tasks.workunit.client.1.smithi152.stdout:8/836: dread - d2/d3/d15/d1c/d84/d87/f108 zero size 2024-03-04T16:04:46.991 INFO:tasks.workunit.client.1.smithi152.stdout:7/638: write d14/d2b/f7e [847812,126446] 0 2024-03-04T16:04:46.992 INFO:tasks.workunit.client.1.smithi152.stdout:6/823: readlink d8/d23/ld8 0 2024-03-04T16:04:46.996 INFO:tasks.workunit.client.1.smithi152.stdout:9/670: mkdir d2/d13/dad/ddf 0 2024-03-04T16:04:46.996 INFO:tasks.workunit.client.1.smithi152.stdout:2/725: dwrite da/d24/d80/f9b [0,4194304] 0 2024-03-04T16:04:46.997 INFO:tasks.workunit.client.1.smithi152.stdout:4/592: fsync d8/d9/d24/d26/f32 0 2024-03-04T16:04:46.998 INFO:tasks.workunit.client.1.smithi152.stdout:4/593: dread - d8/d9/f9e zero size 2024-03-04T16:04:46.998 INFO:tasks.workunit.client.1.smithi152.stdout:9/671: stat d2/d13/d23/d66/l70 0 2024-03-04T16:04:46.999 INFO:tasks.workunit.client.1.smithi152.stdout:0/692: symlink d9/d1b/d54/d80/ldd 0 2024-03-04T16:04:47.001 INFO:tasks.workunit.client.1.smithi152.stdout:4/594: chown d8/d9/d24/d26/f2a 24957 1 2024-03-04T16:04:47.001 INFO:tasks.workunit.client.1.smithi152.stdout:1/709: creat d19/d4b/d53/d57/d85/fe5 x:0 0 0 2024-03-04T16:04:47.003 INFO:tasks.workunit.client.1.smithi152.stdout:3/728: link d6/d60/d8e/dc0/l69 d6/d60/d8e/dc0/db5/de7/lfc 0 2024-03-04T16:04:47.003 INFO:tasks.workunit.client.1.smithi152.stdout:9/672: fdatasync d2/d13/d23/d71/fbb 0 2024-03-04T16:04:47.004 INFO:tasks.workunit.client.1.smithi152.stdout:4/595: write d8/d9/f99 [1059616,18470] 0 2024-03-04T16:04:47.005 INFO:tasks.workunit.client.1.smithi152.stdout:8/837: mkdir d2/d75/d89/dd/d22/d26/d110 0 2024-03-04T16:04:47.005 INFO:tasks.workunit.client.1.smithi152.stdout:1/710: chown d19/d4b/d53/d57/d62/d82 1635 1 2024-03-04T16:04:47.006 INFO:tasks.workunit.client.1.smithi152.stdout:3/729: fsync d6/d14/fbc 0 2024-03-04T16:04:47.008 INFO:tasks.workunit.client.1.smithi152.stdout:4/596: chown d8/d9/d24/d26 6 1 2024-03-04T16:04:47.008 INFO:tasks.workunit.client.1.smithi152.stdout:5/729: dread d6/d8/d1f/d3d/d6f/d70/f71 [0,4194304] 0 2024-03-04T16:04:47.009 INFO:tasks.workunit.client.1.smithi152.stdout:7/639: symlink d14/d21/d31/lb5 0 2024-03-04T16:04:47.013 INFO:tasks.workunit.client.1.smithi152.stdout:4/597: dread d8/d2f/f47 [0,4194304] 0 2024-03-04T16:04:47.015 INFO:tasks.workunit.client.1.smithi152.stdout:2/726: creat da/d24/d80/dd0/ff9 x:0 0 0 2024-03-04T16:04:47.019 INFO:tasks.workunit.client.1.smithi152.stdout:3/730: fdatasync d6/d10/d2d/f40 0 2024-03-04T16:04:47.020 INFO:tasks.workunit.client.1.smithi152.stdout:6/824: symlink d8/d92/d59/d7d/d104/l120 0 2024-03-04T16:04:47.023 INFO:tasks.workunit.client.1.smithi152.stdout:1/711: rename d19/d20/d2f/d65/fae to d19/d4b/d53/d57/d62/dd8/fe6 0 2024-03-04T16:04:47.025 INFO:tasks.workunit.client.1.smithi152.stdout:6/825: dread d8/d92/d59/d7d/d91/fcd [0,4194304] 0 2024-03-04T16:04:47.028 INFO:tasks.workunit.client.1.smithi152.stdout:4/598: creat d8/d4e/d8b/fc2 x:0 0 0 2024-03-04T16:04:47.029 INFO:tasks.workunit.client.1.smithi152.stdout:2/727: symlink da/d5f/d85/d8c/lfa 0 2024-03-04T16:04:47.030 INFO:tasks.workunit.client.1.smithi152.stdout:9/673: dread d2/f61 [0,4194304] 0 2024-03-04T16:04:47.032 INFO:tasks.workunit.client.1.smithi152.stdout:6/826: dwrite d8/d23/d3a/d8d/d96/d10a/f7a [0,4194304] 0 2024-03-04T16:04:47.034 INFO:tasks.workunit.client.1.smithi152.stdout:3/731: unlink d6/d14/d41/d49/f83 0 2024-03-04T16:04:47.039 INFO:tasks.workunit.client.1.smithi152.stdout:3/732: dread d6/d14/d41/d49/d84/fba [0,4194304] 0 2024-03-04T16:04:47.040 INFO:tasks.workunit.client.1.smithi152.stdout:8/838: rename d2/d75/d89/f28 to d2/d75/d89/dd/d22/d26/f111 0 2024-03-04T16:04:47.044 INFO:tasks.workunit.client.1.smithi152.stdout:0/693: rmdir d9/d2c/d2e/d94/dd8 0 2024-03-04T16:04:47.046 INFO:tasks.workunit.client.1.smithi152.stdout:3/733: dwrite d6/f2c [0,4194304] 0 2024-03-04T16:04:47.048 INFO:tasks.workunit.client.1.smithi152.stdout:3/734: readlink d6/d10/d2d/d4d/la9 0 2024-03-04T16:04:47.056 INFO:tasks.workunit.client.1.smithi152.stdout:2/728: read da/f1a [1110282,74787] 0 2024-03-04T16:04:47.060 INFO:tasks.workunit.client.1.smithi152.stdout:6/827: creat d8/da9/f121 x:0 0 0 2024-03-04T16:04:47.072 INFO:tasks.workunit.client.1.smithi152.stdout:7/640: rename d14/d21/d47/fb2 to d14/d71/fb6 0 2024-03-04T16:04:47.076 INFO:tasks.workunit.client.1.smithi152.stdout:1/712: truncate d19/d20/d2f/d6b/f46 2253638 0 2024-03-04T16:04:47.080 INFO:tasks.workunit.client.1.smithi152.stdout:7/641: dwrite d14/d2b/f63 [0,4194304] 0 2024-03-04T16:04:47.095 INFO:tasks.workunit.client.1.smithi152.stdout:8/839: mkdir d2/d3/d21/d2d/d112 0 2024-03-04T16:04:47.096 INFO:tasks.workunit.client.1.smithi152.stdout:8/840: readlink d2/d75/d89/dd/l88 0 2024-03-04T16:04:47.101 INFO:tasks.workunit.client.1.smithi152.stdout:0/694: mkdir d9/d1b/d3c/d84/dde 0 2024-03-04T16:04:47.104 INFO:tasks.workunit.client.1.smithi152.stdout:1/713: creat d19/d4b/d53/d57/d62/fe7 x:0 0 0 2024-03-04T16:04:47.105 INFO:tasks.workunit.client.1.smithi152.stdout:1/714: dread - d19/d4b/d53/d57/d85/fe5 zero size 2024-03-04T16:04:47.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:46 smithi152 ceph-mon[36876]: pgmap v5: 65 pgs: 65 active+clean; 2.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail 2024-03-04T16:04:47.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:46 smithi152 ceph-mon[36876]: mgrmap e29: smithi116.jwdhop(active, since 4s) 2024-03-04T16:04:47.107 INFO:tasks.workunit.client.1.smithi152.stdout:3/735: fsync d6/d10/d2d/d4d/f55 0 2024-03-04T16:04:47.110 INFO:tasks.workunit.client.1.smithi152.stdout:6/828: creat d8/d92/de0/f122 x:0 0 0 2024-03-04T16:04:47.112 INFO:tasks.workunit.client.1.smithi152.stdout:8/841: mknod d2/d3/d21/d2d/d31/c113 0 2024-03-04T16:04:47.113 INFO:tasks.workunit.client.1.smithi152.stdout:7/642: dread d14/d21/d47/f34 [0,4194304] 0 2024-03-04T16:04:47.114 INFO:tasks.workunit.client.1.smithi152.stdout:7/643: truncate d14/d69/d8e/f9f 663733 0 2024-03-04T16:04:47.115 INFO:tasks.workunit.client.1.smithi152.stdout:0/695: creat d9/d1b/d54/d80/fdf x:0 0 0 2024-03-04T16:04:47.115 INFO:tasks.workunit.client.1.smithi152.stdout:1/715: chown d19/d75/da8/caf 86930 1 2024-03-04T16:04:47.118 INFO:tasks.workunit.client.1.smithi152.stdout:3/736: unlink d6/f9c 0 2024-03-04T16:04:47.119 INFO:tasks.workunit.client.1.smithi152.stdout:8/842: mkdir d2/d7f/d114 0 2024-03-04T16:04:47.120 INFO:tasks.workunit.client.1.smithi152.stdout:7/644: mkdir d14/d2b/d83/db7 0 2024-03-04T16:04:47.121 INFO:tasks.workunit.client.1.smithi152.stdout:7/645: write d14/d2b/f7e [2504093,33161] 0 2024-03-04T16:04:47.122 INFO:tasks.workunit.client.1.smithi152.stdout:2/729: getdents da/d24/d47/d73 0 2024-03-04T16:04:47.125 INFO:tasks.workunit.client.1.smithi152.stdout:6/829: creat d8/d10/d53/d9a/d11a/f123 x:0 0 0 2024-03-04T16:04:47.127 INFO:tasks.workunit.client.1.smithi152.stdout:3/737: creat d6/d60/d8a/ffd x:0 0 0 2024-03-04T16:04:47.129 INFO:tasks.workunit.client.1.smithi152.stdout:7/646: symlink d14/d21/d31/lb8 0 2024-03-04T16:04:47.129 INFO:tasks.workunit.client.1.smithi152.stdout:9/674: write d2/d13/f93 [986538,113644] 0 2024-03-04T16:04:47.130 INFO:tasks.workunit.client.1.smithi152.stdout:2/730: fdatasync da/df/d2e/f8f 0 2024-03-04T16:04:47.131 INFO:tasks.workunit.client.1.smithi152.stdout:7/647: write d14/d21/d47/f65 [3552965,77222] 0 2024-03-04T16:04:47.133 INFO:tasks.workunit.client.1.smithi152.stdout:7/648: write d14/d69/fa3 [935439,46111] 0 2024-03-04T16:04:47.133 INFO:tasks.workunit.client.1.smithi152.stdout:0/696: unlink d9/d1b/d3c/d84/d56/c65 0 2024-03-04T16:04:47.134 INFO:tasks.workunit.client.1.smithi152.stdout:0/697: read - d9/d1b/d54/fb7 zero size 2024-03-04T16:04:47.134 INFO:tasks.workunit.client.1.smithi152.stdout:1/716: symlink d19/le8 0 2024-03-04T16:04:47.136 INFO:tasks.workunit.client.1.smithi152.stdout:1/717: readlink d19/d20/d2f/d65/la5 0 2024-03-04T16:04:47.137 INFO:tasks.workunit.client.1.smithi152.stdout:6/830: rename d8/d92/d59/d61/d57/d110 to d8/d92/db1/d124 0 2024-03-04T16:04:47.140 INFO:tasks.workunit.client.1.smithi152.stdout:1/718: dread d19/d3b/d6e/d97/fad [0,4194304] 0 2024-03-04T16:04:47.140 INFO:tasks.workunit.client.1.smithi152.stdout:3/738: creat d6/d60/d8e/ffe x:0 0 0 2024-03-04T16:04:47.142 INFO:tasks.workunit.client.1.smithi152.stdout:3/739: readlink d6/d60/d8e/dc0/da1/lb4 0 2024-03-04T16:04:47.142 INFO:tasks.workunit.client.1.smithi152.stdout:9/675: chown d2/c1b 2107014 1 2024-03-04T16:04:47.149 INFO:tasks.workunit.client.1.smithi152.stdout:2/731: fdatasync da/df/d2e/f90 0 2024-03-04T16:04:47.149 INFO:tasks.workunit.client.1.smithi152.stdout:9/676: chown d2/d13/d23/c72 271643881 1 2024-03-04T16:04:47.150 INFO:tasks.workunit.client.1.smithi152.stdout:7/649: fsync d14/d21/f25 0 2024-03-04T16:04:47.151 INFO:tasks.workunit.client.1.smithi152.stdout:2/732: readlink da/d2c/lb8 0 2024-03-04T16:04:47.154 INFO:tasks.workunit.client.1.smithi152.stdout:0/698: creat d9/d1a/db4/fe0 x:0 0 0 2024-03-04T16:04:47.154 INFO:tasks.workunit.client.1.smithi152.stdout:4/599: dwrite d8/d9/d24/f40 [0,4194304] 0 2024-03-04T16:04:47.156 INFO:tasks.workunit.client.1.smithi152.stdout:6/831: fdatasync d8/d92/d59/f7c 0 2024-03-04T16:04:47.160 INFO:tasks.workunit.client.1.smithi152.stdout:8/843: link d2/d3/d21/d2d/f79 d2/d3/d21/f115 0 2024-03-04T16:04:47.162 INFO:tasks.workunit.client.1.smithi152.stdout:1/719: rename d19/d75/fa0 to d19/d4b/fe9 0 2024-03-04T16:04:47.163 INFO:tasks.workunit.client.1.smithi152.stdout:1/720: dread - d19/d3b/d6e/fdf zero size 2024-03-04T16:04:47.165 INFO:tasks.workunit.client.1.smithi152.stdout:3/740: unlink d6/d60/d8a/fc8 0 2024-03-04T16:04:47.213 INFO:tasks.workunit.client.1.smithi152.stdout:9/677: creat d2/d13/d23/d66/d7d/fe0 x:0 0 0 2024-03-04T16:04:47.217 INFO:tasks.workunit.client.1.smithi152.stdout:9/678: dread d2/d13/f93 [0,4194304] 0 2024-03-04T16:04:47.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:46 smithi116 ceph-mon[29120]: pgmap v5: 65 pgs: 65 active+clean; 2.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail 2024-03-04T16:04:47.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:46 smithi116 ceph-mon[29120]: mgrmap e29: smithi116.jwdhop(active, since 4s) 2024-03-04T16:04:47.226 INFO:tasks.workunit.client.1.smithi152.stdout:7/650: mkdir d14/db9 0 2024-03-04T16:04:47.229 INFO:tasks.workunit.client.1.smithi152.stdout:2/733: creat da/d5f/dbc/ffb x:0 0 0 2024-03-04T16:04:47.231 INFO:tasks.workunit.client.1.smithi152.stdout:0/699: mknod d9/d1b/d54/d69/ce1 0 2024-03-04T16:04:47.236 INFO:tasks.workunit.client.1.smithi152.stdout:3/741: dread d6/f8 [0,4194304] 0 2024-03-04T16:04:47.240 INFO:tasks.workunit.client.1.smithi152.stdout:2/734: fsync da/df/d2e/f30 0 2024-03-04T16:04:47.242 INFO:tasks.workunit.client.1.smithi152.stdout:0/700: symlink d9/d1b/d54/d80/db2/le2 0 2024-03-04T16:04:47.243 INFO:tasks.workunit.client.1.smithi152.stdout:6/832: write d8/d92/d59/d7d/d91/fcd [1652926,77300] 0 2024-03-04T16:04:47.244 INFO:tasks.workunit.client.1.smithi152.stdout:0/701: write d9/d2c/d2e/f6a [918311,47772] 0 2024-03-04T16:04:47.246 INFO:tasks.workunit.client.1.smithi152.stdout:8/844: creat d2/d3/d21/f116 x:0 0 0 2024-03-04T16:04:47.248 INFO:tasks.workunit.client.1.smithi152.stdout:5/730: sync 2024-03-04T16:04:47.250 INFO:tasks.workunit.client.1.smithi152.stdout:9/679: link d2/d13/d2e/c9f d2/d13/d23/ce1 0 2024-03-04T16:04:47.251 INFO:tasks.workunit.client.1.smithi152.stdout:9/680: dread - d2/d13/d23/d66/d7d/da7/fce zero size 2024-03-04T16:04:47.253 INFO:tasks.workunit.client.1.smithi152.stdout:9/681: read - d2/d13/d23/d71/da3/fbc zero size 2024-03-04T16:04:47.256 INFO:tasks.workunit.client.1.smithi152.stdout:1/721: creat d19/d4b/fea x:0 0 0 2024-03-04T16:04:47.256 INFO:tasks.workunit.client.1.smithi152.stdout:3/742: dread d6/d60/f81 [0,4194304] 0 2024-03-04T16:04:47.257 INFO:tasks.workunit.client.1.smithi152.stdout:4/600: dwrite d8/d9/d79/f85 [0,4194304] 0 2024-03-04T16:04:47.258 INFO:tasks.workunit.client.1.smithi152.stdout:4/601: chown d8/d3d 35 1 2024-03-04T16:04:47.259 INFO:tasks.workunit.client.1.smithi152.stdout:4/602: stat d8/d3d 0 2024-03-04T16:04:47.262 INFO:tasks.workunit.client.1.smithi152.stdout:7/651: write d14/d21/d47/d36/f68 [1520287,9655] 0 2024-03-04T16:04:47.262 INFO:tasks.workunit.client.1.smithi152.stdout:7/652: dread - d14/d71/f98 zero size 2024-03-04T16:04:47.331 INFO:tasks.workunit.client.1.smithi152.stdout:1/722: dwrite d19/f1d [4194304,4194304] 0 2024-03-04T16:04:47.332 INFO:tasks.workunit.client.1.smithi152.stdout:4/603: truncate d8/d9/d24/d26/d42/f4b 1257483 0 2024-03-04T16:04:47.334 INFO:tasks.workunit.client.1.smithi152.stdout:2/735: link da/df/f97 da/d24/d81/ffc 0 2024-03-04T16:04:47.337 INFO:tasks.workunit.client.1.smithi152.stdout:7/653: dread - d14/d21/d31/f7c zero size 2024-03-04T16:04:47.341 INFO:tasks.workunit.client.1.smithi152.stdout:7/654: dread d14/d21/d47/d36/f73 [0,4194304] 0 2024-03-04T16:04:47.343 INFO:tasks.workunit.client.1.smithi152.stdout:7/655: write d14/d21/d47/d40/d6e/fa4 [2595285,48209] 0 2024-03-04T16:04:47.356 INFO:tasks.workunit.client.1.smithi152.stdout:3/743: mknod d6/d60/d8e/dc0/da1/de0/cff 0 2024-03-04T16:04:47.363 INFO:tasks.workunit.client.1.smithi152.stdout:5/731: creat d6/d8/d5a/fed x:0 0 0 2024-03-04T16:04:47.364 INFO:tasks.workunit.client.1.smithi152.stdout:6/833: link d8/c11 d8/d23/d3a/c125 0 2024-03-04T16:04:47.367 INFO:tasks.workunit.client.1.smithi152.stdout:7/656: write d14/f6a [945434,101916] 0 2024-03-04T16:04:47.368 INFO:tasks.workunit.client.1.smithi152.stdout:7/657: read d14/d2b/f7e [1861572,95929] 0 2024-03-04T16:04:47.369 INFO:tasks.workunit.client.1.smithi152.stdout:7/658: stat d14/d21/d31/l96 0 2024-03-04T16:04:47.369 INFO:tasks.workunit.client.1.smithi152.stdout:9/682: rmdir d2/d13/d23/d71/da3/dc6 0 2024-03-04T16:04:47.371 INFO:tasks.workunit.client.1.smithi152.stdout:7/659: truncate d14/d2b/d83/fa9 165242 0 2024-03-04T16:04:47.372 INFO:tasks.workunit.client.1.smithi152.stdout:8/845: getdents d2/d3/d21 0 2024-03-04T16:04:47.372 INFO:tasks.workunit.client.1.smithi152.stdout:7/660: chown d14/d21/d47/d36/l66 729573 1 2024-03-04T16:04:47.375 INFO:tasks.workunit.client.1.smithi152.stdout:2/736: dread da/df/d2e/f90 [0,4194304] 0 2024-03-04T16:04:47.376 INFO:tasks.workunit.client.1.smithi152.stdout:2/737: readlink da/d2c/lb8 0 2024-03-04T16:04:47.377 INFO:tasks.workunit.client.1.smithi152.stdout:6/834: mkdir d8/d52/d126 0 2024-03-04T16:04:47.378 INFO:tasks.workunit.client.1.smithi152.stdout:6/835: dread - d8/d8f/db2/f106 zero size 2024-03-04T16:04:47.380 INFO:tasks.workunit.client.1.smithi152.stdout:9/683: mknod d2/d13/d23/d66/d7d/da7/ce2 0 2024-03-04T16:04:47.385 INFO:tasks.workunit.client.1.smithi152.stdout:9/684: dwrite d2/d13/d23/d71/fa1 [0,4194304] 0 2024-03-04T16:04:47.391 INFO:tasks.workunit.client.1.smithi152.stdout:5/732: unlink d6/d8/d5a/d90/c9c 0 2024-03-04T16:04:47.392 INFO:tasks.workunit.client.1.smithi152.stdout:5/733: stat d6/d8/d1f/d3d/d6f/d70/ddb/dde 0 2024-03-04T16:04:47.393 INFO:tasks.workunit.client.1.smithi152.stdout:9/685: dread d2/d13/d23/f40 [0,4194304] 0 2024-03-04T16:04:47.395 INFO:tasks.workunit.client.1.smithi152.stdout:2/738: unlink da/df/d2e/f3a 0 2024-03-04T16:04:47.405 INFO:tasks.workunit.client.1.smithi152.stdout:7/661: getdents d14/db9 0 2024-03-04T16:04:47.412 INFO:tasks.workunit.client.1.smithi152.stdout:2/739: symlink da/d5f/d85/lfd 0 2024-03-04T16:04:47.412 INFO:tasks.workunit.client.1.smithi152.stdout:2/740: dread - da/d24/d81/ffc zero size 2024-03-04T16:04:47.414 INFO:tasks.workunit.client.1.smithi152.stdout:7/662: mknod d14/d71/cba 0 2024-03-04T16:04:47.419 INFO:tasks.workunit.client.1.smithi152.stdout:2/741: dwrite da/d24/d80/f9b [0,4194304] 0 2024-03-04T16:04:47.420 INFO:tasks.workunit.client.1.smithi152.stdout:2/742: write f6 [685053,35677] 0 2024-03-04T16:04:47.421 INFO:tasks.workunit.client.1.smithi152.stdout:2/743: read - da/feb zero size 2024-03-04T16:04:47.424 INFO:tasks.workunit.client.1.smithi152.stdout:2/744: rmdir da/d24/d47/d4f 39 2024-03-04T16:04:47.428 INFO:tasks.workunit.client.1.smithi152.stdout:2/745: mknod da/df/d58/cfe 0 2024-03-04T16:04:47.436 INFO:tasks.workunit.client.1.smithi152.stdout:6/836: creat d8/d92/d59/d61/f127 x:0 0 0 2024-03-04T16:04:47.437 INFO:tasks.workunit.client.1.smithi152.stdout:0/702: write d9/d1b/d3c/d84/fcb [2783568,27089] 0 2024-03-04T16:04:47.439 INFO:tasks.workunit.client.1.smithi152.stdout:0/703: truncate d9/d1b/d54/d80/da8/fd9 462211 0 2024-03-04T16:04:47.439 INFO:tasks.workunit.client.1.smithi152.stdout:6/837: creat d8/d42/dde/dee/f128 x:0 0 0 2024-03-04T16:04:47.443 INFO:tasks.workunit.client.1.smithi152.stdout:6/838: fsync d8/f1f 0 2024-03-04T16:04:47.447 INFO:tasks.workunit.client.1.smithi152.stdout:0/704: getdents d9/d1b/d54/d69 0 2024-03-04T16:04:47.451 INFO:tasks.workunit.client.1.smithi152.stdout:6/839: dwrite d8/d23/d3a/d8d/f107 [0,4194304] 0 2024-03-04T16:04:47.457 INFO:tasks.workunit.client.1.smithi152.stdout:0/705: dwrite d9/d2c/d2e/d7f/d9d/fcd [0,4194304] 0 2024-03-04T16:04:47.461 INFO:tasks.workunit.client.1.smithi152.stdout:6/840: dwrite d8/d42/f4a [4194304,4194304] 0 2024-03-04T16:04:47.467 INFO:tasks.workunit.client.1.smithi152.stdout:6/841: unlink d8/d23/d3a/ff1 0 2024-03-04T16:04:47.471 INFO:tasks.workunit.client.1.smithi152.stdout:4/604: write d8/d2f/f33 [540642,30261] 0 2024-03-04T16:04:47.478 INFO:tasks.workunit.client.1.smithi152.stdout:4/605: rmdir d8/d2f/d6f 39 2024-03-04T16:04:47.484 INFO:tasks.workunit.client.1.smithi152.stdout:4/606: link d8/d2f/d6f/d90/fbd d8/d4e/d86/fc3 0 2024-03-04T16:04:47.489 INFO:tasks.workunit.client.1.smithi152.stdout:1/723: rename d19/d20/c34 to d19/d3b/ceb 0 2024-03-04T16:04:47.490 INFO:tasks.workunit.client.1.smithi152.stdout:8/846: creat d2/d3/d21/f117 x:0 0 0 2024-03-04T16:04:47.492 INFO:tasks.workunit.client.1.smithi152.stdout:1/724: read d19/d3b/d6e/d97/dd7/f3e [3110044,108948] 0 2024-03-04T16:04:47.494 INFO:tasks.workunit.client.1.smithi152.stdout:3/744: rename d6/d14/d93/f89 to d6/d60/d8e/dc0/da1/f100 0 2024-03-04T16:04:47.495 INFO:tasks.workunit.client.1.smithi152.stdout:3/745: dread - d6/d60/d8e/dc0/da1/fdc zero size 2024-03-04T16:04:47.496 INFO:tasks.workunit.client.1.smithi152.stdout:6/842: dread d8/d92/db1/d87/fc2 [0,4194304] 0 2024-03-04T16:04:47.497 INFO:tasks.workunit.client.1.smithi152.stdout:3/746: chown d6/d10/d2d/d4d/f55 0 1 2024-03-04T16:04:47.500 INFO:tasks.workunit.client.1.smithi152.stdout:1/725: symlink d19/d20/d2f/d5d/lec 0 2024-03-04T16:04:47.502 INFO:tasks.workunit.client.1.smithi152.stdout:9/686: write d2/d63/f94 [3113663,85993] 0 2024-03-04T16:04:47.503 INFO:tasks.workunit.client.1.smithi152.stdout:8/847: creat d2/d7f/d114/f118 x:0 0 0 2024-03-04T16:04:47.503 INFO:tasks.workunit.client.1.smithi152.stdout:4/607: dread d8/d2f/f33 [0,4194304] 0 2024-03-04T16:04:47.508 INFO:tasks.workunit.client.1.smithi152.stdout:5/734: dwrite d6/d8/f41 [0,4194304] 0 2024-03-04T16:04:47.517 INFO:tasks.workunit.client.1.smithi152.stdout:7/663: rename d14/d21/f6b to d14/d21/d47/d32/fbb 0 2024-03-04T16:04:47.518 INFO:tasks.workunit.client.1.smithi152.stdout:7/664: chown d14/d21/d47/d36/da7/fb1 7159275 1 2024-03-04T16:04:47.520 INFO:tasks.workunit.client.1.smithi152.stdout:7/665: fsync d14/f6a 0 2024-03-04T16:04:47.521 INFO:tasks.workunit.client.1.smithi152.stdout:4/608: dwrite f7 [0,4194304] 0 2024-03-04T16:04:47.523 INFO:tasks.workunit.client.1.smithi152.stdout:6/843: dread d8/d23/d3a/d8d/d96/fb9 [0,4194304] 0 2024-03-04T16:04:47.523 INFO:tasks.workunit.client.1.smithi152.stdout:5/735: chown d6/d8/f2f 97 1 2024-03-04T16:04:47.526 INFO:tasks.workunit.client.1.smithi152.stdout:6/844: write d8/d92/d75/f105 [814592,118232] 0 2024-03-04T16:04:47.532 INFO:tasks.workunit.client.1.smithi152.stdout:7/666: rmdir d14/d21/d47/d36 39 2024-03-04T16:04:47.536 INFO:tasks.workunit.client.1.smithi152.stdout:4/609: creat d8/d3d/fc4 x:0 0 0 2024-03-04T16:04:47.537 INFO:tasks.workunit.client.1.smithi152.stdout:5/736: unlink d6/d8/d1f/d3d/c89 0 2024-03-04T16:04:47.547 INFO:tasks.workunit.client.1.smithi152.stdout:4/610: creat d8/d89/fc5 x:0 0 0 2024-03-04T16:04:47.548 INFO:tasks.workunit.client.1.smithi152.stdout:4/611: chown d8/d3d/fa1 127748327 1 2024-03-04T16:04:47.552 INFO:tasks.workunit.client.1.smithi152.stdout:2/746: rename da/ff8 to da/d5f/fff 0 2024-03-04T16:04:47.554 INFO:tasks.workunit.client.1.smithi152.stdout:7/667: mkdir d14/d21/d47/d36/da7/dbc 0 2024-03-04T16:04:47.555 INFO:tasks.workunit.client.1.smithi152.stdout:7/668: dread - d14/d21/f90 zero size 2024-03-04T16:04:47.559 INFO:tasks.workunit.client.1.smithi152.stdout:0/706: rename d9/d2c/d2e/l43 to d9/d92/le3 0 2024-03-04T16:04:47.562 INFO:tasks.workunit.client.1.smithi152.stdout:1/726: link d19/d4b/d53/d57/f60 d19/d20/d2f/d6b/fed 0 2024-03-04T16:04:47.563 INFO:tasks.workunit.client.1.smithi152.stdout:4/612: dread d8/d9/d7a/fa6 [0,4194304] 0 2024-03-04T16:04:47.564 INFO:tasks.workunit.client.1.smithi152.stdout:3/747: getdents d6/d60/d8e/dc0/da1 0 2024-03-04T16:04:47.566 INFO:tasks.workunit.client.1.smithi152.stdout:7/669: symlink d14/d21/d47/d36/lbd 0 2024-03-04T16:04:47.567 INFO:tasks.workunit.client.1.smithi152.stdout:2/747: mknod da/d2c/d3f/c100 0 2024-03-04T16:04:47.569 INFO:tasks.workunit.client.1.smithi152.stdout:1/727: stat d19/d4b/d53/d57/d62/dac/cde 0 2024-03-04T16:04:47.572 INFO:tasks.workunit.client.1.smithi152.stdout:3/748: mknod d6/d10/c101 0 2024-03-04T16:04:47.574 INFO:tasks.workunit.client.1.smithi152.stdout:3/749: read d6/f8 [421985,24080] 0 2024-03-04T16:04:47.575 INFO:tasks.workunit.client.1.smithi152.stdout:0/707: symlink d9/d1b/d3c/d84/le4 0 2024-03-04T16:04:47.577 INFO:tasks.workunit.client.1.smithi152.stdout:2/748: dwrite da/fc [0,4194304] 0 2024-03-04T16:04:47.577 INFO:tasks.workunit.client.1.smithi152.stdout:3/750: stat d6/d14/ffa 0 2024-03-04T16:04:47.577 INFO:tasks.workunit.client.1.smithi152.stdout:8/848: write d2/d3/d15/d1c/d84/fa8 [3417573,69918] 0 2024-03-04T16:04:47.580 INFO:tasks.workunit.client.1.smithi152.stdout:4/613: dwrite d8/d9/f28 [0,4194304] 0 2024-03-04T16:04:47.584 INFO:tasks.workunit.client.1.smithi152.stdout:0/708: chown d9/d1b/d54/d69/d6c/dbf/fcf 173 1 2024-03-04T16:04:47.585 INFO:tasks.workunit.client.1.smithi152.stdout:1/728: mkdir d19/d75/dd9/dee 0 2024-03-04T16:04:47.587 INFO:tasks.workunit.client.1.smithi152.stdout:0/709: read d9/d1b/f71 [3180441,32985] 0 2024-03-04T16:04:47.590 INFO:tasks.workunit.client.1.smithi152.stdout:2/749: creat da/d24/d80/d82/da6/f101 x:0 0 0 2024-03-04T16:04:47.592 INFO:tasks.workunit.client.1.smithi152.stdout:6/845: write d8/d92/f48 [1217778,102505] 0 2024-03-04T16:04:47.592 INFO:tasks.workunit.client.1.smithi152.stdout:3/751: truncate d6/fd4 4305736 0 2024-03-04T16:04:47.594 INFO:tasks.workunit.client.1.smithi152.stdout:4/614: fsync d8/d9/d24/d26/f2e 0 2024-03-04T16:04:47.594 INFO:tasks.workunit.client.1.smithi152.stdout:9/687: dwrite d2/d63/d69/f8b [0,4194304] 0 2024-03-04T16:04:47.595 INFO:tasks.workunit.client.1.smithi152.stdout:6/846: dread - d8/d42/f51 zero size 2024-03-04T16:04:47.602 INFO:tasks.workunit.client.1.smithi152.stdout:8/849: chown d2/d3/d21/d2d/d31/l41 0 1 2024-03-04T16:04:47.606 INFO:tasks.workunit.client.1.smithi152.stdout:2/750: dread da/df/d58/d6e/fc8 [0,4194304] 0 2024-03-04T16:04:47.607 INFO:tasks.workunit.client.1.smithi152.stdout:1/729: dread d19/d3b/d6e/d97/fad [0,4194304] 0 2024-03-04T16:04:47.611 INFO:tasks.workunit.client.1.smithi152.stdout:5/737: dwrite d6/d8/d1f/d3d/d61/f6c [0,4194304] 0 2024-03-04T16:04:47.612 INFO:tasks.workunit.client.1.smithi152.stdout:5/738: chown d6/d8/f30 19246 1 2024-03-04T16:04:47.614 INFO:tasks.workunit.client.1.smithi152.stdout:3/752: dwrite d6/d14/d41/d49/fae [0,4194304] 0 2024-03-04T16:04:47.617 INFO:tasks.workunit.client.1.smithi152.stdout:9/688: chown d2/d13/d23/d42/cb6 139201 1 2024-03-04T16:04:47.617 INFO:tasks.workunit.client.1.smithi152.stdout:3/753: stat d6/d14/d93/db7/le1 0 2024-03-04T16:04:47.619 INFO:tasks.workunit.client.1.smithi152.stdout:7/670: truncate d14/f15 1441536 0 2024-03-04T16:04:47.625 INFO:tasks.workunit.client.1.smithi152.stdout:5/739: dwrite d6/d8/d5a/d64/d76/d9f/dc7/dcb/dd1/fda [0,4194304] 0 2024-03-04T16:04:47.628 INFO:tasks.workunit.client.1.smithi152.stdout:5/740: chown d6/d8/f2c 182 1 2024-03-04T16:04:47.635 INFO:tasks.workunit.client.1.smithi152.stdout:2/751: creat da/df/d58/da2/db5/f102 x:0 0 0 2024-03-04T16:04:47.638 INFO:tasks.workunit.client.1.smithi152.stdout:1/730: mkdir d19/d20/d2f/d65/def 0 2024-03-04T16:04:47.639 INFO:tasks.workunit.client.1.smithi152.stdout:1/731: stat d19/d20/d2f/d6b/l3f 0 2024-03-04T16:04:47.640 INFO:tasks.workunit.client.1.smithi152.stdout:1/732: chown d19/d3b/d6e/d97/dd7/d63/c8c 125 1 2024-03-04T16:04:47.641 INFO:tasks.workunit.client.1.smithi152.stdout:4/615: symlink d8/d9/d24/lc6 0 2024-03-04T16:04:47.645 INFO:tasks.workunit.client.1.smithi152.stdout:8/850: dread d2/d3/d21/d2d/d52/d68/d6e/d7d/f8c [0,4194304] 0 2024-03-04T16:04:47.647 INFO:tasks.workunit.client.1.smithi152.stdout:8/851: write d2/d7f/d114/f118 [523023,102255] 0 2024-03-04T16:04:47.653 INFO:tasks.workunit.client.1.smithi152.stdout:0/710: truncate d9/d2c/f6f 3856306 0 2024-03-04T16:04:47.653 INFO:tasks.workunit.client.1.smithi152.stdout:0/711: readlink d9/d1b/d3c/l67 0 2024-03-04T16:04:47.660 INFO:tasks.workunit.client.1.smithi152.stdout:7/671: write d14/d21/d47/d36/f5b [1239510,111960] 0 2024-03-04T16:04:47.661 INFO:tasks.workunit.client.1.smithi152.stdout:6/847: link d8/d10/c26 d8/d52/d58/c129 0 2024-03-04T16:04:47.664 INFO:tasks.workunit.client.1.smithi152.stdout:8/852: symlink d2/d3/d15/d1c/d84/dde/de4/l119 0 2024-03-04T16:04:47.667 INFO:tasks.workunit.client.1.smithi152.stdout:1/733: truncate d19/d3b/d6e/d97/dd7/d63/fb0 655447 0 2024-03-04T16:04:47.669 INFO:tasks.workunit.client.1.smithi152.stdout:4/616: getdents d8/d2f/d7c/d87/dbe 0 2024-03-04T16:04:47.670 INFO:tasks.workunit.client.1.smithi152.stdout:4/617: stat d8/d4e/d8b/cb8 0 2024-03-04T16:04:47.673 INFO:tasks.workunit.client.1.smithi152.stdout:5/741: creat d6/d8/d5a/d64/fee x:0 0 0 2024-03-04T16:04:47.675 INFO:tasks.workunit.client.1.smithi152.stdout:8/853: symlink d2/d7f/d114/l11a 0 2024-03-04T16:04:47.679 INFO:tasks.workunit.client.1.smithi152.stdout:1/734: unlink d19/d3b/d6e/l8f 0 2024-03-04T16:04:47.686 INFO:tasks.workunit.client.1.smithi152.stdout:3/754: getdents d6 0 2024-03-04T16:04:47.690 INFO:tasks.workunit.client.1.smithi152.stdout:4/618: rmdir d8 39 2024-03-04T16:04:47.693 INFO:tasks.workunit.client.1.smithi152.stdout:7/672: mkdir d14/dbe 0 2024-03-04T16:04:47.693 INFO:tasks.workunit.client.1.smithi152.stdout:7/673: stat d14/d21/d47/d36/f68 0 2024-03-04T16:04:47.696 INFO:tasks.workunit.client.1.smithi152.stdout:0/712: truncate d9/d1a/d7c/faa 2873585 0 2024-03-04T16:04:47.698 INFO:tasks.workunit.client.1.smithi152.stdout:6/848: unlink d8/f28 0 2024-03-04T16:04:47.703 INFO:tasks.workunit.client.1.smithi152.stdout:8/854: dread - d2/d3/d21/d2d/d52/d68/d6e/d7d/f9b zero size 2024-03-04T16:04:47.705 INFO:tasks.workunit.client.1.smithi152.stdout:1/735: fdatasync d19/d75/da8/db8/fd2 0 2024-03-04T16:04:47.706 INFO:tasks.workunit.client.1.smithi152.stdout:1/736: dread - d19/d75/da8/db8/fd2 zero size 2024-03-04T16:04:47.706 INFO:tasks.workunit.client.1.smithi152.stdout:1/737: chown d19/d4b/d53/d57/d62 1 1 2024-03-04T16:04:47.707 INFO:tasks.workunit.client.1.smithi152.stdout:1/738: fdatasync d19/d20/d2f/d6b/f5a 0 2024-03-04T16:04:47.708 INFO:tasks.workunit.client.1.smithi152.stdout:9/689: link d2/d13/d96/cba d2/d63/d69/ce3 0 2024-03-04T16:04:47.717 INFO:tasks.workunit.client.1.smithi152.stdout:7/674: chown d14/f9b 66881860 1 2024-03-04T16:04:47.719 INFO:tasks.workunit.client.1.smithi152.stdout:5/742: dwrite d6/d8/d1f/f3b [0,4194304] 0 2024-03-04T16:04:47.721 INFO:tasks.workunit.client.1.smithi152.stdout:0/713: creat d9/d1b/d54/d69/d98/fe5 x:0 0 0 2024-03-04T16:04:47.723 INFO:tasks.workunit.client.1.smithi152.stdout:8/855: dread d2/d75/f8f [0,4194304] 0 2024-03-04T16:04:47.737 INFO:tasks.workunit.client.1.smithi152.stdout:9/690: dread d2/d6e/f7a [0,4194304] 0 2024-03-04T16:04:47.740 INFO:tasks.workunit.client.1.smithi152.stdout:4/619: symlink d8/d4e/lc7 0 2024-03-04T16:04:47.740 INFO:tasks.workunit.client.1.smithi152.stdout:6/849: dwrite d8/d42/fc3 [0,4194304] 0 2024-03-04T16:04:47.741 INFO:tasks.workunit.client.1.smithi152.stdout:9/691: write d2/d13/f15 [1413965,74303] 0 2024-03-04T16:04:47.745 INFO:tasks.workunit.client.1.smithi152.stdout:3/755: truncate d6/f46 2072036 0 2024-03-04T16:04:47.747 INFO:tasks.workunit.client.1.smithi152.stdout:3/756: dread - d6/d60/d8e/ffe zero size 2024-03-04T16:04:47.748 INFO:tasks.workunit.client.1.smithi152.stdout:3/757: chown d6/d60/d8e 2500443 1 2024-03-04T16:04:47.753 INFO:tasks.workunit.client.1.smithi152.stdout:8/856: truncate d2/d3/d21/d2d/d48/f7e 637218 0 2024-03-04T16:04:47.756 INFO:tasks.workunit.client.1.smithi152.stdout:8/857: write d2/d3/d15/d1c/d84/d87/d5d/f7b [98589,40057] 0 2024-03-04T16:04:47.758 INFO:tasks.workunit.client.1.smithi152.stdout:5/743: fdatasync d6/d8/d5a/d8b/f8f 0 2024-03-04T16:04:47.760 INFO:tasks.workunit.client.1.smithi152.stdout:4/620: rmdir d8/d2f/d6f/d90 39 2024-03-04T16:04:47.762 INFO:tasks.workunit.client.1.smithi152.stdout:8/858: dread d2/f80 [0,4194304] 0 2024-03-04T16:04:47.763 INFO:tasks.workunit.client.1.smithi152.stdout:3/758: rename d6/d14/d41/d49/f62 to d6/d60/d8e/dc0/db5/de7/f102 0 2024-03-04T16:04:47.767 INFO:tasks.workunit.client.1.smithi152.stdout:0/714: dread d9/f45 [0,4194304] 0 2024-03-04T16:04:47.767 INFO:tasks.workunit.client.1.smithi152.stdout:3/759: dread d6/d14/d58/dad/fde [0,4194304] 0 2024-03-04T16:04:47.768 INFO:tasks.workunit.client.1.smithi152.stdout:3/760: readlink d6/d10/l66 0 2024-03-04T16:04:47.773 INFO:tasks.workunit.client.1.smithi152.stdout:6/850: mkdir d8/d92/db1/d87/d12a 0 2024-03-04T16:04:47.782 INFO:tasks.workunit.client.1.smithi152.stdout:1/739: link d19/d3b/f92 d19/d20/d2f/ff0 0 2024-03-04T16:04:47.784 INFO:tasks.workunit.client.1.smithi152.stdout:0/715: mknod d9/d2c/d2e/d7f/ce6 0 2024-03-04T16:04:47.786 INFO:tasks.workunit.client.1.smithi152.stdout:3/761: rmdir d6/d10 39 2024-03-04T16:04:47.787 INFO:tasks.workunit.client.1.smithi152.stdout:8/859: write d2/d75/d89/fd4 [1233330,20634] 0 2024-03-04T16:04:47.789 INFO:tasks.workunit.client.1.smithi152.stdout:1/740: dwrite d19/fca [0,4194304] 0 2024-03-04T16:04:47.791 INFO:tasks.workunit.client.1.smithi152.stdout:6/851: creat d8/d52/f12b x:0 0 0 2024-03-04T16:04:47.799 INFO:tasks.workunit.client.1.smithi152.stdout:0/716: mknod d9/d1b/d3c/d84/d56/ce7 0 2024-03-04T16:04:47.802 INFO:tasks.workunit.client.1.smithi152.stdout:0/717: chown d9/d1b/d3c/d84/d56/d5b/ca4 100054 1 2024-03-04T16:04:47.803 INFO:tasks.workunit.client.1.smithi152.stdout:5/744: link d6/fd d6/d8/d5a/d64/d76/fef 0 2024-03-04T16:04:47.804 INFO:tasks.workunit.client.1.smithi152.stdout:1/741: mkdir d19/d20/d2f/d65/def/df1 0 2024-03-04T16:04:47.806 INFO:tasks.workunit.client.1.smithi152.stdout:6/852: unlink d8/d42/dde/le5 0 2024-03-04T16:04:47.808 INFO:tasks.workunit.client.1.smithi152.stdout:1/742: creat d19/d4b/d53/d57/d62/dd8/ff2 x:0 0 0 2024-03-04T16:04:47.809 INFO:tasks.workunit.client.1.smithi152.stdout:5/745: symlink d6/d8/d5a/d64/d76/d9f/db0/dd8/lf0 0 2024-03-04T16:04:47.811 INFO:tasks.workunit.client.1.smithi152.stdout:6/853: creat d8/d23/d98/f12c x:0 0 0 2024-03-04T16:04:47.816 INFO:tasks.workunit.client.1.smithi152.stdout:1/743: dwrite d19/d4b/d53/d57/dd6/fe0 [0,4194304] 0 2024-03-04T16:04:47.818 INFO:tasks.workunit.client.1.smithi152.stdout:0/718: getdents d9/d1a/d7c 0 2024-03-04T16:04:47.825 INFO:tasks.workunit.client.1.smithi152.stdout:0/719: dread d9/d1b/d3c/fa9 [0,4194304] 0 2024-03-04T16:04:47.826 INFO:tasks.workunit.client.1.smithi152.stdout:5/746: write d6/d8/d1f/f3a [261477,34782] 0 2024-03-04T16:04:47.828 INFO:tasks.workunit.client.1.smithi152.stdout:1/744: unlink d19/d4b/d53/d57/d62/dd8/fe6 0 2024-03-04T16:04:47.832 INFO:tasks.workunit.client.1.smithi152.stdout:6/854: rmdir d8/d23/d3a/d71/df7 0 2024-03-04T16:04:47.834 INFO:tasks.workunit.client.1.smithi152.stdout:5/747: mknod d6/d8/d5a/d64/cf1 0 2024-03-04T16:04:47.835 INFO:tasks.workunit.client.1.smithi152.stdout:0/720: dwrite d9/d1b/f6d [4194304,4194304] 0 2024-03-04T16:04:47.835 INFO:tasks.workunit.client.1.smithi152.stdout:5/748: chown d6/c2a 12521259 1 2024-03-04T16:04:47.837 INFO:tasks.workunit.client.1.smithi152.stdout:0/721: write d9/d1b/d54/d80/f9e [2176967,108183] 0 2024-03-04T16:04:47.872 INFO:tasks.workunit.client.1.smithi152.stdout:6/855: unlink d8/d52/f12b 0 2024-03-04T16:04:47.874 INFO:tasks.workunit.client.1.smithi152.stdout:1/745: dread d19/d20/d2f/fa1 [0,4194304] 0 2024-03-04T16:04:47.875 INFO:tasks.workunit.client.1.smithi152.stdout:0/722: mkdir d9/d2c/d2e/d7f/d9d/de8 0 2024-03-04T16:04:47.881 INFO:tasks.workunit.client.1.smithi152.stdout:6/856: dwrite d8/d92/d59/d7d/d91/fcd [0,4194304] 0 2024-03-04T16:04:47.883 INFO:tasks.workunit.client.1.smithi152.stdout:1/746: mknod d19/d4b/cf3 0 2024-03-04T16:04:47.890 INFO:tasks.workunit.client.1.smithi152.stdout:1/747: dread d19/f1d [4194304,4194304] 0 2024-03-04T16:04:47.893 INFO:tasks.workunit.client.1.smithi152.stdout:0/723: dread d9/d2c/d2e/d7f/d9d/da3/faf [0,4194304] 0 2024-03-04T16:04:47.894 INFO:tasks.workunit.client.1.smithi152.stdout:5/749: dwrite d6/d8/d1f/d3d/d6f/f73 [0,4194304] 0 2024-03-04T16:04:47.897 INFO:tasks.workunit.client.1.smithi152.stdout:0/724: mknod d9/d2c/d2e/d7f/ce9 0 2024-03-04T16:04:47.898 INFO:tasks.workunit.client.1.smithi152.stdout:0/725: chown d9/d1b/f1d 956852326 1 2024-03-04T16:04:47.899 INFO:tasks.workunit.client.1.smithi152.stdout:0/726: dread - d9/d1b/d54/fb7 zero size 2024-03-04T16:04:47.903 INFO:tasks.workunit.client.1.smithi152.stdout:5/750: creat d6/d8/d1f/d3d/da4/dc9/ff2 x:0 0 0 2024-03-04T16:04:47.908 INFO:tasks.workunit.client.1.smithi152.stdout:0/727: dwrite d9/d1b/d3c/d84/d56/d5b/f6b [0,4194304] 0 2024-03-04T16:04:47.913 INFO:tasks.workunit.client.1.smithi152.stdout:0/728: symlink d9/d2c/d2e/d94/lea 0 2024-03-04T16:04:47.914 INFO:tasks.workunit.client.1.smithi152.stdout:5/751: dwrite d6/d8/d5a/d64/d76/d9f/dc7/dcb/fd2 [0,4194304] 0 2024-03-04T16:04:47.920 INFO:tasks.workunit.client.1.smithi152.stdout:5/752: chown d6/d8/d1f/d3d/d6f/d70/dc4 13458590 1 2024-03-04T16:04:47.923 INFO:tasks.workunit.client.1.smithi152.stdout:0/729: truncate d9/d1b/d54/d80/da8/fd9 834133 0 2024-03-04T16:04:47.923 INFO:tasks.workunit.client.1.smithi152.stdout:6/857: dwrite d8/d92/d59/f7c [0,4194304] 0 2024-03-04T16:04:47.924 INFO:tasks.workunit.client.1.smithi152.stdout:6/858: stat f7 0 2024-03-04T16:04:47.925 INFO:tasks.workunit.client.1.smithi152.stdout:1/748: dwrite d19/d20/d2f/d6b/fed [0,4194304] 0 2024-03-04T16:04:47.934 INFO:tasks.workunit.client.1.smithi152.stdout:6/859: readlink d8/d92/d59/d7d/d91/ldd 0 2024-03-04T16:04:47.937 INFO:tasks.workunit.client.1.smithi152.stdout:5/753: creat d6/d8/d5a/d64/d76/ff3 x:0 0 0 2024-03-04T16:04:47.940 INFO:tasks.workunit.client.1.smithi152.stdout:0/730: link d9/d1b/d3c/d84/fcb d9/d1b/d54/d69/d98/feb 0 2024-03-04T16:04:47.941 INFO:tasks.workunit.client.1.smithi152.stdout:0/731: dread - d9/fd0 zero size 2024-03-04T16:04:47.944 INFO:tasks.workunit.client.1.smithi152.stdout:6/860: truncate d8/d23/d3a/f3f 704622 0 2024-03-04T16:04:47.944 INFO:tasks.workunit.client.1.smithi152.stdout:5/754: write d6/d8/d1f/d3d/d61/f6c [2906740,104598] 0 2024-03-04T16:04:47.944 INFO:tasks.workunit.client.1.smithi152.stdout:1/749: dwrite d19/d3b/d6e/d97/dd7/da6/fe1 [0,4194304] 0 2024-03-04T16:04:47.949 INFO:tasks.workunit.client.1.smithi152.stdout:0/732: write d9/d1a/d7c/f7e [2045013,117604] 0 2024-03-04T16:04:47.951 INFO:tasks.workunit.client.1.smithi152.stdout:5/755: creat d6/d8/d1f/d3d/d61/ff4 x:0 0 0 2024-03-04T16:04:47.952 INFO:tasks.workunit.client.1.smithi152.stdout:1/750: creat d19/d4b/d53/d57/d62/ff4 x:0 0 0 2024-03-04T16:04:47.953 INFO:tasks.workunit.client.1.smithi152.stdout:6/861: dwrite d8/d8f/db2/f106 [0,4194304] 0 2024-03-04T16:04:47.960 INFO:tasks.workunit.client.1.smithi152.stdout:1/751: read d19/d3b/d6e/d97/dd7/f3a [3613495,73842] 0 2024-03-04T16:04:47.964 INFO:tasks.workunit.client.1.smithi152.stdout:6/862: dwrite d8/f9 [0,4194304] 0 2024-03-04T16:04:47.974 INFO:tasks.workunit.client.1.smithi152.stdout:5/756: rename d6/d8/d5a/d90/dd6/fea to d6/d8/d1f/d3d/d6f/d70/d98/de7/ff5 0 2024-03-04T16:04:47.976 INFO:tasks.workunit.client.1.smithi152.stdout:5/757: truncate d6/d8/d1f/d3d/fc1 2017381 0 2024-03-04T16:04:47.982 INFO:tasks.workunit.client.1.smithi152.stdout:6/863: dread - d8/d23/d93/d99/fc9 zero size 2024-03-04T16:04:47.987 INFO:tasks.workunit.client.1.smithi152.stdout:5/758: creat d6/d8/d1f/d3d/d6f/d70/ddb/ff6 x:0 0 0 2024-03-04T16:04:47.989 INFO:tasks.workunit.client.1.smithi152.stdout:0/733: rmdir d9/d1b/d3c/d84/d56/d5b/d75 0 2024-03-04T16:04:47.992 INFO:tasks.workunit.client.1.smithi152.stdout:6/864: rename d8/d52/d58/cd1 to d8/d92/db1/d87/dca/c12d 0 2024-03-04T16:04:47.992 INFO:tasks.workunit.client.1.smithi152.stdout:0/734: dread d9/d2c/d2e/f31 [0,4194304] 0 2024-03-04T16:04:47.995 INFO:tasks.workunit.client.1.smithi152.stdout:0/735: mknod d9/d2c/d2e/d7f/d9d/da3/cec 0 2024-03-04T16:04:47.997 INFO:tasks.workunit.client.1.smithi152.stdout:0/736: fsync d9/f45 0 2024-03-04T16:04:48.000 INFO:tasks.workunit.client.1.smithi152.stdout:1/752: dwrite d19/d4b/d53/f55 [0,4194304] 0 2024-03-04T16:04:48.002 INFO:tasks.workunit.client.1.smithi152.stdout:0/737: chown d9/c11 12 1 2024-03-04T16:04:48.008 INFO:tasks.workunit.client.1.smithi152.stdout:1/753: creat d19/d20/d2f/d6b/ff5 x:0 0 0 2024-03-04T16:04:48.010 INFO:tasks.workunit.client.1.smithi152.stdout:0/738: unlink d9/c2d 0 2024-03-04T16:04:48.014 INFO:tasks.workunit.client.1.smithi152.stdout:0/739: fdatasync d9/d1b/f6e 0 2024-03-04T16:04:48.026 INFO:tasks.workunit.client.1.smithi152.stdout:0/740: dread d9/d1b/f35 [0,4194304] 0 2024-03-04T16:04:48.029 INFO:tasks.workunit.client.1.smithi152.stdout:0/741: creat d9/d2c/d2e/d7f/d9d/de8/fed x:0 0 0 2024-03-04T16:04:48.031 INFO:tasks.workunit.client.1.smithi152.stdout:0/742: mknod d9/d1b/d54/d69/d98/cee 0 2024-03-04T16:04:48.032 INFO:tasks.workunit.client.1.smithi152.stdout:1/754: write d19/d4b/d53/f8d [708495,127832] 0 2024-03-04T16:04:48.033 INFO:tasks.workunit.client.1.smithi152.stdout:0/743: stat d9/d2c/d2e/d7f/d9d/de8/fed 0 2024-03-04T16:04:48.034 INFO:tasks.workunit.client.1.smithi152.stdout:0/744: dread - d9/d1b/d54/d80/fdf zero size 2024-03-04T16:04:48.037 INFO:tasks.workunit.client.1.smithi152.stdout:1/755: dread d19/d20/f9b [0,4194304] 0 2024-03-04T16:04:48.038 INFO:tasks.workunit.client.1.smithi152.stdout:0/745: rename d9/d1b/d54/d80/fb8 to d9/d2c/d2e/d7f/fef 0 2024-03-04T16:04:48.043 INFO:tasks.workunit.client.1.smithi152.stdout:1/756: link d19/d20/fb2 d19/d75/dc3/ff6 0 2024-03-04T16:04:48.070 INFO:tasks.workunit.client.1.smithi152.stdout:0/746: dwrite d9/d1a/d7c/faa [0,4194304] 0 2024-03-04T16:04:48.074 INFO:tasks.workunit.client.1.smithi152.stdout:0/747: creat d9/d1a/db4/ff0 x:0 0 0 2024-03-04T16:04:48.077 INFO:tasks.workunit.client.1.smithi152.stdout:1/757: dread d19/d4b/fe9 [0,4194304] 0 2024-03-04T16:04:48.078 INFO:tasks.workunit.client.1.smithi152.stdout:0/748: mkdir d9/d1b/d54/df1 0 2024-03-04T16:04:48.080 INFO:tasks.workunit.client.1.smithi152.stdout:1/758: truncate d19/d3b/fcf 1957712 0 2024-03-04T16:04:48.081 INFO:tasks.workunit.client.1.smithi152.stdout:1/759: chown d19/d75/f83 458 1 2024-03-04T16:04:48.082 INFO:tasks.workunit.client.1.smithi152.stdout:0/749: rmdir d9/d1b/d54/d80/da8 39 2024-03-04T16:04:48.084 INFO:tasks.workunit.client.1.smithi152.stdout:1/760: creat d19/d3b/d6e/d97/dd7/d63/ff7 x:0 0 0 2024-03-04T16:04:48.084 INFO:tasks.workunit.client.1.smithi152.stdout:0/750: rmdir d9/d1b/d54/d69/d6c/dbf 39 2024-03-04T16:04:48.086 INFO:tasks.workunit.client.1.smithi152.stdout:1/761: fsync d19/d4b/d53/d57/d62/d82/fc2 0 2024-03-04T16:04:48.089 INFO:tasks.workunit.client.1.smithi152.stdout:1/762: chown d19/d20/c3d 0 1 2024-03-04T16:04:48.090 INFO:tasks.workunit.client.1.smithi152.stdout:1/763: chown d19/d3b/d6e 28370971 1 2024-03-04T16:04:48.100 INFO:tasks.workunit.client.1.smithi152.stdout:1/764: creat d19/d3b/d6e/ff8 x:0 0 0 2024-03-04T16:04:48.103 INFO:tasks.workunit.client.1.smithi152.stdout:0/751: getdents d9/d1b/d54 0 2024-03-04T16:04:48.104 INFO:tasks.workunit.client.1.smithi152.stdout:1/765: symlink d19/d75/dc3/lf9 0 2024-03-04T16:04:48.111 INFO:tasks.workunit.client.1.smithi152.stdout:1/766: rename d19/d75/f83 to d19/d20/d2f/d65/ffa 0 2024-03-04T16:04:48.113 INFO:tasks.workunit.client.1.smithi152.stdout:1/767: symlink d19/d75/da8/db8/lfb 0 2024-03-04T16:04:48.154 INFO:tasks.workunit.client.1.smithi152.stdout:0/752: write d9/d2c/f6f [3486911,65928] 0 2024-03-04T16:04:48.161 INFO:tasks.workunit.client.1.smithi152.stdout:1/768: dwrite d19/d3b/d6e/d97/dd7/d63/d9a/fa2 [0,4194304] 0 2024-03-04T16:04:48.178 INFO:tasks.workunit.client.1.smithi152.stdout:0/753: mknod d9/d2c/cf2 0 2024-03-04T16:04:48.179 INFO:tasks.workunit.client.1.smithi152.stdout:1/769: symlink d19/d4b/d53/d57/d62/dac/lfc 0 2024-03-04T16:04:48.183 INFO:tasks.workunit.client.1.smithi152.stdout:1/770: rename f1 to d19/d20/d2f/d65/def/ffd 0 2024-03-04T16:04:48.184 INFO:tasks.workunit.client.1.smithi152.stdout:1/771: fdatasync d19/d20/d2f/f59 0 2024-03-04T16:04:48.184 INFO:tasks.workunit.client.1.smithi152.stdout:1/772: chown d19/d75/ldb 874657 1 2024-03-04T16:04:48.186 INFO:tasks.workunit.client.1.smithi152.stdout:0/754: getdents d9/d1b/d54/d80 0 2024-03-04T16:04:48.188 INFO:tasks.workunit.client.1.smithi152.stdout:1/773: mkdir d19/d75/dd9/dfe 0 2024-03-04T16:04:48.191 INFO:tasks.workunit.client.1.smithi152.stdout:0/755: creat d9/d1b/d3c/d84/d56/d5b/ff3 x:0 0 0 2024-03-04T16:04:48.193 INFO:tasks.workunit.client.1.smithi152.stdout:0/756: write d9/d1b/f6e [3178580,114567] 0 2024-03-04T16:04:48.195 INFO:tasks.workunit.client.1.smithi152.stdout:0/757: mkdir d9/d2c/d2e/d7f/d9d/de8/df4 0 2024-03-04T16:04:48.201 INFO:tasks.workunit.client.1.smithi152.stdout:0/758: dread d9/d2c/d2e/f32 [0,4194304] 0 2024-03-04T16:04:48.203 INFO:tasks.workunit.client.1.smithi152.stdout:0/759: fdatasync d9/d1b/d54/d69/f9a 0 2024-03-04T16:04:48.205 INFO:tasks.workunit.client.1.smithi152.stdout:0/760: mkdir d9/d1b/d54/df5 0 2024-03-04T16:04:48.211 INFO:tasks.workunit.client.1.smithi152.stdout:0/761: dwrite d9/d2c/d2e/d94/fca [0,4194304] 0 2024-03-04T16:04:48.217 INFO:tasks.workunit.client.1.smithi152.stdout:1/774: write d19/d20/d2f/d6b/f40 [642273,75927] 0 2024-03-04T16:04:48.218 INFO:tasks.workunit.client.1.smithi152.stdout:1/775: readlink d19/d4b/d53/d57/d62/dac/lb5 0 2024-03-04T16:04:48.221 INFO:tasks.workunit.client.1.smithi152.stdout:1/776: mknod d19/d4b/d53/d57/d62/d82/cff 0 2024-03-04T16:04:48.223 INFO:tasks.workunit.client.1.smithi152.stdout:0/762: write d9/d2c/d2e/d7f/d9d/da3/fab [1565429,113227] 0 2024-03-04T16:04:48.224 INFO:tasks.workunit.client.1.smithi152.stdout:1/777: mknod d19/d3b/c100 0 2024-03-04T16:04:48.227 INFO:tasks.workunit.client.1.smithi152.stdout:0/763: chown d9/d1b/d54/d80/da8/fd9 0 1 2024-03-04T16:04:48.228 INFO:tasks.workunit.client.1.smithi152.stdout:1/778: mkdir d19/d3b/d6e/d97/dd7/d63/d101 0 2024-03-04T16:04:48.235 INFO:tasks.workunit.client.1.smithi152.stdout:1/779: creat d19/d4b/d53/d57/f102 x:0 0 0 2024-03-04T16:04:48.250 INFO:tasks.workunit.client.1.smithi152.stdout:0/764: dread d9/d1b/d54/d69/d95/fb5 [0,4194304] 0 2024-03-04T16:04:48.252 INFO:tasks.workunit.client.1.smithi152.stdout:0/765: rmdir d9/d2c/d2e/d7f 39 2024-03-04T16:04:48.255 INFO:tasks.workunit.client.1.smithi152.stdout:1/780: dread f10 [0,4194304] 0 2024-03-04T16:04:48.257 INFO:tasks.workunit.client.1.smithi152.stdout:1/781: chown d19/d20/d2f/d6b/f42 69334 1 2024-03-04T16:04:48.259 INFO:tasks.workunit.client.1.smithi152.stdout:0/766: dwrite d9/d1b/d3c/d84/d56/f63 [0,4194304] 0 2024-03-04T16:04:48.261 INFO:tasks.workunit.client.1.smithi152.stdout:1/782: unlink d19/d3b/d6e/c9c 0 2024-03-04T16:04:48.263 INFO:tasks.workunit.client.1.smithi152.stdout:1/783: read d19/d75/fce [4379467,4814] 0 2024-03-04T16:04:48.275 INFO:tasks.workunit.client.1.smithi152.stdout:0/767: truncate d9/d1b/f6d 5890881 0 2024-03-04T16:04:48.277 INFO:tasks.workunit.client.1.smithi152.stdout:0/768: creat d9/d1a/db4/ff6 x:0 0 0 2024-03-04T16:04:48.283 INFO:tasks.workunit.client.1.smithi152.stdout:1/784: dwrite d19/d4b/d53/d57/f5e [0,4194304] 0 2024-03-04T16:04:48.286 INFO:tasks.workunit.client.1.smithi152.stdout:0/769: dwrite d9/d1b/d54/d69/f9a [4194304,4194304] 0 2024-03-04T16:04:48.299 INFO:tasks.workunit.client.1.smithi152.stdout:1/785: getdents d19/d3b 0 2024-03-04T16:04:48.306 INFO:tasks.workunit.client.1.smithi152.stdout:0/770: dwrite d9/d1b/d3c/d84/f77 [0,4194304] 0 2024-03-04T16:04:48.307 INFO:tasks.workunit.client.1.smithi152.stdout:1/786: getdents d19/d3b/d6e/d97/dd7/d63/d9a 0 2024-03-04T16:04:48.310 INFO:tasks.workunit.client.1.smithi152.stdout:0/771: mknod d9/d2c/d2e/d94/dac/cf7 0 2024-03-04T16:04:48.314 INFO:tasks.workunit.client.1.smithi152.stdout:0/772: fsync d9/f49 0 2024-03-04T16:04:48.316 INFO:tasks.workunit.client.1.smithi152.stdout:1/787: dwrite d19/d3b/d6e/ff8 [0,4194304] 0 2024-03-04T16:04:48.317 INFO:tasks.workunit.client.1.smithi152.stdout:1/788: readlink d19/d20/d2f/l90 0 2024-03-04T16:04:48.318 INFO:tasks.workunit.client.1.smithi152.stdout:1/789: fdatasync d19/d4b/fea 0 2024-03-04T16:04:48.331 INFO:tasks.workunit.client.1.smithi152.stdout:1/790: write d19/f23 [1814299,41811] 0 2024-03-04T16:04:48.334 INFO:tasks.workunit.client.1.smithi152.stdout:0/773: dwrite d9/d1b/d3c/d84/fcb [0,4194304] 0 2024-03-04T16:04:48.334 INFO:tasks.workunit.client.1.smithi152.stdout:1/791: mkdir d19/d75/dc3/d103 0 2024-03-04T16:04:48.335 INFO:tasks.workunit.client.1.smithi152.stdout:0/774: chown d9/d1b/d54/d69/d6c 3347 1 2024-03-04T16:04:48.338 INFO:tasks.workunit.client.1.smithi152.stdout:1/792: read d19/d4b/d53/d5c/f96 [442891,29850] 0 2024-03-04T16:04:48.339 INFO:tasks.workunit.client.1.smithi152.stdout:1/793: chown d19/d75/dc3 0 1 2024-03-04T16:04:48.344 INFO:tasks.workunit.client.1.smithi152.stdout:0/775: mknod d9/d2c/d2e/cf8 0 2024-03-04T16:04:48.347 INFO:tasks.workunit.client.1.smithi152.stdout:1/794: creat d19/d4b/d53/d57/d62/f104 x:0 0 0 2024-03-04T16:04:48.348 INFO:tasks.workunit.client.1.smithi152.stdout:1/795: readlink d19/d20/d2f/lb6 0 2024-03-04T16:04:48.352 INFO:tasks.workunit.client.1.smithi152.stdout:0/776: unlink d9/c1f 0 2024-03-04T16:04:48.356 INFO:tasks.workunit.client.1.smithi152.stdout:0/777: creat d9/d1b/d54/d69/d6c/ff9 x:0 0 0 2024-03-04T16:04:48.448 INFO:tasks.workunit.client.1.smithi152.stdout:0/778: write d9/f16 [9120901,75628] 0 2024-03-04T16:04:48.451 INFO:tasks.workunit.client.1.smithi152.stdout:0/779: creat d9/d1a/d7c/ffa x:0 0 0 2024-03-04T16:04:48.457 INFO:tasks.workunit.client.1.smithi152.stdout:0/780: creat d9/d2c/d2e/ffb x:0 0 0 2024-03-04T16:04:48.462 INFO:tasks.workunit.client.1.smithi152.stdout:0/781: mknod d9/d2c/d2e/d7f/d9d/cfc 0 2024-03-04T16:04:48.474 INFO:tasks.workunit.client.1.smithi152.stdout:0/782: truncate d9/d1b/d3c/d84/d56/f8b 2060699 0 2024-03-04T16:04:48.487 INFO:tasks.workunit.client.1.smithi152.stdout:0/783: dwrite d9/d1b/d3c/d84/d56/f63 [4194304,4194304] 0 2024-03-04T16:04:48.492 INFO:tasks.workunit.client.1.smithi152.stdout:0/784: mkdir d9/d1b/d54/d69/d6c/dbf/dfd 0 2024-03-04T16:04:48.494 INFO:tasks.workunit.client.1.smithi152.stdout:0/785: read d9/f36 [996867,1128] 0 2024-03-04T16:04:48.497 INFO:tasks.workunit.client.1.smithi152.stdout:0/786: link d9/d1a/f79 d9/d1b/d54/d80/ffe 0 2024-03-04T16:04:48.499 INFO:tasks.workunit.client.1.smithi152.stdout:0/787: fsync d9/d2c/d2e/d7f/d9d/fa7 0 2024-03-04T16:04:48.713 INFO:tasks.workunit.client.1.smithi152.stdout:2/752: sync 2024-03-04T16:04:48.716 INFO:tasks.workunit.client.1.smithi152.stdout:2/753: creat da/d2c/d5e/f103 x:0 0 0 2024-03-04T16:04:48.717 INFO:tasks.workunit.client.1.smithi152.stdout:2/754: fsync da/f3c 0 2024-03-04T16:04:48.734 INFO:tasks.workunit.client.1.smithi152.stdout:2/755: write da/d24/f94 [3294562,101304] 0 2024-03-04T16:04:48.738 INFO:tasks.workunit.client.1.smithi152.stdout:2/756: creat da/d5f/d85/d8c/f104 x:0 0 0 2024-03-04T16:04:48.742 INFO:tasks.workunit.client.1.smithi152.stdout:2/757: mknod da/d24/d47/c105 0 2024-03-04T16:04:48.744 INFO:tasks.workunit.client.1.smithi152.stdout:2/758: creat da/d24/d80/f106 x:0 0 0 2024-03-04T16:04:48.747 INFO:tasks.workunit.client.1.smithi152.stdout:2/759: mkdir da/df/d2e/d51/d107 0 2024-03-04T16:04:48.748 INFO:tasks.workunit.client.1.smithi152.stdout:2/760: truncate da/d24/d80/d82/da6/f101 43546 0 2024-03-04T16:04:48.750 INFO:tasks.workunit.client.1.smithi152.stdout:2/761: mknod da/d24/d80/d82/da8/c108 0 2024-03-04T16:04:48.868 INFO:tasks.workunit.client.1.smithi152.stdout:7/675: sync 2024-03-04T16:04:48.870 INFO:tasks.workunit.client.1.smithi152.stdout:7/676: fdatasync f4 0 2024-03-04T16:04:48.887 INFO:tasks.workunit.client.1.smithi152.stdout:7/677: dread d14/f16 [0,4194304] 0 2024-03-04T16:04:48.890 INFO:tasks.workunit.client.1.smithi152.stdout:7/678: creat d14/d21/d47/d36/fbf x:0 0 0 2024-03-04T16:04:48.893 INFO:tasks.workunit.client.1.smithi152.stdout:7/679: read d14/f2f [95084,130823] 0 2024-03-04T16:04:48.899 INFO:tasks.workunit.client.1.smithi152.stdout:7/680: rmdir d14/dbe 0 2024-03-04T16:04:49.082 INFO:tasks.workunit.client.1.smithi152.stdout:9/692: sync 2024-03-04T16:04:49.084 INFO:tasks.workunit.client.1.smithi152.stdout:9/693: read d2/d13/d23/f5f [2750966,15157] 0 2024-03-04T16:04:49.086 INFO:tasks.workunit.client.1.smithi152.stdout:9/694: mknod d2/d13/d23/d42/ce4 0 2024-03-04T16:04:49.093 INFO:tasks.workunit.client.1.smithi152.stdout:9/695: truncate d2/d13/d23/d42/f78 811752 0 2024-03-04T16:04:49.093 INFO:tasks.workunit.client.1.smithi152.stdout:9/696: fsync d2/d63/d69/fdb 0 2024-03-04T16:04:49.105 INFO:tasks.workunit.client.1.smithi152.stdout:9/697: rename d2/d13/d23/d71/da3/ld5 to d2/d63/d69/le5 0 2024-03-04T16:04:49.107 INFO:tasks.workunit.client.1.smithi152.stdout:9/698: creat d2/d13/d38/d79/fe6 x:0 0 0 2024-03-04T16:04:49.108 INFO:tasks.workunit.client.1.smithi152.stdout:9/699: write d2/d13/dad/dd2/fd6 [2474,120846] 0 2024-03-04T16:04:49.111 INFO:tasks.workunit.client.1.smithi152.stdout:9/700: creat d2/d13/d96/fe7 x:0 0 0 2024-03-04T16:04:49.116 INFO:tasks.workunit.client.1.smithi152.stdout:9/701: dwrite d2/d13/d2e/f76 [0,4194304] 0 2024-03-04T16:04:49.117 INFO:tasks.workunit.client.1.smithi152.stdout:9/702: fsync d2/d13/dad/dd2/fd6 0 2024-03-04T16:04:49.119 INFO:tasks.workunit.client.1.smithi152.stdout:9/703: write d2/d13/d23/d66/d7d/f9a [2130526,71472] 0 2024-03-04T16:04:49.123 INFO:tasks.workunit.client.1.smithi152.stdout:9/704: creat d2/d13/d23/d42/fe8 x:0 0 0 2024-03-04T16:04:49.126 INFO:tasks.workunit.client.1.smithi152.stdout:0/788: sync 2024-03-04T16:04:49.126 INFO:tasks.workunit.client.1.smithi152.stdout:3/762: sync 2024-03-04T16:04:49.126 INFO:tasks.workunit.client.1.smithi152.stdout:8/860: sync 2024-03-04T16:04:49.126 INFO:tasks.workunit.client.1.smithi152.stdout:7/681: sync 2024-03-04T16:04:49.126 INFO:tasks.workunit.client.1.smithi152.stdout:1/796: sync 2024-03-04T16:04:49.127 INFO:tasks.workunit.client.1.smithi152.stdout:4/621: sync 2024-03-04T16:04:49.127 INFO:tasks.workunit.client.1.smithi152.stdout:5/759: sync 2024-03-04T16:04:49.127 INFO:tasks.workunit.client.1.smithi152.stdout:2/762: sync 2024-03-04T16:04:49.127 INFO:tasks.workunit.client.1.smithi152.stdout:6/865: sync 2024-03-04T16:04:49.128 INFO:tasks.workunit.client.1.smithi152.stdout:4/622: dread - d8/d3d/fc4 zero size 2024-03-04T16:04:49.128 INFO:tasks.workunit.client.1.smithi152.stdout:9/705: truncate d2/d13/f18 666349 0 2024-03-04T16:04:49.129 INFO:tasks.workunit.client.1.smithi152.stdout:2/763: write da/df/d2e/f4e [3125396,17761] 0 2024-03-04T16:04:49.134 INFO:tasks.workunit.client.1.smithi152.stdout:8/861: chown d2/d3/d21/d2d/d31/lc9 8108832 1 2024-03-04T16:04:49.141 INFO:tasks.workunit.client.1.smithi152.stdout:0/789: dwrite d9/d1b/d54/d69/d98/fe5 [0,4194304] 0 2024-03-04T16:04:49.141 INFO:tasks.workunit.client.1.smithi152.stdout:4/623: dwrite d8/d9/fb2 [0,4194304] 0 2024-03-04T16:04:49.141 INFO:tasks.workunit.client.1.smithi152.stdout:7/682: rename d14/c2e to d14/d2b/d83/cc0 0 2024-03-04T16:04:49.144 INFO:tasks.workunit.client.1.smithi152.stdout:3/763: chown d6/l98 22401 1 2024-03-04T16:04:49.148 INFO:tasks.workunit.client.1.smithi152.stdout:9/706: creat d2/d13/dad/dd2/fe9 x:0 0 0 2024-03-04T16:04:49.149 INFO:tasks.workunit.client.1.smithi152.stdout:9/707: chown d2/d13/d23/d42 2741 1 2024-03-04T16:04:49.152 INFO:tasks.workunit.client.1.smithi152.stdout:2/764: symlink da/df/d2e/d61/l109 0 2024-03-04T16:04:49.154 INFO:tasks.workunit.client.1.smithi152.stdout:2/765: write da/df/d2e/f38 [790043,13113] 0 2024-03-04T16:04:49.164 INFO:tasks.workunit.client.1.smithi152.stdout:7/683: unlink d14/d21/d47/f65 0 2024-03-04T16:04:49.167 INFO:tasks.workunit.client.1.smithi152.stdout:0/790: symlink d9/d2c/lff 0 2024-03-04T16:04:49.171 INFO:tasks.workunit.client.1.smithi152.stdout:5/760: mknod d6/d8/d1f/d3d/d6f/d70/ddb/dde/cf7 0 2024-03-04T16:04:49.173 INFO:tasks.workunit.client.1.smithi152.stdout:0/791: dwrite d9/d1b/d3c/d84/f77 [0,4194304] 0 2024-03-04T16:04:49.180 INFO:tasks.workunit.client.1.smithi152.stdout:1/797: write d19/d4b/f5f [902158,76541] 0 2024-03-04T16:04:49.182 INFO:tasks.workunit.client.1.smithi152.stdout:1/798: readlink d19/d20/d2f/d6b/l4e 0 2024-03-04T16:04:49.185 INFO:tasks.workunit.client.1.smithi152.stdout:6/866: write d8/d23/f25 [3073065,80196] 0 2024-03-04T16:04:49.235 INFO:tasks.workunit.client.1.smithi152.stdout:2/766: creat da/d24/d81/f10a x:0 0 0 2024-03-04T16:04:49.239 INFO:tasks.workunit.client.1.smithi152.stdout:3/764: dread d6/d60/f67 [0,4194304] 0 2024-03-04T16:04:49.240 INFO:tasks.workunit.client.1.smithi152.stdout:5/761: mknod d6/d8/d1f/d3d/d6f/d70/d98/de7/cf8 0 2024-03-04T16:04:49.242 INFO:tasks.workunit.client.1.smithi152.stdout:9/708: mkdir d2/d13/dea 0 2024-03-04T16:04:49.244 INFO:tasks.workunit.client.1.smithi152.stdout:1/799: mknod d19/d4b/d53/c105 0 2024-03-04T16:04:49.245 INFO:tasks.workunit.client.1.smithi152.stdout:6/867: mkdir d8/d10/d53/d12e 0 2024-03-04T16:04:49.247 INFO:tasks.workunit.client.1.smithi152.stdout:2/767: fdatasync da/d5f/f68 0 2024-03-04T16:04:49.248 INFO:tasks.workunit.client.1.smithi152.stdout:4/624: link d8/d9/d24/d26/c8a d8/d9/d7a/cc8 0 2024-03-04T16:04:49.250 INFO:tasks.workunit.client.1.smithi152.stdout:5/762: rmdir d6/d8 39 2024-03-04T16:04:49.253 INFO:tasks.workunit.client.1.smithi152.stdout:1/800: mknod d19/d4b/d53/d57/d62/dd8/c106 0 2024-03-04T16:04:49.255 INFO:tasks.workunit.client.1.smithi152.stdout:9/709: dread d2/d13/d23/d71/f86 [0,4194304] 0 2024-03-04T16:04:49.256 INFO:tasks.workunit.client.1.smithi152.stdout:8/862: getdents d2/d3/d21/d2d 0 2024-03-04T16:04:49.258 INFO:tasks.workunit.client.1.smithi152.stdout:8/863: truncate d2/d75/d89/fe2 1774400 0 2024-03-04T16:04:49.264 INFO:tasks.workunit.client.1.smithi152.stdout:6/868: getdents d8/da9/d111 0 2024-03-04T16:04:49.266 INFO:tasks.workunit.client.1.smithi152.stdout:0/792: dread d9/d1b/d54/d80/f96 [0,4194304] 0 2024-03-04T16:04:49.269 INFO:tasks.workunit.client.1.smithi152.stdout:2/768: link da/df/d2e/d51/fb4 da/d5f/db0/f10b 0 2024-03-04T16:04:49.272 INFO:tasks.workunit.client.1.smithi152.stdout:8/864: truncate d2/d75/d89/dd/d22/f3b 7457622 0 2024-03-04T16:04:49.276 INFO:tasks.workunit.client.1.smithi152.stdout:2/769: readlink da/l7c 0 2024-03-04T16:04:49.280 INFO:tasks.workunit.client.1.smithi152.stdout:5/763: symlink d6/d8/d5a/d64/lf9 0 2024-03-04T16:04:49.282 INFO:tasks.workunit.client.1.smithi152.stdout:1/801: rmdir d19/d75/dd9/dee 0 2024-03-04T16:04:49.283 INFO:tasks.workunit.client.1.smithi152.stdout:9/710: truncate d2/d13/d2e/f3c 1944353 0 2024-03-04T16:04:49.285 INFO:tasks.workunit.client.1.smithi152.stdout:0/793: mknod d9/d2c/d2e/d7f/d9d/de8/df4/c100 0 2024-03-04T16:04:49.286 INFO:tasks.workunit.client.1.smithi152.stdout:4/625: getdents d8/d9/d24/d26 0 2024-03-04T16:04:49.290 INFO:tasks.workunit.client.1.smithi152.stdout:2/770: symlink da/d24/d47/d73/l10c 0 2024-03-04T16:04:49.291 INFO:tasks.workunit.client.1.smithi152.stdout:8/865: rmdir d2/d75/d89/dd/d22/d83 39 2024-03-04T16:04:49.297 INFO:tasks.workunit.client.1.smithi152.stdout:5/764: symlink d6/d8/d5a/d64/d76/d9f/dc7/dcb/dd1/lfa 0 2024-03-04T16:04:49.301 INFO:tasks.workunit.client.1.smithi152.stdout:0/794: write d9/d2c/d2e/d7f/d9d/da3/faf [655833,33850] 0 2024-03-04T16:04:49.302 INFO:tasks.workunit.client.1.smithi152.stdout:2/771: dread - da/d2c/fba zero size 2024-03-04T16:04:49.305 INFO:tasks.workunit.client.1.smithi152.stdout:6/869: getdents d8/d92/d75 0 2024-03-04T16:04:49.309 INFO:tasks.workunit.client.1.smithi152.stdout:6/870: dread d8/d2a/f89 [0,4194304] 0 2024-03-04T16:04:49.311 INFO:tasks.workunit.client.1.smithi152.stdout:3/765: dwrite d6/d14/f7d [0,4194304] 0 2024-03-04T16:04:49.314 INFO:tasks.workunit.client.1.smithi152.stdout:4/626: creat d8/d9/d24/da0/fc9 x:0 0 0 2024-03-04T16:04:49.317 INFO:tasks.workunit.client.1.smithi152.stdout:0/795: rename d9/d2c/d2e/d94/dac/cf7 to d9/d92/c101 0 2024-03-04T16:04:49.322 INFO:tasks.workunit.client.1.smithi152.stdout:5/765: mkdir d6/d8/d1f/d3d/d6f/d70/dfb 0 2024-03-04T16:04:49.322 INFO:tasks.workunit.client.1.smithi152.stdout:9/711: write d2/d13/d2e/f44 [802391,29051] 0 2024-03-04T16:04:49.323 INFO:tasks.workunit.client.1.smithi152.stdout:6/871: mknod d8/d23/d93/c12f 0 2024-03-04T16:04:49.324 INFO:tasks.workunit.client.1.smithi152.stdout:0/796: dwrite d9/d1b/d54/d69/d98/fe5 [0,4194304] 0 2024-03-04T16:04:49.328 INFO:tasks.workunit.client.1.smithi152.stdout:1/802: truncate d19/f23 3097787 0 2024-03-04T16:04:49.328 INFO:tasks.workunit.client.1.smithi152.stdout:0/797: chown d9/d1b/d3c/d84/l5c 581 1 2024-03-04T16:04:49.331 INFO:tasks.workunit.client.1.smithi152.stdout:0/798: dread - d9/fd0 zero size 2024-03-04T16:04:49.332 INFO:tasks.workunit.client.1.smithi152.stdout:1/803: chown d19/d3b/d6e/d97/dd7/d63/d9a 0 1 2024-03-04T16:04:49.332 INFO:tasks.workunit.client.1.smithi152.stdout:8/866: dwrite d2/f37 [0,4194304] 0 2024-03-04T16:04:49.337 INFO:tasks.workunit.client.1.smithi152.stdout:9/712: unlink d2/d13/d23/c26 0 2024-03-04T16:04:49.339 INFO:tasks.workunit.client.1.smithi152.stdout:9/713: write d2/d13/d23/d71/da3/fab [2499255,121868] 0 2024-03-04T16:04:49.340 INFO:tasks.workunit.client.1.smithi152.stdout:6/872: creat d8/d23/d93/d99/f130 x:0 0 0 2024-03-04T16:04:49.341 INFO:tasks.workunit.client.1.smithi152.stdout:3/766: creat d6/d14/d93/def/f103 x:0 0 0 2024-03-04T16:04:49.342 INFO:tasks.workunit.client.1.smithi152.stdout:7/684: sync 2024-03-04T16:04:49.344 INFO:tasks.workunit.client.1.smithi152.stdout:8/867: write d2/d3/d21/dac/f106 [82420,82465] 0 2024-03-04T16:04:49.345 INFO:tasks.workunit.client.1.smithi152.stdout:0/799: unlink d9/d1b/d3c/c5d 0 2024-03-04T16:04:49.346 INFO:tasks.workunit.client.1.smithi152.stdout:0/800: chown d9/d1b/f1d 2240 1 2024-03-04T16:04:49.347 INFO:tasks.workunit.client.1.smithi152.stdout:1/804: truncate d19/d4b/d53/d5c/f96 1426347 0 2024-03-04T16:04:49.353 INFO:tasks.workunit.client.1.smithi152.stdout:9/714: creat d2/d13/d23/d66/feb x:0 0 0 2024-03-04T16:04:49.354 INFO:tasks.workunit.client.1.smithi152.stdout:9/715: fdatasync d2/d13/dad/dd2/fe9 0 2024-03-04T16:04:49.357 INFO:tasks.workunit.client.1.smithi152.stdout:6/873: creat d8/d23/d3a/d71/d9d/f131 x:0 0 0 2024-03-04T16:04:49.357 INFO:tasks.workunit.client.1.smithi152.stdout:4/627: write f0 [4340880,48725] 0 2024-03-04T16:04:49.358 INFO:tasks.workunit.client.1.smithi152.stdout:6/874: chown d8/d42/dde/f77 24676 1 2024-03-04T16:04:49.359 INFO:tasks.workunit.client.1.smithi152.stdout:7/685: rename d14/d21/d47/d32/c52 to d14/d71/cc1 0 2024-03-04T16:04:49.361 INFO:tasks.workunit.client.1.smithi152.stdout:8/868: mknod d2/d3/d15/d1c/d84/d87/d5d/c11b 0 2024-03-04T16:04:49.362 INFO:tasks.workunit.client.1.smithi152.stdout:0/801: rmdir d9/d1b/d54/d80 39 2024-03-04T16:04:49.366 INFO:tasks.workunit.client.1.smithi152.stdout:2/772: getdents da/df 0 2024-03-04T16:04:49.374 INFO:tasks.workunit.client.1.smithi152.stdout:9/716: dread d2/d13/d23/f5b [0,4194304] 0 2024-03-04T16:04:49.381 INFO:tasks.workunit.client.1.smithi152.stdout:5/766: truncate d6/d8/f86 2475984 0 2024-03-04T16:04:49.381 INFO:tasks.workunit.client.1.smithi152.stdout:5/767: chown d6/d8/d1f/d3d/d6f/f95 6 1 2024-03-04T16:04:49.382 INFO:tasks.workunit.client.1.smithi152.stdout:7/686: fsync d14/d2b/f3b 0 2024-03-04T16:04:49.383 INFO:tasks.workunit.client.1.smithi152.stdout:8/869: read d2/d3/d15/d1c/fb5 [3427,92797] 0 2024-03-04T16:04:49.388 INFO:tasks.workunit.client.1.smithi152.stdout:3/767: write d6/f85 [609030,121432] 0 2024-03-04T16:04:49.393 INFO:tasks.workunit.client.1.smithi152.stdout:6/875: write d8/f2f [1172159,129115] 0 2024-03-04T16:04:49.395 INFO:tasks.workunit.client.1.smithi152.stdout:4/628: dwrite d8/d9/d24/d26/f2a [0,4194304] 0 2024-03-04T16:04:49.395 INFO:tasks.workunit.client.1.smithi152.stdout:6/876: fsync d8/d8f/f11b 0 2024-03-04T16:04:49.396 INFO:tasks.workunit.client.1.smithi152.stdout:7/687: dread d14/d21/d31/f57 [0,4194304] 0 2024-03-04T16:04:49.399 INFO:tasks.workunit.client.1.smithi152.stdout:5/768: unlink d6/d8/d1f/d3d/dc0/ce1 0 2024-03-04T16:04:49.402 INFO:tasks.workunit.client.1.smithi152.stdout:5/769: dread - d6/d8/d1f/d3d/d6f/fdf zero size 2024-03-04T16:04:49.405 INFO:tasks.workunit.client.1.smithi152.stdout:7/688: dwrite d14/d21/d47/f91 [4194304,4194304] 0 2024-03-04T16:04:49.408 INFO:tasks.workunit.client.1.smithi152.stdout:3/768: rmdir d6/d60/d8e/dc0/da1 39 2024-03-04T16:04:49.411 INFO:tasks.workunit.client.1.smithi152.stdout:6/877: mkdir d8/d10/d53/d132 0 2024-03-04T16:04:49.419 INFO:tasks.workunit.client.1.smithi152.stdout:0/802: creat d9/f102 x:0 0 0 2024-03-04T16:04:49.424 INFO:tasks.workunit.client.1.smithi152.stdout:7/689: symlink d14/d71/lc2 0 2024-03-04T16:04:49.425 INFO:tasks.workunit.client.1.smithi152.stdout:7/690: write d14/d71/f7d [646612,112926] 0 2024-03-04T16:04:49.426 INFO:tasks.workunit.client.1.smithi152.stdout:1/805: truncate d19/d4b/d53/d5c/f96 1025027 0 2024-03-04T16:04:49.429 INFO:tasks.workunit.client.1.smithi152.stdout:2/773: rename da/d24/d80/d82/da8/f71 to da/d24/d47/f10d 0 2024-03-04T16:04:49.435 INFO:tasks.workunit.client.1.smithi152.stdout:0/803: fdatasync d9/d1b/f35 0 2024-03-04T16:04:49.435 INFO:tasks.workunit.client.1.smithi152.stdout:1/806: dwrite d19/d4b/d53/d57/f88 [0,4194304] 0 2024-03-04T16:04:49.436 INFO:tasks.workunit.client.1.smithi152.stdout:6/878: dread d8/d10/d53/d9a/fe8 [0,4194304] 0 2024-03-04T16:04:49.437 INFO:tasks.workunit.client.1.smithi152.stdout:6/879: write d8/d92/d59/d61/d57/feb [585062,116514] 0 2024-03-04T16:04:49.438 INFO:tasks.workunit.client.1.smithi152.stdout:5/770: write d6/d8/d5a/d90/daf/fc6 [269682,62623] 0 2024-03-04T16:04:49.444 INFO:tasks.workunit.client.1.smithi152.stdout:4/629: rename d8/d9/fb2 to d8/d2f/d6f/d90/fca 0 2024-03-04T16:04:49.445 INFO:tasks.workunit.client.1.smithi152.stdout:8/870: write d2/d3/d21/d2d/f2c [7529273,89553] 0 2024-03-04T16:04:49.445 INFO:tasks.workunit.client.1.smithi152.stdout:7/691: fdatasync d14/d71/f7d 0 2024-03-04T16:04:49.451 INFO:tasks.workunit.client.1.smithi152.stdout:3/769: write d6/d60/d8e/dc0/fa2 [3174522,104253] 0 2024-03-04T16:04:49.456 INFO:tasks.workunit.client.1.smithi152.stdout:1/807: symlink d19/d20/d2f/d5d/l107 0 2024-03-04T16:04:49.462 INFO:tasks.workunit.client.1.smithi152.stdout:6/880: rename d8/d92/d59/d7d/d91/fcd to d8/d92/db1/d87/d12a/f133 0 2024-03-04T16:04:49.462 INFO:tasks.workunit.client.1.smithi152.stdout:6/881: stat d8/d92/d75/cf5 0 2024-03-04T16:04:49.463 INFO:tasks.workunit.client.1.smithi152.stdout:6/882: fsync d8/d92/d59/d61/d57/feb 0 2024-03-04T16:04:49.466 INFO:tasks.workunit.client.1.smithi152.stdout:7/692: unlink d14/d21/d48/fab 0 2024-03-04T16:04:49.471 INFO:tasks.workunit.client.1.smithi152.stdout:0/804: fsync d9/d1b/d3c/d84/d56/f8b 0 2024-03-04T16:04:49.480 INFO:tasks.workunit.client.1.smithi152.stdout:2/774: creat da/df/d2e/f10e x:0 0 0 2024-03-04T16:04:49.482 INFO:tasks.workunit.client.1.smithi152.stdout:2/775: stat da/df/d2e/d61/d8e 0 2024-03-04T16:04:49.482 INFO:tasks.workunit.client.1.smithi152.stdout:4/630: dwrite d8/d4e/f64 [0,4194304] 0 2024-03-04T16:04:49.483 INFO:tasks.workunit.client.1.smithi152.stdout:1/808: dwrite d19/d20/f99 [0,4194304] 0 2024-03-04T16:04:49.487 INFO:tasks.workunit.client.1.smithi152.stdout:8/871: dread d2/d75/d89/dd/d22/d26/f58 [0,4194304] 0 2024-03-04T16:04:49.488 INFO:tasks.workunit.client.1.smithi152.stdout:6/883: unlink d8/d10/d53/lc5 0 2024-03-04T16:04:49.489 INFO:tasks.workunit.client.1.smithi152.stdout:6/884: chown d8/d42/fcf 2460628 1 2024-03-04T16:04:49.490 INFO:tasks.workunit.client.1.smithi152.stdout:7/693: mknod d14/d21/d47/d32/cc3 0 2024-03-04T16:04:49.496 INFO:tasks.workunit.client.1.smithi152.stdout:3/770: mkdir d6/d14/d41/d49/d84/de2/de9/d104 0 2024-03-04T16:04:49.497 INFO:tasks.workunit.client.1.smithi152.stdout:5/771: creat d6/d8/d1f/d3d/d6f/d70/ffc x:0 0 0 2024-03-04T16:04:49.498 INFO:tasks.workunit.client.1.smithi152.stdout:5/772: dread - d6/d8/d1f/d3d/da4/dc9/ff2 zero size 2024-03-04T16:04:49.500 INFO:tasks.workunit.client.1.smithi152.stdout:1/809: unlink fa 0 2024-03-04T16:04:49.504 INFO:tasks.workunit.client.1.smithi152.stdout:6/885: creat d8/d10/d53/d9a/d11a/f134 x:0 0 0 2024-03-04T16:04:49.504 INFO:tasks.workunit.client.1.smithi152.stdout:5/773: dread d6/d8/d1f/d3d/f78 [0,4194304] 0 2024-03-04T16:04:49.506 INFO:tasks.workunit.client.1.smithi152.stdout:7/694: write f7 [3045571,38175] 0 2024-03-04T16:04:49.508 INFO:tasks.workunit.client.1.smithi152.stdout:7/695: readlink d14/d21/d31/l96 0 2024-03-04T16:04:49.508 INFO:tasks.workunit.client.1.smithi152.stdout:3/771: mkdir d6/d14/d58/d105 0 2024-03-04T16:04:49.514 INFO:tasks.workunit.client.1.smithi152.stdout:5/774: dwrite d6/d8/f30 [4194304,4194304] 0 2024-03-04T16:04:49.522 INFO:tasks.workunit.client.1.smithi152.stdout:4/631: rmdir d8/d2f/d7c/d87/dbe 0 2024-03-04T16:04:49.525 INFO:tasks.workunit.client.1.smithi152.stdout:6/886: creat d8/d10/d53/d132/f135 x:0 0 0 2024-03-04T16:04:49.529 INFO:tasks.workunit.client.1.smithi152.stdout:5/775: mknod d6/d8/d5a/d64/cfd 0 2024-03-04T16:04:49.529 INFO:tasks.workunit.client.1.smithi152.stdout:2/776: write da/d24/d47/f10d [1290075,122832] 0 2024-03-04T16:04:49.532 INFO:tasks.workunit.client.1.smithi152.stdout:6/887: mkdir d8/d92/d59/d7d/d104/d136 0 2024-03-04T16:04:49.540 INFO:tasks.workunit.client.1.smithi152.stdout:0/805: dread d9/d1b/f6d [0,4194304] 0 2024-03-04T16:04:49.540 INFO:tasks.workunit.client.1.smithi152.stdout:2/777: dwrite da/d24/f42 [0,4194304] 0 2024-03-04T16:04:49.545 INFO:tasks.workunit.client.1.smithi152.stdout:1/810: dwrite d19/d3b/fcf [0,4194304] 0 2024-03-04T16:04:49.551 INFO:tasks.workunit.client.1.smithi152.stdout:7/696: dread d14/d21/d47/d36/f8b [0,4194304] 0 2024-03-04T16:04:49.557 INFO:tasks.workunit.client.1.smithi152.stdout:2/778: mknod da/d5f/db0/c10f 0 2024-03-04T16:04:49.558 INFO:tasks.workunit.client.1.smithi152.stdout:4/632: getdents d8/d9/d24/d26/d42 0 2024-03-04T16:04:49.561 INFO:tasks.workunit.client.1.smithi152.stdout:5/776: dwrite d6/d8/d5a/d64/d76/fef [12582912,4194304] 0 2024-03-04T16:04:49.561 INFO:tasks.workunit.client.1.smithi152.stdout:4/633: write d8/d2f/d6f/d90/fa5 [809351,64432] 0 2024-03-04T16:04:49.562 INFO:tasks.workunit.client.1.smithi152.stdout:1/811: stat d19/d20/f30 0 2024-03-04T16:04:49.563 INFO:tasks.workunit.client.1.smithi152.stdout:4/634: truncate d8/d3d/f44 4873660 0 2024-03-04T16:04:49.564 INFO:tasks.workunit.client.1.smithi152.stdout:1/812: dread - d19/d4b/d53/d57/d62/dd8/ff2 zero size 2024-03-04T16:04:49.564 INFO:tasks.workunit.client.1.smithi152.stdout:4/635: chown d8/d9/d24/d26/f2a 47750 1 2024-03-04T16:04:49.565 INFO:tasks.workunit.client.1.smithi152.stdout:1/813: chown d19/d4b/d53/d5c 2 1 2024-03-04T16:04:49.567 INFO:tasks.workunit.client.1.smithi152.stdout:1/814: readlink d19/d20/d2f/d6b/l4e 0 2024-03-04T16:04:49.568 INFO:tasks.workunit.client.1.smithi152.stdout:1/815: chown d19/d75/dd9/dfe 445 1 2024-03-04T16:04:49.570 INFO:tasks.workunit.client.1.smithi152.stdout:3/772: dread d6/d10/d2d/f4c [4194304,4194304] 0 2024-03-04T16:04:49.571 INFO:tasks.workunit.client.1.smithi152.stdout:1/816: fsync d19/d4b/d53/d57/f5e 0 2024-03-04T16:04:49.572 INFO:tasks.workunit.client.1.smithi152.stdout:7/697: dread d14/f2f [0,4194304] 0 2024-03-04T16:04:49.574 INFO:tasks.workunit.client.1.smithi152.stdout:5/777: creat d6/d8/d5a/d90/ffe x:0 0 0 2024-03-04T16:04:49.580 INFO:tasks.workunit.client.1.smithi152.stdout:6/888: creat d8/d92/d6f/f137 x:0 0 0 2024-03-04T16:04:49.584 INFO:tasks.workunit.client.1.smithi152.stdout:0/806: creat d9/d1b/d54/d80/da8/f103 x:0 0 0 2024-03-04T16:04:49.592 INFO:tasks.workunit.client.1.smithi152.stdout:7/698: creat d14/d21/d47/d40/fc4 x:0 0 0 2024-03-04T16:04:49.593 INFO:tasks.workunit.client.1.smithi152.stdout:6/889: fsync d8/d10/d53/dd7/fec 0 2024-03-04T16:04:49.596 INFO:tasks.workunit.client.1.smithi152.stdout:5/778: dread d6/d8/d1f/d3d/d6f/d70/f71 [0,4194304] 0 2024-03-04T16:04:49.599 INFO:tasks.workunit.client.1.smithi152.stdout:1/817: dwrite d19/d3b/d6e/d97/dd7/d63/fd0 [0,4194304] 0 2024-03-04T16:04:49.599 INFO:tasks.workunit.client.1.smithi152.stdout:7/699: truncate d14/d71/faf 640603 0 2024-03-04T16:04:49.601 INFO:tasks.workunit.client.1.smithi152.stdout:7/700: stat d14/d2b/f4e 0 2024-03-04T16:04:49.602 INFO:tasks.workunit.client.1.smithi152.stdout:3/773: write d6/db1/fd2 [897051,28012] 0 2024-03-04T16:04:49.604 INFO:tasks.workunit.client.1.smithi152.stdout:4/636: dwrite d8/d9/d24/f74 [0,4194304] 0 2024-03-04T16:04:49.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:49 smithi152 ceph-mon[36876]: pgmap v6: 65 pgs: 65 active+clean; 2.5 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 24 MiB/s rd, 61 MiB/s wr, 218 op/s 2024-03-04T16:04:49.611 INFO:tasks.workunit.client.1.smithi152.stdout:0/807: creat d9/d1b/d54/d80/f104 x:0 0 0 2024-03-04T16:04:49.615 INFO:tasks.workunit.client.1.smithi152.stdout:6/890: dwrite d8/d23/d3a/d71/d9d/fa8 [0,4194304] 0 2024-03-04T16:04:49.619 INFO:tasks.workunit.client.1.smithi152.stdout:4/637: chown d8/cc 4083 1 2024-03-04T16:04:49.626 INFO:tasks.workunit.client.1.smithi152.stdout:1/818: dread d19/d75/dc3/ff6 [0,4194304] 0 2024-03-04T16:04:49.626 INFO:tasks.workunit.client.1.smithi152.stdout:6/891: dwrite d8/d23/d93/d99/f130 [0,4194304] 0 2024-03-04T16:04:49.627 INFO:tasks.workunit.client.1.smithi152.stdout:1/819: chown f10 9743 1 2024-03-04T16:04:49.631 INFO:tasks.workunit.client.1.smithi152.stdout:0/808: symlink d9/d1b/l105 0 2024-03-04T16:04:49.632 INFO:tasks.workunit.client.1.smithi152.stdout:0/809: write d9/f49 [4687617,122241] 0 2024-03-04T16:04:49.634 INFO:tasks.workunit.client.1.smithi152.stdout:5/779: getdents d6/d8/d5a/d90/daf 0 2024-03-04T16:04:49.635 INFO:tasks.workunit.client.1.smithi152.stdout:1/820: fsync d19/d3b/d6e/d97/dd7/f3a 0 2024-03-04T16:04:49.637 INFO:tasks.workunit.client.1.smithi152.stdout:7/701: getdents d14/d71 0 2024-03-04T16:04:49.639 INFO:tasks.workunit.client.1.smithi152.stdout:0/810: creat d9/d1b/d54/d69/f106 x:0 0 0 2024-03-04T16:04:49.639 INFO:tasks.workunit.client.1.smithi152.stdout:6/892: dread d8/f39 [0,4194304] 0 2024-03-04T16:04:49.640 INFO:tasks.workunit.client.1.smithi152.stdout:3/774: link d6/l22 d6/d14/d41/d49/l106 0 2024-03-04T16:04:49.648 INFO:tasks.workunit.client.1.smithi152.stdout:4/638: rename d8/f10 to d8/d2f/fcb 0 2024-03-04T16:04:49.651 INFO:tasks.workunit.client.1.smithi152.stdout:1/821: unlink d19/d20/d2f/fa1 0 2024-03-04T16:04:49.652 INFO:tasks.workunit.client.1.smithi152.stdout:0/811: mkdir d9/d92/d107 0 2024-03-04T16:04:49.654 INFO:tasks.workunit.client.1.smithi152.stdout:3/775: mknod d6/d14/d58/c107 0 2024-03-04T16:04:49.660 INFO:tasks.workunit.client.1.smithi152.stdout:7/702: rename d14/d69/fa3 to d14/d21/d47/d36/da7/fc5 0 2024-03-04T16:04:49.663 INFO:tasks.workunit.client.1.smithi152.stdout:1/822: fsync d19/d20/d2f/d6b/f37 0 2024-03-04T16:04:49.675 INFO:tasks.workunit.client.1.smithi152.stdout:4/639: mkdir d8/d4e/dba/dcc 0 2024-03-04T16:04:49.679 INFO:tasks.workunit.client.1.smithi152.stdout:5/780: getdents d6/d8/d1f/d3d/dc0 0 2024-03-04T16:04:49.680 INFO:tasks.workunit.client.1.smithi152.stdout:1/823: rmdir d19/d20/d2f/d6b 39 2024-03-04T16:04:49.683 INFO:tasks.workunit.client.1.smithi152.stdout:0/812: rename d9/d1b/d54/l8d to d9/d1a/db4/l108 0 2024-03-04T16:04:49.685 INFO:tasks.workunit.client.1.smithi152.stdout:3/776: write d6/d14/f8d [1205802,70372] 0 2024-03-04T16:04:49.686 INFO:tasks.workunit.client.1.smithi152.stdout:5/781: write d6/d8/d5a/d64/d76/d9f/db0/fe5 [2617541,123392] 0 2024-03-04T16:04:49.688 INFO:tasks.workunit.client.1.smithi152.stdout:1/824: mkdir d19/d4b/d53/d108 0 2024-03-04T16:04:49.695 INFO:tasks.workunit.client.1.smithi152.stdout:3/777: unlink d6/d10/d2d/ff6 0 2024-03-04T16:04:49.700 INFO:tasks.workunit.client.1.smithi152.stdout:1/825: unlink d19/d20/d2f/f4d 0 2024-03-04T16:04:49.700 INFO:tasks.workunit.client.1.smithi152.stdout:7/703: link d14/d21/d47/c89 d14/d21/d31/cc6 0 2024-03-04T16:04:49.701 INFO:tasks.workunit.client.1.smithi152.stdout:3/778: dwrite d6/d14/f7d [0,4194304] 0 2024-03-04T16:04:49.704 INFO:tasks.workunit.client.1.smithi152.stdout:0/813: symlink d9/d1b/d54/d80/l109 0 2024-03-04T16:04:49.705 INFO:tasks.workunit.client.1.smithi152.stdout:4/640: creat d8/fcd x:0 0 0 2024-03-04T16:04:49.715 INFO:tasks.workunit.client.1.smithi152.stdout:0/814: creat d9/d1b/d3c/d84/d56/d5b/f10a x:0 0 0 2024-03-04T16:04:49.716 INFO:tasks.workunit.client.1.smithi152.stdout:4/641: creat d8/d2f/d7c/fce x:0 0 0 2024-03-04T16:04:49.724 INFO:tasks.workunit.client.1.smithi152.stdout:4/642: dwrite d8/d2f/d7c/fce [0,4194304] 0 2024-03-04T16:04:49.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:49 smithi116 ceph-mon[29120]: pgmap v6: 65 pgs: 65 active+clean; 2.5 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 24 MiB/s rd, 61 MiB/s wr, 218 op/s 2024-03-04T16:04:49.726 INFO:tasks.workunit.client.1.smithi152.stdout:7/704: rename d14/d2b/f56 to d14/d21/fc7 0 2024-03-04T16:04:49.730 INFO:tasks.workunit.client.1.smithi152.stdout:4/643: mknod d8/d2f/d6f/d84/ccf 0 2024-03-04T16:04:49.733 INFO:tasks.workunit.client.1.smithi152.stdout:7/705: dwrite d14/d71/f7d [0,4194304] 0 2024-03-04T16:04:49.733 INFO:tasks.workunit.client.1.smithi152.stdout:4/644: dread d8/f45 [0,4194304] 0 2024-03-04T16:04:49.737 INFO:tasks.workunit.client.1.smithi152.stdout:7/706: rename d14/f16 to d14/d21/d47/d40/d6e/fc8 0 2024-03-04T16:04:49.744 INFO:tasks.workunit.client.1.smithi152.stdout:5/782: dread d6/d8/f86 [0,4194304] 0 2024-03-04T16:04:49.745 INFO:tasks.workunit.client.1.smithi152.stdout:4/645: dread d8/d9/d24/d26/d42/f4b [0,4194304] 0 2024-03-04T16:04:49.760 INFO:tasks.workunit.client.1.smithi152.stdout:9/717: sync 2024-03-04T16:04:49.760 INFO:tasks.workunit.client.1.smithi152.stdout:9/718: chown d2/d63/d69/f8b 42169892 1 2024-03-04T16:04:49.763 INFO:tasks.workunit.client.1.smithi152.stdout:9/719: truncate d2/d13/dad/dd2/fd6 805822 0 2024-03-04T16:04:49.763 INFO:tasks.workunit.client.1.smithi152.stdout:9/720: chown d2/f2d 21 1 2024-03-04T16:04:49.764 INFO:tasks.workunit.client.1.smithi152.stdout:9/721: fdatasync d2/d13/d2e/f44 0 2024-03-04T16:04:49.765 INFO:tasks.workunit.client.1.smithi152.stdout:3/779: write d6/d10/d2d/fcf [13844,7874] 0 2024-03-04T16:04:49.767 INFO:tasks.workunit.client.1.smithi152.stdout:4/646: dread d8/d2f/d67/f77 [0,4194304] 0 2024-03-04T16:04:49.771 INFO:tasks.workunit.client.1.smithi152.stdout:1/826: write d19/d3b/d6e/d97/dd7/d63/fb0 [756514,26895] 0 2024-03-04T16:04:49.772 INFO:tasks.workunit.client.1.smithi152.stdout:4/647: mkdir d8/d2f/d7c/d87/dd0 0 2024-03-04T16:04:49.774 INFO:tasks.workunit.client.1.smithi152.stdout:4/648: write d8/d2f/f4a [3766225,86013] 0 2024-03-04T16:04:49.776 INFO:tasks.workunit.client.1.smithi152.stdout:7/707: getdents d14/d2b 0 2024-03-04T16:04:49.778 INFO:tasks.workunit.client.1.smithi152.stdout:0/815: dwrite d9/d1b/d54/d80/ffe [0,4194304] 0 2024-03-04T16:04:49.783 INFO:tasks.workunit.client.1.smithi152.stdout:1/827: creat d19/d20/d2f/d65/def/df1/f109 x:0 0 0 2024-03-04T16:04:49.788 INFO:tasks.workunit.client.1.smithi152.stdout:7/708: write d14/d21/d47/d36/da7/fc5 [1343851,69252] 0 2024-03-04T16:04:49.788 INFO:tasks.workunit.client.1.smithi152.stdout:0/816: stat d9/d2c/d2e/d7f/lbe 0 2024-03-04T16:04:49.788 INFO:tasks.workunit.client.1.smithi152.stdout:1/828: dread - d19/d4b/d53/d57/d62/d82/fbe zero size 2024-03-04T16:04:49.789 INFO:tasks.workunit.client.1.smithi152.stdout:0/817: write d9/d2c/d2e/d94/fca [3584382,100932] 0 2024-03-04T16:04:49.790 INFO:tasks.workunit.client.1.smithi152.stdout:9/722: dwrite d2/d13/fd7 [0,4194304] 0 2024-03-04T16:04:49.794 INFO:tasks.workunit.client.1.smithi152.stdout:7/709: fsync d14/d69/d8e/f9f 0 2024-03-04T16:04:49.794 INFO:tasks.workunit.client.1.smithi152.stdout:4/649: getdents d8/d4e/d86 0 2024-03-04T16:04:49.796 INFO:tasks.workunit.client.1.smithi152.stdout:7/710: fdatasync d14/d21/d47/d40/fc4 0 2024-03-04T16:04:49.797 INFO:tasks.workunit.client.1.smithi152.stdout:1/829: unlink d19/d3b/d6e/d97/dd7/l3c 0 2024-03-04T16:04:49.797 INFO:tasks.workunit.client.1.smithi152.stdout:9/723: mknod d2/d13/d23/d71/cec 0 2024-03-04T16:04:49.799 INFO:tasks.workunit.client.1.smithi152.stdout:0/818: mknod d9/d2c/d2e/d7f/d9d/de8/df4/c10b 0 2024-03-04T16:04:49.800 INFO:tasks.workunit.client.1.smithi152.stdout:7/711: write d14/d2b/f63 [655182,48674] 0 2024-03-04T16:04:49.802 INFO:tasks.workunit.client.1.smithi152.stdout:9/724: rmdir d2/d63 39 2024-03-04T16:04:49.806 INFO:tasks.workunit.client.1.smithi152.stdout:4/650: dwrite d8/f25 [4194304,4194304] 0 2024-03-04T16:04:49.808 INFO:tasks.workunit.client.1.smithi152.stdout:1/830: creat d19/d3b/d6e/d97/dd7/d63/f10a x:0 0 0 2024-03-04T16:04:49.810 INFO:tasks.workunit.client.1.smithi152.stdout:1/831: write d19/d4b/d53/d57/d62/fdc [675465,15330] 0 2024-03-04T16:04:49.811 INFO:tasks.workunit.client.1.smithi152.stdout:0/819: mkdir d9/d2c/d2e/d7f/d9d/de8/df4/d10c 0 2024-03-04T16:04:49.813 INFO:tasks.workunit.client.1.smithi152.stdout:7/712: rename d14/d2b to d14/db9/dc9 0 2024-03-04T16:04:49.814 INFO:tasks.workunit.client.1.smithi152.stdout:9/725: truncate d2/d13/d35/f7f 538206 0 2024-03-04T16:04:49.818 INFO:tasks.workunit.client.1.smithi152.stdout:1/832: dread - d19/f2c zero size 2024-03-04T16:04:49.820 INFO:tasks.workunit.client.1.smithi152.stdout:9/726: creat d2/d13/d23/fed x:0 0 0 2024-03-04T16:04:49.824 INFO:tasks.workunit.client.1.smithi152.stdout:1/833: truncate d19/d3b/d6e/d97/dd7/f3e 4666384 0 2024-03-04T16:04:49.824 INFO:tasks.workunit.client.1.smithi152.stdout:1/834: stat d19/d20/d2f 0 2024-03-04T16:04:49.826 INFO:tasks.workunit.client.1.smithi152.stdout:1/835: fsync d19/d3b/d6e/d97/dd7/da6/fe1 0 2024-03-04T16:04:49.826 INFO:tasks.workunit.client.1.smithi152.stdout:7/713: creat d14/d21/d47/fca x:0 0 0 2024-03-04T16:04:49.828 INFO:tasks.workunit.client.1.smithi152.stdout:9/727: symlink d2/d13/lee 0 2024-03-04T16:04:49.828 INFO:tasks.workunit.client.1.smithi152.stdout:1/836: chown d19/d3b/d6e/d97/dd7/d63/d9a/fa2 15 1 2024-03-04T16:04:49.832 INFO:tasks.workunit.client.1.smithi152.stdout:7/714: fsync d14/d21/d31/f64 0 2024-03-04T16:04:49.833 INFO:tasks.workunit.client.1.smithi152.stdout:1/837: dread d19/d4b/d53/d57/f5e [0,4194304] 0 2024-03-04T16:04:49.833 INFO:tasks.workunit.client.1.smithi152.stdout:9/728: symlink d2/d13/d38/d79/lef 0 2024-03-04T16:04:49.835 INFO:tasks.workunit.client.1.smithi152.stdout:1/838: chown d19/d3b/d6e/d97/dd7/d63/d9a/fa2 956561 1 2024-03-04T16:04:49.836 INFO:tasks.workunit.client.1.smithi152.stdout:7/715: dread d14/f9b [0,4194304] 0 2024-03-04T16:04:49.838 INFO:tasks.workunit.client.1.smithi152.stdout:7/716: truncate d14/d69/d8e/fa5 959249 0 2024-03-04T16:04:49.845 INFO:tasks.workunit.client.1.smithi152.stdout:7/717: dwrite d14/db9/dc9/f63 [4194304,4194304] 0 2024-03-04T16:04:49.845 INFO:tasks.workunit.client.1.smithi152.stdout:4/651: dwrite f5 [0,4194304] 0 2024-03-04T16:04:49.848 INFO:tasks.workunit.client.1.smithi152.stdout:9/729: rename d2/d13/d2e/cc7 to d2/d13/d23/d42/cf0 0 2024-03-04T16:04:49.848 INFO:tasks.workunit.client.1.smithi152.stdout:7/718: dread - d14/d21/d47/d32/fb0 zero size 2024-03-04T16:04:49.851 INFO:tasks.workunit.client.1.smithi152.stdout:9/730: chown d2/d13/f52 16 1 2024-03-04T16:04:49.853 INFO:tasks.workunit.client.1.smithi152.stdout:9/731: dread - d2/d13/d23/d66/feb zero size 2024-03-04T16:04:49.858 INFO:tasks.workunit.client.1.smithi152.stdout:7/719: dwrite d14/d21/d47/d36/da7/fc5 [0,4194304] 0 2024-03-04T16:04:49.858 INFO:tasks.workunit.client.1.smithi152.stdout:9/732: chown d2/d63/d69/l87 511839441 1 2024-03-04T16:04:49.861 INFO:tasks.workunit.client.1.smithi152.stdout:1/839: link d19/d20/d2f/d5d/f67 d19/d20/d2f/d65/def/df1/f10b 0 2024-03-04T16:04:49.865 INFO:tasks.workunit.client.1.smithi152.stdout:4/652: symlink d8/d2f/ld1 0 2024-03-04T16:04:49.865 INFO:tasks.workunit.client.1.smithi152.stdout:4/653: readlink d8/d9/d24/d26/d42/l80 0 2024-03-04T16:04:49.866 INFO:tasks.workunit.client.1.smithi152.stdout:9/733: creat d2/d13/d23/d71/ff1 x:0 0 0 2024-03-04T16:04:49.868 INFO:tasks.workunit.client.1.smithi152.stdout:4/654: chown d8/d4e/dba/dcc 64109087 1 2024-03-04T16:04:49.868 INFO:tasks.workunit.client.1.smithi152.stdout:7/720: dwrite d14/d69/d8e/f9f [0,4194304] 0 2024-03-04T16:04:49.871 INFO:tasks.workunit.client.1.smithi152.stdout:7/721: chown d14/d21/d47/l54 1037838 1 2024-03-04T16:04:49.872 INFO:tasks.workunit.client.1.smithi152.stdout:9/734: mknod d2/d13/d2e/cf2 0 2024-03-04T16:04:49.874 INFO:tasks.workunit.client.1.smithi152.stdout:1/840: creat d19/d3b/d6e/f10c x:0 0 0 2024-03-04T16:04:49.874 INFO:tasks.workunit.client.1.smithi152.stdout:4/655: dread - d8/d2f/d6f/d84/fa4 zero size 2024-03-04T16:04:49.878 INFO:tasks.workunit.client.1.smithi152.stdout:1/841: creat d19/d3b/d6e/d97/dd7/f10d x:0 0 0 2024-03-04T16:04:49.885 INFO:tasks.workunit.client.1.smithi152.stdout:4/656: creat d8/d9/d24/fd2 x:0 0 0 2024-03-04T16:04:49.891 INFO:tasks.workunit.client.1.smithi152.stdout:4/657: fdatasync d8/d9/d7a/f7d 0 2024-03-04T16:04:49.894 INFO:tasks.workunit.client.1.smithi152.stdout:4/658: mkdir d8/d9/da2/dd3 0 2024-03-04T16:04:49.897 INFO:tasks.workunit.client.1.smithi152.stdout:4/659: mknod d8/d2f/d6f/cd4 0 2024-03-04T16:04:49.899 INFO:tasks.workunit.client.1.smithi152.stdout:7/722: dwrite d14/d21/d31/f64 [0,4194304] 0 2024-03-04T16:04:49.901 INFO:tasks.workunit.client.1.smithi152.stdout:9/735: truncate d2/d13/d23/d66/d7d/da7/fb8 2012830 0 2024-03-04T16:04:49.901 INFO:tasks.workunit.client.1.smithi152.stdout:4/660: creat d8/d4e/fd5 x:0 0 0 2024-03-04T16:04:49.909 INFO:tasks.workunit.client.1.smithi152.stdout:9/736: mknod d2/d13/d23/d71/d80/dcc/cf3 0 2024-03-04T16:04:49.911 INFO:tasks.workunit.client.1.smithi152.stdout:4/661: rmdir d8/d2f/d7c/d87/dd0 0 2024-03-04T16:04:49.911 INFO:tasks.workunit.client.1.smithi152.stdout:9/737: read d2/d13/d23/f5c [285686,23575] 0 2024-03-04T16:04:49.915 INFO:tasks.workunit.client.1.smithi152.stdout:7/723: dread f4 [0,4194304] 0 2024-03-04T16:04:49.916 INFO:tasks.workunit.client.1.smithi152.stdout:7/724: write f7 [535742,17763] 0 2024-03-04T16:04:49.955 INFO:tasks.workunit.client.1.smithi152.stdout:9/738: mkdir d2/d63/df4 0 2024-03-04T16:04:49.957 INFO:tasks.workunit.client.1.smithi152.stdout:7/725: mknod d14/d21/d47/d36/ccb 0 2024-03-04T16:04:49.958 INFO:tasks.workunit.client.1.smithi152.stdout:4/662: link d8/d4e/fd5 d8/d2f/d6f/d90/fd6 0 2024-03-04T16:04:50.045 INFO:tasks.workunit.client.1.smithi152.stdout:7/726: truncate d14/f9b 527309 0 2024-03-04T16:04:50.045 INFO:tasks.workunit.client.1.smithi152.stdout:4/663: mknod d8/d78/cd7 0 2024-03-04T16:04:50.046 INFO:tasks.workunit.client.1.smithi152.stdout:4/664: readlink d8/d9/d24/d26/d42/l80 0 2024-03-04T16:04:50.047 INFO:tasks.workunit.client.1.smithi152.stdout:7/727: truncate d14/d69/f88 388182 0 2024-03-04T16:04:50.050 INFO:tasks.workunit.client.1.smithi152.stdout:4/665: read d8/d3d/f5f [1469459,65577] 0 2024-03-04T16:04:50.050 INFO:tasks.workunit.client.1.smithi152.stdout:4/666: write d8/d89/fc5 [307766,28323] 0 2024-03-04T16:04:50.051 INFO:tasks.workunit.client.1.smithi152.stdout:9/739: rename d2/l24 to d2/d63/lf5 0 2024-03-04T16:04:50.058 INFO:tasks.workunit.client.1.smithi152.stdout:9/740: dwrite d2/d13/d23/fed [0,4194304] 0 2024-03-04T16:04:50.058 INFO:tasks.workunit.client.1.smithi152.stdout:4/667: mknod d8/cd8 0 2024-03-04T16:04:50.060 INFO:tasks.workunit.client.1.smithi152.stdout:4/668: write d8/d3d/fc4 [239220,6777] 0 2024-03-04T16:04:50.060 INFO:tasks.workunit.client.1.smithi152.stdout:9/741: chown d2/d13/d23/d66/c6b 190122 1 2024-03-04T16:04:50.065 INFO:tasks.workunit.client.1.smithi152.stdout:9/742: unlink d2/d13/f1a 0 2024-03-04T16:04:50.068 INFO:tasks.workunit.client.1.smithi152.stdout:4/669: unlink d8/d4e/l5a 0 2024-03-04T16:04:50.069 INFO:tasks.workunit.client.1.smithi152.stdout:4/670: write d8/d9/d24/da0/fc9 [982694,94298] 0 2024-03-04T16:04:50.074 INFO:tasks.workunit.client.1.smithi152.stdout:9/743: mkdir d2/d13/df6 0 2024-03-04T16:04:50.077 INFO:tasks.workunit.client.1.smithi152.stdout:8/872: sync 2024-03-04T16:04:50.084 INFO:tasks.workunit.client.1.smithi152.stdout:8/873: dwrite d2/d3/d21/d2d/f79 [4194304,4194304] 0 2024-03-04T16:04:50.098 INFO:tasks.workunit.client.1.smithi152.stdout:7/728: dwrite d14/d21/d31/f7c [0,4194304] 0 2024-03-04T16:04:50.105 INFO:tasks.workunit.client.1.smithi152.stdout:7/729: creat d14/d21/d47/d36/fcc x:0 0 0 2024-03-04T16:04:50.114 INFO:tasks.workunit.client.1.smithi152.stdout:7/730: rename d14/d71/faf to d14/d21/d47/fcd 0 2024-03-04T16:04:50.115 INFO:tasks.workunit.client.1.smithi152.stdout:7/731: write d14/d71/f98 [278311,93138] 0 2024-03-04T16:04:50.116 INFO:tasks.workunit.client.1.smithi152.stdout:7/732: chown d14/db9/dc9/c85 3789586 1 2024-03-04T16:04:50.117 INFO:tasks.workunit.client.1.smithi152.stdout:7/733: stat d14/db9/dc9/d83 0 2024-03-04T16:04:50.117 INFO:tasks.workunit.client.1.smithi152.stdout:7/734: readlink d14/d21/d31/l96 0 2024-03-04T16:04:50.121 INFO:tasks.workunit.client.1.smithi152.stdout:7/735: creat d14/d69/fce x:0 0 0 2024-03-04T16:04:50.123 INFO:tasks.workunit.client.1.smithi152.stdout:7/736: write d14/d21/d47/f76 [1415406,105819] 0 2024-03-04T16:04:50.123 INFO:tasks.workunit.client.1.smithi152.stdout:4/671: truncate d8/d2f/f4a 1362318 0 2024-03-04T16:04:50.124 INFO:tasks.workunit.client.1.smithi152.stdout:7/737: write d14/d21/d47/d36/f8b [4864719,39105] 0 2024-03-04T16:04:50.127 INFO:tasks.workunit.client.1.smithi152.stdout:4/672: creat d8/d2f/d6f/d90/fd9 x:0 0 0 2024-03-04T16:04:50.130 INFO:tasks.workunit.client.1.smithi152.stdout:7/738: rmdir d14/d69/d8e 39 2024-03-04T16:04:50.131 INFO:tasks.workunit.client.1.smithi152.stdout:7/739: stat d14/d21/d47/d40/l87 0 2024-03-04T16:04:50.133 INFO:tasks.workunit.client.1.smithi152.stdout:4/673: rename d8/d4e/lc7 to d8/d2f/d6f/d90/lda 0 2024-03-04T16:04:50.137 INFO:tasks.workunit.client.1.smithi152.stdout:9/744: truncate d2/d13/d38/f58 1251179 0 2024-03-04T16:04:50.137 INFO:tasks.workunit.client.1.smithi152.stdout:7/740: mkdir d14/d21/d47/d36/dcf 0 2024-03-04T16:04:50.145 INFO:tasks.workunit.client.1.smithi152.stdout:4/674: dwrite d8/d2f/d7c/f7f [4194304,4194304] 0 2024-03-04T16:04:50.154 INFO:tasks.workunit.client.1.smithi152.stdout:9/745: truncate d2/d13/d35/f7f 28244 0 2024-03-04T16:04:50.165 INFO:tasks.workunit.client.1.smithi152.stdout:4/675: rmdir d8/d4e/d86 39 2024-03-04T16:04:50.165 INFO:tasks.workunit.client.1.smithi152.stdout:9/746: readlink d2/l2c 0 2024-03-04T16:04:50.170 INFO:tasks.workunit.client.1.smithi152.stdout:8/874: dwrite d2/d3/d21/d2d/f50 [4194304,4194304] 0 2024-03-04T16:04:50.196 INFO:tasks.workunit.client.1.smithi152.stdout:2/779: sync 2024-03-04T16:04:50.227 INFO:tasks.workunit.client.1.smithi152.stdout:7/741: write d14/f2f [1658127,128140] 0 2024-03-04T16:04:50.228 INFO:tasks.workunit.client.1.smithi152.stdout:7/742: chown d14/d71/cc1 30 1 2024-03-04T16:04:50.229 INFO:tasks.workunit.client.1.smithi152.stdout:7/743: stat d14/d21 0 2024-03-04T16:04:50.246 INFO:tasks.workunit.client.1.smithi152.stdout:2/780: mknod da/d2c/c110 0 2024-03-04T16:04:50.251 INFO:tasks.workunit.client.1.smithi152.stdout:9/747: truncate d2/d13/d2e/f3c 638377 0 2024-03-04T16:04:50.254 INFO:tasks.workunit.client.1.smithi152.stdout:8/875: fdatasync d2/d3/d15/f4f 0 2024-03-04T16:04:50.256 INFO:tasks.workunit.client.1.smithi152.stdout:2/781: readlink da/d24/d47/d4f/l6c 0 2024-03-04T16:04:50.256 INFO:tasks.workunit.client.1.smithi152.stdout:6/893: sync 2024-03-04T16:04:50.257 INFO:tasks.workunit.client.1.smithi152.stdout:5/783: sync 2024-03-04T16:04:50.257 INFO:tasks.workunit.client.1.smithi152.stdout:3/780: sync 2024-03-04T16:04:50.258 INFO:tasks.workunit.client.1.smithi152.stdout:6/894: chown d8/d23/d3a/d8d/f107 92254999 1 2024-03-04T16:04:50.259 INFO:tasks.workunit.client.1.smithi152.stdout:7/744: fsync d14/d21/d47/f94 0 2024-03-04T16:04:50.262 INFO:tasks.workunit.client.1.smithi152.stdout:9/748: truncate d2/d13/d35/f7c 866458 0 2024-03-04T16:04:50.265 INFO:tasks.workunit.client.1.smithi152.stdout:8/876: creat d2/d3/d21/d2d/d52/dcc/dd6/f11c x:0 0 0 2024-03-04T16:04:50.268 INFO:tasks.workunit.client.1.smithi152.stdout:2/782: read da/fb [321708,115629] 0 2024-03-04T16:04:50.269 INFO:tasks.workunit.client.1.smithi152.stdout:5/784: symlink d6/d8/d1f/d3d/d6f/d70/d98/de7/lff 0 2024-03-04T16:04:50.270 INFO:tasks.workunit.client.1.smithi152.stdout:2/783: write da/df/d58/da2/db5/f102 [578618,84039] 0 2024-03-04T16:04:50.274 INFO:tasks.workunit.client.1.smithi152.stdout:8/877: dwrite d2/d3/d15/fc1 [0,4194304] 0 2024-03-04T16:04:50.274 INFO:tasks.workunit.client.1.smithi152.stdout:4/676: getdents d8/d4e/d8b 0 2024-03-04T16:04:50.275 INFO:tasks.workunit.client.1.smithi152.stdout:8/878: stat d2/d3/d15/d1c/f1d 0 2024-03-04T16:04:50.277 INFO:tasks.workunit.client.1.smithi152.stdout:7/745: symlink d14/db9/dc9/d83/ld0 0 2024-03-04T16:04:50.281 INFO:tasks.workunit.client.1.smithi152.stdout:5/785: creat d6/d8/d5a/d64/d76/d9f/dc7/dcb/f100 x:0 0 0 2024-03-04T16:04:50.283 INFO:tasks.workunit.client.1.smithi152.stdout:2/784: creat da/df/d2e/d61/f111 x:0 0 0 2024-03-04T16:04:50.284 INFO:tasks.workunit.client.1.smithi152.stdout:4/677: readlink d8/l17 0 2024-03-04T16:04:50.285 INFO:tasks.workunit.client.1.smithi152.stdout:8/879: creat d2/d3/d15/d1c/f11d x:0 0 0 2024-03-04T16:04:50.286 INFO:tasks.workunit.client.1.smithi152.stdout:8/880: stat l0 0 2024-03-04T16:04:50.291 INFO:tasks.workunit.client.1.smithi152.stdout:4/678: dwrite d8/d3d/fc4 [0,4194304] 0 2024-03-04T16:04:50.292 INFO:tasks.workunit.client.1.smithi152.stdout:4/679: truncate d8/d9/f99 1323966 0 2024-03-04T16:04:50.301 INFO:tasks.workunit.client.1.smithi152.stdout:8/881: symlink d2/d3/d21/d2d/d31/dff/l11e 0 2024-03-04T16:04:50.302 INFO:tasks.workunit.client.1.smithi152.stdout:8/882: stat d2/d3/d9e/fda 0 2024-03-04T16:04:50.302 INFO:tasks.workunit.client.1.smithi152.stdout:8/883: chown d2/d3/d15/d1c/f1d 53017 1 2024-03-04T16:04:50.303 INFO:tasks.workunit.client.1.smithi152.stdout:7/746: mkdir d14/d69/dd1 0 2024-03-04T16:04:50.304 INFO:tasks.workunit.client.1.smithi152.stdout:7/747: fsync d14/d21/d47/d36/da7/fad 0 2024-03-04T16:04:50.305 INFO:tasks.workunit.client.1.smithi152.stdout:4/680: creat d8/d2f/d7c/fdb x:0 0 0 2024-03-04T16:04:50.306 INFO:tasks.workunit.client.1.smithi152.stdout:9/749: link d2/d13/d23/d66/d7d/c81 d2/d13/dad/ddf/cf7 0 2024-03-04T16:04:50.306 INFO:tasks.workunit.client.1.smithi152.stdout:4/681: chown d8/d4e/dba 16798 1 2024-03-04T16:04:50.308 INFO:tasks.workunit.client.1.smithi152.stdout:9/750: write d2/d13/d23/d71/ff1 [1033043,122328] 0 2024-03-04T16:04:50.308 INFO:tasks.workunit.client.1.smithi152.stdout:2/785: getdents da/df/d2e/dc6 0 2024-03-04T16:04:50.309 INFO:tasks.workunit.client.1.smithi152.stdout:9/751: readlink d2/d13/d23/d66/l70 0 2024-03-04T16:04:50.309 INFO:tasks.workunit.client.1.smithi152.stdout:8/884: chown d2/d3/f4b 24 1 2024-03-04T16:04:50.310 INFO:tasks.workunit.client.1.smithi152.stdout:9/752: write d2/d13/d96/fe7 [813791,89690] 0 2024-03-04T16:04:50.313 INFO:tasks.workunit.client.1.smithi152.stdout:7/748: creat d14/db9/fd2 x:0 0 0 2024-03-04T16:04:50.315 INFO:tasks.workunit.client.1.smithi152.stdout:4/682: creat d8/d2f/d67/fdc x:0 0 0 2024-03-04T16:04:50.321 INFO:tasks.workunit.client.1.smithi152.stdout:4/683: creat d8/d2f/d6f/d84/fdd x:0 0 0 2024-03-04T16:04:50.324 INFO:tasks.workunit.client.1.smithi152.stdout:8/885: symlink d2/d3/l11f 0 2024-03-04T16:04:50.325 INFO:tasks.workunit.client.1.smithi152.stdout:5/786: rmdir d6/d8/d1f/d3d/d6f/d70 39 2024-03-04T16:04:50.327 INFO:tasks.workunit.client.1.smithi152.stdout:6/895: write d8/d10/d53/dd7/fec [3577851,47144] 0 2024-03-04T16:04:50.329 INFO:tasks.workunit.client.1.smithi152.stdout:2/786: rename da/d2c/ddd/ff1 to da/df/d2e/d51/f112 0 2024-03-04T16:04:50.331 INFO:tasks.workunit.client.1.smithi152.stdout:5/787: dwrite d6/d50/f59 [0,4194304] 0 2024-03-04T16:04:50.332 INFO:tasks.workunit.client.1.smithi152.stdout:5/788: fdatasync d6/f68 0 2024-03-04T16:04:50.333 INFO:tasks.workunit.client.1.smithi152.stdout:5/789: chown d6/d8/d1f/d3d/f4f 182 1 2024-03-04T16:04:50.334 INFO:tasks.workunit.client.1.smithi152.stdout:8/886: unlink d2/d3/d21/d2d/d52/dcc/ce9 0 2024-03-04T16:04:50.340 INFO:tasks.workunit.client.1.smithi152.stdout:6/896: stat d8/d23/c2b 0 2024-03-04T16:04:50.342 INFO:tasks.workunit.client.1.smithi152.stdout:6/897: write d8/d23/d3a/d71/d9d/fa8 [4217091,14513] 0 2024-03-04T16:04:50.343 INFO:tasks.workunit.client.1.smithi152.stdout:6/898: write d8/f2f [836322,92306] 0 2024-03-04T16:04:50.346 INFO:tasks.workunit.client.1.smithi152.stdout:4/684: creat d8/d9/fde x:0 0 0 2024-03-04T16:04:50.347 INFO:tasks.workunit.client.1.smithi152.stdout:7/749: rename d14/d21/c39 to d14/d9d/cd3 0 2024-03-04T16:04:50.348 INFO:tasks.workunit.client.1.smithi152.stdout:7/750: chown d14/d21/d47/d36/dcf 985 1 2024-03-04T16:04:50.351 INFO:tasks.workunit.client.1.smithi152.stdout:5/790: fdatasync d6/d8/d1f/d3d/d6f/d70/d98/fb7 0 2024-03-04T16:04:50.352 INFO:tasks.workunit.client.1.smithi152.stdout:8/887: getdents d2/d3/d21/d2d/d52/dfa 0 2024-03-04T16:04:50.357 INFO:tasks.workunit.client.1.smithi152.stdout:8/888: symlink d2/d3/l120 0 2024-03-04T16:04:50.362 INFO:tasks.workunit.client.1.smithi152.stdout:7/751: getdents d14/d21/d47/d32 0 2024-03-04T16:04:50.365 INFO:tasks.workunit.client.1.smithi152.stdout:8/889: creat d2/d3/d21/dc0/f121 x:0 0 0 2024-03-04T16:04:50.369 INFO:tasks.workunit.client.1.smithi152.stdout:7/752: creat d14/d21/d47/d36/da7/dbc/fd4 x:0 0 0 2024-03-04T16:04:50.374 INFO:tasks.workunit.client.1.smithi152.stdout:7/753: mkdir d14/d21/d47/dd5 0 2024-03-04T16:04:50.377 INFO:tasks.workunit.client.1.smithi152.stdout:9/753: write d2/d63/f95 [925712,121524] 0 2024-03-04T16:04:50.378 INFO:tasks.workunit.client.1.smithi152.stdout:9/754: chown d2/d13/d38 839 1 2024-03-04T16:04:50.381 INFO:tasks.workunit.client.1.smithi152.stdout:9/755: creat d2/d13/dad/ddf/ff8 x:0 0 0 2024-03-04T16:04:50.382 INFO:tasks.workunit.client.1.smithi152.stdout:9/756: chown d2/d63 8 1 2024-03-04T16:04:50.387 INFO:tasks.workunit.client.1.smithi152.stdout:9/757: getdents d2/d13/d23/d66/d7d 0 2024-03-04T16:04:50.389 INFO:tasks.workunit.client.1.smithi152.stdout:2/787: dwrite da/d24/f32 [0,4194304] 0 2024-03-04T16:04:50.391 INFO:tasks.workunit.client.1.smithi152.stdout:9/758: mkdir d2/d13/d38/df9 0 2024-03-04T16:04:50.393 INFO:tasks.workunit.client.1.smithi152.stdout:2/788: mknod da/df/d2e/c113 0 2024-03-04T16:04:50.394 INFO:tasks.workunit.client.1.smithi152.stdout:2/789: stat da/d24/f65 0 2024-03-04T16:04:50.395 INFO:tasks.workunit.client.1.smithi152.stdout:4/685: dwrite d8/d9/f50 [0,4194304] 0 2024-03-04T16:04:50.396 INFO:tasks.workunit.client.1.smithi152.stdout:2/790: dread - da/df/f97 zero size 2024-03-04T16:04:50.399 INFO:tasks.workunit.client.1.smithi152.stdout:9/759: creat d2/d63/ffa x:0 0 0 2024-03-04T16:04:50.400 INFO:tasks.workunit.client.1.smithi152.stdout:9/760: write d2/d13/d23/d42/fe8 [524748,92484] 0 2024-03-04T16:04:50.405 INFO:tasks.workunit.client.1.smithi152.stdout:2/791: mkdir da/d5f/db0/d114 0 2024-03-04T16:04:50.405 INFO:tasks.workunit.client.1.smithi152.stdout:8/890: dwrite d2/d3/d15/d1c/d84/d87/d5d/f44 [0,4194304] 0 2024-03-04T16:04:50.410 INFO:tasks.workunit.client.1.smithi152.stdout:4/686: creat d8/d9/dc1/fdf x:0 0 0 2024-03-04T16:04:50.415 INFO:tasks.workunit.client.1.smithi152.stdout:9/761: mknod d2/d6e/cfb 0 2024-03-04T16:04:50.417 INFO:tasks.workunit.client.1.smithi152.stdout:8/891: mkdir d2/d3/d21/d2d/d52/d122 0 2024-03-04T16:04:50.419 INFO:tasks.workunit.client.1.smithi152.stdout:8/892: truncate d2/d7f/d114/f118 1007646 0 2024-03-04T16:04:50.419 INFO:tasks.workunit.client.1.smithi152.stdout:4/687: creat d8/d58/fe0 x:0 0 0 2024-03-04T16:04:50.423 INFO:tasks.workunit.client.1.smithi152.stdout:9/762: fdatasync d2/d13/d96/fa4 0 2024-03-04T16:04:50.434 INFO:tasks.workunit.client.1.smithi152.stdout:2/792: write da/d24/d80/fd3 [405229,90698] 0 2024-03-04T16:04:50.435 INFO:tasks.workunit.client.1.smithi152.stdout:2/793: chown da/d2c/l62 6765 1 2024-03-04T16:04:50.435 INFO:tasks.workunit.client.1.smithi152.stdout:4/688: mknod d8/d9/d24/ce1 0 2024-03-04T16:04:50.438 INFO:tasks.workunit.client.1.smithi152.stdout:1/842: sync 2024-03-04T16:04:50.438 INFO:tasks.workunit.client.1.smithi152.stdout:0/820: sync 2024-03-04T16:04:50.440 INFO:tasks.workunit.client.1.smithi152.stdout:1/843: fsync d19/d4b/d53/d57/f102 0 2024-03-04T16:04:50.441 INFO:tasks.workunit.client.1.smithi152.stdout:2/794: unlink da/d24/d47/d4f/l55 0 2024-03-04T16:04:50.444 INFO:tasks.workunit.client.1.smithi152.stdout:4/689: write d8/d2f/d6f/d90/fca [439675,21121] 0 2024-03-04T16:04:50.445 INFO:tasks.workunit.client.1.smithi152.stdout:4/690: readlink d8/d9/d24/d26/d42/l80 0 2024-03-04T16:04:50.446 INFO:tasks.workunit.client.1.smithi152.stdout:9/763: getdents d2/d13/d23/d71 0 2024-03-04T16:04:50.448 INFO:tasks.workunit.client.1.smithi152.stdout:0/821: creat d9/d2c/d2e/d94/dac/f10d x:0 0 0 2024-03-04T16:04:50.449 INFO:tasks.workunit.client.1.smithi152.stdout:8/893: dread d2/d75/d89/f2e [0,4194304] 0 2024-03-04T16:04:50.450 INFO:tasks.workunit.client.1.smithi152.stdout:1/844: unlink d19/d75/dc3/lf9 0 2024-03-04T16:04:50.452 INFO:tasks.workunit.client.1.smithi152.stdout:2/795: symlink da/d24/d47/d4f/l115 0 2024-03-04T16:04:50.457 INFO:tasks.workunit.client.1.smithi152.stdout:0/822: readlink d9/d92/le3 0 2024-03-04T16:04:50.458 INFO:tasks.workunit.client.1.smithi152.stdout:1/845: dwrite d19/d3b/d6e/d97/dd7/d63/f10a [0,4194304] 0 2024-03-04T16:04:50.459 INFO:tasks.workunit.client.1.smithi152.stdout:0/823: readlink d9/d1b/d54/d69/d6c/dbf/ld2 0 2024-03-04T16:04:50.460 INFO:tasks.workunit.client.1.smithi152.stdout:8/894: unlink d2/d3/d21/dac/f106 0 2024-03-04T16:04:50.464 INFO:tasks.workunit.client.1.smithi152.stdout:4/691: dread d8/d9/d7a/fa6 [0,4194304] 0 2024-03-04T16:04:50.465 INFO:tasks.workunit.client.1.smithi152.stdout:9/764: write d2/d13/f93 [1183151,76087] 0 2024-03-04T16:04:50.467 INFO:tasks.workunit.client.1.smithi152.stdout:2/796: mkdir da/df/d2e/d51/d8d/d9a/d116 0 2024-03-04T16:04:50.468 INFO:tasks.workunit.client.1.smithi152.stdout:2/797: fsync da/d5f/dbc/ffb 0 2024-03-04T16:04:50.471 INFO:tasks.workunit.client.1.smithi152.stdout:1/846: dread - d19/d4b/d53/fc1 zero size 2024-03-04T16:04:50.471 INFO:tasks.workunit.client.1.smithi152.stdout:1/847: chown d19/d75/da8 14924 1 2024-03-04T16:04:50.489 INFO:tasks.workunit.client.1.smithi152.stdout:4/692: mknod d8/d2f/ce2 0 2024-03-04T16:04:50.492 INFO:tasks.workunit.client.1.smithi152.stdout:0/824: truncate d9/d2c/d2e/d94/fca 2321372 0 2024-03-04T16:04:50.495 INFO:tasks.workunit.client.1.smithi152.stdout:2/798: write da/d24/d47/d4f/fc9 [4757008,52822] 0 2024-03-04T16:04:50.495 INFO:tasks.workunit.client.1.smithi152.stdout:8/895: dwrite d2/d75/d89/dd/d22/d6c/f70 [0,4194304] 0 2024-03-04T16:04:50.499 INFO:tasks.workunit.client.1.smithi152.stdout:0/825: write d9/d1b/d54/d80/da8/fd9 [461360,123794] 0 2024-03-04T16:04:50.500 INFO:tasks.workunit.client.1.smithi152.stdout:8/896: readlink d2/d3/d21/d2d/d48/l4c 0 2024-03-04T16:04:50.501 INFO:tasks.workunit.client.1.smithi152.stdout:9/765: dread d2/d13/d38/d79/f89 [0,4194304] 0 2024-03-04T16:04:50.501 INFO:tasks.workunit.client.1.smithi152.stdout:1/848: dwrite d19/d3b/f49 [0,4194304] 0 2024-03-04T16:04:50.504 INFO:tasks.workunit.client.1.smithi152.stdout:4/693: mkdir d8/de3 0 2024-03-04T16:04:50.511 INFO:tasks.workunit.client.1.smithi152.stdout:2/799: dwrite da/d2c/d3f/f9d [0,4194304] 0 2024-03-04T16:04:50.512 INFO:tasks.workunit.client.1.smithi152.stdout:2/800: readlink da/d24/d47/d73/lbd 0 2024-03-04T16:04:50.513 INFO:tasks.workunit.client.1.smithi152.stdout:2/801: dread - da/d2c/d5e/fe3 zero size 2024-03-04T16:04:50.516 INFO:tasks.workunit.client.1.smithi152.stdout:9/766: truncate d2/d13/f52 3242826 0 2024-03-04T16:04:50.516 INFO:tasks.workunit.client.1.smithi152.stdout:8/897: chown d2/d3/d21/d2d/d48/f7e 27508 1 2024-03-04T16:04:50.517 INFO:tasks.workunit.client.1.smithi152.stdout:0/826: creat d9/d2c/d2e/d7f/d9d/f10e x:0 0 0 2024-03-04T16:04:50.520 INFO:tasks.workunit.client.1.smithi152.stdout:4/694: mknod d8/ce4 0 2024-03-04T16:04:50.525 INFO:tasks.workunit.client.1.smithi152.stdout:0/827: write d9/d1b/d54/d80/fdf [570170,45637] 0 2024-03-04T16:04:50.529 INFO:tasks.workunit.client.1.smithi152.stdout:2/802: link da/l7c da/df/l117 0 2024-03-04T16:04:50.532 INFO:tasks.workunit.client.1.smithi152.stdout:0/828: mkdir d9/d1b/d54/d69/d98/d10f 0 2024-03-04T16:04:50.537 INFO:tasks.workunit.client.1.smithi152.stdout:0/829: dread d9/d1a/f79 [0,4194304] 0 2024-03-04T16:04:50.539 INFO:tasks.workunit.client.1.smithi152.stdout:1/849: write d19/d20/d2f/d5d/f67 [107746,55740] 0 2024-03-04T16:04:50.542 INFO:tasks.workunit.client.1.smithi152.stdout:8/898: dread d2/d3/d9e/fda [0,4194304] 0 2024-03-04T16:04:50.543 INFO:tasks.workunit.client.1.smithi152.stdout:9/767: write d2/d13/d23/f33 [6387907,27462] 0 2024-03-04T16:04:50.546 INFO:tasks.workunit.client.1.smithi152.stdout:1/850: chown d19/d4b/d53/d57/cc7 14 1 2024-03-04T16:04:50.547 INFO:tasks.workunit.client.1.smithi152.stdout:1/851: truncate d19/d20/d2f/fcc 908807 0 2024-03-04T16:04:50.552 INFO:tasks.workunit.client.1.smithi152.stdout:0/830: creat d9/d2c/d2e/f110 x:0 0 0 2024-03-04T16:04:50.561 INFO:tasks.workunit.client.1.smithi152.stdout:0/831: mknod d9/d2c/d2e/d7f/d9d/de8/df4/c111 0 2024-03-04T16:04:50.562 INFO:tasks.workunit.client.1.smithi152.stdout:1/852: write d19/d4b/d53/d57/d85/fa3 [640453,50095] 0 2024-03-04T16:04:50.564 INFO:tasks.workunit.client.1.smithi152.stdout:2/803: symlink da/df/d2e/d51/l118 0 2024-03-04T16:04:50.565 INFO:tasks.workunit.client.1.smithi152.stdout:9/768: truncate d2/f61 3406142 0 2024-03-04T16:04:50.565 INFO:tasks.workunit.client.1.smithi152.stdout:8/899: dread d2/d75/d89/f45 [0,4194304] 0 2024-03-04T16:04:50.566 INFO:tasks.workunit.client.1.smithi152.stdout:0/832: creat d9/d1a/d7c/f112 x:0 0 0 2024-03-04T16:04:50.568 INFO:tasks.workunit.client.1.smithi152.stdout:1/853: fdatasync d19/d20/fb2 0 2024-03-04T16:04:50.568 INFO:tasks.workunit.client.1.smithi152.stdout:2/804: mknod da/d2c/ddd/c119 0 2024-03-04T16:04:50.569 INFO:tasks.workunit.client.1.smithi152.stdout:1/854: chown d19/d20/fb2 0 1 2024-03-04T16:04:50.569 INFO:tasks.workunit.client.1.smithi152.stdout:2/805: chown da/d5f/d85/lfd 464201 1 2024-03-04T16:04:50.571 INFO:tasks.workunit.client.1.smithi152.stdout:9/769: truncate d2/d13/d35/f8c 481349 0 2024-03-04T16:04:50.572 INFO:tasks.workunit.client.1.smithi152.stdout:0/833: creat d9/d1b/d54/d80/da8/f113 x:0 0 0 2024-03-04T16:04:50.575 INFO:tasks.workunit.client.1.smithi152.stdout:1/855: mkdir d19/d3b/d6e/d97/dd7/d63/d9a/d10e 0 2024-03-04T16:04:50.577 INFO:tasks.workunit.client.1.smithi152.stdout:1/856: chown d19/d4b/d53/d108 92433029 1 2024-03-04T16:04:50.579 INFO:tasks.workunit.client.1.smithi152.stdout:8/900: dwrite d2/d3/d21/f10a [0,4194304] 0 2024-03-04T16:04:50.582 INFO:tasks.workunit.client.1.smithi152.stdout:2/806: rename da/df/d2e/d61 to da/d24/d47/d4f/d11a 0 2024-03-04T16:04:50.584 INFO:tasks.workunit.client.1.smithi152.stdout:9/770: fdatasync d2/f28 0 2024-03-04T16:04:50.590 INFO:tasks.workunit.client.1.smithi152.stdout:8/901: fsync d2/d75/f9f 0 2024-03-04T16:04:50.593 INFO:tasks.workunit.client.1.smithi152.stdout:0/834: symlink d9/d92/d107/l114 0 2024-03-04T16:04:50.600 INFO:tasks.workunit.client.1.smithi152.stdout:8/902: mknod d2/d75/d89/dd/d22/d6c/d86/c123 0 2024-03-04T16:04:50.606 INFO:tasks.workunit.client.1.smithi152.stdout:8/903: creat d2/d3/d21/d2d/f124 x:0 0 0 2024-03-04T16:04:50.606 INFO:tasks.workunit.client.1.smithi152.stdout:1/857: write d19/d3b/d6e/d97/dd7/f3a [3821603,56624] 0 2024-03-04T16:04:50.613 INFO:tasks.workunit.client.1.smithi152.stdout:1/858: creat d19/d4b/d53/d57/d62/dd8/f10f x:0 0 0 2024-03-04T16:04:50.617 INFO:tasks.workunit.client.1.smithi152.stdout:0/835: truncate d9/d2c/d2e/d7f/d9d/fb0 189407 0 2024-03-04T16:04:50.620 INFO:tasks.workunit.client.1.smithi152.stdout:9/771: dwrite d2/d13/d23/d66/d7d/da7/fb8 [0,4194304] 0 2024-03-04T16:04:50.623 INFO:tasks.workunit.client.1.smithi152.stdout:1/859: dwrite d19/d4b/f5f [0,4194304] 0 2024-03-04T16:04:50.630 INFO:tasks.workunit.client.1.smithi152.stdout:8/904: dwrite d2/f80 [0,4194304] 0 2024-03-04T16:04:50.631 INFO:tasks.workunit.client.1.smithi152.stdout:9/772: creat d2/d13/dad/ddf/ffc x:0 0 0 2024-03-04T16:04:50.634 INFO:tasks.workunit.client.1.smithi152.stdout:1/860: unlink d19/d4b/d53/d57/d62/dd8/ff2 0 2024-03-04T16:04:50.635 INFO:tasks.workunit.client.1.smithi152.stdout:1/861: write d19/d3b/f49 [1776265,67299] 0 2024-03-04T16:04:50.636 INFO:tasks.workunit.client.1.smithi152.stdout:8/905: symlink d2/d3/d15/d1c/d84/d87/d5d/l125 0 2024-03-04T16:04:50.700 INFO:tasks.workunit.client.1.smithi152.stdout:8/906: truncate d2/d75/d89/dd/f93 2580759 0 2024-03-04T16:04:50.700 INFO:tasks.workunit.client.1.smithi152.stdout:8/907: chown d2/d75/d89/dd/l88 0 1 2024-03-04T16:04:50.706 INFO:tasks.workunit.client.1.smithi152.stdout:8/908: fdatasync d2/d3/d21/f1b 0 2024-03-04T16:04:50.708 INFO:tasks.workunit.client.1.smithi152.stdout:1/862: getdents d19/d75 0 2024-03-04T16:04:50.708 INFO:tasks.workunit.client.1.smithi152.stdout:9/773: dwrite d2/d13/d35/f7c [0,4194304] 0 2024-03-04T16:04:50.709 INFO:tasks.workunit.client.1.smithi152.stdout:1/863: chown d19/d75/dc3/fdd 86728208 1 2024-03-04T16:04:50.718 INFO:tasks.workunit.client.1.smithi152.stdout:1/864: mkdir d19/d20/d2f/d65/def/d110 0 2024-03-04T16:04:50.722 INFO:tasks.workunit.client.1.smithi152.stdout:9/774: write d2/d13/d23/d71/f86 [1035665,21399] 0 2024-03-04T16:04:50.724 INFO:tasks.workunit.client.1.smithi152.stdout:1/865: mkdir d19/d4b/d53/d57/d62/d82/d111 0 2024-03-04T16:04:50.725 INFO:tasks.workunit.client.1.smithi152.stdout:8/909: creat d2/d75/d89/dd/f126 x:0 0 0 2024-03-04T16:04:50.726 INFO:tasks.workunit.client.1.smithi152.stdout:9/775: chown d2/d13/d35/lca 1073073241 1 2024-03-04T16:04:50.728 INFO:tasks.workunit.client.1.smithi152.stdout:9/776: fdatasync d2/d13/d35/f34 0 2024-03-04T16:04:50.730 INFO:tasks.workunit.client.1.smithi152.stdout:8/910: truncate d2/d75/d89/dd/d22/d6c/d86/f107 177527 0 2024-03-04T16:04:50.731 INFO:tasks.workunit.client.1.smithi152.stdout:1/866: dwrite d19/d3b/d6e/f10c [0,4194304] 0 2024-03-04T16:04:50.731 INFO:tasks.workunit.client.1.smithi152.stdout:8/911: stat d2/d7f/cb3 0 2024-03-04T16:04:50.734 INFO:tasks.workunit.client.1.smithi152.stdout:9/777: truncate d2/d63/fc0 376062 0 2024-03-04T16:04:50.741 INFO:tasks.workunit.client.1.smithi152.stdout:1/867: dwrite d19/d4b/d53/d57/d62/ff4 [0,4194304] 0 2024-03-04T16:04:50.744 INFO:tasks.workunit.client.1.smithi152.stdout:9/778: creat d2/d13/df6/ffd x:0 0 0 2024-03-04T16:04:50.749 INFO:tasks.workunit.client.1.smithi152.stdout:8/912: rename d2/d75/d89/dd/d22/d83/d96 to d2/d3/d21/d2d/d52/deb/d127 0 2024-03-04T16:04:50.757 INFO:tasks.workunit.client.1.smithi152.stdout:8/913: mknod d2/d3/d21/d2d/c128 0 2024-03-04T16:04:50.760 INFO:tasks.workunit.client.1.smithi152.stdout:1/868: mknod d19/d20/d2f/d6b/c112 0 2024-03-04T16:04:50.760 INFO:tasks.workunit.client.1.smithi152.stdout:1/869: stat d19/d75/fa7 0 2024-03-04T16:04:50.764 INFO:tasks.workunit.client.1.smithi152.stdout:8/914: creat d2/d75/d89/dd/d22/d83/dd9/f129 x:0 0 0 2024-03-04T16:04:50.770 INFO:tasks.workunit.client.1.smithi152.stdout:9/779: write d2/d13/d35/f8c [645357,117704] 0 2024-03-04T16:04:50.771 INFO:tasks.workunit.client.1.smithi152.stdout:1/870: truncate d19/d20/fb2 2843090 0 2024-03-04T16:04:50.773 INFO:tasks.workunit.client.1.smithi152.stdout:1/871: read d19/d20/d2f/d5d/f95 [96291,1509] 0 2024-03-04T16:04:50.779 INFO:tasks.workunit.client.1.smithi152.stdout:9/780: creat d2/ffe x:0 0 0 2024-03-04T16:04:50.784 INFO:tasks.workunit.client.1.smithi152.stdout:8/915: dread d2/d3/d21/f1b [0,4194304] 0 2024-03-04T16:04:50.786 INFO:tasks.workunit.client.1.smithi152.stdout:9/781: truncate d2/d13/d23/fc4 111860 0 2024-03-04T16:04:50.786 INFO:tasks.workunit.client.1.smithi152.stdout:9/782: readlink d2/d13/d23/d71/da3/lc8 0 2024-03-04T16:04:50.789 INFO:tasks.workunit.client.1.smithi152.stdout:8/916: write d2/d75/d89/fe2 [1395126,102094] 0 2024-03-04T16:04:50.791 INFO:tasks.workunit.client.1.smithi152.stdout:8/917: chown d2/d3/d21/f10a 13 1 2024-03-04T16:04:50.793 INFO:tasks.workunit.client.1.smithi152.stdout:1/872: truncate d19/d20/d2f/d6b/f40 518200 0 2024-03-04T16:04:50.798 INFO:tasks.workunit.client.1.smithi152.stdout:9/783: dread d2/d13/d23/f5b [0,4194304] 0 2024-03-04T16:04:50.804 INFO:tasks.workunit.client.1.smithi152.stdout:1/873: dwrite d19/d3b/d6e/d97/dd7/d63/d9a/fa2 [0,4194304] 0 2024-03-04T16:04:50.809 INFO:tasks.workunit.client.1.smithi152.stdout:8/918: dread d2/d3/d21/d2d/f1e [0,4194304] 0 2024-03-04T16:04:50.811 INFO:tasks.workunit.client.1.smithi152.stdout:1/874: unlink d19/d20/ce2 0 2024-03-04T16:04:50.813 INFO:tasks.workunit.client.1.smithi152.stdout:1/875: dread d19/d20/f9b [0,4194304] 0 2024-03-04T16:04:50.813 INFO:tasks.workunit.client.1.smithi152.stdout:8/919: dread d2/d75/d89/f2e [0,4194304] 0 2024-03-04T16:04:50.817 INFO:tasks.workunit.client.1.smithi152.stdout:9/784: write d2/d13/d23/f5c [1252395,118264] 0 2024-03-04T16:04:50.820 INFO:tasks.workunit.client.1.smithi152.stdout:8/920: rename d2/d3/d21/d2d/d52/d68/d6e/d7d/df9 to d2/d3/d21/d2d/d52/dfa/d12a 0 2024-03-04T16:04:50.823 INFO:tasks.workunit.client.1.smithi152.stdout:1/876: write d19/d20/d2f/d5d/fb9 [230988,126365] 0 2024-03-04T16:04:50.827 INFO:tasks.workunit.client.1.smithi152.stdout:8/921: write d2/d3/d21/d2d/d48/db8/f104 [148334,95051] 0 2024-03-04T16:04:50.831 INFO:tasks.workunit.client.1.smithi152.stdout:1/877: dwrite d19/d3b/d6e/d97/dd7/d63/f10a [0,4194304] 0 2024-03-04T16:04:50.831 INFO:tasks.workunit.client.1.smithi152.stdout:3/781: sync 2024-03-04T16:04:50.832 INFO:tasks.workunit.client.1.smithi152.stdout:9/785: link d2/d13/d38/d79/lef d2/d13/d38/df9/lff 0 2024-03-04T16:04:50.836 INFO:tasks.workunit.client.1.smithi152.stdout:8/922: fsync d2/d3/d21/d2d/d52/deb/d127/da6/fcd 0 2024-03-04T16:04:50.837 INFO:tasks.workunit.client.1.smithi152.stdout:8/923: fsync d2/d3/d21/d2d/f79 0 2024-03-04T16:04:50.850 INFO:tasks.workunit.client.1.smithi152.stdout:8/924: mknod d2/d3/d21/d2d/d52/d122/c12b 0 2024-03-04T16:04:50.851 INFO:tasks.workunit.client.1.smithi152.stdout:8/925: chown d2/d3/f40 343025 1 2024-03-04T16:04:50.855 INFO:tasks.workunit.client.1.smithi152.stdout:9/786: truncate d2/d13/d23/d66/d7d/da7/fb8 373913 0 2024-03-04T16:04:50.857 INFO:tasks.workunit.client.1.smithi152.stdout:1/878: getdents d19/d75/dd9 0 2024-03-04T16:04:50.858 INFO:tasks.workunit.client.1.smithi152.stdout:3/782: truncate d6/f1a 1651299 0 2024-03-04T16:04:50.861 INFO:tasks.workunit.client.1.smithi152.stdout:1/879: dread d19/d3b/d6e/f10c [0,4194304] 0 2024-03-04T16:04:50.867 INFO:tasks.workunit.client.1.smithi152.stdout:9/787: mknod d2/d13/d23/d66/d7d/da7/c100 0 2024-03-04T16:04:50.867 INFO:tasks.workunit.client.1.smithi152.stdout:9/788: stat d2/d13/d38/d79/fe6 0 2024-03-04T16:04:50.875 INFO:tasks.workunit.client.1.smithi152.stdout:3/783: rename d6/d14/d41/d49/d84/deb to d6/d14/d41/d49/d84/de2/d108 0 2024-03-04T16:04:50.876 INFO:tasks.workunit.client.1.smithi152.stdout:8/926: dwrite d2/d3/d21/f5f [0,4194304] 0 2024-03-04T16:04:50.877 INFO:tasks.workunit.client.1.smithi152.stdout:8/927: readlink d2/d75/ld5 0 2024-03-04T16:04:50.888 INFO:tasks.workunit.client.1.smithi152.stdout:8/928: fdatasync d2/d75/d89/dd/d22/d83/ff1 0 2024-03-04T16:04:50.890 INFO:tasks.workunit.client.1.smithi152.stdout:1/880: creat d19/d4b/d53/d57/d62/d82/d111/f113 x:0 0 0 2024-03-04T16:04:50.898 INFO:tasks.workunit.client.1.smithi152.stdout:8/929: mkdir d2/d3/d21/dac/d12c 0 2024-03-04T16:04:50.901 INFO:tasks.workunit.client.1.smithi152.stdout:1/881: dread - d19/d20/d2f/d6b/ff5 zero size 2024-03-04T16:04:50.901 INFO:tasks.workunit.client.1.smithi152.stdout:3/784: write d6/d14/f15 [390466,3661] 0 2024-03-04T16:04:50.903 INFO:tasks.workunit.client.1.smithi152.stdout:9/789: link d2/d13/d96/fe7 d2/f101 0 2024-03-04T16:04:50.904 INFO:tasks.workunit.client.1.smithi152.stdout:8/930: mknod d2/d3/d21/d2d/d52/deb/d127/da6/c12d 0 2024-03-04T16:04:50.908 INFO:tasks.workunit.client.1.smithi152.stdout:9/790: readlink d2/d13/d35/l55 0 2024-03-04T16:04:50.910 INFO:tasks.workunit.client.1.smithi152.stdout:8/931: symlink d2/d3/d21/d2d/d52/d122/l12e 0 2024-03-04T16:04:50.912 INFO:tasks.workunit.client.1.smithi152.stdout:9/791: mkdir d2/d13/d23/d71/d102 0 2024-03-04T16:04:50.915 INFO:tasks.workunit.client.1.smithi152.stdout:8/932: rename d2/d75/la7 to d2/d3/d15/d1c/d84/d87/l12f 0 2024-03-04T16:04:50.918 INFO:tasks.workunit.client.1.smithi152.stdout:8/933: creat d2/d75/f130 x:0 0 0 2024-03-04T16:04:50.922 INFO:tasks.workunit.client.1.smithi152.stdout:9/792: creat d2/d13/d38/df9/f103 x:0 0 0 2024-03-04T16:04:50.923 INFO:tasks.workunit.client.1.smithi152.stdout:1/882: dread d19/d3b/d6e/d97/dd7/f3e [0,4194304] 0 2024-03-04T16:04:50.923 INFO:tasks.workunit.client.1.smithi152.stdout:8/934: mknod d2/d75/d89/dd/dfc/c131 0 2024-03-04T16:04:50.927 INFO:tasks.workunit.client.1.smithi152.stdout:3/785: dwrite d6/d10/d2d/d4d/fc3 [0,4194304] 0 2024-03-04T16:04:50.929 INFO:tasks.workunit.client.1.smithi152.stdout:9/793: symlink d2/d13/d23/d42/db1/l104 0 2024-03-04T16:04:50.932 INFO:tasks.workunit.client.1.smithi152.stdout:8/935: symlink d2/d3/d15/d1c/d84/d87/d5d/l132 0 2024-03-04T16:04:50.933 INFO:tasks.workunit.client.1.smithi152.stdout:8/936: stat d2/d3/d15/d1c/d84/d87/d5d/l132 0 2024-03-04T16:04:50.935 INFO:tasks.workunit.client.1.smithi152.stdout:3/786: creat d6/d60/f109 x:0 0 0 2024-03-04T16:04:50.939 INFO:tasks.workunit.client.1.smithi152.stdout:9/794: mknod d2/d13/d35/c105 0 2024-03-04T16:04:50.944 INFO:tasks.workunit.client.1.smithi152.stdout:8/937: unlink d2/d75/d89/dd/d22/cec 0 2024-03-04T16:04:50.947 INFO:tasks.workunit.client.1.smithi152.stdout:1/883: link d19/d20/d2f/lb6 d19/d3b/d6e/d97/dd7/da6/l114 0 2024-03-04T16:04:50.948 INFO:tasks.workunit.client.1.smithi152.stdout:9/795: chown d2/d63/d69/l8a 27353 1 2024-03-04T16:04:50.952 INFO:tasks.workunit.client.1.smithi152.stdout:8/938: dwrite d2/d3/d21/dac/fdb [0,4194304] 0 2024-03-04T16:04:50.953 INFO:tasks.workunit.client.1.smithi152.stdout:9/796: symlink d2/d13/d2e/dcd/l106 0 2024-03-04T16:04:50.964 INFO:tasks.workunit.client.1.smithi152.stdout:8/939: dwrite d2/d75/d89/dd/d22/d6c/d86/f107 [0,4194304] 0 2024-03-04T16:04:50.965 INFO:tasks.workunit.client.1.smithi152.stdout:3/787: dread d6/f96 [0,4194304] 0 2024-03-04T16:04:50.967 INFO:tasks.workunit.client.1.smithi152.stdout:9/797: link d2/d63/fc0 d2/d13/d23/d66/d7d/f107 0 2024-03-04T16:04:50.972 INFO:tasks.workunit.client.1.smithi152.stdout:8/940: mknod d2/d3/d21/d2d/d52/dfa/c133 0 2024-03-04T16:04:50.976 INFO:tasks.workunit.client.1.smithi152.stdout:3/788: symlink d6/d60/d8e/dc0/da1/l10a 0 2024-03-04T16:04:50.977 INFO:tasks.workunit.client.1.smithi152.stdout:8/941: mknod d2/c134 0 2024-03-04T16:04:50.979 INFO:tasks.workunit.client.1.smithi152.stdout:8/942: read - d2/d3/d21/d2d/d52/dcc/dd6/f11c zero size 2024-03-04T16:04:50.986 INFO:tasks.workunit.client.1.smithi152.stdout:8/943: dwrite d2/d3/d15/d1c/f11d [0,4194304] 0 2024-03-04T16:04:50.993 INFO:tasks.workunit.client.1.smithi152.stdout:3/789: dwrite d6/d10/f21 [0,4194304] 0 2024-03-04T16:04:50.998 INFO:tasks.workunit.client.1.smithi152.stdout:8/944: dwrite d2/d3/d21/d2d/f124 [0,4194304] 0 2024-03-04T16:04:51.001 INFO:tasks.workunit.client.1.smithi152.stdout:3/790: link d6/d60/d8a/fca d6/d14/d58/dad/f10b 0 2024-03-04T16:04:51.008 INFO:tasks.workunit.client.1.smithi152.stdout:3/791: creat d6/d60/d8a/f10c x:0 0 0 2024-03-04T16:04:51.012 INFO:tasks.workunit.client.1.smithi152.stdout:3/792: creat d6/d14/d41/d49/d84/de2/d108/f10d x:0 0 0 2024-03-04T16:04:51.022 INFO:tasks.workunit.client.1.smithi152.stdout:6/899: sync 2024-03-04T16:04:51.024 INFO:tasks.workunit.client.1.smithi152.stdout:6/900: symlink d8/d92/db1/d87/dca/l138 0 2024-03-04T16:04:51.029 INFO:tasks.workunit.client.1.smithi152.stdout:6/901: rename d8/d23/d93/fbc to d8/d23/d3a/d71/f139 0 2024-03-04T16:04:51.037 INFO:tasks.workunit.client.1.smithi152.stdout:8/945: dread d2/d3/d15/d1c/f1d [0,4194304] 0 2024-03-04T16:04:51.040 INFO:tasks.workunit.client.1.smithi152.stdout:8/946: write d2/d75/d89/dd/d22/f3b [8605945,14680] 0 2024-03-04T16:04:51.043 INFO:tasks.workunit.client.1.smithi152.stdout:3/793: dwrite d6/f3f [0,4194304] 0 2024-03-04T16:04:51.053 INFO:tasks.workunit.client.1.smithi152.stdout:6/902: dread d8/d8f/db2/fc7 [0,4194304] 0 2024-03-04T16:04:51.062 INFO:tasks.workunit.client.1.smithi152.stdout:3/794: dwrite d6/d10/d2d/f4c [4194304,4194304] 0 2024-03-04T16:04:51.068 INFO:tasks.workunit.client.1.smithi152.stdout:3/795: rename d6/d10/c78 to d6/d14/d41/d49/d84/de2/d108/c10e 0 2024-03-04T16:04:51.071 INFO:tasks.workunit.client.1.smithi152.stdout:3/796: rename d6/d60/d8a/f10c to d6/d14/d41/d49/d84/f10f 0 2024-03-04T16:04:51.073 INFO:tasks.workunit.client.1.smithi152.stdout:8/947: truncate d2/d75/d89/dd/d22/d26/f111 1799799 0 2024-03-04T16:04:51.075 INFO:tasks.workunit.client.1.smithi152.stdout:8/948: write d2/d75/d89/dd/d22/d6c/d86/f107 [3935537,14130] 0 2024-03-04T16:04:51.080 INFO:tasks.workunit.client.1.smithi152.stdout:3/797: creat d6/d14/f110 x:0 0 0 2024-03-04T16:04:51.081 INFO:tasks.workunit.client.1.smithi152.stdout:3/798: readlink d6/d10/d2d/l33 0 2024-03-04T16:04:51.087 INFO:tasks.workunit.client.1.smithi152.stdout:6/903: dread d8/d23/f25 [0,4194304] 0 2024-03-04T16:04:51.089 INFO:tasks.workunit.client.1.smithi152.stdout:8/949: dread d2/d3/d15/d1c/d84/d87/fd1 [0,4194304] 0 2024-03-04T16:04:51.089 INFO:tasks.workunit.client.1.smithi152.stdout:6/904: read d8/d23/d3a/f8b [116576,53257] 0 2024-03-04T16:04:51.091 INFO:tasks.workunit.client.1.smithi152.stdout:8/950: read d2/d75/d89/dd/f93 [2357371,128449] 0 2024-03-04T16:04:51.092 INFO:tasks.workunit.client.1.smithi152.stdout:6/905: creat d8/d92/d59/d61/d57/f13a x:0 0 0 2024-03-04T16:04:51.094 INFO:tasks.workunit.client.1.smithi152.stdout:8/951: creat d2/d3/d21/dc0/f135 x:0 0 0 2024-03-04T16:04:51.101 INFO:tasks.workunit.client.1.smithi152.stdout:8/952: dwrite d2/d3/d15/d1c/d84/d87/f108 [0,4194304] 0 2024-03-04T16:04:51.104 INFO:tasks.workunit.client.1.smithi152.stdout:8/953: creat d2/d3/d21/d2d/d52/deb/f136 x:0 0 0 2024-03-04T16:04:51.113 INFO:tasks.workunit.client.1.smithi152.stdout:3/799: write d6/f3a [2247134,70348] 0 2024-03-04T16:04:51.114 INFO:tasks.workunit.client.1.smithi152.stdout:3/800: stat d6/d14/d58/c107 0 2024-03-04T16:04:51.115 INFO:tasks.workunit.client.1.smithi152.stdout:3/801: write d6/d14/f15 [5194923,63285] 0 2024-03-04T16:04:51.118 INFO:tasks.workunit.client.1.smithi152.stdout:3/802: truncate d6/d10/d2d/f4b 14960 0 2024-03-04T16:04:51.119 INFO:tasks.workunit.client.1.smithi152.stdout:3/803: chown d6/d14/d58/dad/fd1 2338076 1 2024-03-04T16:04:51.129 INFO:tasks.workunit.client.1.smithi152.stdout:3/804: mkdir d6/d14/d58/de4/d111 0 2024-03-04T16:04:51.137 INFO:tasks.workunit.client.1.smithi152.stdout:3/805: fsync d6/f42 0 2024-03-04T16:04:51.139 INFO:tasks.workunit.client.1.smithi152.stdout:8/954: dwrite d2/d75/d89/f45 [0,4194304] 0 2024-03-04T16:04:51.142 INFO:tasks.workunit.client.1.smithi152.stdout:8/955: chown d2/d75/d89/dd/d22/d26/fcb 1141 1 2024-03-04T16:04:51.145 INFO:tasks.workunit.client.1.smithi152.stdout:8/956: truncate d2/d3/d21/d2d/f1e 4471975 0 2024-03-04T16:04:51.146 INFO:tasks.workunit.client.1.smithi152.stdout:3/806: read d6/f9 [1052582,19998] 0 2024-03-04T16:04:51.147 INFO:tasks.workunit.client.1.smithi152.stdout:3/807: write d6/d14/f15 [2572393,78128] 0 2024-03-04T16:04:51.148 INFO:tasks.workunit.client.1.smithi152.stdout:8/957: read - d2/d3/d21/d2d/d52/d68/ffe zero size 2024-03-04T16:04:51.173 INFO:tasks.workunit.client.1.smithi152.stdout:8/958: write d2/d3/d21/d2d/d48/f7e [1132157,64133] 0 2024-03-04T16:04:51.239 INFO:tasks.workunit.client.1.smithi152.stdout:8/959: creat d2/d3/d15/d1c/d84/dde/de4/f137 x:0 0 0 2024-03-04T16:04:51.240 INFO:tasks.workunit.client.1.smithi152.stdout:8/960: chown d2/d3/d21/d2d/d52/deb/d127/da6/fcd 7593589 1 2024-03-04T16:04:51.246 INFO:tasks.workunit.client.1.smithi152.stdout:8/961: mkdir d2/d75/d89/dd/d22/d83/dd9/d138 0 2024-03-04T16:04:51.247 INFO:tasks.workunit.client.1.smithi152.stdout:8/962: write d2/d3/d21/d2d/d48/f7e [623505,56635] 0 2024-03-04T16:04:51.259 INFO:tasks.workunit.client.1.smithi152.stdout:5/791: sync 2024-03-04T16:04:51.259 INFO:tasks.workunit.client.1.smithi152.stdout:5/792: chown d6/d8/d1f/d88/fcf 123290908 1 2024-03-04T16:04:51.261 INFO:tasks.workunit.client.1.smithi152.stdout:5/793: dread - d6/d8/d1f/d3d/d6f/d70/d98/de7/ff5 zero size 2024-03-04T16:04:51.261 INFO:tasks.workunit.client.1.smithi152.stdout:5/794: readlink d6/l47 0 2024-03-04T16:04:51.264 INFO:tasks.workunit.client.1.smithi152.stdout:5/795: rmdir d6/d8/d1f/d3d/da4/dc9 39 2024-03-04T16:04:51.266 INFO:tasks.workunit.client.1.smithi152.stdout:5/796: symlink d6/d8/d1f/d3d/d6f/d70/ddb/dde/l101 0 2024-03-04T16:04:51.271 INFO:tasks.workunit.client.1.smithi152.stdout:5/797: rmdir d6/d8/d5a/d64/d76 39 2024-03-04T16:04:51.280 INFO:tasks.workunit.client.1.smithi152.stdout:8/963: truncate d2/d3/d15/d1c/d84/fa8 4098795 0 2024-03-04T16:04:51.304 INFO:tasks.workunit.client.1.smithi152.stdout:5/798: dread d6/d8/d1f/d88/fcf [0,4194304] 0 2024-03-04T16:04:51.305 INFO:tasks.workunit.client.1.smithi152.stdout:8/964: unlink d2/d3/d15/f4f 0 2024-03-04T16:04:51.307 INFO:tasks.workunit.client.1.smithi152.stdout:5/799: chown d6/d8/d5a/d64/d76/d9f/l45 525641 1 2024-03-04T16:04:51.308 INFO:tasks.workunit.client.1.smithi152.stdout:8/965: rmdir d2/d75/d89/dd/d22/d6c 39 2024-03-04T16:04:51.308 INFO:tasks.workunit.client.1.smithi152.stdout:5/800: chown d6/d8/d1f/d3d/da4 111701598 1 2024-03-04T16:04:51.309 INFO:tasks.workunit.client.1.smithi152.stdout:8/966: stat d2/d3/d15/d1c/d84/d87/d5d/l125 0 2024-03-04T16:04:51.312 INFO:tasks.workunit.client.1.smithi152.stdout:8/967: symlink d2/d75/l139 0 2024-03-04T16:04:51.323 INFO:tasks.workunit.client.1.smithi152.stdout:8/968: rmdir d2/d75/d89/dd/d22/d26/d110 0 2024-03-04T16:04:51.329 INFO:tasks.workunit.client.1.smithi152.stdout:5/801: dwrite d6/d8/d5a/d64/d76/d9f/f53 [0,4194304] 0 2024-03-04T16:04:51.335 INFO:tasks.workunit.client.1.smithi152.stdout:5/802: unlink d6/d8/d1f/f3a 0 2024-03-04T16:04:51.342 INFO:tasks.workunit.client.1.smithi152.stdout:5/803: mkdir d6/d8/d1f/d3d/d6f/d102 0 2024-03-04T16:04:51.343 INFO:tasks.workunit.client.1.smithi152.stdout:5/804: chown d6/d8/d1f/d3d/da4/dc9/lb5 52715 1 2024-03-04T16:04:51.344 INFO:tasks.workunit.client.1.smithi152.stdout:5/805: stat d6/d8/d1f/d3d/d6f/d70/d98/fa9 0 2024-03-04T16:04:51.346 INFO:tasks.workunit.client.1.smithi152.stdout:5/806: symlink d6/d8/d5a/d90/dd6/l103 0 2024-03-04T16:04:51.348 INFO:tasks.workunit.client.1.smithi152.stdout:5/807: chown d6/d8/d5a/d64/d76/d9f/f54 7118520 1 2024-03-04T16:04:51.354 INFO:tasks.workunit.client.1.smithi152.stdout:5/808: write d6/d50/fc8 [4906674,19361] 0 2024-03-04T16:04:51.354 INFO:tasks.workunit.client.1.smithi152.stdout:5/809: chown d6/d8/d5a 6654 1 2024-03-04T16:04:51.357 INFO:tasks.workunit.client.1.smithi152.stdout:5/810: creat d6/d8/d5a/d82/f104 x:0 0 0 2024-03-04T16:04:51.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:51 smithi152 ceph-mon[36876]: pgmap v7: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 42 MiB/s rd, 101 MiB/s wr, 254 op/s 2024-03-04T16:04:51.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:51 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:51.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:51 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:51.362 INFO:tasks.workunit.client.1.smithi152.stdout:5/811: dwrite d6/d50/f59 [0,4194304] 0 2024-03-04T16:04:51.367 INFO:tasks.workunit.client.1.smithi152.stdout:5/812: creat d6/d8/d5a/f105 x:0 0 0 2024-03-04T16:04:51.383 INFO:tasks.workunit.client.1.smithi152.stdout:5/813: dwrite d6/d8/d1f/d3d/d6f/d70/d98/fb7 [0,4194304] 0 2024-03-04T16:04:51.388 INFO:tasks.workunit.client.1.smithi152.stdout:5/814: chown d6/d8/d5a/c80 29781 1 2024-03-04T16:04:51.390 INFO:tasks.workunit.client.1.smithi152.stdout:5/815: rmdir d6/d8/d1f/d3d/da4/dc9 39 2024-03-04T16:04:51.409 INFO:tasks.workunit.client.1.smithi152.stdout:5/816: dread d6/d8/d1f/d3d/d6f/dbb/de3/fb9 [0,4194304] 0 2024-03-04T16:04:51.416 INFO:tasks.workunit.client.1.smithi152.stdout:5/817: rmdir d6/d8/dbe 0 2024-03-04T16:04:51.422 INFO:tasks.workunit.client.1.smithi152.stdout:5/818: dwrite d6/d8/d5a/d90/daf/fc6 [0,4194304] 0 2024-03-04T16:04:51.426 INFO:tasks.workunit.client.1.smithi152.stdout:5/819: mkdir d6/d8/d5a/d90/d106 0 2024-03-04T16:04:51.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:51 smithi116 ceph-mon[29120]: pgmap v7: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 42 MiB/s rd, 101 MiB/s wr, 254 op/s 2024-03-04T16:04:51.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:51 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:51.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:51 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:51.666 INFO:tasks.workunit.client.1.smithi152.stdout:7/754: sync 2024-03-04T16:04:51.669 INFO:tasks.workunit.client.1.smithi152.stdout:7/755: creat d14/d69/dd1/fd6 x:0 0 0 2024-03-04T16:04:51.671 INFO:tasks.workunit.client.1.smithi152.stdout:7/756: chown d14/f15 484877497 1 2024-03-04T16:04:51.678 INFO:tasks.workunit.client.1.smithi152.stdout:7/757: dwrite d14/d69/d8e/f9f [0,4194304] 0 2024-03-04T16:04:51.680 INFO:tasks.workunit.client.1.smithi152.stdout:7/758: stat d14/c2d 0 2024-03-04T16:04:51.684 INFO:tasks.workunit.client.1.smithi152.stdout:7/759: creat d14/d71/fd7 x:0 0 0 2024-03-04T16:04:51.684 INFO:tasks.workunit.client.1.smithi152.stdout:7/760: chown d14/d21/d47/d40 215460716 1 2024-03-04T16:04:51.686 INFO:tasks.workunit.client.1.smithi152.stdout:7/761: mkdir d14/d21/d47/d36/da7/dd8 0 2024-03-04T16:04:51.688 INFO:tasks.workunit.client.1.smithi152.stdout:7/762: fdatasync d14/d21/d47/f34 0 2024-03-04T16:04:51.693 INFO:tasks.workunit.client.1.smithi152.stdout:7/763: symlink d14/d21/d47/d36/da7/dbc/ld9 0 2024-03-04T16:04:51.696 INFO:tasks.workunit.client.1.smithi152.stdout:7/764: creat d14/d21/d47/d36/dcf/fda x:0 0 0 2024-03-04T16:04:51.710 INFO:tasks.workunit.client.1.smithi152.stdout:7/765: dwrite d14/db9/dc9/f4e [0,4194304] 0 2024-03-04T16:04:51.711 INFO:tasks.workunit.client.1.smithi152.stdout:7/766: fsync d14/d21/d47/d32/fb0 0 2024-03-04T16:04:51.716 INFO:tasks.workunit.client.1.smithi152.stdout:0/836: sync 2024-03-04T16:04:51.717 INFO:tasks.workunit.client.1.smithi152.stdout:2/807: sync 2024-03-04T16:04:51.717 INFO:tasks.workunit.client.1.smithi152.stdout:8/969: sync 2024-03-04T16:04:51.717 INFO:tasks.workunit.client.1.smithi152.stdout:4/695: sync 2024-03-04T16:04:51.717 INFO:tasks.workunit.client.1.smithi152.stdout:1/884: sync 2024-03-04T16:04:51.717 INFO:tasks.workunit.client.1.smithi152.stdout:6/906: sync 2024-03-04T16:04:51.717 INFO:tasks.workunit.client.1.smithi152.stdout:3/808: sync 2024-03-04T16:04:51.717 INFO:tasks.workunit.client.1.smithi152.stdout:9/798: sync 2024-03-04T16:04:51.717 INFO:tasks.workunit.client.1.smithi152.stdout:5/820: sync 2024-03-04T16:04:51.718 INFO:tasks.workunit.client.1.smithi152.stdout:7/767: rename d14/d21/d47/d36/da7/fc5 to d14/d69/fdb 0 2024-03-04T16:04:51.721 INFO:tasks.workunit.client.1.smithi152.stdout:3/809: chown d6/d14/d41/d49/d84/de2/d108/f10d 1 1 2024-03-04T16:04:51.723 INFO:tasks.workunit.client.1.smithi152.stdout:2/808: write da/d2c/d5e/f103 [765404,100083] 0 2024-03-04T16:04:51.723 INFO:tasks.workunit.client.1.smithi152.stdout:9/799: creat d2/d6e/f108 x:0 0 0 2024-03-04T16:04:51.725 INFO:tasks.workunit.client.1.smithi152.stdout:6/907: creat d8/d42/dde/f13b x:0 0 0 2024-03-04T16:04:51.727 INFO:tasks.workunit.client.1.smithi152.stdout:6/908: chown d8/d10/d53/d9a/fe8 184 1 2024-03-04T16:04:51.727 INFO:tasks.workunit.client.1.smithi152.stdout:1/885: symlink d19/d4b/d53/d57/d62/dac/l115 0 2024-03-04T16:04:51.728 INFO:tasks.workunit.client.1.smithi152.stdout:8/970: mknod d2/d3/d21/d2d/d52/deb/c13a 0 2024-03-04T16:04:51.728 INFO:tasks.workunit.client.1.smithi152.stdout:6/909: chown d8/d10/d53/d9a/ced 75565063 1 2024-03-04T16:04:51.731 INFO:tasks.workunit.client.1.smithi152.stdout:0/837: read - d9/d2c/d2e/d7f/d9d/fbd zero size 2024-03-04T16:04:51.737 INFO:tasks.workunit.client.1.smithi152.stdout:5/821: symlink d6/d8/l107 0 2024-03-04T16:04:51.739 INFO:tasks.workunit.client.1.smithi152.stdout:3/810: creat d6/d60/d8e/dc0/da1/f112 x:0 0 0 2024-03-04T16:04:51.741 INFO:tasks.workunit.client.1.smithi152.stdout:2/809: mknod da/d24/d80/d82/c11b 0 2024-03-04T16:04:51.743 INFO:tasks.workunit.client.1.smithi152.stdout:2/810: chown da/d5f/dbc/dd5 10906805 1 2024-03-04T16:04:51.743 INFO:tasks.workunit.client.1.smithi152.stdout:9/800: rename d2/l2c to d2/d13/dea/l109 0 2024-03-04T16:04:51.744 INFO:tasks.workunit.client.1.smithi152.stdout:4/696: dwrite d8/d9/f60 [0,4194304] 0 2024-03-04T16:04:51.753 INFO:tasks.workunit.client.1.smithi152.stdout:0/838: rmdir d9/d1b/d54/d69/d98 39 2024-03-04T16:04:51.754 INFO:tasks.workunit.client.1.smithi152.stdout:7/768: symlink d14/d21/d47/dd5/ldc 0 2024-03-04T16:04:51.755 INFO:tasks.workunit.client.1.smithi152.stdout:5/822: mkdir d6/d8/d1f/d3d/d6f/dbb/de3/d108 0 2024-03-04T16:04:51.761 INFO:tasks.workunit.client.1.smithi152.stdout:1/886: rename d19/d4b/d53/d57/d62/dac/lb5 to d19/d75/da8/db8/l116 0 2024-03-04T16:04:51.763 INFO:tasks.workunit.client.1.smithi152.stdout:9/801: unlink d2/d13/f15 0 2024-03-04T16:04:51.766 INFO:tasks.workunit.client.1.smithi152.stdout:6/910: write d8/d42/fcf [25886,6429] 0 2024-03-04T16:04:51.766 INFO:tasks.workunit.client.1.smithi152.stdout:6/911: chown d8/d92/f48 184967 1 2024-03-04T16:04:51.767 INFO:tasks.workunit.client.1.smithi152.stdout:4/697: mknod d8/d78/ce5 0 2024-03-04T16:04:51.772 INFO:tasks.workunit.client.1.smithi152.stdout:0/839: stat d9/d1b/d54/fb7 0 2024-03-04T16:04:51.772 INFO:tasks.workunit.client.1.smithi152.stdout:8/971: dwrite d2/d75/d89/dd/dfc/ffd [0,4194304] 0 2024-03-04T16:04:51.773 INFO:tasks.workunit.client.1.smithi152.stdout:8/972: stat d2/d3/d21/f5f 0 2024-03-04T16:04:51.774 INFO:tasks.workunit.client.1.smithi152.stdout:7/769: creat d14/d9d/fdd x:0 0 0 2024-03-04T16:04:51.775 INFO:tasks.workunit.client.1.smithi152.stdout:7/770: stat d14/d21/d47/dd5/ldc 0 2024-03-04T16:04:51.778 INFO:tasks.workunit.client.1.smithi152.stdout:3/811: rename d6/d60/d8e/dc0/faf to d6/d14/d41/d49/d84/de2/de9/d104/f113 0 2024-03-04T16:04:51.783 INFO:tasks.workunit.client.1.smithi152.stdout:0/840: dread d9/d2c/d2e/d7f/f9f [0,4194304] 0 2024-03-04T16:04:51.783 INFO:tasks.workunit.client.1.smithi152.stdout:1/887: chown d19/d20/d2f/d6b/f2d 211 1 2024-03-04T16:04:51.785 INFO:tasks.workunit.client.1.smithi152.stdout:9/802: symlink d2/d13/d96/l10a 0 2024-03-04T16:04:51.788 INFO:tasks.workunit.client.1.smithi152.stdout:6/912: symlink d8/d23/d93/d99/df6/l13c 0 2024-03-04T16:04:51.794 INFO:tasks.workunit.client.1.smithi152.stdout:3/812: creat d6/d14/d41/f114 x:0 0 0 2024-03-04T16:04:51.802 INFO:tasks.workunit.client.1.smithi152.stdout:4/698: mkdir d8/d9/da2/dd3/de6 0 2024-03-04T16:04:51.803 INFO:tasks.workunit.client.1.smithi152.stdout:1/888: write d19/d3b/d6e/fdf [831288,80929] 0 2024-03-04T16:04:51.804 INFO:tasks.workunit.client.1.smithi152.stdout:6/913: creat d8/d42/dde/f13d x:0 0 0 2024-03-04T16:04:51.804 INFO:tasks.workunit.client.1.smithi152.stdout:9/803: write d2/d13/d23/f40 [894416,94880] 0 2024-03-04T16:04:51.807 INFO:tasks.workunit.client.1.smithi152.stdout:8/973: symlink d2/d3/d15/d1c/d84/l13b 0 2024-03-04T16:04:51.807 INFO:tasks.workunit.client.1.smithi152.stdout:9/804: write d2/d13/d23/f40 [795581,112522] 0 2024-03-04T16:04:51.808 INFO:tasks.workunit.client.1.smithi152.stdout:5/823: creat d6/d8/d5a/d64/f109 x:0 0 0 2024-03-04T16:04:51.812 INFO:tasks.workunit.client.1.smithi152.stdout:4/699: mkdir d8/d9/da2/dd3/de7 0 2024-03-04T16:04:51.817 INFO:tasks.workunit.client.1.smithi152.stdout:8/974: mkdir d2/d75/d89/dd/d22/ddf/d13c 0 2024-03-04T16:04:51.824 INFO:tasks.workunit.client.1.smithi152.stdout:2/811: rename da/d24 to da/df/d2e/d51/d11c 0 2024-03-04T16:04:51.827 INFO:tasks.workunit.client.1.smithi152.stdout:0/841: dread d9/d1b/d54/d80/db2/fb9 [0,4194304] 0 2024-03-04T16:04:51.829 INFO:tasks.workunit.client.1.smithi152.stdout:4/700: rmdir d8/d9/d7a 39 2024-03-04T16:04:51.830 INFO:tasks.workunit.client.1.smithi152.stdout:1/889: dread d19/d20/d2f/d6b/f5a [0,4194304] 0 2024-03-04T16:04:51.830 INFO:tasks.workunit.client.1.smithi152.stdout:0/842: chown d9/d92/lc9 251540 1 2024-03-04T16:04:51.832 INFO:tasks.workunit.client.1.smithi152.stdout:7/771: getdents d14/d21/d47/d36 0 2024-03-04T16:04:51.835 INFO:tasks.workunit.client.1.smithi152.stdout:8/975: unlink d2/d3/d21/d2d/d52/d68/d6e/d7d/f8c 0 2024-03-04T16:04:51.835 INFO:tasks.workunit.client.1.smithi152.stdout:6/914: dread d8/d8f/da3/fac [0,4194304] 0 2024-03-04T16:04:51.839 INFO:tasks.workunit.client.1.smithi152.stdout:1/890: dwrite d19/d4b/f5f [0,4194304] 0 2024-03-04T16:04:51.839 INFO:tasks.workunit.client.1.smithi152.stdout:6/915: write d8/d42/dde/dee/f128 [593367,58080] 0 2024-03-04T16:04:51.839 INFO:tasks.workunit.client.1.smithi152.stdout:9/805: rename d2/d13/d23/d66/d7d/fbf to d2/d13/d23/d71/d102/f10b 0 2024-03-04T16:04:51.841 INFO:tasks.workunit.client.1.smithi152.stdout:3/813: write d6/d5c/fc6 [717358,10996] 0 2024-03-04T16:04:51.841 INFO:tasks.workunit.client.1.smithi152.stdout:2/812: truncate da/d2c/f75 159025 0 2024-03-04T16:04:51.844 INFO:tasks.workunit.client.1.smithi152.stdout:4/701: mknod d8/d9/d79/ce8 0 2024-03-04T16:04:51.844 INFO:tasks.workunit.client.1.smithi152.stdout:9/806: fsync d2/d13/d23/d66/feb 0 2024-03-04T16:04:51.845 INFO:tasks.workunit.client.1.smithi152.stdout:1/891: dread - d19/d3b/d6e/d97/dd7/f10d zero size 2024-03-04T16:04:51.848 INFO:tasks.workunit.client.1.smithi152.stdout:5/824: dwrite d6/d50/f83 [0,4194304] 0 2024-03-04T16:04:51.849 INFO:tasks.workunit.client.1.smithi152.stdout:4/702: readlink d8/d78/l96 0 2024-03-04T16:04:51.853 INFO:tasks.workunit.client.1.smithi152.stdout:6/916: fdatasync d8/d10/f44 0 2024-03-04T16:04:51.853 INFO:tasks.workunit.client.1.smithi152.stdout:9/807: readlink d2/d13/d23/d66/d7d/l7e 0 2024-03-04T16:04:51.857 INFO:tasks.workunit.client.1.smithi152.stdout:9/808: write d2/d13/d38/df9/f103 [27821,9585] 0 2024-03-04T16:04:51.858 INFO:tasks.workunit.client.1.smithi152.stdout:8/976: fsync d2/d3/d21/d2d/d52/d68/ffe 0 2024-03-04T16:04:51.860 INFO:tasks.workunit.client.1.smithi152.stdout:7/772: rename d14/d21/d47/dd5 to d14/d21/d31/dde 0 2024-03-04T16:04:51.861 INFO:tasks.workunit.client.1.smithi152.stdout:4/703: dread - d8/f65 zero size 2024-03-04T16:04:51.985 INFO:tasks.workunit.client.1.smithi152.stdout:4/704: dwrite d8/d3d/f44 [0,4194304] 0 2024-03-04T16:04:51.994 INFO:tasks.workunit.client.1.smithi152.stdout:2/813: mkdir da/df/d2e/d51/d11c/d47/d11d 0 2024-03-04T16:04:51.998 INFO:tasks.workunit.client.1.smithi152.stdout:5/825: symlink d6/d8/d1f/d3d/d6f/dbb/de3/d108/l10a 0 2024-03-04T16:04:51.999 INFO:tasks.workunit.client.1.smithi152.stdout:5/826: chown d6/d8/d1f/d3d/d6f/d70/d98/fa9 22 1 2024-03-04T16:04:52.001 INFO:tasks.workunit.client.1.smithi152.stdout:7/773: unlink d14/d71/fb4 0 2024-03-04T16:04:52.001 INFO:tasks.workunit.client.1.smithi152.stdout:0/843: write d9/d2c/d2e/d94/dac/fc8 [2546108,87462] 0 2024-03-04T16:04:52.006 INFO:tasks.workunit.client.1.smithi152.stdout:3/814: rename d6/d60/d8e/dc0/da1/lcd to d6/d14/d41/d49/d84/de2/l115 0 2024-03-04T16:04:52.010 INFO:tasks.workunit.client.1.smithi152.stdout:4/705: mkdir d8/d9/da2/dd3/de9 0 2024-03-04T16:04:52.015 INFO:tasks.workunit.client.1.smithi152.stdout:9/809: dread d2/d63/f95 [0,4194304] 0 2024-03-04T16:04:52.018 INFO:tasks.workunit.client.1.smithi152.stdout:3/815: dread - d6/d14/d41/d49/d84/de2/de9/d104/f113 zero size 2024-03-04T16:04:52.019 INFO:tasks.workunit.client.1.smithi152.stdout:8/977: dwrite d2/d3/d21/d2d/d48/f103 [0,4194304] 0 2024-03-04T16:04:52.019 INFO:tasks.workunit.client.1.smithi152.stdout:9/810: chown d2/d13/d35/la6 12 1 2024-03-04T16:04:52.028 INFO:tasks.workunit.client.1.smithi152.stdout:5/827: mknod d6/d8/c10b 0 2024-03-04T16:04:52.030 INFO:tasks.workunit.client.1.smithi152.stdout:4/706: symlink d8/d2f/lea 0 2024-03-04T16:04:52.031 INFO:tasks.workunit.client.1.smithi152.stdout:5/828: rmdir d6/d8/d5a/d64/d76/d9f/db0 39 2024-03-04T16:04:52.034 INFO:tasks.workunit.client.1.smithi152.stdout:2/814: truncate da/df/d2e/f38 2623571 0 2024-03-04T16:04:52.036 INFO:tasks.workunit.client.1.smithi152.stdout:6/917: rename d8/d23/d93/d99/cb6 to d8/d23/d3a/d8d/c13e 0 2024-03-04T16:04:52.036 INFO:tasks.workunit.client.1.smithi152.stdout:0/844: dread d9/d2c/f2f [0,4194304] 0 2024-03-04T16:04:52.039 INFO:tasks.workunit.client.1.smithi152.stdout:5/829: creat d6/d8/d1f/d3d/d6f/d70/dc4/f10c x:0 0 0 2024-03-04T16:04:52.040 INFO:tasks.workunit.client.1.smithi152.stdout:2/815: creat da/d2c/f11e x:0 0 0 2024-03-04T16:04:52.045 INFO:tasks.workunit.client.1.smithi152.stdout:7/774: rename d14/d21/d47/d36/da7/dd8 to d14/d21/d47/d36/dcf/ddf 0 2024-03-04T16:04:52.047 INFO:tasks.workunit.client.1.smithi152.stdout:7/775: read d14/d21/d47/fcd [593541,20859] 0 2024-03-04T16:04:52.053 INFO:tasks.workunit.client.1.smithi152.stdout:5/830: fdatasync d6/d8/d5a/d90/dd6/fd9 0 2024-03-04T16:04:52.054 INFO:tasks.workunit.client.1.smithi152.stdout:6/918: dread d8/d10/f44 [0,4194304] 0 2024-03-04T16:04:52.054 INFO:tasks.workunit.client.1.smithi152.stdout:4/707: dwrite f3 [0,4194304] 0 2024-03-04T16:04:52.056 INFO:tasks.workunit.client.1.smithi152.stdout:2/816: chown da/df/fae 2 1 2024-03-04T16:04:52.056 INFO:tasks.workunit.client.1.smithi152.stdout:0/845: dwrite d9/d1b/d54/d80/f96 [0,4194304] 0 2024-03-04T16:04:52.067 INFO:tasks.workunit.client.1.smithi152.stdout:9/811: rename d2/d13/df6 to d2/d13/d35/d10c 0 2024-03-04T16:04:52.072 INFO:tasks.workunit.client.1.smithi152.stdout:7/776: creat d14/d9d/fe0 x:0 0 0 2024-03-04T16:04:52.083 INFO:tasks.workunit.client.1.smithi152.stdout:2/817: dwrite da/df/d58/d6e/f89 [0,4194304] 0 2024-03-04T16:04:52.092 INFO:tasks.workunit.client.1.smithi152.stdout:2/818: rmdir da/d5f/d85/dca 39 2024-03-04T16:04:52.095 INFO:tasks.workunit.client.1.smithi152.stdout:9/812: link d2/d13/d23/d71/da3/fab d2/d6e/f10d 0 2024-03-04T16:04:52.098 INFO:tasks.workunit.client.1.smithi152.stdout:5/831: creat d6/d8/d1f/d3d/d6f/d70/f10d x:0 0 0 2024-03-04T16:04:52.099 INFO:tasks.workunit.client.1.smithi152.stdout:4/708: dwrite d8/d3d/fa1 [0,4194304] 0 2024-03-04T16:04:52.102 INFO:tasks.workunit.client.1.smithi152.stdout:4/709: truncate d8/d4e/d8b/fa9 75988 0 2024-03-04T16:04:52.102 INFO:tasks.workunit.client.1.smithi152.stdout:2/819: mkdir da/df/d58/d6e/d11f 0 2024-03-04T16:04:52.108 INFO:tasks.workunit.client.1.smithi152.stdout:8/978: rename d2/d75/d89/dd/l11 to d2/d3/d21/l13d 0 2024-03-04T16:04:52.108 INFO:tasks.workunit.client.1.smithi152.stdout:8/979: fdatasync d2/d3/d15/fc1 0 2024-03-04T16:04:52.112 INFO:tasks.workunit.client.1.smithi152.stdout:7/777: write d14/d21/d31/f57 [3588099,127471] 0 2024-03-04T16:04:52.113 INFO:tasks.workunit.client.1.smithi152.stdout:2/820: truncate da/df/d2e/f90 48148 0 2024-03-04T16:04:52.114 INFO:tasks.workunit.client.1.smithi152.stdout:2/821: stat da/df/d58/c63 0 2024-03-04T16:04:52.115 INFO:tasks.workunit.client.1.smithi152.stdout:2/822: chown f6 6040 1 2024-03-04T16:04:52.117 INFO:tasks.workunit.client.1.smithi152.stdout:2/823: write da/feb [905298,109002] 0 2024-03-04T16:04:52.118 INFO:tasks.workunit.client.1.smithi152.stdout:8/980: dwrite d2/d75/d89/dd/dfc/ffd [0,4194304] 0 2024-03-04T16:04:52.120 INFO:tasks.workunit.client.1.smithi152.stdout:3/816: rename d6/d5c/c8b to d6/d60/d8e/dc0/db5/de6/c116 0 2024-03-04T16:04:52.122 INFO:tasks.workunit.client.1.smithi152.stdout:3/817: read d6/d10/d2d/d4d/fc3 [1483207,42724] 0 2024-03-04T16:04:52.128 INFO:tasks.workunit.client.1.smithi152.stdout:9/813: fsync d2/d13/d23/d71/da3/fab 0 2024-03-04T16:04:52.131 INFO:tasks.workunit.client.1.smithi152.stdout:4/710: mkdir d8/d2f/d6f/deb 0 2024-03-04T16:04:52.137 INFO:tasks.workunit.client.1.smithi152.stdout:0/846: rename d9/d1b/d3c/d84/f77 to d9/d1a/d7c/f115 0 2024-03-04T16:04:52.140 INFO:tasks.workunit.client.1.smithi152.stdout:3/818: creat d6/d60/d8e/dc0/da1/f117 x:0 0 0 2024-03-04T16:04:52.141 INFO:tasks.workunit.client.1.smithi152.stdout:8/981: dwrite d2/d75/d89/dd/d22/d26/f38 [4194304,4194304] 0 2024-03-04T16:04:52.143 INFO:tasks.workunit.client.1.smithi152.stdout:9/814: mkdir d2/d13/d23/d71/d80/d10e 0 2024-03-04T16:04:52.144 INFO:tasks.workunit.client.1.smithi152.stdout:4/711: symlink d8/d9/d79/lec 0 2024-03-04T16:04:52.145 INFO:tasks.workunit.client.1.smithi152.stdout:2/824: dread da/f77 [0,4194304] 0 2024-03-04T16:04:52.145 INFO:tasks.workunit.client.1.smithi152.stdout:7/778: truncate d14/f9b 409449 0 2024-03-04T16:04:52.146 INFO:tasks.workunit.client.1.smithi152.stdout:7/779: write d14/d69/f88 [168566,61572] 0 2024-03-04T16:04:52.148 INFO:tasks.workunit.client.1.smithi152.stdout:0/847: unlink c3 0 2024-03-04T16:04:52.152 INFO:tasks.workunit.client.1.smithi152.stdout:3/819: chown d6/d60/d8e/dc0/l69 1 1 2024-03-04T16:04:52.155 INFO:tasks.workunit.client.1.smithi152.stdout:9/815: symlink d2/d13/d23/l10f 0 2024-03-04T16:04:52.155 INFO:tasks.workunit.client.1.smithi152.stdout:5/832: dwrite d6/d8/d5a/d64/d76/d9f/f54 [0,4194304] 0 2024-03-04T16:04:52.159 INFO:tasks.workunit.client.1.smithi152.stdout:3/820: dread d6/f3f [0,4194304] 0 2024-03-04T16:04:52.159 INFO:tasks.workunit.client.1.smithi152.stdout:4/712: rmdir d8/d9/d24 39 2024-03-04T16:04:52.161 INFO:tasks.workunit.client.1.smithi152.stdout:2/825: rmdir da/d5f/dbc 39 2024-03-04T16:04:52.162 INFO:tasks.workunit.client.1.smithi152.stdout:2/826: read - da/d2c/f11e zero size 2024-03-04T16:04:52.163 INFO:tasks.workunit.client.1.smithi152.stdout:2/827: chown da/df/d2e/d51/d8d/d9a/ca7 3 1 2024-03-04T16:04:52.165 INFO:tasks.workunit.client.1.smithi152.stdout:7/780: rename d14/d21/d47/d36/da7/dbc/ld9 to d14/d21/d31/dde/le1 0 2024-03-04T16:04:52.167 INFO:tasks.workunit.client.1.smithi152.stdout:7/781: write d14/db9/dc9/f60 [2199574,36270] 0 2024-03-04T16:04:52.168 INFO:tasks.workunit.client.1.smithi152.stdout:7/782: truncate d14/d21/d31/f33 1657538 0 2024-03-04T16:04:52.181 INFO:tasks.workunit.client.1.smithi152.stdout:0/848: dread d9/d1b/d3c/d84/d56/f8b [0,4194304] 0 2024-03-04T16:04:52.182 INFO:tasks.workunit.client.1.smithi152.stdout:7/783: dwrite d14/db9/dc9/f60 [0,4194304] 0 2024-03-04T16:04:52.183 INFO:tasks.workunit.client.1.smithi152.stdout:7/784: stat d14/db9/dc9/f3b 0 2024-03-04T16:04:52.188 INFO:tasks.workunit.client.1.smithi152.stdout:2/828: rmdir da/df/d2e/dc6 39 2024-03-04T16:04:52.237 INFO:tasks.workunit.client.1.smithi152.stdout:3/821: write d6/d10/d2d/fec [623750,128300] 0 2024-03-04T16:04:52.241 INFO:tasks.workunit.client.1.smithi152.stdout:8/982: dwrite d2/d75/d89/dd/d22/d6c/d86/fd3 [0,4194304] 0 2024-03-04T16:04:52.247 INFO:tasks.workunit.client.1.smithi152.stdout:1/892: sync 2024-03-04T16:04:52.260 INFO:tasks.workunit.client.1.smithi152.stdout:0/849: mknod d9/d92/c116 0 2024-03-04T16:04:52.261 INFO:tasks.workunit.client.1.smithi152.stdout:7/785: mknod d14/d69/d8e/ce2 0 2024-03-04T16:04:52.309 INFO:tasks.workunit.client.1.smithi152.stdout:4/713: truncate d8/d9/d7a/f8f 1571857 0 2024-03-04T16:04:52.310 INFO:tasks.workunit.client.1.smithi152.stdout:2/829: dread - da/df/d2e/d51/d11c/d47/d4f/d11a/d8e/f93 zero size 2024-03-04T16:04:52.310 INFO:tasks.workunit.client.1.smithi152.stdout:4/714: chown d8/d2f/d6f/d90/fca 6425530 1 2024-03-04T16:04:52.317 INFO:tasks.workunit.client.1.smithi152.stdout:1/893: creat d19/d20/d2f/d65/def/f117 x:0 0 0 2024-03-04T16:04:52.318 INFO:tasks.workunit.client.1.smithi152.stdout:1/894: truncate d19/d4b/d53/d57/f102 848466 0 2024-03-04T16:04:52.321 INFO:tasks.workunit.client.1.smithi152.stdout:7/786: symlink d14/d69/dd1/le3 0 2024-03-04T16:04:52.328 INFO:tasks.workunit.client.1.smithi152.stdout:4/715: mkdir d8/d58/ded 0 2024-03-04T16:04:52.329 INFO:tasks.workunit.client.1.smithi152.stdout:7/787: dwrite d14/d21/d47/d36/dcf/fda [0,4194304] 0 2024-03-04T16:04:52.331 INFO:tasks.workunit.client.1.smithi152.stdout:3/822: symlink d6/l118 0 2024-03-04T16:04:52.336 INFO:tasks.workunit.client.1.smithi152.stdout:7/788: dread d14/d21/d31/f33 [0,4194304] 0 2024-03-04T16:04:52.338 INFO:tasks.workunit.client.1.smithi152.stdout:7/789: write d14/d21/d31/f33 [1219256,74152] 0 2024-03-04T16:04:52.343 INFO:tasks.workunit.client.1.smithi152.stdout:1/895: truncate d19/d20/fb2 3480089 0 2024-03-04T16:04:52.349 INFO:tasks.workunit.client.1.smithi152.stdout:1/896: dwrite d19/d3b/d6e/ff8 [0,4194304] 0 2024-03-04T16:04:52.355 INFO:tasks.workunit.client.1.smithi152.stdout:3/823: link d6/d10/c5f d6/d10/c119 0 2024-03-04T16:04:52.355 INFO:tasks.workunit.client.1.smithi152.stdout:3/824: chown d6/d60/f109 398 1 2024-03-04T16:04:52.359 INFO:tasks.workunit.client.1.smithi152.stdout:3/825: creat d6/d14/d58/de4/d111/f11a x:0 0 0 2024-03-04T16:04:52.361 INFO:tasks.workunit.client.1.smithi152.stdout:3/826: truncate d6/d60/d8a/fca 1209420 0 2024-03-04T16:04:52.365 INFO:tasks.workunit.client.1.smithi152.stdout:3/827: creat d6/d14/f11b x:0 0 0 2024-03-04T16:04:52.368 INFO:tasks.workunit.client.1.smithi152.stdout:3/828: creat d6/d14/d93/f11c x:0 0 0 2024-03-04T16:04:52.369 INFO:tasks.workunit.client.1.smithi152.stdout:7/790: creat d14/db9/dc9/fe4 x:0 0 0 2024-03-04T16:04:52.372 INFO:tasks.workunit.client.1.smithi152.stdout:7/791: dread d14/d21/d31/f33 [0,4194304] 0 2024-03-04T16:04:52.372 INFO:tasks.workunit.client.1.smithi152.stdout:2/830: mknod da/d5f/d85/c120 0 2024-03-04T16:04:52.374 INFO:tasks.workunit.client.1.smithi152.stdout:3/829: creat d6/f11d x:0 0 0 2024-03-04T16:04:52.374 INFO:tasks.workunit.client.1.smithi152.stdout:5/833: write d6/d8/f24 [5263550,20142] 0 2024-03-04T16:04:52.376 INFO:tasks.workunit.client.1.smithi152.stdout:2/831: fsync da/df/d2e/d51/d11c/d47/f6b 0 2024-03-04T16:04:52.383 INFO:tasks.workunit.client.1.smithi152.stdout:2/832: dwrite da/df/d2e/d51/d11c/d80/f9b [0,4194304] 0 2024-03-04T16:04:52.391 INFO:tasks.workunit.client.1.smithi152.stdout:7/792: dread d14/d21/d47/d32/fbb [0,4194304] 0 2024-03-04T16:04:52.470 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:52 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:52.472 INFO:tasks.workunit.client.1.smithi152.stdout:6/919: sync 2024-03-04T16:04:52.473 INFO:tasks.workunit.client.1.smithi152.stdout:3/830: creat d6/d14/d41/d49/d84/de2/de9/d104/f11e x:0 0 0 2024-03-04T16:04:52.477 INFO:tasks.workunit.client.1.smithi152.stdout:5/834: symlink d6/d8/d1f/d3d/da4/dc9/l10e 0 2024-03-04T16:04:52.481 INFO:tasks.workunit.client.1.smithi152.stdout:7/793: mkdir d14/db9/dc9/d83/de5 0 2024-03-04T16:04:52.481 INFO:tasks.workunit.client.1.smithi152.stdout:7/794: stat d14/d69/dd1/le3 0 2024-03-04T16:04:52.489 INFO:tasks.workunit.client.1.smithi152.stdout:5/835: creat d6/d8/d1f/d3d/dc0/f10f x:0 0 0 2024-03-04T16:04:52.489 INFO:tasks.workunit.client.1.smithi152.stdout:7/795: dwrite f7 [0,4194304] 0 2024-03-04T16:04:52.491 INFO:tasks.workunit.client.1.smithi152.stdout:5/836: dread - d6/d8/d1f/d3d/d6f/d70/d98/de7/ff5 zero size 2024-03-04T16:04:52.495 INFO:tasks.workunit.client.1.smithi152.stdout:7/796: mkdir d14/d21/d47/d36/da7/dbc/de6 0 2024-03-04T16:04:52.499 INFO:tasks.workunit.client.1.smithi152.stdout:7/797: mkdir d14/d69/d8e/de7 0 2024-03-04T16:04:52.502 INFO:tasks.workunit.client.1.smithi152.stdout:9/816: write d2/d13/fd7 [2626464,79472] 0 2024-03-04T16:04:52.504 INFO:tasks.workunit.client.1.smithi152.stdout:9/817: dread - d2/d13/d23/d71/da3/fbc zero size 2024-03-04T16:04:52.506 INFO:tasks.workunit.client.1.smithi152.stdout:9/818: symlink d2/d13/d35/l110 0 2024-03-04T16:04:52.511 INFO:tasks.workunit.client.1.smithi152.stdout:9/819: truncate d2/d13/d23/f4c 368485 0 2024-03-04T16:04:52.516 INFO:tasks.workunit.client.1.smithi152.stdout:9/820: truncate d2/f101 836880 0 2024-03-04T16:04:52.534 INFO:tasks.workunit.client.1.smithi152.stdout:4/716: truncate d8/f25 6257819 0 2024-03-04T16:04:52.535 INFO:tasks.workunit.client.1.smithi152.stdout:4/717: creat d8/d89/fee x:0 0 0 2024-03-04T16:04:52.537 INFO:tasks.workunit.client.1.smithi152.stdout:0/850: creat d9/d2c/d2e/d7f/f117 x:0 0 0 2024-03-04T16:04:52.549 INFO:tasks.workunit.client.1.smithi152.stdout:3/831: dwrite d6/d14/d58/f9a [0,4194304] 0 2024-03-04T16:04:52.556 INFO:tasks.workunit.client.1.smithi152.stdout:5/837: dwrite d6/d8/f56 [0,4194304] 0 2024-03-04T16:04:52.559 INFO:tasks.workunit.client.1.smithi152.stdout:3/832: fsync d6/d14/f37 0 2024-03-04T16:04:52.559 INFO:tasks.workunit.client.1.smithi152.stdout:2/833: dwrite da/df/d2e/f90 [0,4194304] 0 2024-03-04T16:04:52.565 INFO:tasks.workunit.client.1.smithi152.stdout:6/920: rename d8/d8f/f11b to d8/f13f 0 2024-03-04T16:04:52.565 INFO:tasks.workunit.client.1.smithi152.stdout:9/821: write d2/fb [471084,31197] 0 2024-03-04T16:04:52.573 INFO:tasks.workunit.client.1.smithi152.stdout:3/833: symlink d6/d14/d41/d49/d84/de2/d108/l11f 0 2024-03-04T16:04:52.575 INFO:tasks.workunit.client.1.smithi152.stdout:7/798: rename d14/d21/d47/d36/l66 to d14/d21/d47/d32/le8 0 2024-03-04T16:04:52.576 INFO:tasks.workunit.client.1.smithi152.stdout:6/921: dread - d8/d92/d59/d61/fcb zero size 2024-03-04T16:04:52.584 INFO:tasks.workunit.client.1.smithi152.stdout:4/718: rename d8/de3 to d8/d2f/d6f/deb/def 0 2024-03-04T16:04:52.593 INFO:tasks.workunit.client.1.smithi152.stdout:0/851: rename d9/d2c/d2e/l52 to d9/d1b/d3c/d84/dde/l118 0 2024-03-04T16:04:52.601 INFO:tasks.workunit.client.1.smithi152.stdout:7/799: dread d14/d69/fdb [0,4194304] 0 2024-03-04T16:04:52.603 INFO:tasks.workunit.client.1.smithi152.stdout:0/852: dwrite d9/f102 [0,4194304] 0 2024-03-04T16:04:52.603 INFO:tasks.workunit.client.1.smithi152.stdout:7/800: dread - d14/d69/fce zero size 2024-03-04T16:04:52.604 INFO:tasks.workunit.client.1.smithi152.stdout:6/922: dread d8/d23/d3a/d8d/d96/fb9 [0,4194304] 0 2024-03-04T16:04:52.605 INFO:tasks.workunit.client.1.smithi152.stdout:9/822: rename d2/d63 to d2/d13/dad/dd2/d111 0 2024-03-04T16:04:52.609 INFO:tasks.workunit.client.1.smithi152.stdout:5/838: dwrite d6/d8/d5a/d8b/fb1 [0,4194304] 0 2024-03-04T16:04:52.612 INFO:tasks.workunit.client.1.smithi152.stdout:6/923: rmdir d8/d42 39 2024-03-04T16:04:52.612 INFO:tasks.workunit.client.1.smithi152.stdout:3/834: write d6/d60/d8e/dc0/fc1 [68169,103234] 0 2024-03-04T16:04:52.612 INFO:tasks.workunit.client.1.smithi152.stdout:0/853: chown d9/l12 51744 1 2024-03-04T16:04:52.614 INFO:tasks.workunit.client.1.smithi152.stdout:9/823: creat d2/d13/d38/d79/f112 x:0 0 0 2024-03-04T16:04:52.615 INFO:tasks.workunit.client.1.smithi152.stdout:2/834: dwrite da/df/d58/d6e/fc8 [0,4194304] 0 2024-03-04T16:04:52.615 INFO:tasks.workunit.client.1.smithi152.stdout:3/835: chown d6/d5c/f97 270 1 2024-03-04T16:04:52.617 INFO:tasks.workunit.client.1.smithi152.stdout:3/836: chown d6/d60/d8e/df0/cf2 752 1 2024-03-04T16:04:52.618 INFO:tasks.workunit.client.1.smithi152.stdout:2/835: read da/df/d2e/d51/d11c/d47/d4f/fc9 [1993740,2144] 0 2024-03-04T16:04:52.631 INFO:tasks.workunit.client.1.smithi152.stdout:0/854: symlink d9/d1b/d3c/d84/d56/d5b/d7a/l119 0 2024-03-04T16:04:52.631 INFO:tasks.workunit.client.1.smithi152.stdout:4/719: dwrite d8/d2f/d7c/d87/f97 [0,4194304] 0 2024-03-04T16:04:52.634 INFO:tasks.workunit.client.1.smithi152.stdout:7/801: rename d14/d69/d8e/c92 to d14/d69/ce9 0 2024-03-04T16:04:52.644 INFO:tasks.workunit.client.1.smithi152.stdout:4/720: rename d8/d2f/d6f/d90/daf to d8/d2f/d7c/d87/df0 0 2024-03-04T16:04:52.660 INFO:tasks.workunit.client.1.smithi152.stdout:0/855: mknod d9/d1b/d3c/d84/c11a 0 2024-03-04T16:04:52.661 INFO:tasks.workunit.client.1.smithi152.stdout:3/837: link d6/f1a d6/d10/d2d/f120 0 2024-03-04T16:04:52.661 INFO:tasks.workunit.client.1.smithi152.stdout:7/802: mknod d14/db9/dc9/d83/de5/cea 0 2024-03-04T16:04:52.663 INFO:tasks.workunit.client.1.smithi152.stdout:4/721: mkdir d8/d4e/dba/dcc/df1 0 2024-03-04T16:04:52.666 INFO:tasks.workunit.client.1.smithi152.stdout:5/839: truncate d6/d50/f59 3261810 0 2024-03-04T16:04:52.668 INFO:tasks.workunit.client.1.smithi152.stdout:7/803: creat d14/d21/d47/d36/feb x:0 0 0 2024-03-04T16:04:52.668 INFO:tasks.workunit.client.1.smithi152.stdout:0/856: mkdir d9/d1b/d11b 0 2024-03-04T16:04:52.670 INFO:tasks.workunit.client.1.smithi152.stdout:5/840: creat d6/d8/d1f/d88/f110 x:0 0 0 2024-03-04T16:04:52.671 INFO:tasks.workunit.client.1.smithi152.stdout:7/804: chown d14/d21/c28 21622833 1 2024-03-04T16:04:52.676 INFO:tasks.workunit.client.1.smithi152.stdout:0/857: dwrite d9/f49 [4194304,4194304] 0 2024-03-04T16:04:52.679 INFO:tasks.workunit.client.1.smithi152.stdout:7/805: rmdir d14/d21/d47/d36/da0 0 2024-03-04T16:04:52.684 INFO:tasks.workunit.client.1.smithi152.stdout:9/824: write d2/d13/d38/f58 [1767000,5009] 0 2024-03-04T16:04:52.689 INFO:tasks.workunit.client.1.smithi152.stdout:0/858: rename d9/d1a/db4/ff0 to d9/d1b/d54/d69/f11c 0 2024-03-04T16:04:52.690 INFO:tasks.workunit.client.1.smithi152.stdout:0/859: dread - d9/d1b/d54/d69/f11c zero size 2024-03-04T16:04:52.692 INFO:tasks.workunit.client.1.smithi152.stdout:4/722: dread d8/d2f/f9c [0,4194304] 0 2024-03-04T16:04:52.693 INFO:tasks.workunit.client.1.smithi152.stdout:3/838: dread d6/f2c [0,4194304] 0 2024-03-04T16:04:52.705 INFO:tasks.workunit.client.1.smithi152.stdout:5/841: write d6/d8/d1f/d3d/d6f/d70/fe9 [779049,46308] 0 2024-03-04T16:04:52.706 INFO:tasks.workunit.client.1.smithi152.stdout:9/825: dread d2/d13/fd7 [0,4194304] 0 2024-03-04T16:04:52.712 INFO:tasks.workunit.client.1.smithi152.stdout:7/806: dwrite d14/db9/dc9/f58 [0,4194304] 0 2024-03-04T16:04:52.713 INFO:tasks.workunit.client.1.smithi152.stdout:9/826: truncate d2/d13/dad/dd2/d111/d69/f90 2445265 0 2024-03-04T16:04:52.714 INFO:tasks.workunit.client.1.smithi152.stdout:9/827: chown d2/d13/dad/dd2/d111/d69/l8a 969002500 1 2024-03-04T16:04:52.716 INFO:tasks.workunit.client.1.smithi152.stdout:0/860: write d9/d1b/d54/fb7 [683054,111824] 0 2024-03-04T16:04:52.718 INFO:tasks.workunit.client.1.smithi152.stdout:3/839: write d6/d14/d58/dad/fd1 [2055665,116617] 0 2024-03-04T16:04:52.722 INFO:tasks.workunit.client.1.smithi152.stdout:5/842: creat d6/f111 x:0 0 0 2024-03-04T16:04:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:52 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:52 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:52 smithi116 ceph-mon[29120]: Standby manager daemon smithi152.aawieo started 2024-03-04T16:04:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:52 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3281042048' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/crt"}]: dispatch 2024-03-04T16:04:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:52 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3281042048' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:04:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:52 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3281042048' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/key"}]: dispatch 2024-03-04T16:04:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:52 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3281042048' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:04:52.726 INFO:tasks.workunit.client.1.smithi152.stdout:7/807: link d14/db9/dc9/d83/fa9 d14/d21/d48/fec 0 2024-03-04T16:04:52.727 INFO:tasks.workunit.client.1.smithi152.stdout:4/723: dwrite d8/d9/d24/d26/d42/f6b [0,4194304] 0 2024-03-04T16:04:52.733 INFO:tasks.workunit.client.1.smithi152.stdout:3/840: dwrite d6/d60/d8e/dc0/fc1 [0,4194304] 0 2024-03-04T16:04:52.733 INFO:tasks.workunit.client.1.smithi152.stdout:9/828: link d2/l74 d2/d13/d35/l113 0 2024-03-04T16:04:52.741 INFO:tasks.workunit.client.1.smithi152.stdout:5/843: dwrite d6/d8/d5a/fed [0,4194304] 0 2024-03-04T16:04:52.744 INFO:tasks.workunit.client.1.smithi152.stdout:7/808: creat d14/d21/d47/fed x:0 0 0 2024-03-04T16:04:52.745 INFO:tasks.workunit.client.1.smithi152.stdout:4/724: dwrite d8/fcd [0,4194304] 0 2024-03-04T16:04:52.747 INFO:tasks.workunit.client.1.smithi152.stdout:3/841: unlink d6/d10/f27 0 2024-03-04T16:04:52.749 INFO:tasks.workunit.client.1.smithi152.stdout:9/829: fdatasync d2/d13/d23/d66/d7d/f107 0 2024-03-04T16:04:52.751 INFO:tasks.workunit.client.1.smithi152.stdout:7/809: creat d14/d21/d31/fee x:0 0 0 2024-03-04T16:04:52.752 INFO:tasks.workunit.client.1.smithi152.stdout:0/861: dwrite d9/d1b/d54/d80/db2/fb9 [0,4194304] 0 2024-03-04T16:04:52.754 INFO:tasks.workunit.client.1.smithi152.stdout:0/862: dread - d9/d1b/d54/d69/d6c/ff9 zero size 2024-03-04T16:04:52.758 INFO:tasks.workunit.client.1.smithi152.stdout:4/725: mkdir d8/d4e/dba/dcc/df2 0 2024-03-04T16:04:52.762 INFO:tasks.workunit.client.1.smithi152.stdout:3/842: creat d6/d14/d41/d49/d84/de2/d108/f121 x:0 0 0 2024-03-04T16:04:52.768 INFO:tasks.workunit.client.1.smithi152.stdout:0/863: fsync d9/f36 0 2024-03-04T16:04:52.769 INFO:tasks.workunit.client.1.smithi152.stdout:0/864: chown d9/d1b/d54/d80/f104 0 1 2024-03-04T16:04:52.770 INFO:tasks.workunit.client.1.smithi152.stdout:4/726: readlink d8/d4e/d86/lab 0 2024-03-04T16:04:52.774 INFO:tasks.workunit.client.1.smithi152.stdout:9/830: unlink d2/d13/d23/c5a 0 2024-03-04T16:04:52.776 INFO:tasks.workunit.client.1.smithi152.stdout:3/843: creat d6/d14/d41/d49/d84/de2/d108/f122 x:0 0 0 2024-03-04T16:04:52.777 INFO:tasks.workunit.client.1.smithi152.stdout:9/831: stat d2/d13/d23/d71/d80/caa 0 2024-03-04T16:04:52.778 INFO:tasks.workunit.client.1.smithi152.stdout:3/844: chown d6/f96 333 1 2024-03-04T16:04:52.779 INFO:tasks.workunit.client.1.smithi152.stdout:0/865: dwrite d9/d2c/d2e/f110 [0,4194304] 0 2024-03-04T16:04:52.780 INFO:tasks.workunit.client.1.smithi152.stdout:4/727: dwrite d8/d2f/d7c/fce [0,4194304] 0 2024-03-04T16:04:52.785 INFO:tasks.workunit.client.1.smithi152.stdout:9/832: symlink d2/d13/d23/d71/da3/l114 0 2024-03-04T16:04:52.786 INFO:tasks.workunit.client.1.smithi152.stdout:0/866: chown d9/d92/c101 222 1 2024-03-04T16:04:52.790 INFO:tasks.workunit.client.1.smithi152.stdout:3/845: rename d6/d14/d93/c92 to d6/d14/d93/db7/c123 0 2024-03-04T16:04:52.791 INFO:tasks.workunit.client.1.smithi152.stdout:4/728: mknod d8/d2f/d6f/deb/cf3 0 2024-03-04T16:04:52.793 INFO:tasks.workunit.client.1.smithi152.stdout:9/833: dwrite d2/d13/dad/dd2/d111/d69/fdb [0,4194304] 0 2024-03-04T16:04:52.794 INFO:tasks.workunit.client.1.smithi152.stdout:3/846: truncate d6/d14/d58/de4/d111/f11a 44576 0 2024-03-04T16:04:52.803 INFO:tasks.workunit.client.1.smithi152.stdout:4/729: fsync d8/d9/d24/f40 0 2024-03-04T16:04:52.803 INFO:tasks.workunit.client.1.smithi152.stdout:3/847: dread - d6/d60/d8e/fd8 zero size 2024-03-04T16:04:52.804 INFO:tasks.workunit.client.1.smithi152.stdout:9/834: dread - d2/d13/d23/d71/d80/fa8 zero size 2024-03-04T16:04:52.805 INFO:tasks.workunit.client.1.smithi152.stdout:7/810: link d14/d21/l27 d14/d21/d47/lef 0 2024-03-04T16:04:52.812 INFO:tasks.workunit.client.1.smithi152.stdout:3/848: creat d6/d14/d58/f124 x:0 0 0 2024-03-04T16:04:52.813 INFO:tasks.workunit.client.1.smithi152.stdout:3/849: write d6/d60/d8e/dc0/da1/f112 [522111,67284] 0 2024-03-04T16:04:52.815 INFO:tasks.workunit.client.1.smithi152.stdout:9/835: symlink d2/d13/dad/l115 0 2024-03-04T16:04:52.817 INFO:tasks.workunit.client.1.smithi152.stdout:7/811: fsync d14/d21/d47/f8c 0 2024-03-04T16:04:52.819 INFO:tasks.workunit.client.1.smithi152.stdout:0/867: write d9/d1b/d3c/d84/d56/f8b [2708362,110979] 0 2024-03-04T16:04:52.821 INFO:tasks.workunit.client.1.smithi152.stdout:0/868: write d9/d1b/d54/d80/db2/fb9 [4178003,4806] 0 2024-03-04T16:04:52.823 INFO:tasks.workunit.client.1.smithi152.stdout:9/836: mknod d2/d13/dad/dd2/c116 0 2024-03-04T16:04:52.827 INFO:tasks.workunit.client.1.smithi152.stdout:4/730: rename d8/d4e/d8b/fc2 to d8/d9/ff4 0 2024-03-04T16:04:52.837 INFO:tasks.workunit.client.1.smithi152.stdout:3/850: dread d6/d60/d8e/dc0/fc7 [0,4194304] 0 2024-03-04T16:04:52.848 INFO:tasks.workunit.client.1.smithi152.stdout:9/837: dread d2/d13/d23/d71/fa1 [0,4194304] 0 2024-03-04T16:04:52.848 INFO:tasks.workunit.client.1.smithi152.stdout:0/869: mknod d9/d2c/d2e/d7f/c11d 0 2024-03-04T16:04:52.851 INFO:tasks.workunit.client.1.smithi152.stdout:9/838: mknod d2/d13/dad/dd2/c117 0 2024-03-04T16:04:52.852 INFO:tasks.workunit.client.1.smithi152.stdout:0/870: mkdir d9/d1a/db4/d11e 0 2024-03-04T16:04:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:52 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:52 smithi152 ceph-mon[36876]: Standby manager daemon smithi152.aawieo started 2024-03-04T16:04:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:52 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3281042048' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/crt"}]: dispatch 2024-03-04T16:04:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:52 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3281042048' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:04:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:52 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3281042048' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/key"}]: dispatch 2024-03-04T16:04:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:52 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3281042048' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:04:52.859 INFO:tasks.workunit.client.1.smithi152.stdout:0/871: dwrite d9/d1b/d54/d80/f104 [0,4194304] 0 2024-03-04T16:04:52.860 INFO:tasks.workunit.client.1.smithi152.stdout:7/812: rename d14/db9/dc9/c85 to d14/d69/cf0 0 2024-03-04T16:04:52.866 INFO:tasks.workunit.client.1.smithi152.stdout:0/872: symlink d9/d1b/d3c/d84/d56/d5b/l11f 0 2024-03-04T16:04:52.867 INFO:tasks.workunit.client.1.smithi152.stdout:0/873: chown d9/d2c/d2e/l55 470 1 2024-03-04T16:04:52.868 INFO:tasks.workunit.client.1.smithi152.stdout:7/813: creat d14/db9/dc9/d83/ff1 x:0 0 0 2024-03-04T16:04:52.868 INFO:tasks.workunit.client.1.smithi152.stdout:4/731: truncate d8/d2f/f33 1650922 0 2024-03-04T16:04:52.869 INFO:tasks.workunit.client.1.smithi152.stdout:3/851: write d6/d10/d2d/f40 [2486212,17660] 0 2024-03-04T16:04:52.870 INFO:tasks.workunit.client.1.smithi152.stdout:9/839: rename d2/l64 to d2/d13/d38/df9/l118 0 2024-03-04T16:04:52.874 INFO:tasks.workunit.client.1.smithi152.stdout:8/983: sync 2024-03-04T16:04:52.874 INFO:tasks.workunit.client.1.smithi152.stdout:1/897: sync 2024-03-04T16:04:52.876 INFO:tasks.workunit.client.1.smithi152.stdout:8/984: write d2/d75/d89/dd/d22/d83/dd9/f129 [931428,121325] 0 2024-03-04T16:04:53.045 INFO:tasks.workunit.client.1.smithi152.stdout:0/874: mkdir d9/d1b/d54/d80/da8/d120 0 2024-03-04T16:04:53.046 INFO:tasks.workunit.client.1.smithi152.stdout:0/875: readlink d9/d92/d107/l114 0 2024-03-04T16:04:53.046 INFO:tasks.workunit.client.1.smithi152.stdout:3/852: creat d6/d10/f125 x:0 0 0 2024-03-04T16:04:53.052 INFO:tasks.workunit.client.1.smithi152.stdout:9/840: symlink d2/d6e/l119 0 2024-03-04T16:04:53.052 INFO:tasks.workunit.client.1.smithi152.stdout:4/732: rename d8/c41 to d8/d9/dc1/cf5 0 2024-03-04T16:04:53.054 INFO:tasks.workunit.client.1.smithi152.stdout:0/876: truncate d9/d92/fa2 639728 0 2024-03-04T16:04:53.055 INFO:tasks.workunit.client.1.smithi152.stdout:3/853: creat d6/d14/d58/de4/f126 x:0 0 0 2024-03-04T16:04:53.055 INFO:tasks.workunit.client.1.smithi152.stdout:3/854: chown d6/d60/d8e/dc0/da1/fc2 14159285 1 2024-03-04T16:04:53.057 INFO:tasks.workunit.client.1.smithi152.stdout:1/898: mkdir d19/d3b/d6e/d118 0 2024-03-04T16:04:53.060 INFO:tasks.workunit.client.1.smithi152.stdout:7/814: rename d14/d21/c28 to d14/d21/d47/d32/cf2 0 2024-03-04T16:04:53.061 INFO:tasks.workunit.client.1.smithi152.stdout:7/815: fdatasync d14/d21/d47/d36/fcc 0 2024-03-04T16:04:53.062 INFO:tasks.workunit.client.1.smithi152.stdout:7/816: write d14/db9/dc9/fe4 [753152,83186] 0 2024-03-04T16:04:53.062 INFO:tasks.workunit.client.1.smithi152.stdout:0/877: mkdir d9/d1a/db4/d121 0 2024-03-04T16:04:53.071 INFO:tasks.workunit.client.1.smithi152.stdout:1/899: symlink d19/d20/d2f/d5d/l119 0 2024-03-04T16:04:53.078 INFO:tasks.workunit.client.1.smithi152.stdout:7/817: truncate d14/d21/d47/d36/f74 824446 0 2024-03-04T16:04:53.082 INFO:tasks.workunit.client.1.smithi152.stdout:4/733: link d8/d9/d24/l6a d8/d78/lf6 0 2024-03-04T16:04:53.083 INFO:tasks.workunit.client.1.smithi152.stdout:0/878: creat d9/d1b/d54/d69/d6c/dbf/dfd/f122 x:0 0 0 2024-03-04T16:04:53.086 INFO:tasks.workunit.client.1.smithi152.stdout:1/900: unlink d19/f23 0 2024-03-04T16:04:53.090 INFO:tasks.workunit.client.1.smithi152.stdout:0/879: creat d9/d2c/d2e/d94/dac/f123 x:0 0 0 2024-03-04T16:04:53.094 INFO:tasks.workunit.client.1.smithi152.stdout:1/901: creat d19/d20/d2f/d5d/f11a x:0 0 0 2024-03-04T16:04:53.095 INFO:tasks.workunit.client.1.smithi152.stdout:4/734: symlink d8/d4e/dba/dcc/df1/lf7 0 2024-03-04T16:04:53.096 INFO:tasks.workunit.client.1.smithi152.stdout:9/841: write d2/d13/f52 [374483,121934] 0 2024-03-04T16:04:53.099 INFO:tasks.workunit.client.1.smithi152.stdout:3/855: dwrite d6/d14/d41/d49/d84/fba [0,4194304] 0 2024-03-04T16:04:53.101 INFO:tasks.workunit.client.1.smithi152.stdout:4/735: creat d8/d78/ff8 x:0 0 0 2024-03-04T16:04:53.102 INFO:tasks.workunit.client.1.smithi152.stdout:4/736: read f3 [3460383,83063] 0 2024-03-04T16:04:53.103 INFO:tasks.workunit.client.1.smithi152.stdout:4/737: fdatasync d8/d9/d24/f74 0 2024-03-04T16:04:53.104 INFO:tasks.workunit.client.1.smithi152.stdout:8/985: write d2/d3/d21/d2d/d52/d68/d6e/d7d/f10b [607498,88308] 0 2024-03-04T16:04:53.105 INFO:tasks.workunit.client.1.smithi152.stdout:4/738: write d8/d9/d24/da0/fc9 [705782,47331] 0 2024-03-04T16:04:53.105 INFO:tasks.workunit.client.1.smithi152.stdout:7/818: link d14/d21/d47/lef d14/d71/lf3 0 2024-03-04T16:04:53.110 INFO:tasks.workunit.client.1.smithi152.stdout:3/856: creat d6/d14/d93/db7/f127 x:0 0 0 2024-03-04T16:04:53.113 INFO:tasks.workunit.client.1.smithi152.stdout:0/880: symlink d9/d1b/d54/df1/l124 0 2024-03-04T16:04:53.118 INFO:tasks.workunit.client.1.smithi152.stdout:4/739: rmdir d8/d2f/d7c/d87 39 2024-03-04T16:04:53.122 INFO:tasks.workunit.client.1.smithi152.stdout:7/819: readlink d14/d21/l27 0 2024-03-04T16:04:53.124 INFO:tasks.workunit.client.1.smithi152.stdout:4/740: dread d8/d9/d24/da0/fc9 [0,4194304] 0 2024-03-04T16:04:53.125 INFO:tasks.workunit.client.1.smithi152.stdout:4/741: dread - d8/d2f/d6f/d90/fd9 zero size 2024-03-04T16:04:53.129 INFO:tasks.workunit.client.1.smithi152.stdout:0/881: dread d9/d1a/d7c/f115 [0,4194304] 0 2024-03-04T16:04:53.132 INFO:tasks.workunit.client.1.smithi152.stdout:9/842: mknod d2/d13/d23/d42/db1/c11a 0 2024-03-04T16:04:53.135 INFO:tasks.workunit.client.1.smithi152.stdout:4/742: read d8/d2f/d7c/fb1 [3279487,112332] 0 2024-03-04T16:04:53.136 INFO:tasks.workunit.client.1.smithi152.stdout:3/857: link d6/d10/d2d/f120 d6/d14/dd7/f128 0 2024-03-04T16:04:53.138 INFO:tasks.workunit.client.1.smithi152.stdout:0/882: unlink d9/d1a/f8a 0 2024-03-04T16:04:53.139 INFO:tasks.workunit.client.1.smithi152.stdout:0/883: write d9/d2c/d2e/d7f/d9d/fcd [1869522,8381] 0 2024-03-04T16:04:53.142 INFO:tasks.workunit.client.1.smithi152.stdout:9/843: dread - d2/d13/dad/dd2/d111/fd9 zero size 2024-03-04T16:04:53.144 INFO:tasks.workunit.client.1.smithi152.stdout:7/820: creat d14/db9/dc9/d83/db7/ff4 x:0 0 0 2024-03-04T16:04:53.146 INFO:tasks.workunit.client.1.smithi152.stdout:4/743: creat d8/d9/d79/ff9 x:0 0 0 2024-03-04T16:04:53.150 INFO:tasks.workunit.client.1.smithi152.stdout:0/884: unlink d9/d1b/d54/d69/d6c/dbf/cd1 0 2024-03-04T16:04:53.152 INFO:tasks.workunit.client.1.smithi152.stdout:3/858: dwrite d6/d60/d8e/dc0/da1/fc2 [0,4194304] 0 2024-03-04T16:04:53.153 INFO:tasks.workunit.client.1.smithi152.stdout:9/844: rename d2/d13/d38/d79/f112 to d2/d6e/f11b 0 2024-03-04T16:04:53.158 INFO:tasks.workunit.client.1.smithi152.stdout:4/744: write d8/d9/d24/da0/fc9 [1701619,3860] 0 2024-03-04T16:04:53.167 INFO:tasks.workunit.client.1.smithi152.stdout:7/821: mkdir d14/d21/df5 0 2024-03-04T16:04:53.168 INFO:tasks.workunit.client.1.smithi152.stdout:9/845: mknod d2/d13/d23/d66/d7d/da7/c11c 0 2024-03-04T16:04:53.171 INFO:tasks.workunit.client.1.smithi152.stdout:7/822: readlink d14/d21/d47/d40/l99 0 2024-03-04T16:04:53.172 INFO:tasks.workunit.client.1.smithi152.stdout:9/846: creat d2/d13/d23/f11d x:0 0 0 2024-03-04T16:04:53.176 INFO:tasks.workunit.client.1.smithi152.stdout:4/745: link d8/d9/d24/fd2 d8/d4e/d8b/ffa 0 2024-03-04T16:04:53.185 INFO:tasks.workunit.client.1.smithi152.stdout:8/986: dwrite d2/d7f/f98 [0,4194304] 0 2024-03-04T16:04:53.188 INFO:tasks.workunit.client.1.smithi152.stdout:4/746: truncate d8/d2f/d67/fdc 773265 0 2024-03-04T16:04:53.189 INFO:tasks.workunit.client.1.smithi152.stdout:3/859: dread d6/d60/d8e/dc0/f8f [0,4194304] 0 2024-03-04T16:04:53.189 INFO:tasks.workunit.client.1.smithi152.stdout:7/823: dwrite d14/d21/d47/d36/feb [0,4194304] 0 2024-03-04T16:04:53.190 INFO:tasks.workunit.client.1.smithi152.stdout:4/747: dread - d8/d4e/d86/fc3 zero size 2024-03-04T16:04:53.201 INFO:tasks.workunit.client.1.smithi152.stdout:4/748: dwrite d8/d9/f50 [0,4194304] 0 2024-03-04T16:04:53.212 INFO:tasks.workunit.client.1.smithi152.stdout:6/924: sync 2024-03-04T16:04:53.242 INFO:tasks.workunit.client.1.smithi152.stdout:0/885: write d9/d1a/f79 [4755439,63162] 0 2024-03-04T16:04:53.243 INFO:tasks.workunit.client.1.smithi152.stdout:0/886: fsync d9/d1b/d3c/d84/d56/f63 0 2024-03-04T16:04:53.270 INFO:tasks.workunit.client.1.smithi152.stdout:5/844: sync 2024-03-04T16:04:53.271 INFO:tasks.workunit.client.1.smithi152.stdout:2/836: sync 2024-03-04T16:04:53.271 INFO:tasks.workunit.client.1.smithi152.stdout:1/902: sync 2024-03-04T16:04:53.272 INFO:tasks.workunit.client.1.smithi152.stdout:1/903: truncate d19/d4b/d53/d57/f88 4312638 0 2024-03-04T16:04:53.280 INFO:tasks.workunit.client.1.smithi152.stdout:9/847: rename d2/d13/d38/d79/lef to d2/d13/d23/d71/d80/d10e/l11e 0 2024-03-04T16:04:53.282 INFO:tasks.workunit.client.1.smithi152.stdout:9/848: read d2/d13/d35/f34 [2439087,4130] 0 2024-03-04T16:04:53.287 INFO:tasks.workunit.client.1.smithi152.stdout:4/749: mknod d8/d9/da2/dd3/cfb 0 2024-03-04T16:04:53.291 INFO:tasks.workunit.client.1.smithi152.stdout:2/837: unlink da/df/d2e/d51/l8a 0 2024-03-04T16:04:53.294 INFO:tasks.workunit.client.1.smithi152.stdout:8/987: rmdir d2/d3/d21/dc0 39 2024-03-04T16:04:53.296 INFO:tasks.workunit.client.1.smithi152.stdout:9/849: creat d2/d13/d38/f11f x:0 0 0 2024-03-04T16:04:53.297 INFO:tasks.workunit.client.1.smithi152.stdout:6/925: symlink d8/d92/d59/d7d/d104/d136/l140 0 2024-03-04T16:04:53.298 INFO:tasks.workunit.client.1.smithi152.stdout:6/926: chown d8/d10/l2d 206296360 1 2024-03-04T16:04:53.301 INFO:tasks.workunit.client.1.smithi152.stdout:1/904: symlink d19/d3b/d6e/d118/l11b 0 2024-03-04T16:04:53.304 INFO:tasks.workunit.client.1.smithi152.stdout:9/850: creat d2/d13/d23/d42/db1/f120 x:0 0 0 2024-03-04T16:04:53.310 INFO:tasks.workunit.client.1.smithi152.stdout:1/905: mknod d19/d4b/d53/d57/c11c 0 2024-03-04T16:04:53.311 INFO:tasks.workunit.client.1.smithi152.stdout:8/988: creat d2/d3/d21/dac/d12c/f13e x:0 0 0 2024-03-04T16:04:53.313 INFO:tasks.workunit.client.1.smithi152.stdout:9/851: unlink d2/c3e 0 2024-03-04T16:04:53.314 INFO:tasks.workunit.client.1.smithi152.stdout:2/838: getdents da/df/d2e/d51/d11c/d80/d82/da6 0 2024-03-04T16:04:53.317 INFO:tasks.workunit.client.1.smithi152.stdout:6/927: dread d8/d52/d58/dd0/f11d [0,4194304] 0 2024-03-04T16:04:53.317 INFO:tasks.workunit.client.1.smithi152.stdout:9/852: fsync d2/d13/d23/d66/d7d/fb9 0 2024-03-04T16:04:53.318 INFO:tasks.workunit.client.1.smithi152.stdout:9/853: dread - d2/d13/d23/d42/db1/f120 zero size 2024-03-04T16:04:53.320 INFO:tasks.workunit.client.1.smithi152.stdout:2/839: symlink da/df/d2e/d51/d11c/d80/d82/da8/l121 0 2024-03-04T16:04:53.321 INFO:tasks.workunit.client.1.smithi152.stdout:8/989: mknod d2/d3/d21/dc0/c13f 0 2024-03-04T16:04:53.324 INFO:tasks.workunit.client.1.smithi152.stdout:2/840: dread - da/df/d58/da2/db5/fd2 zero size 2024-03-04T16:04:53.326 INFO:tasks.workunit.client.1.smithi152.stdout:8/990: read d2/d3/d21/d2d/d52/d68/f72 [1164742,9655] 0 2024-03-04T16:04:53.327 INFO:tasks.workunit.client.1.smithi152.stdout:2/841: mkdir da/df/d2e/d51/d11c/d47/d122 0 2024-03-04T16:04:53.333 INFO:tasks.workunit.client.1.smithi152.stdout:2/842: mkdir da/d5f/dbc/d123 0 2024-03-04T16:04:53.335 INFO:tasks.workunit.client.1.smithi152.stdout:2/843: dread - da/d5f/dbc/fcc zero size 2024-03-04T16:04:53.339 INFO:tasks.workunit.client.1.smithi152.stdout:2/844: creat da/df/d35/d92/f124 x:0 0 0 2024-03-04T16:04:53.341 INFO:tasks.workunit.client.1.smithi152.stdout:2/845: creat da/d5f/db0/f125 x:0 0 0 2024-03-04T16:04:53.344 INFO:tasks.workunit.client.1.smithi152.stdout:3/860: write d6/d10/d2d/f3b [1365974,59528] 0 2024-03-04T16:04:53.349 INFO:tasks.workunit.client.1.smithi152.stdout:3/861: mkdir d6/d60/d8e/dc0/db5/d129 0 2024-03-04T16:04:53.353 INFO:tasks.workunit.client.1.smithi152.stdout:2/846: dwrite da/df/d58/d6e/fc8 [0,4194304] 0 2024-03-04T16:04:53.361 INFO:tasks.workunit.client.1.smithi152.stdout:1/906: write d19/f1a [3930,76525] 0 2024-03-04T16:04:53.366 INFO:tasks.workunit.client.1.smithi152.stdout:7/824: dwrite d14/d21/d47/d40/f43 [0,4194304] 0 2024-03-04T16:04:53.366 INFO:tasks.workunit.client.1.smithi152.stdout:1/907: readlink l12 0 2024-03-04T16:04:53.369 INFO:tasks.workunit.client.1.smithi152.stdout:0/887: dwrite d9/d1b/f2a [0,4194304] 0 2024-03-04T16:04:53.370 INFO:tasks.workunit.client.1.smithi152.stdout:3/862: truncate d6/fd4 491372 0 2024-03-04T16:04:53.377 INFO:tasks.workunit.client.1.smithi152.stdout:5/845: dwrite d6/d8/d1f/d3d/d6f/d70/f75 [0,4194304] 0 2024-03-04T16:04:53.377 INFO:tasks.workunit.client.1.smithi152.stdout:3/863: fsync d6/f11d 0 2024-03-04T16:04:53.380 INFO:tasks.workunit.client.1.smithi152.stdout:4/750: dwrite d8/d2f/d6f/d90/fbd [0,4194304] 0 2024-03-04T16:04:53.385 INFO:tasks.workunit.client.1.smithi152.stdout:7/825: mkdir d14/db9/dc9/d83/db7/df6 0 2024-03-04T16:04:53.389 INFO:tasks.workunit.client.1.smithi152.stdout:8/991: dwrite d2/d3/d15/d1c/f56 [0,4194304] 0 2024-03-04T16:04:53.389 INFO:tasks.workunit.client.1.smithi152.stdout:9/854: dwrite d2/d13/d23/d71/d80/fa8 [0,4194304] 0 2024-03-04T16:04:53.390 INFO:tasks.workunit.client.1.smithi152.stdout:0/888: truncate d9/d1a/d7c/f7e 1027053 0 2024-03-04T16:04:53.392 INFO:tasks.workunit.client.1.smithi152.stdout:7/826: truncate d14/d21/d47/fca 197003 0 2024-03-04T16:04:53.393 INFO:tasks.workunit.client.1.smithi152.stdout:4/751: mknod d8/d4e/d8b/cfc 0 2024-03-04T16:04:53.393 INFO:tasks.workunit.client.1.smithi152.stdout:5/846: truncate d6/d8/f74 758417 0 2024-03-04T16:04:53.394 INFO:tasks.workunit.client.1.smithi152.stdout:2/847: getdents da/df/d2e/d51/d11c/d47/d4f/d11a/d8e 0 2024-03-04T16:04:53.397 INFO:tasks.workunit.client.1.smithi152.stdout:9/855: creat d2/d13/d35/f121 x:0 0 0 2024-03-04T16:04:53.398 INFO:tasks.workunit.client.1.smithi152.stdout:4/752: symlink d8/d9/d7a/lfd 0 2024-03-04T16:04:53.399 INFO:tasks.workunit.client.1.smithi152.stdout:0/889: truncate d9/f26 904141 0 2024-03-04T16:04:53.401 INFO:tasks.workunit.client.1.smithi152.stdout:5/847: creat d6/d8/d1f/d3d/da4/dc9/f112 x:0 0 0 2024-03-04T16:04:53.401 INFO:tasks.workunit.client.1.smithi152.stdout:2/848: symlink da/df/d58/l126 0 2024-03-04T16:04:53.405 INFO:tasks.workunit.client.1.smithi152.stdout:7/827: dwrite d14/d69/d8e/f9f [4194304,4194304] 0 2024-03-04T16:04:53.409 INFO:tasks.workunit.client.1.smithi152.stdout:9/856: getdents d2/d13/d35/d10c 0 2024-03-04T16:04:53.412 INFO:tasks.workunit.client.1.smithi152.stdout:0/890: mknod d9/d1b/d54/d80/da8/d120/c125 0 2024-03-04T16:04:53.414 INFO:tasks.workunit.client.1.smithi152.stdout:2/849: mknod da/df/d2e/d51/d11c/d47/d4f/d11a/c127 0 2024-03-04T16:04:53.415 INFO:tasks.workunit.client.1.smithi152.stdout:4/753: link d8/d9/d24/lad d8/d9/dc1/lfe 0 2024-03-04T16:04:53.416 INFO:tasks.workunit.client.1.smithi152.stdout:9/857: symlink d2/d13/dad/l122 0 2024-03-04T16:04:53.416 INFO:tasks.workunit.client.1.smithi152.stdout:5/848: dwrite d6/d8/d5a/d64/f109 [0,4194304] 0 2024-03-04T16:04:53.420 INFO:tasks.workunit.client.1.smithi152.stdout:5/849: chown d6/d8/d5a/d64/d76/fef 829024 1 2024-03-04T16:04:53.421 INFO:tasks.workunit.client.1.smithi152.stdout:5/850: chown d6/d8/d1f/d3d/f85 828258 1 2024-03-04T16:04:53.425 INFO:tasks.workunit.client.1.smithi152.stdout:1/908: dread d19/d20/d2f/d6b/f4a [0,4194304] 0 2024-03-04T16:04:53.426 INFO:tasks.workunit.client.1.smithi152.stdout:0/891: fsync d9/d2c/d2e/f32 0 2024-03-04T16:04:53.429 INFO:tasks.workunit.client.1.smithi152.stdout:0/892: dread - d9/d2c/d2e/ffb zero size 2024-03-04T16:04:53.429 INFO:tasks.workunit.client.1.smithi152.stdout:9/858: dwrite d2/d13/d23/d71/fbb [0,4194304] 0 2024-03-04T16:04:53.432 INFO:tasks.workunit.client.1.smithi152.stdout:5/851: mkdir d6/d8/d5a/d90/d113 0 2024-03-04T16:04:53.432 INFO:tasks.workunit.client.1.smithi152.stdout:9/859: chown d2/d13/d23/d71 5 1 2024-03-04T16:04:53.434 INFO:tasks.workunit.client.1.smithi152.stdout:9/860: write d2/d13/d38/d79/fe6 [747685,48230] 0 2024-03-04T16:04:53.436 INFO:tasks.workunit.client.1.smithi152.stdout:4/754: symlink d8/d4e/lff 0 2024-03-04T16:04:53.437 INFO:tasks.workunit.client.1.smithi152.stdout:1/909: creat d19/d4b/d53/d57/d62/f11d x:0 0 0 2024-03-04T16:04:53.438 INFO:tasks.workunit.client.1.smithi152.stdout:3/864: write d6/d60/d8e/dc0/db5/fcc [1167990,109256] 0 2024-03-04T16:04:53.442 INFO:tasks.workunit.client.1.smithi152.stdout:5/852: creat d6/d8/d1f/d3d/d6f/d70/ddb/dde/f114 x:0 0 0 2024-03-04T16:04:53.446 INFO:tasks.workunit.client.1.smithi152.stdout:5/853: fsync d6/d8/d5a/d64/d76/d9f/dc7/dcb/f100 0 2024-03-04T16:04:53.446 INFO:tasks.workunit.client.1.smithi152.stdout:9/861: creat d2/d13/d23/d66/f123 x:0 0 0 2024-03-04T16:04:53.448 INFO:tasks.workunit.client.1.smithi152.stdout:0/893: symlink d9/d1b/d54/l126 0 2024-03-04T16:04:53.450 INFO:tasks.workunit.client.1.smithi152.stdout:3/865: stat d6/cc9 0 2024-03-04T16:04:53.456 INFO:tasks.workunit.client.1.smithi152.stdout:1/910: rename d19/d20/d2f/l90 to d19/d4b/d53/l11e 0 2024-03-04T16:04:53.458 INFO:tasks.workunit.client.1.smithi152.stdout:0/894: dwrite d9/d1b/d54/d80/f104 [0,4194304] 0 2024-03-04T16:04:53.460 INFO:tasks.workunit.client.1.smithi152.stdout:7/828: dwrite d14/d21/f25 [0,4194304] 0 2024-03-04T16:04:53.464 INFO:tasks.workunit.client.1.smithi152.stdout:9/862: unlink d2/d13/f18 0 2024-03-04T16:04:53.464 INFO:tasks.workunit.client.1.smithi152.stdout:4/755: mkdir d8/d2f/d100 0 2024-03-04T16:04:53.467 INFO:tasks.workunit.client.1.smithi152.stdout:2/850: write da/d2c/f4b [956011,92030] 0 2024-03-04T16:04:53.471 INFO:tasks.workunit.client.1.smithi152.stdout:5/854: creat d6/d8/d5a/d64/d76/d9f/db0/dd8/f115 x:0 0 0 2024-03-04T16:04:53.478 INFO:tasks.workunit.client.1.smithi152.stdout:2/851: dread da/df/d58/da2/db5/f102 [0,4194304] 0 2024-03-04T16:04:53.478 INFO:tasks.workunit.client.1.smithi152.stdout:4/756: creat d8/d9/da2/f101 x:0 0 0 2024-03-04T16:04:53.481 INFO:tasks.workunit.client.1.smithi152.stdout:3/866: truncate d6/d10/d2d/f4b 305957 0 2024-03-04T16:04:53.489 INFO:tasks.workunit.client.1.smithi152.stdout:9/863: dread d2/d6e/f10d [0,4194304] 0 2024-03-04T16:04:53.492 INFO:tasks.workunit.client.1.smithi152.stdout:4/757: mknod d8/d2f/d6f/deb/def/c102 0 2024-03-04T16:04:53.495 INFO:tasks.workunit.client.1.smithi152.stdout:3/867: readlink d6/l22 0 2024-03-04T16:04:53.496 INFO:tasks.workunit.client.1.smithi152.stdout:1/911: dwrite d19/d20/d2f/d65/fbd [0,4194304] 0 2024-03-04T16:04:53.496 INFO:tasks.workunit.client.1.smithi152.stdout:0/895: creat d9/d1b/d54/f127 x:0 0 0 2024-03-04T16:04:53.497 INFO:tasks.workunit.client.1.smithi152.stdout:2/852: symlink da/df/d2e/d51/d11c/l128 0 2024-03-04T16:04:53.498 INFO:tasks.workunit.client.1.smithi152.stdout:1/912: stat d19/d75/da8 0 2024-03-04T16:04:53.498 INFO:tasks.workunit.client.1.smithi152.stdout:2/853: chown da/df/d2e 216 1 2024-03-04T16:04:53.499 INFO:tasks.workunit.client.1.smithi152.stdout:9/864: unlink d2/d6e/cfb 0 2024-03-04T16:04:53.500 INFO:tasks.workunit.client.1.smithi152.stdout:0/896: write d9/d2c/d2e/d7f/d9d/fa7 [4683908,74757] 0 2024-03-04T16:04:53.501 INFO:tasks.workunit.client.1.smithi152.stdout:7/829: dwrite d14/d21/d47/d36/f68 [0,4194304] 0 2024-03-04T16:04:53.503 INFO:tasks.workunit.client.1.smithi152.stdout:7/830: read d14/d21/d47/d36/f68 [638510,114827] 0 2024-03-04T16:04:53.513 INFO:tasks.workunit.client.1.smithi152.stdout:2/854: dwrite da/df/d2e/d51/d11c/d80/fd3 [0,4194304] 0 2024-03-04T16:04:53.514 INFO:tasks.workunit.client.1.smithi152.stdout:2/855: dread - da/d5f/dbc/ffb zero size 2024-03-04T16:04:53.521 INFO:tasks.workunit.client.1.smithi152.stdout:9/865: symlink d2/d13/dad/l124 0 2024-03-04T16:04:53.522 INFO:tasks.workunit.client.1.smithi152.stdout:0/897: chown d9/d1b/d3c/d84/d56/d5b/c66 682 1 2024-03-04T16:04:53.524 INFO:tasks.workunit.client.1.smithi152.stdout:7/831: truncate d14/d21/d47/d36/f82 4564171 0 2024-03-04T16:04:53.527 INFO:tasks.workunit.client.1.smithi152.stdout:4/758: unlink d8/f25 0 2024-03-04T16:04:53.528 INFO:tasks.workunit.client.1.smithi152.stdout:5/855: getdents d6/d8/d1f/d3d/d6f 0 2024-03-04T16:04:53.530 INFO:tasks.workunit.client.1.smithi152.stdout:2/856: read da/f1c [3131884,3699] 0 2024-03-04T16:04:53.530 INFO:tasks.workunit.client.1.smithi152.stdout:9/866: mkdir d2/d13/d35/d10c/d125 0 2024-03-04T16:04:53.533 INFO:tasks.workunit.client.1.smithi152.stdout:4/759: write d8/d2f/d6f/d90/fbd [328380,58857] 0 2024-03-04T16:04:53.534 INFO:tasks.workunit.client.1.smithi152.stdout:3/868: dwrite d6/d14/f90 [4194304,4194304] 0 2024-03-04T16:04:53.535 INFO:tasks.workunit.client.1.smithi152.stdout:0/898: dwrite d9/d2c/d2e/f110 [4194304,4194304] 0 2024-03-04T16:04:53.540 INFO:tasks.workunit.client.1.smithi152.stdout:0/899: dread d9/f102 [0,4194304] 0 2024-03-04T16:04:53.541 INFO:tasks.workunit.client.1.smithi152.stdout:0/900: fdatasync d9/fd0 0 2024-03-04T16:04:53.544 INFO:tasks.workunit.client.1.smithi152.stdout:9/867: fsync d2/f8d 0 2024-03-04T16:04:53.551 INFO:tasks.workunit.client.1.smithi152.stdout:7/832: mkdir d14/d21/d47/d36/da7/dbc/de6/df7 0 2024-03-04T16:04:53.554 INFO:tasks.workunit.client.1.smithi152.stdout:2/857: dwrite da/f77 [0,4194304] 0 2024-03-04T16:04:53.557 INFO:tasks.workunit.client.1.smithi152.stdout:3/869: mknod d6/d14/d41/d49/d84/de2/d108/c12a 0 2024-03-04T16:04:53.560 INFO:tasks.workunit.client.1.smithi152.stdout:1/913: getdents d19/d20/d2f/d6b 0 2024-03-04T16:04:53.562 INFO:tasks.workunit.client.1.smithi152.stdout:0/901: mkdir d9/d1b/d54/d69/d6c/dbf/dfd/d128 0 2024-03-04T16:04:53.567 INFO:tasks.workunit.client.1.smithi152.stdout:9/868: symlink d2/d13/dea/l126 0 2024-03-04T16:04:53.571 INFO:tasks.workunit.client.1.smithi152.stdout:2/858: creat da/df/d58/f129 x:0 0 0 2024-03-04T16:04:53.575 INFO:tasks.workunit.client.1.smithi152.stdout:9/869: dwrite d2/d13/d2e/f76 [0,4194304] 0 2024-03-04T16:04:53.577 INFO:tasks.workunit.client.1.smithi152.stdout:4/760: creat d8/d4e/f103 x:0 0 0 2024-03-04T16:04:53.580 INFO:tasks.workunit.client.1.smithi152.stdout:5/856: link d6/d8/c58 d6/d8/d5a/d82/c116 0 2024-03-04T16:04:53.583 INFO:tasks.workunit.client.1.smithi152.stdout:9/870: symlink d2/d6e/l127 0 2024-03-04T16:04:53.585 INFO:tasks.workunit.client.1.smithi152.stdout:1/914: write d19/d20/f9b [1496268,76241] 0 2024-03-04T16:04:53.585 INFO:tasks.workunit.client.1.smithi152.stdout:2/859: truncate da/df/d2e/d51/d11c/d80/d82/ff2 119553 0 2024-03-04T16:04:53.589 INFO:tasks.workunit.client.1.smithi152.stdout:3/870: link d6/d14/ffa d6/d14/d41/d49/d84/de2/de9/f12b 0 2024-03-04T16:04:53.598 INFO:tasks.workunit.client.1.smithi152.stdout:4/761: mkdir d8/d89/db3/d104 0 2024-03-04T16:04:53.598 INFO:tasks.workunit.client.1.smithi152.stdout:9/871: dread d2/d13/d23/d42/db1/fb2 [0,4194304] 0 2024-03-04T16:04:53.599 INFO:tasks.workunit.client.1.smithi152.stdout:3/871: rmdir d6/d14/d93/def 39 2024-03-04T16:04:53.599 INFO:tasks.workunit.client.1.smithi152.stdout:9/872: stat d2/d13/d96/l10a 0 2024-03-04T16:04:53.601 INFO:tasks.workunit.client.1.smithi152.stdout:7/833: dread d14/d21/d47/d36/f82 [0,4194304] 0 2024-03-04T16:04:53.608 INFO:tasks.workunit.client.1.smithi152.stdout:4/762: mkdir d8/d3d/d105 0 2024-03-04T16:04:53.610 INFO:tasks.workunit.client.1.smithi152.stdout:2/860: dread da/d5f/f7a [0,4194304] 0 2024-03-04T16:04:53.610 INFO:tasks.workunit.client.1.smithi152.stdout:3/872: rmdir d6/d60 39 2024-03-04T16:04:53.614 INFO:tasks.workunit.client.1.smithi152.stdout:9/873: chown d2/c5 2440347 1 2024-03-04T16:04:53.618 INFO:tasks.workunit.client.1.smithi152.stdout:5/857: dwrite d6/f38 [0,4194304] 0 2024-03-04T16:04:53.620 INFO:tasks.workunit.client.1.smithi152.stdout:1/915: write d19/d20/d2f/d65/ffa [343181,42945] 0 2024-03-04T16:04:53.622 INFO:tasks.workunit.client.1.smithi152.stdout:2/861: truncate da/d5f/dbc/fcc 851919 0 2024-03-04T16:04:53.623 INFO:tasks.workunit.client.1.smithi152.stdout:2/862: write da/d5f/db0/f125 [771975,111624] 0 2024-03-04T16:04:53.624 INFO:tasks.workunit.client.1.smithi152.stdout:3/873: dread d6/d14/d41/d49/f5a [0,4194304] 0 2024-03-04T16:04:53.627 INFO:tasks.workunit.client.1.smithi152.stdout:4/763: creat d8/d2f/d100/f106 x:0 0 0 2024-03-04T16:04:53.629 INFO:tasks.workunit.client.1.smithi152.stdout:9/874: dwrite d2/d13/fd7 [0,4194304] 0 2024-03-04T16:04:53.673 INFO:tasks.workunit.client.1.smithi152.stdout:1/916: creat d19/d3b/d6e/dc6/f11f x:0 0 0 2024-03-04T16:04:53.676 INFO:tasks.workunit.client.1.smithi152.stdout:2/863: fdatasync da/df/f37 0 2024-03-04T16:04:53.677 INFO:tasks.workunit.client.1.smithi152.stdout:4/764: creat d8/d9/d79/f107 x:0 0 0 2024-03-04T16:04:53.678 INFO:tasks.workunit.client.1.smithi152.stdout:4/765: stat d8/d2f/d6f/d84/ccf 0 2024-03-04T16:04:53.680 INFO:tasks.workunit.client.1.smithi152.stdout:9/875: symlink d2/d13/d2e/dcd/l128 0 2024-03-04T16:04:53.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/876: chown d2/d13/d2e/c62 52698 1 2024-03-04T16:04:53.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/877: stat d2/d13/d35/d10c/d125 0 2024-03-04T16:04:53.683 INFO:tasks.workunit.client.1.smithi152.stdout:7/834: link d14/c75 d14/db9/dc9/d83/db7/cf8 0 2024-03-04T16:04:53.687 INFO:tasks.workunit.client.1.smithi152.stdout:3/874: creat d6/d60/d8e/df0/f12c x:0 0 0 2024-03-04T16:04:53.689 INFO:tasks.workunit.client.1.smithi152.stdout:5/858: dwrite d6/d8/d5a/d64/d76/fa1 [0,4194304] 0 2024-03-04T16:04:53.691 INFO:tasks.workunit.client.1.smithi152.stdout:5/859: dread - d6/d8/d1f/d3d/d6f/d70/d98/de7/ff5 zero size 2024-03-04T16:04:53.691 INFO:tasks.workunit.client.1.smithi152.stdout:1/917: read d19/d4b/d53/f55 [3310126,28149] 0 2024-03-04T16:04:53.692 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:53 smithi116 ceph-mon[29120]: pgmap v8: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 32 MiB/s rd, 78 MiB/s wr, 197 op/s 2024-03-04T16:04:53.692 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:53 smithi116 ceph-mon[29120]: mgrmap e30: smithi116.jwdhop(active, since 10s), standbys: smithi152.aawieo 2024-03-04T16:04:53.692 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:53 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi152.aawieo", "id": "smithi152.aawieo"}]: dispatch 2024-03-04T16:04:53.692 INFO:tasks.workunit.client.1.smithi152.stdout:1/918: stat d19/d4b/d53/d57/dd6 0 2024-03-04T16:04:53.697 INFO:tasks.workunit.client.1.smithi152.stdout:4/766: creat d8/d9/da2/f108 x:0 0 0 2024-03-04T16:04:53.697 INFO:tasks.workunit.client.1.smithi152.stdout:7/835: mkdir d14/d21/d47/d36/df9 0 2024-03-04T16:04:53.700 INFO:tasks.workunit.client.1.smithi152.stdout:3/875: read d6/d14/d58/f6b [6881360,53418] 0 2024-03-04T16:04:53.701 INFO:tasks.workunit.client.1.smithi152.stdout:5/860: truncate d6/d8/d5a/d64/f99 562969 0 2024-03-04T16:04:53.703 INFO:tasks.workunit.client.1.smithi152.stdout:2/864: symlink da/d5f/db0/d114/l12a 0 2024-03-04T16:04:53.704 INFO:tasks.workunit.client.1.smithi152.stdout:2/865: fsync da/d5f/d85/d8c/f104 0 2024-03-04T16:04:53.705 INFO:tasks.workunit.client.1.smithi152.stdout:4/767: rename d8/d58/fe0 to d8/d2f/d6f/d84/f109 0 2024-03-04T16:04:53.709 INFO:tasks.workunit.client.1.smithi152.stdout:1/919: creat d19/d4b/d53/d108/f120 x:0 0 0 2024-03-04T16:04:53.709 INFO:tasks.workunit.client.1.smithi152.stdout:3/876: dwrite d6/d14/d93/f11c [0,4194304] 0 2024-03-04T16:04:53.710 INFO:tasks.workunit.client.1.smithi152.stdout:5/861: creat d6/d8/d1f/d3d/d6f/d70/ddb/dde/f117 x:0 0 0 2024-03-04T16:04:53.711 INFO:tasks.workunit.client.1.smithi152.stdout:3/877: read d6/f85 [119332,97984] 0 2024-03-04T16:04:53.721 INFO:tasks.workunit.client.1.smithi152.stdout:1/920: creat d19/d4b/d53/f121 x:0 0 0 2024-03-04T16:04:53.724 INFO:tasks.workunit.client.1.smithi152.stdout:4/768: dread d8/d3d/f4c [0,4194304] 0 2024-03-04T16:04:53.725 INFO:tasks.workunit.client.1.smithi152.stdout:4/769: stat d8/d78/l9f 0 2024-03-04T16:04:53.726 INFO:tasks.workunit.client.1.smithi152.stdout:5/862: symlink d6/d8/d5a/d64/d76/d9f/dc7/l118 0 2024-03-04T16:04:53.727 INFO:tasks.workunit.client.1.smithi152.stdout:7/836: write d14/db9/dc9/d83/f9c [1009775,72946] 0 2024-03-04T16:04:53.728 INFO:tasks.workunit.client.1.smithi152.stdout:2/866: dread da/df/d2e/d51/d11c/d47/f5a [0,4194304] 0 2024-03-04T16:04:53.728 INFO:tasks.workunit.client.1.smithi152.stdout:3/878: fsync d6/ff1 0 2024-03-04T16:04:53.729 INFO:tasks.workunit.client.1.smithi152.stdout:3/879: chown d6/d10/d2d/f36 80181 1 2024-03-04T16:04:53.731 INFO:tasks.workunit.client.1.smithi152.stdout:1/921: creat d19/d4b/d53/d57/d62/d82/d111/f122 x:0 0 0 2024-03-04T16:04:53.731 INFO:tasks.workunit.client.1.smithi152.stdout:3/880: fdatasync d6/d14/f90 0 2024-03-04T16:04:53.731 INFO:tasks.workunit.client.1.smithi152.stdout:1/922: chown d19/d20/d2f/d6b/l25 61 1 2024-03-04T16:04:53.732 INFO:tasks.workunit.client.1.smithi152.stdout:5/863: symlink d6/d8/d1f/l119 0 2024-03-04T16:04:53.735 INFO:tasks.workunit.client.1.smithi152.stdout:4/770: mknod d8/d9/da2/dd3/de9/c10a 0 2024-03-04T16:04:53.736 INFO:tasks.workunit.client.1.smithi152.stdout:3/881: creat d6/d60/d8e/dc0/db5/f12d x:0 0 0 2024-03-04T16:04:53.736 INFO:tasks.workunit.client.1.smithi152.stdout:4/771: write d8/d9/dc1/fdf [121483,54927] 0 2024-03-04T16:04:53.743 INFO:tasks.workunit.client.1.smithi152.stdout:4/772: rename d8/d4e/lff to d8/d58/ded/l10b 0 2024-03-04T16:04:53.744 INFO:tasks.workunit.client.1.smithi152.stdout:4/773: write d8/d4e/f64 [2856798,126755] 0 2024-03-04T16:04:53.745 INFO:tasks.workunit.client.1.smithi152.stdout:2/867: write da/df/d58/da2/db5/fe6 [9010097,31373] 0 2024-03-04T16:04:53.747 INFO:tasks.workunit.client.1.smithi152.stdout:7/837: creat d14/d69/ffa x:0 0 0 2024-03-04T16:04:53.748 INFO:tasks.workunit.client.1.smithi152.stdout:5/864: mknod d6/d8/d5a/d64/d76/c11a 0 2024-03-04T16:04:53.749 INFO:tasks.workunit.client.1.smithi152.stdout:1/923: write d19/d20/d2f/f76 [913478,17222] 0 2024-03-04T16:04:53.750 INFO:tasks.workunit.client.1.smithi152.stdout:1/924: write d19/d3b/d6e/d97/fad [2749241,125064] 0 2024-03-04T16:04:53.752 INFO:tasks.workunit.client.1.smithi152.stdout:4/774: symlink d8/d9/da2/l10c 0 2024-03-04T16:04:53.753 INFO:tasks.workunit.client.1.smithi152.stdout:4/775: chown d8/d9 27577 1 2024-03-04T16:04:53.754 INFO:tasks.workunit.client.1.smithi152.stdout:4/776: chown d8/d9/da2/dd3/de9 11 1 2024-03-04T16:04:53.754 INFO:tasks.workunit.client.1.smithi152.stdout:7/838: rmdir d14/d9d 39 2024-03-04T16:04:53.756 INFO:tasks.workunit.client.1.smithi152.stdout:7/839: fdatasync d14/db9/dc9/fe4 0 2024-03-04T16:04:53.756 INFO:tasks.workunit.client.1.smithi152.stdout:5/865: symlink d6/d8/d5a/d90/l11b 0 2024-03-04T16:04:53.762 INFO:tasks.workunit.client.1.smithi152.stdout:3/882: link d6/d10/c63 d6/d60/d8e/dc0/da1/de0/c12e 0 2024-03-04T16:04:53.764 INFO:tasks.workunit.client.1.smithi152.stdout:4/777: symlink d8/d2f/d100/l10d 0 2024-03-04T16:04:53.764 INFO:tasks.workunit.client.1.smithi152.stdout:2/868: dread da/d2c/f96 [0,4194304] 0 2024-03-04T16:04:53.765 INFO:tasks.workunit.client.1.smithi152.stdout:4/778: dread d8/f30 [0,4194304] 0 2024-03-04T16:04:53.768 INFO:tasks.workunit.client.1.smithi152.stdout:2/869: dread da/d2c/d5e/f103 [0,4194304] 0 2024-03-04T16:04:53.768 INFO:tasks.workunit.client.1.smithi152.stdout:5/866: mknod d6/d8/d5a/d64/d76/d9f/dc7/dcb/dd1/c11c 0 2024-03-04T16:04:53.770 INFO:tasks.workunit.client.1.smithi152.stdout:1/925: mknod d19/c123 0 2024-03-04T16:04:53.773 INFO:tasks.workunit.client.1.smithi152.stdout:3/883: rmdir d6/d14 39 2024-03-04T16:04:53.779 INFO:tasks.workunit.client.1.smithi152.stdout:5/867: creat d6/d8/d5a/d90/dd6/f11d x:0 0 0 2024-03-04T16:04:53.779 INFO:tasks.workunit.client.1.smithi152.stdout:1/926: truncate d19/d4b/d53/d57/f60 2618940 0 2024-03-04T16:04:53.784 INFO:tasks.workunit.client.1.smithi152.stdout:7/840: rmdir d14/d69/d8e/de7 0 2024-03-04T16:04:53.785 INFO:tasks.workunit.client.1.smithi152.stdout:2/870: mkdir da/df/d2e/d51/d8d/d9a/d116/d12b 0 2024-03-04T16:04:53.789 INFO:tasks.workunit.client.1.smithi152.stdout:3/884: chown d6/d14/d58/cf5 73978621 1 2024-03-04T16:04:53.793 INFO:tasks.workunit.client.1.smithi152.stdout:2/871: unlink da/df/d2e/d51/d11c/l128 0 2024-03-04T16:04:53.793 INFO:tasks.workunit.client.1.smithi152.stdout:4/779: write d8/f3e [594213,113911] 0 2024-03-04T16:04:53.794 INFO:tasks.workunit.client.1.smithi152.stdout:1/927: mknod d19/d4b/c124 0 2024-03-04T16:04:53.797 INFO:tasks.workunit.client.1.smithi152.stdout:3/885: mknod d6/d60/d8e/dc0/db5/de6/c12f 0 2024-03-04T16:04:53.798 INFO:tasks.workunit.client.1.smithi152.stdout:7/841: truncate d14/d21/fc7 663631 0 2024-03-04T16:04:53.799 INFO:tasks.workunit.client.1.smithi152.stdout:3/886: read d6/d10/d2d/f3b [779989,36353] 0 2024-03-04T16:04:53.800 INFO:tasks.workunit.client.1.smithi152.stdout:7/842: write d14/d21/d47/d36/f8b [4025624,88414] 0 2024-03-04T16:04:53.801 INFO:tasks.workunit.client.1.smithi152.stdout:7/843: write d14/db9/dc9/fe4 [1557909,111540] 0 2024-03-04T16:04:53.802 INFO:tasks.workunit.client.1.smithi152.stdout:4/780: mkdir d8/d9/dc1/d10e 0 2024-03-04T16:04:53.804 INFO:tasks.workunit.client.1.smithi152.stdout:7/844: read d14/d21/d47/d36/f73 [3943858,97478] 0 2024-03-04T16:04:53.805 INFO:tasks.workunit.client.1.smithi152.stdout:5/868: dread d6/d8/f9a [0,4194304] 0 2024-03-04T16:04:53.807 INFO:tasks.workunit.client.1.smithi152.stdout:3/887: truncate d6/d14/d93/db7/fc5 191388 0 2024-03-04T16:04:53.808 INFO:tasks.workunit.client.1.smithi152.stdout:2/872: mknod da/df/d2e/c12c 0 2024-03-04T16:04:53.808 INFO:tasks.workunit.client.1.smithi152.stdout:3/888: fsync d6/d14/f8d 0 2024-03-04T16:04:53.809 INFO:tasks.workunit.client.1.smithi152.stdout:3/889: chown d6/d60/d8e/df0 112 1 2024-03-04T16:04:53.811 INFO:tasks.workunit.client.1.smithi152.stdout:7/845: chown d14/d69/ce9 870 1 2024-03-04T16:04:53.820 INFO:tasks.workunit.client.1.smithi152.stdout:3/890: mknod d6/d14/d41/d49/d84/de2/de9/d104/c130 0 2024-03-04T16:04:53.822 INFO:tasks.workunit.client.1.smithi152.stdout:7/846: creat d14/d21/d47/d36/dcf/ddf/ffb x:0 0 0 2024-03-04T16:04:53.826 INFO:tasks.workunit.client.1.smithi152.stdout:1/928: getdents d19/d20/d2f/d5d 0 2024-03-04T16:04:53.829 INFO:tasks.workunit.client.1.smithi152.stdout:1/929: chown d19/d20/d2f/d65/def/d110 1948027425 1 2024-03-04T16:04:53.830 INFO:tasks.workunit.client.1.smithi152.stdout:7/847: creat d14/d21/d47/d36/da7/dbc/de6/ffc x:0 0 0 2024-03-04T16:04:53.830 INFO:tasks.workunit.client.1.smithi152.stdout:4/781: dwrite d8/d9/d24/d26/f2e [0,4194304] 0 2024-03-04T16:04:53.833 INFO:tasks.workunit.client.1.smithi152.stdout:2/873: creat da/d5f/f12d x:0 0 0 2024-03-04T16:04:53.833 INFO:tasks.workunit.client.1.smithi152.stdout:5/869: dwrite d6/f39 [0,4194304] 0 2024-03-04T16:04:53.835 INFO:tasks.workunit.client.1.smithi152.stdout:4/782: creat d8/d9/da2/f10f x:0 0 0 2024-03-04T16:04:53.839 INFO:tasks.workunit.client.1.smithi152.stdout:3/891: creat d6/d60/d8e/dc0/f131 x:0 0 0 2024-03-04T16:04:53.841 INFO:tasks.workunit.client.1.smithi152.stdout:5/870: dread d6/d8/d1f/d88/fac [0,4194304] 0 2024-03-04T16:04:53.844 INFO:tasks.workunit.client.1.smithi152.stdout:3/892: symlink d6/d60/d8e/df0/l132 0 2024-03-04T16:04:53.848 INFO:tasks.workunit.client.1.smithi152.stdout:4/783: mkdir d8/d89/db3/d104/d110 0 2024-03-04T16:04:53.856 INFO:tasks.workunit.client.1.smithi152.stdout:7/848: dwrite d14/d21/d47/f97 [0,4194304] 0 2024-03-04T16:04:53.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:53 smithi152 ceph-mon[36876]: pgmap v8: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 32 MiB/s rd, 78 MiB/s wr, 197 op/s 2024-03-04T16:04:53.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:53 smithi152 ceph-mon[36876]: mgrmap e30: smithi116.jwdhop(active, since 10s), standbys: smithi152.aawieo 2024-03-04T16:04:53.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:53 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi152.aawieo", "id": "smithi152.aawieo"}]: dispatch 2024-03-04T16:04:53.861 INFO:tasks.workunit.client.1.smithi152.stdout:7/849: mkdir d14/db9/dfd 0 2024-03-04T16:04:53.863 INFO:tasks.workunit.client.1.smithi152.stdout:7/850: chown d14/d21/d47/d40/fc4 11420 1 2024-03-04T16:04:53.864 INFO:tasks.workunit.client.1.smithi152.stdout:5/871: dwrite d6/d8/f2c [0,4194304] 0 2024-03-04T16:04:53.864 INFO:tasks.workunit.client.1.smithi152.stdout:3/893: write d6/d14/d58/f7e [3458691,11291] 0 2024-03-04T16:04:53.868 INFO:tasks.workunit.client.1.smithi152.stdout:3/894: chown d6/d14/d58/f6b 4625 1 2024-03-04T16:04:53.872 INFO:tasks.workunit.client.1.smithi152.stdout:3/895: readlink d6/d60/l7f 0 2024-03-04T16:04:53.872 INFO:tasks.workunit.client.1.smithi152.stdout:4/784: dwrite d8/d9/d24/d26/d42/fa7 [0,4194304] 0 2024-03-04T16:04:53.873 INFO:tasks.workunit.client.1.smithi152.stdout:7/851: unlink d14/d21/d47/lef 0 2024-03-04T16:04:53.880 INFO:tasks.workunit.client.1.smithi152.stdout:5/872: dwrite d6/d8/d5a/d90/daf/fc6 [0,4194304] 0 2024-03-04T16:04:53.884 INFO:tasks.workunit.client.1.smithi152.stdout:4/785: dwrite d8/d2f/d7c/fdb [0,4194304] 0 2024-03-04T16:04:53.970 INFO:tasks.workunit.client.1.smithi152.stdout:3/896: truncate d6/d14/d41/d49/d84/de2/de9/d104/f113 687009 0 2024-03-04T16:04:53.971 INFO:tasks.workunit.client.1.smithi152.stdout:5/873: fdatasync d6/d8/d1f/d3d/d6f/d70/d98/de7/ff5 0 2024-03-04T16:04:53.972 INFO:tasks.workunit.client.1.smithi152.stdout:4/786: dread - d8/d9/da2/f101 zero size 2024-03-04T16:04:53.973 INFO:tasks.workunit.client.1.smithi152.stdout:5/874: chown d6/d8/d5a/d64/d76/fef 758 1 2024-03-04T16:04:53.974 INFO:tasks.workunit.client.1.smithi152.stdout:5/875: readlink d6/l47 0 2024-03-04T16:04:53.979 INFO:tasks.workunit.client.1.smithi152.stdout:5/876: symlink d6/d8/d1f/d3d/d6f/dbb/l11e 0 2024-03-04T16:04:53.981 INFO:tasks.workunit.client.1.smithi152.stdout:7/852: dread d14/d21/d31/f7c [0,4194304] 0 2024-03-04T16:04:53.983 INFO:tasks.workunit.client.1.smithi152.stdout:3/897: getdents d6/d10/d2d 0 2024-03-04T16:04:53.984 INFO:tasks.workunit.client.1.smithi152.stdout:3/898: chown d6/d60/d8a/ce8 36692 1 2024-03-04T16:04:53.984 INFO:tasks.workunit.client.1.smithi152.stdout:5/877: creat d6/d8/d5a/d90/dd6/f11f x:0 0 0 2024-03-04T16:04:53.986 INFO:tasks.workunit.client.1.smithi152.stdout:7/853: rename d14/d71/fd7 to d14/db9/dc9/d83/db7/ffe 0 2024-03-04T16:04:53.997 INFO:tasks.workunit.client.1.smithi152.stdout:3/899: write d6/d14/f7d [2275868,66890] 0 2024-03-04T16:04:53.999 INFO:tasks.workunit.client.1.smithi152.stdout:5/878: write d6/d8/f41 [2750825,122239] 0 2024-03-04T16:04:54.001 INFO:tasks.workunit.client.1.smithi152.stdout:4/787: dwrite d8/d78/f82 [4194304,4194304] 0 2024-03-04T16:04:54.003 INFO:tasks.workunit.client.1.smithi152.stdout:7/854: write d14/f2c [1303089,93008] 0 2024-03-04T16:04:54.005 INFO:tasks.workunit.client.1.smithi152.stdout:7/855: write d14/d69/ffa [637757,110997] 0 2024-03-04T16:04:54.005 INFO:tasks.workunit.client.1.smithi152.stdout:3/900: creat d6/d14/d58/de4/f133 x:0 0 0 2024-03-04T16:04:54.006 INFO:tasks.workunit.client.1.smithi152.stdout:3/901: chown d6/d10/l66 0 1 2024-03-04T16:04:54.011 INFO:tasks.workunit.client.1.smithi152.stdout:7/856: mknod d14/d21/d47/d36/dcf/cff 0 2024-03-04T16:04:54.012 INFO:tasks.workunit.client.1.smithi152.stdout:3/902: creat d6/db1/f134 x:0 0 0 2024-03-04T16:04:54.013 INFO:tasks.workunit.client.1.smithi152.stdout:5/879: creat d6/d8/d1f/d3d/d6f/d102/f120 x:0 0 0 2024-03-04T16:04:54.014 INFO:tasks.workunit.client.1.smithi152.stdout:4/788: symlink d8/d4e/dba/dcc/df2/l111 0 2024-03-04T16:04:54.015 INFO:tasks.workunit.client.1.smithi152.stdout:7/857: creat d14/d69/dd1/f100 x:0 0 0 2024-03-04T16:04:54.016 INFO:tasks.workunit.client.1.smithi152.stdout:7/858: write d14/d21/d47/fed [288287,106508] 0 2024-03-04T16:04:54.016 INFO:tasks.workunit.client.1.smithi152.stdout:7/859: fsync d14/d71/f98 0 2024-03-04T16:04:54.018 INFO:tasks.workunit.client.1.smithi152.stdout:3/903: fsync d6/d14/dd7/fdb 0 2024-03-04T16:04:54.019 INFO:tasks.workunit.client.1.smithi152.stdout:5/880: creat d6/d8/d1f/d3d/d6f/dbb/f121 x:0 0 0 2024-03-04T16:04:54.024 INFO:tasks.workunit.client.1.smithi152.stdout:6/928: sync 2024-03-04T16:04:54.024 INFO:tasks.workunit.client.1.smithi152.stdout:6/929: chown d8/d92/d75 75130269 1 2024-03-04T16:04:54.024 INFO:tasks.workunit.client.1.smithi152.stdout:7/860: fsync d14/d69/fdb 0 2024-03-04T16:04:54.025 INFO:tasks.workunit.client.1.smithi152.stdout:7/861: chown d14/d69/d8e/fa5 3284188 1 2024-03-04T16:04:54.026 INFO:tasks.workunit.client.1.smithi152.stdout:3/904: creat d6/d60/d8e/dc0/da1/de0/f135 x:0 0 0 2024-03-04T16:04:54.030 INFO:tasks.workunit.client.1.smithi152.stdout:6/930: rename d8/d92/d59/f7f to d8/d92/d59/f141 0 2024-03-04T16:04:54.033 INFO:tasks.workunit.client.1.smithi152.stdout:5/881: write d6/d8/fa3 [259033,101709] 0 2024-03-04T16:04:54.036 INFO:tasks.workunit.client.1.smithi152.stdout:4/789: dwrite d8/d2f/d6f/f73 [0,4194304] 0 2024-03-04T16:04:54.037 INFO:tasks.workunit.client.1.smithi152.stdout:5/882: read d6/d8/d5a/d64/d76/fef [13294449,96780] 0 2024-03-04T16:04:54.039 INFO:tasks.workunit.client.1.smithi152.stdout:5/883: truncate d6/d8/d1f/d88/f110 996722 0 2024-03-04T16:04:54.042 INFO:tasks.workunit.client.1.smithi152.stdout:4/790: dwrite d8/d9/fde [0,4194304] 0 2024-03-04T16:04:54.045 INFO:tasks.workunit.client.1.smithi152.stdout:3/905: write d6/d60/d8e/dc0/f88 [164883,96613] 0 2024-03-04T16:04:54.046 INFO:tasks.workunit.client.1.smithi152.stdout:3/906: readlink d6/d60/d8a/lb2 0 2024-03-04T16:04:54.047 INFO:tasks.workunit.client.1.smithi152.stdout:3/907: write d6/d14/f90 [9133123,117778] 0 2024-03-04T16:04:54.051 INFO:tasks.workunit.client.1.smithi152.stdout:5/884: rename d6/d8/d1f/d3d/f85 to d6/d8/d5a/d64/d76/d9f/db0/dd8/f122 0 2024-03-04T16:04:54.051 INFO:tasks.workunit.client.1.smithi152.stdout:6/931: dread d8/f1f [0,4194304] 0 2024-03-04T16:04:54.052 INFO:tasks.workunit.client.1.smithi152.stdout:4/791: creat d8/d2f/d6f/deb/def/f112 x:0 0 0 2024-03-04T16:04:54.062 INFO:tasks.workunit.client.1.smithi152.stdout:3/908: write d6/d60/d8e/dc0/fc1 [5039560,30796] 0 2024-03-04T16:04:54.063 INFO:tasks.workunit.client.1.smithi152.stdout:5/885: getdents d6/d8/d1f/d88 0 2024-03-04T16:04:54.063 INFO:tasks.workunit.client.1.smithi152.stdout:3/909: read - d6/d14/d58/de4/f133 zero size 2024-03-04T16:04:54.065 INFO:tasks.workunit.client.1.smithi152.stdout:6/932: write d8/d92/d59/fef [3255961,121429] 0 2024-03-04T16:04:54.065 INFO:tasks.workunit.client.1.smithi152.stdout:3/910: rmdir d6/d10/d2d 39 2024-03-04T16:04:54.074 INFO:tasks.workunit.client.1.smithi152.stdout:4/792: dread d8/f16 [4194304,4194304] 0 2024-03-04T16:04:54.080 INFO:tasks.workunit.client.1.smithi152.stdout:4/793: link d8/d9/d24/f74 d8/d89/db3/d104/f113 0 2024-03-04T16:04:54.085 INFO:tasks.workunit.client.1.smithi152.stdout:6/933: dwrite d8/d23/d3a/f3f [0,4194304] 0 2024-03-04T16:04:54.090 INFO:tasks.workunit.client.1.smithi152.stdout:4/794: getdents d8/d9/d24/da0 0 2024-03-04T16:04:54.091 INFO:tasks.workunit.client.1.smithi152.stdout:6/934: symlink d8/d92/d59/d61/d57/l142 0 2024-03-04T16:04:54.097 INFO:tasks.workunit.client.1.smithi152.stdout:6/935: mknod d8/d92/d6f/c143 0 2024-03-04T16:04:54.100 INFO:tasks.workunit.client.1.smithi152.stdout:6/936: creat d8/d42/f144 x:0 0 0 2024-03-04T16:04:54.112 INFO:tasks.workunit.client.1.smithi152.stdout:4/795: dwrite d8/d2f/d7c/fb1 [0,4194304] 0 2024-03-04T16:04:54.116 INFO:tasks.workunit.client.1.smithi152.stdout:4/796: creat d8/d9/d7a/f114 x:0 0 0 2024-03-04T16:04:54.117 INFO:tasks.workunit.client.1.smithi152.stdout:4/797: chown d8/d9/da2/f108 40590932 1 2024-03-04T16:04:54.123 INFO:tasks.workunit.client.1.smithi152.stdout:6/937: write d8/d92/d59/fc8 [365961,50892] 0 2024-03-04T16:04:54.126 INFO:tasks.workunit.client.1.smithi152.stdout:6/938: fdatasync d8/d92/d59/d61/fcb 0 2024-03-04T16:04:54.132 INFO:tasks.workunit.client.1.smithi152.stdout:6/939: link d8/d92/d59/d61/f127 d8/d42/dde/f145 0 2024-03-04T16:04:54.134 INFO:tasks.workunit.client.1.smithi152.stdout:6/940: mknod d8/d92/d59/d7d/d91/c146 0 2024-03-04T16:04:54.140 INFO:tasks.workunit.client.1.smithi152.stdout:6/941: creat d8/d92/d59/d7d/d104/d136/f147 x:0 0 0 2024-03-04T16:04:54.147 INFO:tasks.workunit.client.1.smithi152.stdout:6/942: mkdir d8/d23/d148 0 2024-03-04T16:04:54.154 INFO:tasks.workunit.client.1.smithi152.stdout:6/943: read d8/d2a/f34 [3337407,92778] 0 2024-03-04T16:04:54.165 INFO:tasks.workunit.client.1.smithi152.stdout:4/798: write d8/d89/db3/d104/f113 [4249089,53644] 0 2024-03-04T16:04:54.169 INFO:tasks.workunit.client.1.smithi152.stdout:4/799: mkdir d8/d9/d24/d26/d42/d115 0 2024-03-04T16:04:54.175 INFO:tasks.workunit.client.1.smithi152.stdout:4/800: dwrite d8/d9/da2/f10f [0,4194304] 0 2024-03-04T16:04:54.179 INFO:tasks.workunit.client.1.smithi152.stdout:4/801: creat d8/d89/db3/f116 x:0 0 0 2024-03-04T16:04:54.236 INFO:tasks.workunit.client.1.smithi152.stdout:8/992: sync 2024-03-04T16:04:54.244 INFO:tasks.workunit.client.1.smithi152.stdout:6/944: dwrite d8/d92/d59/d61/f78 [0,4194304] 0 2024-03-04T16:04:54.262 INFO:tasks.workunit.client.1.smithi152.stdout:4/802: creat d8/d9/d24/d26/f117 x:0 0 0 2024-03-04T16:04:54.263 INFO:tasks.workunit.client.1.smithi152.stdout:8/993: creat d2/d3/d15/d1c/d84/d87/d5d/f140 x:0 0 0 2024-03-04T16:04:54.265 INFO:tasks.workunit.client.1.smithi152.stdout:6/945: mkdir d8/d92/d149 0 2024-03-04T16:04:54.266 INFO:tasks.workunit.client.1.smithi152.stdout:4/803: creat d8/d58/ded/f118 x:0 0 0 2024-03-04T16:04:54.266 INFO:tasks.workunit.client.1.smithi152.stdout:4/804: chown d8/d3d/f44 6654513 1 2024-03-04T16:04:54.267 INFO:tasks.workunit.client.1.smithi152.stdout:4/805: chown d8/d2f/d6f/d90/fd9 727723 1 2024-03-04T16:04:54.268 INFO:tasks.workunit.client.1.smithi152.stdout:8/994: unlink d2/cb0 0 2024-03-04T16:04:54.269 INFO:tasks.workunit.client.1.smithi152.stdout:6/946: chown d8/d23/fe9 2897 1 2024-03-04T16:04:54.269 INFO:tasks.workunit.client.1.smithi152.stdout:4/806: symlink d8/d4e/dba/dcc/l119 0 2024-03-04T16:04:54.272 INFO:tasks.workunit.client.1.smithi152.stdout:4/807: unlink d8/d4e/d8b/cfc 0 2024-03-04T16:04:54.273 INFO:tasks.workunit.client.1.smithi152.stdout:4/808: dread - d8/d2f/d6f/d90/fd9 zero size 2024-03-04T16:04:54.274 INFO:tasks.workunit.client.1.smithi152.stdout:8/995: unlink d2/d3/d15/f6a 0 2024-03-04T16:04:54.276 INFO:tasks.workunit.client.1.smithi152.stdout:4/809: symlink d8/d2f/d6f/deb/def/l11a 0 2024-03-04T16:04:54.318 INFO:tasks.workunit.client.1.smithi152.stdout:4/810: write d8/f30 [1237770,20417] 0 2024-03-04T16:04:54.323 INFO:tasks.workunit.client.1.smithi152.stdout:8/996: dwrite d2/d75/d89/dd/d22/d26/f102 [0,4194304] 0 2024-03-04T16:04:54.326 INFO:tasks.workunit.client.1.smithi152.stdout:4/811: rename d8/d9/ff4 to d8/d9/d24/da0/f11b 0 2024-03-04T16:04:54.332 INFO:tasks.workunit.client.1.smithi152.stdout:8/997: rename d2/d3/d21/d2d/d31/l77 to d2/d3/d15/d1c/l141 0 2024-03-04T16:04:54.340 INFO:tasks.workunit.client.1.smithi152.stdout:8/998: unlink d2/d75/d89/f45 0 2024-03-04T16:04:54.344 INFO:tasks.workunit.client.1.smithi152.stdout:4/812: link d8/d3d/l9d d8/d2f/d7c/d87/l11c 0 2024-03-04T16:04:54.345 INFO:tasks.workunit.client.1.smithi152.stdout:8/999: mkdir d2/d3/d15/d1c/d84/d87/d142 0 2024-03-04T16:04:54.346 INFO:tasks.workunit.client.1.smithi152.stdout:4/813: mknod d8/d9/d7a/c11d 0 2024-03-04T16:04:54.361 INFO:tasks.workunit.client.1.smithi152.stdout:4/814: dread f7 [0,4194304] 0 2024-03-04T16:04:54.363 INFO:tasks.workunit.client.1.smithi152.stdout:4/815: rename d8/d9/c43 to d8/d78/c11e 0 2024-03-04T16:04:54.364 INFO:tasks.workunit.client.1.smithi152.stdout:4/816: fsync d8/f16 0 2024-03-04T16:04:54.366 INFO:tasks.workunit.client.1.smithi152.stdout:4/817: creat d8/d58/ded/f11f x:0 0 0 2024-03-04T16:04:54.367 INFO:tasks.workunit.client.1.smithi152.stdout:4/818: dread - d8/d58/ded/f11f zero size 2024-03-04T16:04:54.368 INFO:tasks.workunit.client.1.smithi152.stdout:0/902: sync 2024-03-04T16:04:54.372 INFO:tasks.workunit.client.1.smithi152.stdout:0/903: mkdir d9/d1b/d3c/d84/dde/d129 0 2024-03-04T16:04:54.374 INFO:tasks.workunit.client.1.smithi152.stdout:4/819: creat d8/f120 x:0 0 0 2024-03-04T16:04:54.378 INFO:tasks.workunit.client.1.smithi152.stdout:4/820: mkdir d8/d2f/d6f/d90/d121 0 2024-03-04T16:04:54.381 INFO:tasks.workunit.client.1.smithi152.stdout:4/821: rename d8/f3e to d8/d2f/d7c/d87/df0/f122 0 2024-03-04T16:04:54.384 INFO:tasks.workunit.client.1.smithi152.stdout:4/822: unlink d8/d4e/f64 0 2024-03-04T16:04:54.390 INFO:tasks.workunit.client.1.smithi152.stdout:4/823: rename d8/d9/d24/d26/c8a to d8/d2f/c123 0 2024-03-04T16:04:54.392 INFO:tasks.workunit.client.1.smithi152.stdout:0/904: dwrite d9/d1b/d54/d69/d95/fb5 [4194304,4194304] 0 2024-03-04T16:04:54.393 INFO:tasks.workunit.client.1.smithi152.stdout:4/824: creat d8/d9/da2/dd3/de6/f124 x:0 0 0 2024-03-04T16:04:54.397 INFO:tasks.workunit.client.1.smithi152.stdout:0/905: creat d9/d1a/d7c/f12a x:0 0 0 2024-03-04T16:04:54.407 INFO:tasks.workunit.client.1.smithi152.stdout:0/906: getdents d9/d2c/d2e/d7f/d9d/de8/df4 0 2024-03-04T16:04:54.407 INFO:tasks.workunit.client.1.smithi152.stdout:4/825: write d8/d9/f9e [23230,85410] 0 2024-03-04T16:04:54.407 INFO:tasks.workunit.client.1.smithi152.stdout:0/907: stat d9/d2c/d2e/d7f/f117 0 2024-03-04T16:04:54.409 INFO:tasks.workunit.client.1.smithi152.stdout:4/826: fdatasync d8/d9/f9e 0 2024-03-04T16:04:54.409 INFO:tasks.workunit.client.1.smithi152.stdout:0/908: read - d9/d1a/d7c/ffa zero size 2024-03-04T16:04:54.417 INFO:tasks.workunit.client.1.smithi152.stdout:0/909: write d9/d1a/d7c/fc6 [422525,64193] 0 2024-03-04T16:04:54.427 INFO:tasks.workunit.client.1.smithi152.stdout:4/827: dread d8/d9/fae [0,4194304] 0 2024-03-04T16:04:54.428 INFO:tasks.workunit.client.1.smithi152.stdout:4/828: write d8/f120 [63940,23138] 0 2024-03-04T16:04:54.431 INFO:tasks.workunit.client.1.smithi152.stdout:4/829: unlink d8/d2f/d6f/deb/cf3 0 2024-03-04T16:04:54.432 INFO:tasks.workunit.client.1.smithi152.stdout:4/830: fsync d8/d2f/d6f/d90/fca 0 2024-03-04T16:04:54.434 INFO:tasks.workunit.client.1.smithi152.stdout:0/910: dread d9/f45 [0,4194304] 0 2024-03-04T16:04:54.437 INFO:tasks.workunit.client.1.smithi152.stdout:0/911: rmdir d9/d1b/d3c/d84/d56/d5b 39 2024-03-04T16:04:54.440 INFO:tasks.workunit.client.1.smithi152.stdout:4/831: dread d8/d4e/f57 [0,4194304] 0 2024-03-04T16:04:54.442 INFO:tasks.workunit.client.1.smithi152.stdout:4/832: fdatasync d8/d9/d24/fd2 0 2024-03-04T16:04:54.445 INFO:tasks.workunit.client.1.smithi152.stdout:4/833: fsync d8/d9/d24/f4d 0 2024-03-04T16:04:54.452 INFO:tasks.workunit.client.1.smithi152.stdout:4/834: write d8/d2f/d6f/d90/fd6 [1002078,115244] 0 2024-03-04T16:04:54.454 INFO:tasks.workunit.client.1.smithi152.stdout:4/835: mknod d8/d2f/d6f/d90/d121/c125 0 2024-03-04T16:04:54.456 INFO:tasks.workunit.client.1.smithi152.stdout:4/836: truncate d8/d9/d24/d26/d42/f81 2863721 0 2024-03-04T16:04:54.458 INFO:tasks.workunit.client.1.smithi152.stdout:4/837: rename d8/d2f/l35 to d8/d9/d79/l126 0 2024-03-04T16:04:54.460 INFO:tasks.workunit.client.1.smithi152.stdout:4/838: mknod d8/d4e/dba/dcc/c127 0 2024-03-04T16:04:54.462 INFO:tasks.workunit.client.1.smithi152.stdout:4/839: fdatasync d8/d9/d24/d26/d42/f81 0 2024-03-04T16:04:54.480 INFO:tasks.workunit.client.1.smithi152.stdout:4/840: dread d8/d89/db3/d104/f113 [0,4194304] 0 2024-03-04T16:04:54.484 INFO:tasks.workunit.client.1.smithi152.stdout:4/841: mknod d8/d9/dc1/d10e/c128 0 2024-03-04T16:04:54.485 INFO:tasks.workunit.client.1.smithi152.stdout:4/842: creat d8/d9/d7a/f129 x:0 0 0 2024-03-04T16:04:54.580 INFO:tasks.workunit.client.1.smithi152.stdout:9/878: sync 2024-03-04T16:04:54.581 INFO:tasks.workunit.client.1.smithi152.stdout:9/879: fsync d2/d13/dad/dd2/d111/f94 0 2024-03-04T16:04:54.591 INFO:tasks.workunit.client.1.smithi152.stdout:9/880: write d2/d13/d35/f7f [726764,21057] 0 2024-03-04T16:04:54.594 INFO:tasks.workunit.client.1.smithi152.stdout:9/881: link d2/d13/dad/dd2/d111/c8e d2/d13/d38/df9/c129 0 2024-03-04T16:04:54.600 INFO:tasks.workunit.client.1.smithi152.stdout:9/882: truncate d2/d13/d23/d71/da3/fab 4667445 0 2024-03-04T16:04:54.606 INFO:tasks.workunit.client.1.smithi152.stdout:9/883: dwrite d2/d13/d23/d66/f123 [0,4194304] 0 2024-03-04T16:04:54.610 INFO:tasks.workunit.client.1.smithi152.stdout:9/884: symlink d2/d13/d23/d42/l12a 0 2024-03-04T16:04:54.611 INFO:tasks.workunit.client.1.smithi152.stdout:9/885: fdatasync d2/d13/dad/dd2/d111/fc0 0 2024-03-04T16:04:54.615 INFO:tasks.workunit.client.1.smithi152.stdout:9/886: readlink d2/d13/d38/lc3 0 2024-03-04T16:04:54.615 INFO:tasks.workunit.client.1.smithi152.stdout:9/887: chown d2/d13/dad/dd2/d111/d69/ce3 16 1 2024-03-04T16:04:54.622 INFO:tasks.workunit.client.1.smithi152.stdout:9/888: dwrite d2/d13/d2e/f44 [0,4194304] 0 2024-03-04T16:04:54.623 INFO:tasks.workunit.client.1.smithi152.stdout:9/889: fsync d2/d13/dad/dd2/fe9 0 2024-03-04T16:04:54.643 INFO:tasks.workunit.client.1.smithi152.stdout:9/890: dread d2/f8d [0,4194304] 0 2024-03-04T16:04:54.645 INFO:tasks.workunit.client.1.smithi152.stdout:9/891: creat d2/d13/d35/d10c/f12b x:0 0 0 2024-03-04T16:04:54.651 INFO:tasks.workunit.client.1.smithi152.stdout:9/892: write d2/d13/d23/d66/d7d/f9a [4159462,42507] 0 2024-03-04T16:04:54.659 INFO:tasks.workunit.client.1.smithi152.stdout:9/893: write d2/d13/d2e/f4f [1282194,101085] 0 2024-03-04T16:04:54.660 INFO:tasks.workunit.client.1.smithi152.stdout:9/894: write d2/d13/dad/dd2/fd6 [1666811,83169] 0 2024-03-04T16:04:54.676 INFO:tasks.workunit.client.1.smithi152.stdout:9/895: dread d2/d13/d23/f33 [4194304,4194304] 0 2024-03-04T16:04:54.679 INFO:tasks.workunit.client.1.smithi152.stdout:9/896: creat d2/d13/d23/d71/d80/f12c x:0 0 0 2024-03-04T16:04:54.681 INFO:tasks.workunit.client.1.smithi152.stdout:9/897: creat d2/d13/d23/d42/db1/f12d x:0 0 0 2024-03-04T16:04:54.688 INFO:tasks.workunit.client.1.smithi152.stdout:9/898: write d2/d13/d35/f5d [1207033,99477] 0 2024-03-04T16:04:54.690 INFO:tasks.workunit.client.1.smithi152.stdout:9/899: chown d2/d13/d23/d66/d7d/da7/c100 32 1 2024-03-04T16:04:54.691 INFO:tasks.workunit.client.1.smithi152.stdout:9/900: chown d2/d13/dad/dd2/d111/l68 17 1 2024-03-04T16:04:54.692 INFO:tasks.workunit.client.1.smithi152.stdout:9/901: chown d2/d13/d35/lca 47 1 2024-03-04T16:04:54.694 INFO:tasks.workunit.client.1.smithi152.stdout:9/902: dread - d2/d13/dad/dd2/d111/d69/f97 zero size 2024-03-04T16:04:54.695 INFO:tasks.workunit.client.1.smithi152.stdout:9/903: write d2/d13/d23/f11d [4116,98926] 0 2024-03-04T16:04:54.697 INFO:tasks.workunit.client.1.smithi152.stdout:9/904: mkdir d2/d6e/d12e 0 2024-03-04T16:04:54.703 INFO:tasks.workunit.client.1.smithi152.stdout:9/905: getdents d2/d13/d23/d71/da3 0 2024-03-04T16:04:54.705 INFO:tasks.workunit.client.1.smithi152.stdout:9/906: mkdir d2/d13/d23/d71/da3/d12f 0 2024-03-04T16:04:54.707 INFO:tasks.workunit.client.1.smithi152.stdout:9/907: rmdir d2/d13/dea 39 2024-03-04T16:04:54.710 INFO:tasks.workunit.client.1.smithi152.stdout:9/908: creat d2/d13/dad/f130 x:0 0 0 2024-03-04T16:04:54.712 INFO:tasks.workunit.client.1.smithi152.stdout:9/909: symlink d2/d13/dad/ddf/l131 0 2024-03-04T16:04:54.717 INFO:tasks.workunit.client.1.smithi152.stdout:9/910: truncate d2/d13/d23/d71/d102/f10b 96065 0 2024-03-04T16:04:54.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:54 smithi116 ceph-mon[29120]: pgmap v9: 65 pgs: 65 active+clean; 2.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 46 MiB/s rd, 109 MiB/s wr, 286 op/s 2024-03-04T16:04:54.729 INFO:tasks.workunit.client.1.smithi152.stdout:9/911: write d2/f28 [1405684,58170] 0 2024-03-04T16:04:54.734 INFO:tasks.workunit.client.1.smithi152.stdout:9/912: creat d2/d13/f132 x:0 0 0 2024-03-04T16:04:54.735 INFO:tasks.workunit.client.1.smithi152.stdout:9/913: write d2/d13/d35/f7c [4376326,60015] 0 2024-03-04T16:04:54.750 INFO:tasks.workunit.client.1.smithi152.stdout:1/930: sync 2024-03-04T16:04:54.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:54 smithi152 ceph-mon[36876]: pgmap v9: 65 pgs: 65 active+clean; 2.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 46 MiB/s rd, 109 MiB/s wr, 286 op/s 2024-03-04T16:04:54.904 INFO:tasks.workunit.client.1.smithi152.stdout:2/874: sync 2024-03-04T16:04:54.905 INFO:tasks.workunit.client.1.smithi152.stdout:2/875: dread - da/df/d2e/d51/d11c/d81/ffc zero size 2024-03-04T16:04:54.906 INFO:tasks.workunit.client.1.smithi152.stdout:2/876: chown da/df/d2e/f90 14742524 1 2024-03-04T16:04:54.906 INFO:tasks.workunit.client.1.smithi152.stdout:2/877: dread - da/df/d2e/d51/d11c/d80/d82/da6/fe5 zero size 2024-03-04T16:04:54.913 INFO:tasks.workunit.client.1.smithi152.stdout:2/878: write da/df/d2e/d51/d11c/d80/d82/da8/f5c [3164710,92965] 0 2024-03-04T16:04:54.920 INFO:tasks.workunit.client.1.smithi152.stdout:2/879: dwrite da/df/d2e/d51/f112 [0,4194304] 0 2024-03-04T16:04:54.923 INFO:tasks.workunit.client.1.smithi152.stdout:2/880: creat da/df/d2e/d51/d11c/d80/dd0/f12e x:0 0 0 2024-03-04T16:04:54.925 INFO:tasks.workunit.client.1.smithi152.stdout:2/881: truncate da/df/d2e/d51/d11c/d47/fe0 365701 0 2024-03-04T16:04:54.925 INFO:tasks.workunit.client.1.smithi152.stdout:2/882: chown da/f1a 5085881 1 2024-03-04T16:04:54.929 INFO:tasks.workunit.client.1.smithi152.stdout:2/883: symlink da/df/d2e/d51/d11c/d47/l12f 0 2024-03-04T16:04:54.930 INFO:tasks.workunit.client.1.smithi152.stdout:2/884: chown da/df/d2e/c12c 25373 1 2024-03-04T16:04:54.935 INFO:tasks.workunit.client.1.smithi152.stdout:2/885: link da/df/d2e/l98 da/df/d58/da2/db5/l130 0 2024-03-04T16:04:54.936 INFO:tasks.workunit.client.1.smithi152.stdout:2/886: readlink da/d2c/lb8 0 2024-03-04T16:04:54.939 INFO:tasks.workunit.client.1.smithi152.stdout:2/887: symlink da/df/l131 0 2024-03-04T16:04:54.943 INFO:tasks.workunit.client.1.smithi152.stdout:2/888: mkdir da/df/d2e/d51/d11c/d47/d122/d132 0 2024-03-04T16:04:54.957 INFO:tasks.workunit.client.1.smithi152.stdout:2/889: dread da/df/d2e/f9c [0,4194304] 0 2024-03-04T16:04:54.959 INFO:tasks.workunit.client.1.smithi152.stdout:2/890: fsync da/d2c/d5e/fe3 0 2024-03-04T16:04:54.962 INFO:tasks.workunit.client.1.smithi152.stdout:2/891: dread da/d5f/db0/f125 [0,4194304] 0 2024-03-04T16:04:54.964 INFO:tasks.workunit.client.1.smithi152.stdout:2/892: fdatasync da/d2c/d3f/f86 0 2024-03-04T16:04:54.967 INFO:tasks.workunit.client.1.smithi152.stdout:2/893: creat da/df/d2e/d51/d8d/d9a/d116/f133 x:0 0 0 2024-03-04T16:04:54.968 INFO:tasks.workunit.client.1.smithi152.stdout:2/894: readlink da/df/d2e/ldb 0 2024-03-04T16:04:54.971 INFO:tasks.workunit.client.1.smithi152.stdout:7/862: sync 2024-03-04T16:04:54.971 INFO:tasks.workunit.client.1.smithi152.stdout:2/895: getdents da/df/d2e/d51/d11c/d80/d82/da6 0 2024-03-04T16:04:54.973 INFO:tasks.workunit.client.1.smithi152.stdout:7/863: mknod d14/d21/d47/d36/da7/dbc/de6/c101 0 2024-03-04T16:04:54.974 INFO:tasks.workunit.client.1.smithi152.stdout:2/896: mknod da/df/d2e/d51/d11c/d47/d4f/d11a/c134 0 2024-03-04T16:04:54.975 INFO:tasks.workunit.client.1.smithi152.stdout:2/897: write da/df/d2e/d51/d11c/d80/dd0/f12e [447529,74276] 0 2024-03-04T16:04:54.980 INFO:tasks.workunit.client.1.smithi152.stdout:7/864: getdents d14/d69/d8e 0 2024-03-04T16:04:54.983 INFO:tasks.workunit.client.1.smithi152.stdout:2/898: link da/df/d2e/d51/cf4 da/df/d35/d92/c135 0 2024-03-04T16:04:54.984 INFO:tasks.workunit.client.1.smithi152.stdout:7/865: creat d14/d21/d47/d36/da7/dbc/de6/f102 x:0 0 0 2024-03-04T16:04:54.984 INFO:tasks.workunit.client.1.smithi152.stdout:7/866: write d14/d69/dd1/fd6 [1029165,12287] 0 2024-03-04T16:04:54.987 INFO:tasks.workunit.client.1.smithi152.stdout:7/867: symlink d14/d21/d47/d36/dcf/ddf/l103 0 2024-03-04T16:04:54.988 INFO:tasks.workunit.client.1.smithi152.stdout:7/868: read d14/d21/d31/f33 [1440873,69783] 0 2024-03-04T16:04:54.995 INFO:tasks.workunit.client.1.smithi152.stdout:7/869: creat d14/d21/f104 x:0 0 0 2024-03-04T16:04:54.996 INFO:tasks.workunit.client.1.smithi152.stdout:7/870: stat d14/d69/dd1 0 2024-03-04T16:04:55.001 INFO:tasks.workunit.client.1.smithi152.stdout:7/871: dwrite d14/db9/dc9/f4e [0,4194304] 0 2024-03-04T16:04:55.002 INFO:tasks.workunit.client.1.smithi152.stdout:7/872: chown d14/d21/d47/d36 23600984 1 2024-03-04T16:04:55.009 INFO:tasks.workunit.client.1.smithi152.stdout:2/899: dwrite da/df/d2e/d51/d11c/d80/d82/ff2 [0,4194304] 0 2024-03-04T16:04:55.015 INFO:tasks.workunit.client.1.smithi152.stdout:7/873: write d14/d21/d48/f4b [3160919,102294] 0 2024-03-04T16:04:55.020 INFO:tasks.workunit.client.1.smithi152.stdout:2/900: dwrite da/df/d2e/d51/d11c/d47/fe0 [0,4194304] 0 2024-03-04T16:04:55.020 INFO:tasks.workunit.client.1.smithi152.stdout:3/911: sync 2024-03-04T16:04:55.020 INFO:tasks.workunit.client.1.smithi152.stdout:6/947: sync 2024-03-04T16:04:55.021 INFO:tasks.workunit.client.1.smithi152.stdout:5/886: sync 2024-03-04T16:04:55.021 INFO:tasks.workunit.client.1.smithi152.stdout:4/843: sync 2024-03-04T16:04:55.021 INFO:tasks.workunit.client.1.smithi152.stdout:9/914: sync 2024-03-04T16:04:55.021 INFO:tasks.workunit.client.1.smithi152.stdout:0/912: sync 2024-03-04T16:04:55.021 INFO:tasks.workunit.client.1.smithi152.stdout:1/931: sync 2024-03-04T16:04:55.022 INFO:tasks.workunit.client.1.smithi152.stdout:3/912: chown d6/d14/d58/de4/d111 79351461 1 2024-03-04T16:04:55.023 INFO:tasks.workunit.client.1.smithi152.stdout:6/948: chown d8/d52/d58/c10b 2358288 1 2024-03-04T16:04:55.023 INFO:tasks.workunit.client.1.smithi152.stdout:9/915: write d2/d13/f52 [590384,21285] 0 2024-03-04T16:04:55.024 INFO:tasks.workunit.client.1.smithi152.stdout:2/901: chown da/d5f/cc5 255 1 2024-03-04T16:04:55.027 INFO:tasks.workunit.client.1.smithi152.stdout:5/887: truncate d6/d8/d5a/d82/f104 321998 0 2024-03-04T16:04:55.027 INFO:tasks.workunit.client.1.smithi152.stdout:4/844: symlink d8/d9/da2/dd3/de9/l12a 0 2024-03-04T16:04:55.027 INFO:tasks.workunit.client.1.smithi152.stdout:3/913: write d6/ff1 [155030,127031] 0 2024-03-04T16:04:55.031 INFO:tasks.workunit.client.1.smithi152.stdout:7/874: write d14/d21/d47/f34 [257279,30535] 0 2024-03-04T16:04:55.033 INFO:tasks.workunit.client.1.smithi152.stdout:1/932: mkdir d19/d4b/d53/d57/d125 0 2024-03-04T16:04:55.035 INFO:tasks.workunit.client.1.smithi152.stdout:1/933: chown d19/d20/d2f/d6b/f40 15126 1 2024-03-04T16:04:55.036 INFO:tasks.workunit.client.1.smithi152.stdout:5/888: dwrite d6/d8/fa3 [0,4194304] 0 2024-03-04T16:04:55.037 INFO:tasks.workunit.client.1.smithi152.stdout:2/902: rmdir da/df/d35/d92 39 2024-03-04T16:04:55.038 INFO:tasks.workunit.client.1.smithi152.stdout:1/934: write d19/d3b/f92 [1827538,33195] 0 2024-03-04T16:04:55.041 INFO:tasks.workunit.client.1.smithi152.stdout:4/845: symlink d8/d89/db3/d104/l12b 0 2024-03-04T16:04:55.045 INFO:tasks.workunit.client.1.smithi152.stdout:3/914: truncate d6/d14/d41/d49/f5a 1254168 0 2024-03-04T16:04:55.048 INFO:tasks.workunit.client.1.smithi152.stdout:7/875: rmdir d14/d21/d48 39 2024-03-04T16:04:55.052 INFO:tasks.workunit.client.1.smithi152.stdout:5/889: truncate d6/fd 5564786 0 2024-03-04T16:04:55.061 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-04T16:04:55.062 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-04T16:04:55.063 INFO:tasks.workunit.client.1.smithi152.stdout:6/949: dwrite d8/d42/dde/f145 [0,4194304] 0 2024-03-04T16:04:55.063 INFO:tasks.workunit.client.1.smithi152.stdout:1/935: unlink d19/d3b/d6e/fdf 0 2024-03-04T16:04:55.063 INFO:tasks.workunit.client.1.smithi152.stdout:9/916: write d2/d13/d23/d71/da3/fab [2234279,35245] 0 2024-03-04T16:04:55.063 INFO:tasks.workunit.client.1.smithi152.stdout:4/846: rmdir d8/d2f 39 2024-03-04T16:04:55.063 INFO:tasks.workunit.client.1.smithi152.stdout:4/847: chown d8/d9/d24/d26/d42/l6e 6171 1 2024-03-04T16:04:55.064 INFO:tasks.workunit.client.1.smithi152.stdout:9/917: write d2/d13/d35/d10c/ffd [546908,105512] 0 2024-03-04T16:04:55.064 INFO:tasks.workunit.client.1.smithi152.stdout:0/913: rmdir d9/d1b/d54/d69/d6c/dbf/dfd/d128 0 2024-03-04T16:04:55.064 INFO:tasks.workunit.client.1.smithi152.stdout:7/876: mkdir d14/db9/dc9/d83/d105 0 2024-03-04T16:04:55.065 INFO:tasks.workunit.client.1.smithi152.stdout:6/950: creat d8/d23/d98/f14a x:0 0 0 2024-03-04T16:04:55.069 INFO:tasks.workunit.client.1.smithi152.stdout:0/914: dwrite d9/d2c/d2e/d7f/d9d/da3/faf [4194304,4194304] 0 2024-03-04T16:04:55.072 INFO:tasks.workunit.client.1.smithi152.stdout:0/915: write d9/d1b/d54/d80/f96 [1730467,66520] 0 2024-03-04T16:04:55.084 INFO:tasks.workunit.client.1.smithi152.stdout:5/890: truncate d6/d8/f74 25304 0 2024-03-04T16:04:55.085 INFO:tasks.workunit.client.1.smithi152.stdout:6/951: readlink d8/d92/d6f/l72 0 2024-03-04T16:04:55.087 INFO:tasks.workunit.client.1.smithi152.stdout:6/952: write d8/d23/d98/f14a [944106,25775] 0 2024-03-04T16:04:55.088 INFO:tasks.workunit.client.1.smithi152.stdout:5/891: dread d6/f68 [0,4194304] 0 2024-03-04T16:04:55.090 INFO:tasks.workunit.client.1.smithi152.stdout:2/903: truncate da/f77 2464874 0 2024-03-04T16:04:55.091 INFO:tasks.workunit.client.1.smithi152.stdout:4/848: symlink d8/d2f/d6f/deb/def/l12c 0 2024-03-04T16:04:55.096 INFO:tasks.workunit.client.1.smithi152.stdout:7/877: mknod d14/d9d/c106 0 2024-03-04T16:04:55.098 INFO:tasks.workunit.client.1.smithi152.stdout:9/918: dwrite d2/d13/d23/d66/d7d/da7/fdc [0,4194304] 0 2024-03-04T16:04:55.098 INFO:tasks.workunit.client.1.smithi152.stdout:6/953: read - d8/d23/d98/f12c zero size 2024-03-04T16:04:55.099 INFO:tasks.workunit.client.1.smithi152.stdout:5/892: creat d6/d8/d5a/d90/f123 x:0 0 0 2024-03-04T16:04:55.100 INFO:tasks.workunit.client.1.smithi152.stdout:0/916: dread d9/d2c/d2e/d94/dac/fc8 [0,4194304] 0 2024-03-04T16:04:55.107 INFO:tasks.workunit.client.1.smithi152.stdout:6/954: dwrite d8/d23/d3a/d71/d9d/fa8 [4194304,4194304] 0 2024-03-04T16:04:55.112 INFO:tasks.workunit.client.1.smithi152.stdout:3/915: link d6/d60/d8e/dc0/fc7 d6/f136 0 2024-03-04T16:04:55.113 INFO:tasks.workunit.client.1.smithi152.stdout:1/936: dwrite d19/d75/dc3/ff6 [0,4194304] 0 2024-03-04T16:04:55.120 INFO:tasks.workunit.client.1.smithi152.stdout:0/917: creat d9/d1b/d54/d69/d6c/dbf/dfd/f12b x:0 0 0 2024-03-04T16:04:55.122 INFO:tasks.workunit.client.1.smithi152.stdout:2/904: mknod da/df/d2e/d51/d11c/c136 0 2024-03-04T16:04:55.124 INFO:tasks.workunit.client.1.smithi152.stdout:2/905: read da/f1a [1303891,17165] 0 2024-03-04T16:04:55.126 INFO:tasks.workunit.client.1.smithi152.stdout:6/955: rmdir d8/d23/d3a/d71/d9d 39 2024-03-04T16:04:55.127 INFO:tasks.workunit.client.1.smithi152.stdout:6/956: chown d8/d92/d59/d61/d57/feb 288114 1 2024-03-04T16:04:55.131 INFO:tasks.workunit.client.1.smithi152.stdout:1/937: read d19/d20/d2f/f94 [1534463,38185] 0 2024-03-04T16:04:55.132 INFO:tasks.workunit.client.1.smithi152.stdout:3/916: dread d6/f3f [0,4194304] 0 2024-03-04T16:04:55.133 INFO:tasks.workunit.client.1.smithi152.stdout:5/893: mkdir d6/d8/d5a/d64/d124 0 2024-03-04T16:04:55.134 INFO:tasks.workunit.client.1.smithi152.stdout:5/894: readlink d6/d8/d1f/d3d/da4/dc9/lb5 0 2024-03-04T16:04:55.135 INFO:tasks.workunit.client.1.smithi152.stdout:5/895: chown d6/d50/f7e 782071 1 2024-03-04T16:04:55.136 INFO:tasks.workunit.client.1.smithi152.stdout:5/896: write d6/d8/f2c [3576477,23444] 0 2024-03-04T16:04:55.136 INFO:tasks.workunit.client.1.smithi152.stdout:6/957: creat d8/d8f/da3/f14b x:0 0 0 2024-03-04T16:04:55.138 INFO:tasks.workunit.client.1.smithi152.stdout:4/849: write d8/d9/d7a/f7d [694130,19029] 0 2024-03-04T16:04:55.139 INFO:tasks.workunit.client.1.smithi152.stdout:4/850: readlink d8/d4e/d86/lab 0 2024-03-04T16:04:55.139 INFO:tasks.workunit.client.1.smithi152.stdout:3/917: symlink d6/d14/d41/d49/d84/de3/l137 0 2024-03-04T16:04:55.140 INFO:tasks.workunit.client.1.smithi152.stdout:3/918: chown d6/d10/f125 27857 1 2024-03-04T16:04:55.142 INFO:tasks.workunit.client.1.smithi152.stdout:0/918: rename d9/d1b/d54/d80/l88 to d9/d1b/d3c/d84/dde/l12c 0 2024-03-04T16:04:55.143 INFO:tasks.workunit.client.1.smithi152.stdout:2/906: mkdir da/d2c/d3f/d137 0 2024-03-04T16:04:55.146 INFO:tasks.workunit.client.1.smithi152.stdout:3/919: dwrite d6/d14/d58/dad/fcb [0,4194304] 0 2024-03-04T16:04:55.146 INFO:tasks.workunit.client.1.smithi152.stdout:3/920: stat d6/cc9 0 2024-03-04T16:04:55.148 INFO:tasks.workunit.client.1.smithi152.stdout:1/938: mknod d19/d4b/d53/d57/c126 0 2024-03-04T16:04:55.150 INFO:tasks.workunit.client.1.smithi152.stdout:4/851: truncate d8/d4e/d8b/ffa 665796 0 2024-03-04T16:04:55.158 INFO:tasks.workunit.client.1.smithi152.stdout:1/939: dwrite d19/d20/d2f/d6b/f5a [0,4194304] 0 2024-03-04T16:04:55.160 INFO:tasks.workunit.client.1.smithi152.stdout:1/940: chown d19/d3b/d6e/d97/dd7/f4c 2630 1 2024-03-04T16:04:55.166 INFO:tasks.workunit.client.1.smithi152.stdout:3/921: rename d6/d14/d58/fb0 to d6/d14/d41/d49/d84/de2/de9/d104/f138 0 2024-03-04T16:04:55.168 INFO:tasks.workunit.client.1.smithi152.stdout:4/852: stat d8/d9/da2/cb0 0 2024-03-04T16:04:55.171 INFO:tasks.workunit.client.1.smithi152.stdout:7/878: write d14/d21/d47/d36/f74 [170297,83341] 0 2024-03-04T16:04:55.173 INFO:tasks.workunit.client.1.smithi152.stdout:1/941: rmdir d19/d4b/d53/d57/dd6 39 2024-03-04T16:04:55.173 INFO:tasks.workunit.client.1.smithi152.stdout:7/879: write d14/d21/d47/d40/f43 [5122471,68259] 0 2024-03-04T16:04:55.175 INFO:tasks.workunit.client.1.smithi152.stdout:7/880: write d14/d69/dd1/fd6 [891995,61956] 0 2024-03-04T16:04:55.178 INFO:tasks.workunit.client.1.smithi152.stdout:4/853: rmdir d8/d58/ded 39 2024-03-04T16:04:55.179 INFO:tasks.workunit.client.1.smithi152.stdout:7/881: chown d14/db9/dc9/f7e 80641 1 2024-03-04T16:04:55.184 INFO:tasks.workunit.client.1.smithi152.stdout:9/919: dwrite d2/d13/d23/fc4 [0,4194304] 0 2024-03-04T16:04:55.188 INFO:tasks.workunit.client.1.smithi152.stdout:7/882: fdatasync d14/d21/d47/d36/dcf/fda 0 2024-03-04T16:04:55.191 INFO:tasks.workunit.client.1.smithi152.stdout:7/883: chown d14/d69/fce 107307218 1 2024-03-04T16:04:55.192 INFO:tasks.workunit.client.1.smithi152.stdout:7/884: chown f4 205 1 2024-03-04T16:04:55.193 INFO:tasks.workunit.client.1.smithi152.stdout:1/942: dwrite d19/d3b/d6e/d97/dd7/d63/d9a/fa2 [0,4194304] 0 2024-03-04T16:04:55.194 INFO:tasks.workunit.client.1.smithi152.stdout:4/854: fsync d8/d9/d24/d26/f32 0 2024-03-04T16:04:55.197 INFO:tasks.workunit.client.1.smithi152.stdout:7/885: read d14/db9/dc9/d83/fa9 [56031,10760] 0 2024-03-04T16:04:55.199 INFO:tasks.workunit.client.1.smithi152.stdout:9/920: rename d2/d13/d38/c82 to d2/d13/d38/d79/c133 0 2024-03-04T16:04:55.201 INFO:tasks.workunit.client.1.smithi152.stdout:9/921: fdatasync d2/ffe 0 2024-03-04T16:04:55.202 INFO:tasks.workunit.client.1.smithi152.stdout:4/855: dwrite d8/d9/d24/d26/f2e [0,4194304] 0 2024-03-04T16:04:55.203 INFO:tasks.workunit.client.1.smithi152.stdout:1/943: rmdir d19/d20/d2f/d5d 39 2024-03-04T16:04:55.211 INFO:tasks.workunit.client.1.smithi152.stdout:3/922: link d6/dd9/fdd d6/d60/d8e/f139 0 2024-03-04T16:04:55.212 INFO:tasks.workunit.client.1.smithi152.stdout:9/922: readlink d2/d13/dad/dd2/d111/lf5 0 2024-03-04T16:04:55.213 INFO:tasks.workunit.client.1.smithi152.stdout:4/856: creat d8/d4e/dba/dcc/df2/f12d x:0 0 0 2024-03-04T16:04:55.214 INFO:tasks.workunit.client.1.smithi152.stdout:1/944: mkdir d19/d20/d2f/d65/def/df1/d127 0 2024-03-04T16:04:55.214 INFO:tasks.workunit.client.1.smithi152.stdout:5/897: write d6/d8/d5a/d90/daf/fbf [852343,79878] 0 2024-03-04T16:04:55.216 INFO:tasks.workunit.client.1.smithi152.stdout:5/898: chown d6/d8/d5a/d90/dd6/f11d 14710 1 2024-03-04T16:04:55.218 INFO:tasks.workunit.client.1.smithi152.stdout:9/923: chown d2/d6e/lcf 96 1 2024-03-04T16:04:55.218 INFO:tasks.workunit.client.1.smithi152.stdout:5/899: read d6/d8/d1f/d88/fac [2155936,87353] 0 2024-03-04T16:04:55.222 INFO:tasks.workunit.client.1.smithi152.stdout:1/945: symlink d19/d3b/l128 0 2024-03-04T16:04:55.223 INFO:tasks.workunit.client.1.smithi152.stdout:1/946: fsync d19/d20/d2f/d65/ffa 0 2024-03-04T16:04:55.229 INFO:tasks.workunit.client.1.smithi152.stdout:0/919: write d9/d1b/d3c/d84/d56/d5b/f93 [604570,121993] 0 2024-03-04T16:04:55.230 INFO:tasks.workunit.client.1.smithi152.stdout:4/857: fdatasync d8/d2f/f4a 0 2024-03-04T16:04:55.234 INFO:tasks.workunit.client.1.smithi152.stdout:4/858: write d8/d9/d24/d26/d42/f81 [3422503,95567] 0 2024-03-04T16:04:55.234 INFO:tasks.workunit.client.1.smithi152.stdout:1/947: creat d19/d4b/d53/d57/d85/f129 x:0 0 0 2024-03-04T16:04:55.235 INFO:tasks.workunit.client.1.smithi152.stdout:2/907: dwrite da/d5f/dbc/fcc [0,4194304] 0 2024-03-04T16:04:55.239 INFO:tasks.workunit.client.1.smithi152.stdout:0/920: dwrite d9/d2c/d2e/f32 [0,4194304] 0 2024-03-04T16:04:55.249 INFO:tasks.workunit.client.1.smithi152.stdout:7/886: write d14/d21/f90 [135598,115451] 0 2024-03-04T16:04:55.249 INFO:tasks.workunit.client.1.smithi152.stdout:1/948: mkdir d19/d3b/d6e/d97/dd7/d63/d12a 0 2024-03-04T16:04:55.250 INFO:tasks.workunit.client.1.smithi152.stdout:9/924: write d2/d13/d38/d79/f89 [4582309,99574] 0 2024-03-04T16:04:55.256 INFO:tasks.workunit.client.1.smithi152.stdout:9/925: write d2/d13/d2e/f44 [642143,83090] 0 2024-03-04T16:04:55.258 INFO:tasks.workunit.client.1.smithi152.stdout:3/923: dwrite d6/d10/d2d/f3b [0,4194304] 0 2024-03-04T16:04:55.261 INFO:tasks.workunit.client.1.smithi152.stdout:5/900: dwrite d6/d8/d1f/d3d/da4/fba [0,4194304] 0 2024-03-04T16:04:55.269 INFO:tasks.workunit.client.1.smithi152.stdout:9/926: dwrite d2/d13/d2e/f4f [0,4194304] 0 2024-03-04T16:04:55.294 INFO:tasks.workunit.client.1.smithi152.stdout:2/908: creat da/d5f/dbc/d123/f138 x:0 0 0 2024-03-04T16:04:55.295 INFO:tasks.workunit.client.1.smithi152.stdout:4/859: truncate d8/d2f/d6f/d90/fca 290130 0 2024-03-04T16:04:55.296 INFO:tasks.workunit.client.1.smithi152.stdout:7/887: readlink d14/d21/d47/d32/l41 0 2024-03-04T16:04:55.302 INFO:tasks.workunit.client.1.smithi152.stdout:0/921: truncate d9/d1b/d54/d69/d98/fc7 2197261 0 2024-03-04T16:04:55.303 INFO:tasks.workunit.client.1.smithi152.stdout:7/888: dwrite d14/db9/dc9/fe4 [0,4194304] 0 2024-03-04T16:04:55.305 INFO:tasks.workunit.client.1.smithi152.stdout:3/924: creat d6/d14/d41/d49/d84/de3/f13a x:0 0 0 2024-03-04T16:04:55.310 INFO:tasks.workunit.client.1.smithi152.stdout:2/909: mkdir da/d5f/dbc/d123/d139 0 2024-03-04T16:04:55.313 INFO:tasks.workunit.client.1.smithi152.stdout:1/949: mkdir d19/d20/d2f/d65/def/df1/d127/d12b 0 2024-03-04T16:04:55.316 INFO:tasks.workunit.client.1.smithi152.stdout:0/922: read - d9/d1a/db4/ff6 zero size 2024-03-04T16:04:55.319 INFO:tasks.workunit.client.1.smithi152.stdout:3/925: creat d6/d60/f13b x:0 0 0 2024-03-04T16:04:55.319 INFO:tasks.workunit.client.1.smithi152.stdout:5/901: symlink d6/d8/d1f/d3d/l125 0 2024-03-04T16:04:55.321 INFO:tasks.workunit.client.1.smithi152.stdout:9/927: unlink d2/d13/d38/df9/c129 0 2024-03-04T16:04:55.324 INFO:tasks.workunit.client.1.smithi152.stdout:1/950: mknod d19/d3b/c12c 0 2024-03-04T16:04:55.326 INFO:tasks.workunit.client.1.smithi152.stdout:1/951: chown d19/d75/ldb 163406 1 2024-03-04T16:04:55.329 INFO:tasks.workunit.client.1.smithi152.stdout:3/926: creat d6/d5c/f13c x:0 0 0 2024-03-04T16:04:55.330 INFO:tasks.workunit.client.1.smithi152.stdout:9/928: dwrite d2/d13/dad/ddf/ff8 [0,4194304] 0 2024-03-04T16:04:55.334 INFO:tasks.workunit.client.1.smithi152.stdout:4/860: rename d8/d4e/dba/dcc/c127 to d8/d58/c12e 0 2024-03-04T16:04:55.334 INFO:tasks.workunit.client.1.smithi152.stdout:1/952: dread d19/d4b/d53/d57/d85/fa3 [0,4194304] 0 2024-03-04T16:04:55.337 INFO:tasks.workunit.client.1.smithi152.stdout:7/889: link d14/d71/cac d14/d21/d31/c107 0 2024-03-04T16:04:55.338 INFO:tasks.workunit.client.1.smithi152.stdout:7/890: readlink d14/d21/d47/d32/l72 0 2024-03-04T16:04:55.342 INFO:tasks.workunit.client.1.smithi152.stdout:0/923: dread d9/d1b/f4f [0,4194304] 0 2024-03-04T16:04:55.343 INFO:tasks.workunit.client.1.smithi152.stdout:9/929: symlink d2/d13/d38/df9/l134 0 2024-03-04T16:04:55.346 INFO:tasks.workunit.client.1.smithi152.stdout:4/861: dread - d8/d2f/d6f/d90/f92 zero size 2024-03-04T16:04:55.348 INFO:tasks.workunit.client.1.smithi152.stdout:4/862: write d8/d3d/f44 [3945714,73585] 0 2024-03-04T16:04:55.350 INFO:tasks.workunit.client.1.smithi152.stdout:4/863: write d8/d4e/dba/dcc/df2/f12d [537317,52024] 0 2024-03-04T16:04:55.357 INFO:tasks.workunit.client.1.smithi152.stdout:3/927: link d6/d14/f110 d6/d60/d8a/f13d 0 2024-03-04T16:04:55.366 INFO:tasks.workunit.client.1.smithi152.stdout:4/864: dread d8/d4e/d86/fc3 [0,4194304] 0 2024-03-04T16:04:55.366 INFO:tasks.workunit.client.1.smithi152.stdout:4/865: chown d8/l17 69 1 2024-03-04T16:04:55.368 INFO:tasks.workunit.client.1.smithi152.stdout:1/953: mknod d19/d20/c12d 0 2024-03-04T16:04:55.369 INFO:tasks.workunit.client.1.smithi152.stdout:5/902: truncate d6/d8/d1f/d3d/d6f/d70/fe9 548784 0 2024-03-04T16:04:55.372 INFO:tasks.workunit.client.1.smithi152.stdout:3/928: mkdir d6/d10/d2d/d6e/d13e 0 2024-03-04T16:04:55.373 INFO:tasks.workunit.client.1.smithi152.stdout:0/924: mknod d9/d2c/d2e/c12d 0 2024-03-04T16:04:55.376 INFO:tasks.workunit.client.1.smithi152.stdout:2/910: rename da/df/d2e/d51/d11c/d80/f106 to da/df/f13a 0 2024-03-04T16:04:55.378 INFO:tasks.workunit.client.1.smithi152.stdout:7/891: write d14/d21/d47/f94 [926236,123661] 0 2024-03-04T16:04:55.379 INFO:tasks.workunit.client.1.smithi152.stdout:4/866: mkdir d8/d2f/d7c/d87/d12f 0 2024-03-04T16:04:55.380 INFO:tasks.workunit.client.1.smithi152.stdout:1/954: creat d19/d3b/d6e/d97/dd7/d63/f12e x:0 0 0 2024-03-04T16:04:55.382 INFO:tasks.workunit.client.1.smithi152.stdout:5/903: creat d6/d8/d1f/d3d/d6f/f126 x:0 0 0 2024-03-04T16:04:55.384 INFO:tasks.workunit.client.1.smithi152.stdout:3/929: truncate d6/f1c 4598135 0 2024-03-04T16:04:55.385 INFO:tasks.workunit.client.1.smithi152.stdout:0/925: fsync d9/d2c/d2e/f6a 0 2024-03-04T16:04:55.388 INFO:tasks.workunit.client.1.smithi152.stdout:3/930: write d6/d10/d2d/f4c [1949642,60234] 0 2024-03-04T16:04:55.390 INFO:tasks.workunit.client.1.smithi152.stdout:9/930: dwrite d2/d13/d23/d66/d7d/fe0 [0,4194304] 0 2024-03-04T16:04:55.391 INFO:tasks.workunit.client.1.smithi152.stdout:9/931: stat d2/d13/d35/f121 0 2024-03-04T16:04:55.392 INFO:tasks.workunit.client.1.smithi152.stdout:1/955: fsync f10 0 2024-03-04T16:04:55.397 INFO:tasks.workunit.client.1.smithi152.stdout:9/932: dread d2/d13/dad/ddf/ff8 [0,4194304] 0 2024-03-04T16:04:55.403 INFO:tasks.workunit.client.1.smithi152.stdout:0/926: mkdir d9/d1b/d54/d80/da8/d12e 0 2024-03-04T16:04:55.404 INFO:tasks.workunit.client.1.smithi152.stdout:1/956: dwrite d19/d4b/d53/d57/d62/d82/d111/f122 [0,4194304] 0 2024-03-04T16:04:55.405 INFO:tasks.workunit.client.1.smithi152.stdout:2/911: rmdir da/df/d2e/dc6 39 2024-03-04T16:04:55.409 INFO:tasks.workunit.client.1.smithi152.stdout:7/892: unlink d14/d21/d47/d32/le8 0 2024-03-04T16:04:55.409 INFO:tasks.workunit.client.1.smithi152.stdout:6/958: sync 2024-03-04T16:04:55.410 INFO:tasks.workunit.client.1.smithi152.stdout:4/867: mkdir d8/d2f/d130 0 2024-03-04T16:04:55.414 INFO:tasks.workunit.client.1.smithi152.stdout:3/931: mknod d6/d14/d41/d49/d84/de2/d108/c13f 0 2024-03-04T16:04:55.416 INFO:tasks.workunit.client.1.smithi152.stdout:7/893: dwrite d14/db9/fd2 [0,4194304] 0 2024-03-04T16:04:55.422 INFO:tasks.workunit.client.1.smithi152.stdout:1/957: creat d19/d3b/f12f x:0 0 0 2024-03-04T16:04:55.430 INFO:tasks.workunit.client.1.smithi152.stdout:4/868: symlink d8/d89/l131 0 2024-03-04T16:04:55.434 INFO:tasks.workunit.client.1.smithi152.stdout:3/932: fsync d6/d60/d8e/fd8 0 2024-03-04T16:04:55.438 INFO:tasks.workunit.client.1.smithi152.stdout:9/933: link d2/d13/d23/d66/d7d/da7/fdc d2/d13/d23/d71/d80/d10e/f135 0 2024-03-04T16:04:55.441 INFO:tasks.workunit.client.1.smithi152.stdout:7/894: dwrite d14/d21/d31/f7c [0,4194304] 0 2024-03-04T16:04:55.442 INFO:tasks.workunit.client.1.smithi152.stdout:2/912: symlink da/df/d58/l13b 0 2024-03-04T16:04:55.450 INFO:tasks.workunit.client.1.smithi152.stdout:6/959: dwrite d8/d92/d59/d61/d57/feb [4194304,4194304] 0 2024-03-04T16:04:55.451 INFO:tasks.workunit.client.1.smithi152.stdout:4/869: creat d8/d89/db3/f132 x:0 0 0 2024-03-04T16:04:55.456 INFO:tasks.workunit.client.1.smithi152.stdout:9/934: chown d2/d13/dea 891552 1 2024-03-04T16:04:55.458 INFO:tasks.workunit.client.1.smithi152.stdout:9/935: stat d2/d13/d23/d71/da3/fab 0 2024-03-04T16:04:55.459 INFO:tasks.workunit.client.1.smithi152.stdout:9/936: dread - d2/d13/d23/d71/da3/fd3 zero size 2024-03-04T16:04:55.460 INFO:tasks.workunit.client.1.smithi152.stdout:0/927: rename d9/d1b/d54/d69/ce1 to d9/d1b/d3c/d84/c12f 0 2024-03-04T16:04:55.462 INFO:tasks.workunit.client.1.smithi152.stdout:2/913: mknod da/d2c/d5e/c13c 0 2024-03-04T16:04:55.465 INFO:tasks.workunit.client.1.smithi152.stdout:1/958: write d19/d20/d2f/d6b/f40 [790807,51059] 0 2024-03-04T16:04:55.467 INFO:tasks.workunit.client.1.smithi152.stdout:9/937: rmdir d2/d13/dad/dd2/d111 39 2024-03-04T16:04:55.468 INFO:tasks.workunit.client.1.smithi152.stdout:0/928: rmdir d9/d92/d107 39 2024-03-04T16:04:55.471 INFO:tasks.workunit.client.1.smithi152.stdout:2/914: rmdir da/df/d2e/d51 39 2024-03-04T16:04:55.474 INFO:tasks.workunit.client.1.smithi152.stdout:5/904: write d6/d8/d1f/d3d/d6f/d70/ffc [398373,32234] 0 2024-03-04T16:04:55.478 INFO:tasks.workunit.client.1.smithi152.stdout:0/929: dwrite d9/d2c/d2e/d94/dac/f123 [0,4194304] 0 2024-03-04T16:04:55.479 INFO:tasks.workunit.client.1.smithi152.stdout:3/933: dread d6/d10/d2d/d4d/fc3 [0,4194304] 0 2024-03-04T16:04:55.479 INFO:tasks.workunit.client.1.smithi152.stdout:4/870: dread d8/d3d/fc4 [0,4194304] 0 2024-03-04T16:04:55.488 INFO:tasks.workunit.client.1.smithi152.stdout:1/959: symlink d19/d4b/d53/d57/l130 0 2024-03-04T16:04:55.490 INFO:tasks.workunit.client.1.smithi152.stdout:4/871: dread - d8/d9/d24/fb5 zero size 2024-03-04T16:04:55.490 INFO:tasks.workunit.client.1.smithi152.stdout:6/960: dwrite d8/d92/fad [0,4194304] 0 2024-03-04T16:04:55.492 INFO:tasks.workunit.client.1.smithi152.stdout:4/872: fsync d8/d4e/fd5 0 2024-03-04T16:04:55.495 INFO:tasks.workunit.client.1.smithi152.stdout:3/934: mknod d6/d14/d41/d49/d84/de2/d108/c140 0 2024-03-04T16:04:55.496 INFO:tasks.workunit.client.1.smithi152.stdout:2/915: creat da/d5f/d85/f13d x:0 0 0 2024-03-04T16:04:55.500 INFO:tasks.workunit.client.1.smithi152.stdout:4/873: mkdir d8/d9/dc1/d10e/d133 0 2024-03-04T16:04:55.504 INFO:tasks.workunit.client.1.smithi152.stdout:1/960: creat d19/d20/d2f/d65/def/df1/d127/d12b/f131 x:0 0 0 2024-03-04T16:04:55.505 INFO:tasks.workunit.client.1.smithi152.stdout:5/905: dwrite d6/d8/d5a/d64/d76/d9f/f51 [0,4194304] 0 2024-03-04T16:04:55.506 INFO:tasks.workunit.client.1.smithi152.stdout:3/935: rename d6/d14/fc4 to d6/d60/d8e/dc0/db5/de7/f141 0 2024-03-04T16:04:55.509 INFO:tasks.workunit.client.1.smithi152.stdout:2/916: truncate da/df/f97 552963 0 2024-03-04T16:04:55.511 INFO:tasks.workunit.client.1.smithi152.stdout:1/961: creat d19/d4b/d53/d57/d62/d82/d111/f132 x:0 0 0 2024-03-04T16:04:55.516 INFO:tasks.workunit.client.1.smithi152.stdout:1/962: write d19/d75/dc3/ff6 [1994513,119455] 0 2024-03-04T16:04:55.518 INFO:tasks.workunit.client.1.smithi152.stdout:0/930: dread d9/d1b/d3c/d84/d56/f63 [4194304,4194304] 0 2024-03-04T16:04:55.519 INFO:tasks.workunit.client.1.smithi152.stdout:5/906: dread d6/d8/d5a/f87 [0,4194304] 0 2024-03-04T16:04:55.519 INFO:tasks.workunit.client.1.smithi152.stdout:6/961: creat d8/d92/db1/d87/f14c x:0 0 0 2024-03-04T16:04:55.520 INFO:tasks.workunit.client.1.smithi152.stdout:3/936: rename d6/d10/d2d/c30 to d6/d14/d93/def/c142 0 2024-03-04T16:04:55.523 INFO:tasks.workunit.client.1.smithi152.stdout:6/962: dread d8/d92/d59/fc8 [0,4194304] 0 2024-03-04T16:04:55.524 INFO:tasks.workunit.client.1.smithi152.stdout:1/963: creat d19/d3b/d6e/d97/dd7/f133 x:0 0 0 2024-03-04T16:04:55.527 INFO:tasks.workunit.client.1.smithi152.stdout:2/917: mknod da/df/d2e/dc6/c13e 0 2024-03-04T16:04:55.528 INFO:tasks.workunit.client.1.smithi152.stdout:0/931: mknod d9/d2c/d2e/d7f/d9d/c130 0 2024-03-04T16:04:55.538 INFO:tasks.workunit.client.1.smithi152.stdout:4/874: truncate d8/f30 1220526 0 2024-03-04T16:04:55.541 INFO:tasks.workunit.client.1.smithi152.stdout:3/937: truncate d6/d14/d58/f71 1431563 0 2024-03-04T16:04:55.541 INFO:tasks.workunit.client.1.smithi152.stdout:3/938: readlink d6/l98 0 2024-03-04T16:04:55.544 INFO:tasks.workunit.client.1.smithi152.stdout:4/875: dwrite d8/d9/d7a/f129 [0,4194304] 0 2024-03-04T16:04:55.546 INFO:tasks.workunit.client.1.smithi152.stdout:2/918: fsync da/df/d2e/d51/d11c/d47/f6b 0 2024-03-04T16:04:55.549 INFO:tasks.workunit.client.1.smithi152.stdout:6/963: write d8/d2a/f34 [687071,85011] 0 2024-03-04T16:04:55.550 INFO:tasks.workunit.client.1.smithi152.stdout:1/964: write d19/d4b/d53/d57/d62/d82/f86 [247419,98549] 0 2024-03-04T16:04:55.553 INFO:tasks.workunit.client.1.smithi152.stdout:1/965: write d19/d3b/d6e/d97/dd7/d63/d9a/fa2 [5239265,1798] 0 2024-03-04T16:04:55.553 INFO:tasks.workunit.client.1.smithi152.stdout:2/919: dread da/df/d2e/f9c [4194304,4194304] 0 2024-03-04T16:04:55.554 INFO:tasks.workunit.client.1.smithi152.stdout:1/966: chown d19/d4b/d53/f121 1025581 1 2024-03-04T16:04:55.557 INFO:tasks.workunit.client.1.smithi152.stdout:4/876: mkdir d8/d2f/d6f/deb/d134 0 2024-03-04T16:04:55.557 INFO:tasks.workunit.client.1.smithi152.stdout:6/964: symlink d8/d42/dde/dee/l14d 0 2024-03-04T16:04:55.559 INFO:tasks.workunit.client.1.smithi152.stdout:2/920: symlink da/d2c/ddd/l13f 0 2024-03-04T16:04:55.564 INFO:tasks.workunit.client.1.smithi152.stdout:3/939: write d6/d10/f39 [572671,6848] 0 2024-03-04T16:04:55.567 INFO:tasks.workunit.client.1.smithi152.stdout:4/877: mkdir d8/d3d/d105/d135 0 2024-03-04T16:04:55.569 INFO:tasks.workunit.client.1.smithi152.stdout:2/921: symlink da/df/d2e/d51/d8d/d9a/d116/d12b/l140 0 2024-03-04T16:04:55.573 INFO:tasks.workunit.client.1.smithi152.stdout:4/878: fdatasync d8/d2f/d6f/d90/fa5 0 2024-03-04T16:04:55.574 INFO:tasks.workunit.client.1.smithi152.stdout:4/879: chown d8/d9/d24/d26/d42 1629446 1 2024-03-04T16:04:55.576 INFO:tasks.workunit.client.1.smithi152.stdout:6/965: truncate d8/d92/d59/d61/d57/feb 5527687 0 2024-03-04T16:04:55.577 INFO:tasks.workunit.client.1.smithi152.stdout:1/967: dwrite d19/d4b/d53/fc1 [0,4194304] 0 2024-03-04T16:04:55.579 INFO:tasks.workunit.client.1.smithi152.stdout:3/940: symlink d6/d10/d2d/d6e/d13e/l143 0 2024-03-04T16:04:55.585 INFO:tasks.workunit.client.1.smithi152.stdout:2/922: dwrite da/df/d2e/d51/d11c/f32 [4194304,4194304] 0 2024-03-04T16:04:55.587 INFO:tasks.workunit.client.1.smithi152.stdout:2/923: write da/df/d2e/d51/d11c/d80/d82/ff2 [1186369,105731] 0 2024-03-04T16:04:55.591 INFO:tasks.workunit.client.1.smithi152.stdout:1/968: fsync d19/d4b/d53/f8d 0 2024-03-04T16:04:55.593 INFO:tasks.workunit.client.1.smithi152.stdout:3/941: chown d6/d14/d41/d49/d84/lda 59940071 1 2024-03-04T16:04:55.595 INFO:tasks.workunit.client.1.smithi152.stdout:6/966: write d8/d92/db1/fd5 [4059774,76006] 0 2024-03-04T16:04:55.599 INFO:tasks.workunit.client.1.smithi152.stdout:2/924: unlink da/d2c/ddd/c119 0 2024-03-04T16:04:55.601 INFO:tasks.workunit.client.1.smithi152.stdout:1/969: readlink d19/d3b/d6e/dc6/le4 0 2024-03-04T16:04:55.602 INFO:tasks.workunit.client.1.smithi152.stdout:3/942: creat d6/d14/d41/d49/d84/de2/f144 x:0 0 0 2024-03-04T16:04:55.607 INFO:tasks.workunit.client.1.smithi152.stdout:1/970: dwrite d19/d3b/d6e/d97/dd7/f3a [0,4194304] 0 2024-03-04T16:04:55.716 INFO:tasks.workunit.client.1.smithi152.stdout:2/925: symlink da/df/d2e/d51/d11c/d80/d82/da8/l141 0 2024-03-04T16:04:55.718 INFO:tasks.workunit.client.1.smithi152.stdout:3/943: truncate d6/d60/d8e/fa8 2397437 0 2024-03-04T16:04:55.719 INFO:tasks.workunit.client.1.smithi152.stdout:1/971: mkdir d19/d4b/d53/d57/d62/d82/d111/d134 0 2024-03-04T16:04:55.720 INFO:tasks.workunit.client.1.smithi152.stdout:4/880: link d8/d78/l9f d8/d58/ded/l136 0 2024-03-04T16:04:55.721 INFO:tasks.workunit.client.1.smithi152.stdout:6/967: write d8/d42/dde/f77 [1483634,97409] 0 2024-03-04T16:04:55.721 INFO:tasks.workunit.client.1.smithi152.stdout:4/881: write d8/f120 [1036656,5663] 0 2024-03-04T16:04:55.723 INFO:tasks.workunit.client.1.smithi152.stdout:4/882: chown d8/d9/d24/d26/l31 251297035 1 2024-03-04T16:04:55.724 INFO:tasks.workunit.client.1.smithi152.stdout:4/883: read - d8/d89/db3/f132 zero size 2024-03-04T16:04:55.728 INFO:tasks.workunit.client.1.smithi152.stdout:2/926: rename da/d2c/fa3 to da/d5f/d85/dca/f142 0 2024-03-04T16:04:55.730 INFO:tasks.workunit.client.1.smithi152.stdout:4/884: dwrite d8/d3d/fa1 [4194304,4194304] 0 2024-03-04T16:04:55.731 INFO:tasks.workunit.client.1.smithi152.stdout:3/944: fdatasync d6/d14/d93/db7/fc5 0 2024-03-04T16:04:55.732 INFO:tasks.workunit.client.1.smithi152.stdout:4/885: write d8/d2f/d6f/d90/fd6 [2020366,115005] 0 2024-03-04T16:04:55.733 INFO:tasks.workunit.client.1.smithi152.stdout:4/886: fdatasync d8/d9/fde 0 2024-03-04T16:04:55.735 INFO:tasks.workunit.client.1.smithi152.stdout:1/972: creat d19/d75/dc3/d103/f135 x:0 0 0 2024-03-04T16:04:55.743 INFO:tasks.workunit.client.1.smithi152.stdout:6/968: dwrite d8/d23/d93/d99/f130 [0,4194304] 0 2024-03-04T16:04:55.748 INFO:tasks.workunit.client.1.smithi152.stdout:4/887: symlink d8/d58/ded/l137 0 2024-03-04T16:04:55.751 INFO:tasks.workunit.client.1.smithi152.stdout:2/927: creat da/df/d2e/d51/d11c/d47/d122/d132/f143 x:0 0 0 2024-03-04T16:04:55.752 INFO:tasks.workunit.client.1.smithi152.stdout:6/969: symlink d8/d92/d59/d7d/l14e 0 2024-03-04T16:04:55.762 INFO:tasks.workunit.client.1.smithi152.stdout:3/945: getdents d6/d14/d41/d49 0 2024-03-04T16:04:55.763 INFO:tasks.workunit.client.1.smithi152.stdout:3/946: write d6/d10/f39 [277261,51725] 0 2024-03-04T16:04:55.763 INFO:tasks.workunit.client.1.smithi152.stdout:6/970: rename d8/d92/db1 to d8/d23/d3a/d8d/d14f 0 2024-03-04T16:04:55.766 INFO:tasks.workunit.client.1.smithi152.stdout:3/947: readlink d6/d14/l26 0 2024-03-04T16:04:55.766 INFO:tasks.workunit.client.1.smithi152.stdout:6/971: truncate d8/d8f/da3/fba 901371 0 2024-03-04T16:04:55.768 INFO:tasks.workunit.client.1.smithi152.stdout:3/948: fdatasync d6/f96 0 2024-03-04T16:04:55.770 INFO:tasks.workunit.client.1.smithi152.stdout:6/972: creat d8/d2a/f150 x:0 0 0 2024-03-04T16:04:55.771 INFO:tasks.workunit.client.1.smithi152.stdout:4/888: dread d8/d78/f82 [0,4194304] 0 2024-03-04T16:04:55.771 INFO:tasks.workunit.client.1.smithi152.stdout:3/949: symlink d6/d14/d41/ddf/l145 0 2024-03-04T16:04:55.773 INFO:tasks.workunit.client.1.smithi152.stdout:6/973: unlink d8/d52/d58/dd0/f11d 0 2024-03-04T16:04:55.775 INFO:tasks.workunit.client.1.smithi152.stdout:3/950: fdatasync d6/d14/d41/fa3 0 2024-03-04T16:04:55.776 INFO:tasks.workunit.client.1.smithi152.stdout:3/951: dread - d6/d10/d2d/f6f zero size 2024-03-04T16:04:55.777 INFO:tasks.workunit.client.1.smithi152.stdout:6/974: rename d8/d23/d3a/c109 to d8/d42/dde/dee/c151 0 2024-03-04T16:04:55.778 INFO:tasks.workunit.client.1.smithi152.stdout:1/973: write d19/d20/d2f/d5d/f95 [1578392,68624] 0 2024-03-04T16:04:55.780 INFO:tasks.workunit.client.1.smithi152.stdout:3/952: chown d6/f4a 19184 1 2024-03-04T16:04:55.782 INFO:tasks.workunit.client.1.smithi152.stdout:1/974: creat d19/d75/da8/f136 x:0 0 0 2024-03-04T16:04:55.784 INFO:tasks.workunit.client.1.smithi152.stdout:2/928: dwrite da/df/fae [0,4194304] 0 2024-03-04T16:04:55.791 INFO:tasks.workunit.client.1.smithi152.stdout:3/953: symlink d6/d14/d58/d105/l146 0 2024-03-04T16:04:55.791 INFO:tasks.workunit.client.1.smithi152.stdout:3/954: read d6/d5c/fc6 [427480,58512] 0 2024-03-04T16:04:55.801 INFO:tasks.workunit.client.1.smithi152.stdout:6/975: link d8/d8f/da3/c100 d8/d23/d3a/d8d/d14f/d87/c152 0 2024-03-04T16:04:55.804 INFO:tasks.workunit.client.1.smithi152.stdout:6/976: mkdir d8/da9/d153 0 2024-03-04T16:04:55.806 INFO:tasks.workunit.client.1.smithi152.stdout:6/977: mkdir d8/d23/d93/d154 0 2024-03-04T16:04:55.809 INFO:tasks.workunit.client.1.smithi152.stdout:2/929: dwrite da/df/d58/f79 [0,4194304] 0 2024-03-04T16:04:55.813 INFO:tasks.workunit.client.1.smithi152.stdout:6/978: fdatasync d8/d42/dde/f73 0 2024-03-04T16:04:55.815 INFO:tasks.workunit.client.1.smithi152.stdout:1/975: dwrite d19/d4b/f51 [0,4194304] 0 2024-03-04T16:04:55.820 INFO:tasks.workunit.client.1.smithi152.stdout:3/955: dwrite d6/f4a [0,4194304] 0 2024-03-04T16:04:55.820 INFO:tasks.workunit.client.1.smithi152.stdout:2/930: dread - da/df/d2e/f10e zero size 2024-03-04T16:04:55.824 INFO:tasks.workunit.client.1.smithi152.stdout:6/979: unlink d8/d23/d93/d99/df6/f54 0 2024-03-04T16:04:55.832 INFO:tasks.workunit.client.1.smithi152.stdout:6/980: dread d8/d8f/da3/fba [0,4194304] 0 2024-03-04T16:04:55.834 INFO:tasks.workunit.client.1.smithi152.stdout:6/981: truncate d8/f12 937330 0 2024-03-04T16:04:55.836 INFO:tasks.workunit.client.1.smithi152.stdout:6/982: unlink d8/d23/d93/d99/l11c 0 2024-03-04T16:04:55.838 INFO:tasks.workunit.client.1.smithi152.stdout:3/956: dread d6/fd4 [0,4194304] 0 2024-03-04T16:04:55.839 INFO:tasks.workunit.client.1.smithi152.stdout:3/957: readlink d6/d14/d41/d49/d84/lda 0 2024-03-04T16:04:55.841 INFO:tasks.workunit.client.1.smithi152.stdout:3/958: creat d6/d60/d8e/f147 x:0 0 0 2024-03-04T16:04:55.843 INFO:tasks.workunit.client.1.smithi152.stdout:3/959: creat d6/d14/d93/db7/f148 x:0 0 0 2024-03-04T16:04:55.846 INFO:tasks.workunit.client.1.smithi152.stdout:3/960: rename d6/f2c to d6/d14/dd7/f149 0 2024-03-04T16:04:55.847 INFO:tasks.workunit.client.1.smithi152.stdout:3/961: fdatasync d6/d60/d8e/dc0/f5b 0 2024-03-04T16:04:55.850 INFO:tasks.workunit.client.1.smithi152.stdout:2/931: dread da/d2c/d5e/f69 [0,4194304] 0 2024-03-04T16:04:55.852 INFO:tasks.workunit.client.1.smithi152.stdout:2/932: dread da/df/d2e/fa1 [0,4194304] 0 2024-03-04T16:04:55.854 INFO:tasks.workunit.client.1.smithi152.stdout:2/933: symlink da/df/l144 0 2024-03-04T16:04:55.854 INFO:tasks.workunit.client.1.smithi152.stdout:2/934: dread - da/d2c/d5e/fe3 zero size 2024-03-04T16:04:55.857 INFO:tasks.workunit.client.1.smithi152.stdout:2/935: creat da/df/d2e/d51/d11c/d47/d11d/f145 x:0 0 0 2024-03-04T16:04:55.859 INFO:tasks.workunit.client.1.smithi152.stdout:2/936: creat da/df/d2e/d51/d8d/d9a/f146 x:0 0 0 2024-03-04T16:04:55.860 INFO:tasks.workunit.client.1.smithi152.stdout:2/937: fdatasync da/f45 0 2024-03-04T16:04:55.862 INFO:tasks.workunit.client.1.smithi152.stdout:2/938: truncate da/d2c/fdc 443440 0 2024-03-04T16:04:55.863 INFO:tasks.workunit.client.1.smithi152.stdout:3/962: dread d6/f136 [0,4194304] 0 2024-03-04T16:04:55.864 INFO:tasks.workunit.client.1.smithi152.stdout:2/939: read da/d2c/d5e/f103 [84816,90296] 0 2024-03-04T16:04:55.867 INFO:tasks.workunit.client.1.smithi152.stdout:2/940: rmdir da/d2c/de1 0 2024-03-04T16:04:55.869 INFO:tasks.workunit.client.1.smithi152.stdout:2/941: read da/df/f13 [3929440,77320] 0 2024-03-04T16:04:55.871 INFO:tasks.workunit.client.1.smithi152.stdout:2/942: write da/df/d2e/d51/d11c/d80/f9b [4294183,127158] 0 2024-03-04T16:04:55.879 INFO:tasks.workunit.client.1.smithi152.stdout:1/976: write d19/d20/f30 [883101,3648] 0 2024-03-04T16:04:55.880 INFO:tasks.workunit.client.1.smithi152.stdout:1/977: chown d19/d4b/f5f 621 1 2024-03-04T16:04:55.888 INFO:tasks.workunit.client.1.smithi152.stdout:6/983: dwrite d8/d10/d53/f5b [0,4194304] 0 2024-03-04T16:04:55.891 INFO:tasks.workunit.client.1.smithi152.stdout:6/984: mknod d8/d92/d59/d7d/d91/c155 0 2024-03-04T16:04:55.894 INFO:tasks.workunit.client.1.smithi152.stdout:6/985: dread d8/d42/fcf [0,4194304] 0 2024-03-04T16:04:55.900 INFO:tasks.workunit.client.1.smithi152.stdout:1/978: dread d19/d3b/d6e/f10c [0,4194304] 0 2024-03-04T16:04:55.900 INFO:tasks.workunit.client.1.smithi152.stdout:6/986: truncate d8/d23/d3a/d8d/d14f/d87/fdc 877324 0 2024-03-04T16:04:55.900 INFO:tasks.workunit.client.1.smithi152.stdout:2/943: write da/df/f97 [956881,8310] 0 2024-03-04T16:04:55.903 INFO:tasks.workunit.client.1.smithi152.stdout:3/963: dwrite d6/d10/d2d/f34 [0,4194304] 0 2024-03-04T16:04:55.904 INFO:tasks.workunit.client.1.smithi152.stdout:3/964: stat d6/d5c/f13c 0 2024-03-04T16:04:55.907 INFO:tasks.workunit.client.1.smithi152.stdout:2/944: symlink da/d5f/db0/d114/l147 0 2024-03-04T16:04:55.907 INFO:tasks.workunit.client.1.smithi152.stdout:6/987: truncate d8/d92/d59/d7d/d91/ff0 1215774 0 2024-03-04T16:04:55.909 INFO:tasks.workunit.client.1.smithi152.stdout:3/965: stat d6/d14/d41/d49/l5e 0 2024-03-04T16:04:55.911 INFO:tasks.workunit.client.1.smithi152.stdout:6/988: fsync d8/d23/d3a/d71/d9d/fa8 0 2024-03-04T16:04:55.917 INFO:tasks.workunit.client.1.smithi152.stdout:1/979: write d19/d20/d2f/f59 [782321,34998] 0 2024-03-04T16:04:55.917 INFO:tasks.workunit.client.1.smithi152.stdout:6/989: rmdir d8/d23/d93/d99/df6 39 2024-03-04T16:04:55.917 INFO:tasks.workunit.client.1.smithi152.stdout:1/980: stat d19/d3b/d6e/d97/dd7/d63/fb0 0 2024-03-04T16:04:55.919 INFO:tasks.workunit.client.1.smithi152.stdout:3/966: creat d6/d60/d8e/dc0/db5/d129/f14a x:0 0 0 2024-03-04T16:04:55.920 INFO:tasks.workunit.client.1.smithi152.stdout:1/981: mknod d19/d75/c137 0 2024-03-04T16:04:55.924 INFO:tasks.workunit.client.1.smithi152.stdout:1/982: creat d19/d3b/d6e/d97/dd7/d63/f138 x:0 0 0 2024-03-04T16:04:55.925 INFO:tasks.workunit.client.1.smithi152.stdout:6/990: symlink d8/d23/d3a/d8d/l156 0 2024-03-04T16:04:55.927 INFO:tasks.workunit.client.1.smithi152.stdout:1/983: creat d19/d3b/d6e/d97/dd7/d63/d9a/f139 x:0 0 0 2024-03-04T16:04:55.930 INFO:tasks.workunit.client.1.smithi152.stdout:1/984: chown d19/d20/d2f/d65/def/ffd 0 1 2024-03-04T16:04:55.930 INFO:tasks.workunit.client.1.smithi152.stdout:1/985: chown d19/d75/da8/f136 6416602 1 2024-03-04T16:04:55.931 INFO:tasks.workunit.client.1.smithi152.stdout:6/991: creat d8/d23/d3a/d71/f157 x:0 0 0 2024-03-04T16:04:55.933 INFO:tasks.workunit.client.1.smithi152.stdout:3/967: dread d6/d14/d58/dad/f10b [0,4194304] 0 2024-03-04T16:04:55.933 INFO:tasks.workunit.client.1.smithi152.stdout:6/992: chown d8/d52/d58/ca6 5 1 2024-03-04T16:04:55.933 INFO:tasks.workunit.client.1.smithi152.stdout:2/945: dread da/fb [0,4194304] 0 2024-03-04T16:04:55.934 INFO:tasks.workunit.client.1.smithi152.stdout:1/986: creat d19/d3b/d6e/d97/dd7/d63/d9a/f13a x:0 0 0 2024-03-04T16:04:55.938 INFO:tasks.workunit.client.1.smithi152.stdout:6/993: creat d8/d92/f158 x:0 0 0 2024-03-04T16:04:55.939 INFO:tasks.workunit.client.1.smithi152.stdout:9/938: sync 2024-03-04T16:04:55.939 INFO:tasks.workunit.client.1.smithi152.stdout:7/895: sync 2024-03-04T16:04:55.940 INFO:tasks.workunit.client.1.smithi152.stdout:7/896: read - d14/d71/fb6 zero size 2024-03-04T16:04:55.941 INFO:tasks.workunit.client.1.smithi152.stdout:3/968: write d6/d10/d2d/fcf [943442,36880] 0 2024-03-04T16:04:55.941 INFO:tasks.workunit.client.1.smithi152.stdout:1/987: dwrite d19/d20/d2f/d6b/f40 [0,4194304] 0 2024-03-04T16:04:55.943 INFO:tasks.workunit.client.1.smithi152.stdout:2/946: write da/f45 [2133826,22954] 0 2024-03-04T16:04:55.948 INFO:tasks.workunit.client.1.smithi152.stdout:1/988: fsync d19/d4b/d53/d57/d62/d82/d111/f132 0 2024-03-04T16:04:55.948 INFO:tasks.workunit.client.1.smithi152.stdout:2/947: creat da/d5f/dbc/dd5/f148 x:0 0 0 2024-03-04T16:04:55.953 INFO:tasks.workunit.client.1.smithi152.stdout:6/994: getdents d8/d92/d59/d7d/d104/d136 0 2024-03-04T16:04:55.956 INFO:tasks.workunit.client.1.smithi152.stdout:9/939: dwrite d2/d13/d23/fc4 [0,4194304] 0 2024-03-04T16:04:55.956 INFO:tasks.workunit.client.1.smithi152.stdout:1/989: mkdir d19/d3b/d6e/d97/d13b 0 2024-03-04T16:04:55.957 INFO:tasks.workunit.client.1.smithi152.stdout:3/969: dwrite d6/d60/f109 [0,4194304] 0 2024-03-04T16:04:55.959 INFO:tasks.workunit.client.1.smithi152.stdout:3/970: stat d6/d60/d8e/dc0/fc1 0 2024-03-04T16:04:55.959 INFO:tasks.workunit.client.1.smithi152.stdout:1/990: dread - d19/d3b/d6e/d97/dd7/d63/d9a/f13a zero size 2024-03-04T16:04:55.959 INFO:tasks.workunit.client.1.smithi152.stdout:3/971: chown d6/d14/d41 20 1 2024-03-04T16:04:55.962 INFO:tasks.workunit.client.1.smithi152.stdout:2/948: mkdir da/df/d2e/d51/d11c/d47/d149 0 2024-03-04T16:04:55.969 INFO:tasks.workunit.client.1.smithi152.stdout:7/897: dread f7 [0,4194304] 0 2024-03-04T16:04:55.972 INFO:tasks.workunit.client.1.smithi152.stdout:9/940: write d2/d13/d23/fed [2738896,91870] 0 2024-03-04T16:04:55.976 INFO:tasks.workunit.client.1.smithi152.stdout:7/898: rename d14/d21/d47/d36/da7/dbc/fd4 to d14/d21/f108 0 2024-03-04T16:04:55.977 INFO:tasks.workunit.client.1.smithi152.stdout:9/941: symlink d2/d13/d23/d42/l136 0 2024-03-04T16:04:55.977 INFO:tasks.workunit.client.1.smithi152.stdout:7/899: chown d14/db9/dc9/d83/db7/ff4 2664 1 2024-03-04T16:04:55.990 INFO:tasks.workunit.client.1.smithi152.stdout:1/991: dread d19/d20/d2f/d6b/f2d [0,4194304] 0 2024-03-04T16:04:55.995 INFO:tasks.workunit.client.1.smithi152.stdout:1/992: link d19/d20/d2f/d6b/c112 d19/d75/da8/c13c 0 2024-03-04T16:04:55.995 INFO:tasks.workunit.client.1.smithi152.stdout:9/942: dwrite d2/fd [0,4194304] 0 2024-03-04T16:04:55.996 INFO:tasks.workunit.client.1.smithi152.stdout:1/993: readlink d19/d20/d2f/d5d/l119 0 2024-03-04T16:04:55.998 INFO:tasks.workunit.client.1.smithi152.stdout:1/994: symlink d19/d4b/d53/d5c/l13d 0 2024-03-04T16:04:56.000 INFO:tasks.workunit.client.1.smithi152.stdout:9/943: truncate d2/d13/d96/fa4 2607616 0 2024-03-04T16:04:56.002 INFO:tasks.workunit.client.1.smithi152.stdout:9/944: read d2/d13/d23/d71/ff1 [598996,70289] 0 2024-03-04T16:04:56.005 INFO:tasks.workunit.client.1.smithi152.stdout:9/945: mknod d2/d13/d35/d10c/c137 0 2024-03-04T16:04:56.007 INFO:tasks.workunit.client.1.smithi152.stdout:9/946: mknod d2/d13/d2e/dcd/c138 0 2024-03-04T16:04:56.016 INFO:tasks.workunit.client.1.smithi152.stdout:9/947: rename d2/d13/d23/d71/fa1 to d2/d13/d35/d10c/d125/f139 0 2024-03-04T16:04:56.016 INFO:tasks.workunit.client.1.smithi152.stdout:1/995: dwrite d19/d4b/d53/d57/d62/f104 [0,4194304] 0 2024-03-04T16:04:56.021 INFO:tasks.workunit.client.1.smithi152.stdout:1/996: truncate d19/f1d 4234113 0 2024-03-04T16:04:56.023 INFO:tasks.workunit.client.1.smithi152.stdout:9/948: fdatasync d2/d13/d23/d42/f78 0 2024-03-04T16:04:56.024 INFO:tasks.workunit.client.1.smithi152.stdout:9/949: fdatasync d2/d6e/f10d 0 2024-03-04T16:04:56.026 INFO:tasks.workunit.client.1.smithi152.stdout:1/997: creat d19/d20/d2f/d65/def/f13e x:0 0 0 2024-03-04T16:04:56.033 INFO:tasks.workunit.client.1.smithi152.stdout:1/998: symlink d19/d3b/d6e/d97/dd7/da6/l13f 0 2024-03-04T16:04:56.033 INFO:tasks.workunit.client.1.smithi152.stdout:1/999: chown d19/d4b/d53/d57/d62/d82/cff 90202 1 2024-03-04T16:04:56.111 INFO:tasks.workunit.client.1.smithi152.stdout:5/907: sync 2024-03-04T16:04:56.111 INFO:tasks.workunit.client.1.smithi152.stdout:0/932: sync 2024-03-04T16:04:56.113 INFO:tasks.workunit.client.1.smithi152.stdout:9/950: creat d2/d13/dad/dd2/f13a x:0 0 0 2024-03-04T16:04:56.115 INFO:tasks.workunit.client.1.smithi152.stdout:9/951: dread - d2/d6e/f11b zero size 2024-03-04T16:04:56.124 INFO:tasks.workunit.client.1.smithi152.stdout:9/952: mkdir d2/d13/d35/d13b 0 2024-03-04T16:04:56.126 INFO:tasks.workunit.client.1.smithi152.stdout:5/908: creat d6/d8/d1f/d3d/d6f/d70/d98/f127 x:0 0 0 2024-03-04T16:04:56.128 INFO:tasks.workunit.client.1.smithi152.stdout:0/933: link d9/d2c/d2e/d7f/d9d/de8/fed d9/d2c/f131 0 2024-03-04T16:04:56.132 INFO:tasks.workunit.client.1.smithi152.stdout:5/909: mkdir d6/d8/d1f/d3d/d6f/d70/dc4/d128 0 2024-03-04T16:04:56.136 INFO:tasks.workunit.client.1.smithi152.stdout:5/910: mknod d6/d8/d1f/d3d/d6f/d70/d98/c129 0 2024-03-04T16:04:56.141 INFO:tasks.workunit.client.1.smithi152.stdout:5/911: mkdir d6/d8/d5a/d64/d124/d12a 0 2024-03-04T16:04:56.148 INFO:tasks.workunit.client.1.smithi152.stdout:9/953: write d2/d13/d23/f56 [283206,23443] 0 2024-03-04T16:04:56.155 INFO:tasks.workunit.client.1.smithi152.stdout:9/954: rename d2/d13/dad/dd2/d111/d69/l8a to d2/d13/d23/d66/d7d/da7/l13c 0 2024-03-04T16:04:56.159 INFO:tasks.workunit.client.1.smithi152.stdout:9/955: creat d2/d13/d23/d71/f13d x:0 0 0 2024-03-04T16:04:56.161 INFO:tasks.workunit.client.1.smithi152.stdout:9/956: fdatasync d2/d13/d23/fa0 0 2024-03-04T16:04:56.172 INFO:tasks.workunit.client.1.smithi152.stdout:9/957: dwrite d2/d6e/f7a [0,4194304] 0 2024-03-04T16:04:56.174 INFO:tasks.workunit.client.1.smithi152.stdout:9/958: write d2/d13/d38/f58 [469281,11163] 0 2024-03-04T16:04:56.186 INFO:tasks.workunit.client.1.smithi152.stdout:9/959: write d2/d13/d96/fe7 [1832254,90027] 0 2024-03-04T16:04:56.187 INFO:tasks.workunit.client.1.smithi152.stdout:9/960: dread - d2/d13/d35/f121 zero size 2024-03-04T16:04:56.191 INFO:tasks.workunit.client.1.smithi152.stdout:9/961: rename d2/d13/fd7 to d2/d13/d23/d66/f13e 0 2024-03-04T16:04:56.193 INFO:tasks.workunit.client.1.smithi152.stdout:9/962: write d2/d6e/f11b [833186,74923] 0 2024-03-04T16:04:56.199 INFO:tasks.workunit.client.1.smithi152.stdout:9/963: write d2/d13/dad/dd2/d111/f6a [764767,100538] 0 2024-03-04T16:04:56.201 INFO:tasks.workunit.client.1.smithi152.stdout:9/964: fdatasync d2/d13/d23/d42/fe8 0 2024-03-04T16:04:56.202 INFO:tasks.workunit.client.1.smithi152.stdout:9/965: fdatasync d2/d13/d23/f56 0 2024-03-04T16:04:56.206 INFO:tasks.workunit.client.1.smithi152.stdout:9/966: dwrite d2/f28 [0,4194304] 0 2024-03-04T16:04:56.207 INFO:tasks.workunit.client.1.smithi152.stdout:9/967: chown d2/d13/d23/d71/d80/caa 602 1 2024-03-04T16:04:56.212 INFO:tasks.workunit.client.1.smithi152.stdout:9/968: mkdir d2/d13/d23/d71/d80/d13f 0 2024-03-04T16:04:56.212 INFO:tasks.workunit.client.1.smithi152.stdout:9/969: getdents d2/d13/d35/d13b 0 2024-03-04T16:04:56.215 INFO:tasks.workunit.client.1.smithi152.stdout:9/970: unlink d2/d13/dad/dd2/d111/d69/fc1 0 2024-03-04T16:04:56.216 INFO:tasks.workunit.client.1.smithi152.stdout:9/971: read - d2/d13/d23/d42/db1/f120 zero size 2024-03-04T16:04:56.217 INFO:tasks.workunit.client.1.smithi152.stdout:9/972: stat d2/d6e/f10d 0 2024-03-04T16:04:56.221 INFO:tasks.workunit.client.1.smithi152.stdout:9/973: getdents d2/d6e 0 2024-03-04T16:04:56.222 INFO:tasks.workunit.client.1.smithi152.stdout:9/974: read - d2/d13/d23/d66/d7d/fb9 zero size 2024-03-04T16:04:56.228 INFO:tasks.workunit.client.1.smithi152.stdout:9/975: dwrite d2/d13/d35/d10c/f12b [0,4194304] 0 2024-03-04T16:04:56.237 INFO:tasks.workunit.client.1.smithi152.stdout:9/976: mknod d2/d13/dad/dd2/d111/c140 0 2024-03-04T16:04:56.238 INFO:tasks.workunit.client.1.smithi152.stdout:9/977: chown d2/d13/dad/dd2/d111/f95 278 1 2024-03-04T16:04:56.238 INFO:tasks.workunit.client.1.smithi152.stdout:9/978: chown d2/d13/d23/d71/da3/d12f 88 1 2024-03-04T16:04:56.239 INFO:tasks.workunit.client.1.smithi152.stdout:9/979: chown d2/d13/d23/d66/d7d/da7/fdc 1275170 1 2024-03-04T16:04:56.240 INFO:tasks.workunit.client.1.smithi152.stdout:9/980: creat d2/d13/d35/d13b/f141 x:0 0 0 2024-03-04T16:04:56.246 INFO:tasks.workunit.client.1.smithi152.stdout:9/981: link d2/d13/d35/d10c/d125/f139 d2/d13/f142 0 2024-03-04T16:04:56.254 INFO:tasks.workunit.client.1.smithi152.stdout:9/982: fsync d2/d13/d23/d42/db1/fb2 0 2024-03-04T16:04:56.272 INFO:tasks.workunit.client.1.smithi152.stdout:4/889: sync 2024-03-04T16:04:56.273 INFO:tasks.workunit.client.1.smithi152.stdout:4/890: truncate d8/d89/db3/f116 586002 0 2024-03-04T16:04:56.300 INFO:tasks.workunit.client.1.smithi152.stdout:9/983: mknod d2/d13/d23/d66/d7d/c143 0 2024-03-04T16:04:56.304 INFO:tasks.workunit.client.1.smithi152.stdout:9/984: rename d2/d13/d23/d71/d80/caa to d2/d13/d23/d71/da3/c144 0 2024-03-04T16:04:56.304 INFO:tasks.workunit.client.1.smithi152.stdout:9/985: chown d2/f101 5 1 2024-03-04T16:04:56.306 INFO:tasks.workunit.client.1.smithi152.stdout:4/891: getdents d8/d9/d24 0 2024-03-04T16:04:56.389 INFO:tasks.workunit.client.1.smithi152.stdout:4/892: rmdir d8/d2f/d6f/deb/d134 0 2024-03-04T16:04:56.393 INFO:tasks.workunit.client.1.smithi152.stdout:4/893: dwrite d8/d2f/d6f/f73 [0,4194304] 0 2024-03-04T16:04:56.455 INFO:tasks.workunit.client.1.smithi152.stdout:6/995: sync 2024-03-04T16:04:56.456 INFO:tasks.workunit.client.1.smithi152.stdout:4/894: mknod d8/d4e/dba/dcc/c138 0 2024-03-04T16:04:56.457 INFO:tasks.workunit.client.1.smithi152.stdout:4/895: chown d8/d2f/d7c/d87/la3 23 1 2024-03-04T16:04:56.462 INFO:tasks.workunit.client.1.smithi152.stdout:4/896: creat d8/f139 x:0 0 0 2024-03-04T16:04:56.463 INFO:tasks.workunit.client.1.smithi152.stdout:4/897: chown d8/d9/l46 1436 1 2024-03-04T16:04:56.465 INFO:tasks.workunit.client.1.smithi152.stdout:4/898: fsync d8/f3b 0 2024-03-04T16:04:56.468 INFO:tasks.workunit.client.1.smithi152.stdout:4/899: rmdir d8/d9/da2 39 2024-03-04T16:04:56.476 INFO:tasks.workunit.client.1.smithi152.stdout:6/996: write d8/f12 [1808528,86173] 0 2024-03-04T16:04:56.477 INFO:tasks.workunit.client.1.smithi152.stdout:4/900: fsync d8/f139 0 2024-03-04T16:04:56.480 INFO:tasks.workunit.client.1.smithi152.stdout:6/997: creat d8/d42/dde/dee/f159 x:0 0 0 2024-03-04T16:04:56.481 INFO:tasks.workunit.client.1.smithi152.stdout:4/901: creat d8/d2f/d7c/d87/df0/f13a x:0 0 0 2024-03-04T16:04:56.482 INFO:tasks.workunit.client.1.smithi152.stdout:4/902: fdatasync d8/d58/ded/f11f 0 2024-03-04T16:04:56.483 INFO:tasks.workunit.client.1.smithi152.stdout:6/998: truncate d8/f13f 643514 0 2024-03-04T16:04:56.483 INFO:tasks.workunit.client.1.smithi152.stdout:6/999: readlink d8/d10/l49 0 2024-03-04T16:04:56.484 INFO:tasks.workunit.client.1.smithi152.stdout:4/903: dread d8/d9/d24/d26/d42/f4b [0,4194304] 0 2024-03-04T16:04:56.490 INFO:tasks.workunit.client.1.smithi152.stdout:4/904: dwrite d8/d58/ded/f11f [0,4194304] 0 2024-03-04T16:04:56.505 INFO:tasks.workunit.client.1.smithi152.stdout:4/905: dwrite d8/d9/d24/d26/f2a [4194304,4194304] 0 2024-03-04T16:04:56.507 INFO:tasks.workunit.client.1.smithi152.stdout:4/906: stat d8/d9/d7a/f7d 0 2024-03-04T16:04:56.518 INFO:tasks.workunit.client.1.smithi152.stdout:4/907: dwrite d8/d2f/d6f/d84/fa4 [0,4194304] 0 2024-03-04T16:04:56.518 INFO:tasks.workunit.client.1.smithi152.stdout:4/908: chown d8/d9/d24 0 1 2024-03-04T16:04:56.520 INFO:tasks.workunit.client.1.smithi152.stdout:4/909: mknod d8/d9/d7a/c13b 0 2024-03-04T16:04:56.529 INFO:tasks.workunit.client.1.smithi152.stdout:4/910: dread d8/f37 [4194304,4194304] 0 2024-03-04T16:04:56.537 INFO:tasks.workunit.client.1.smithi152.stdout:4/911: write d8/d9/d79/f85 [3441026,33742] 0 2024-03-04T16:04:56.538 INFO:tasks.workunit.client.1.smithi152.stdout:4/912: chown d8/d2f/d6f/deb/def/l12c 35465406 1 2024-03-04T16:04:56.540 INFO:tasks.workunit.client.1.smithi152.stdout:4/913: getdents d8/d9/dc1/d10e/d133 0 2024-03-04T16:04:56.542 INFO:tasks.workunit.client.1.smithi152.stdout:4/914: creat d8/d4e/d8b/f13c x:0 0 0 2024-03-04T16:04:56.544 INFO:tasks.workunit.client.1.smithi152.stdout:4/915: mknod d8/d4e/dba/dcc/df2/c13d 0 2024-03-04T16:04:56.546 INFO:tasks.workunit.client.1.smithi152.stdout:4/916: dread d8/d89/db3/f116 [0,4194304] 0 2024-03-04T16:04:56.546 INFO:tasks.workunit.client.1.smithi152.stdout:4/917: readlink d8/d58/l83 0 2024-03-04T16:04:56.548 INFO:tasks.workunit.client.1.smithi152.stdout:4/918: read d8/d89/db3/d104/f113 [2970517,90137] 0 2024-03-04T16:04:56.551 INFO:tasks.workunit.client.1.smithi152.stdout:4/919: rename d8/d9/f50 to d8/d9/d24/d26/d42/f13e 0 2024-03-04T16:04:56.565 INFO:tasks.workunit.client.1.smithi152.stdout:4/920: dread d8/d2f/d7c/fce [0,4194304] 0 2024-03-04T16:04:56.565 INFO:tasks.workunit.client.1.smithi152.stdout:3/972: sync 2024-03-04T16:04:56.568 INFO:tasks.workunit.client.1.smithi152.stdout:3/973: creat d6/d60/d8e/dc0/db5/d129/f14b x:0 0 0 2024-03-04T16:04:56.580 INFO:tasks.workunit.client.1.smithi152.stdout:3/974: dwrite d6/d60/d8e/f139 [0,4194304] 0 2024-03-04T16:04:56.585 INFO:tasks.workunit.client.1.smithi152.stdout:3/975: rename d6/d10/l3d to d6/d14/d58/de4/l14c 0 2024-03-04T16:04:56.586 INFO:tasks.workunit.client.1.smithi152.stdout:3/976: write d6/d60/d8e/dc0/da1/f112 [349509,61081] 0 2024-03-04T16:04:56.587 INFO:tasks.workunit.client.1.smithi152.stdout:3/977: chown d6/d14/dd7/fdb 1408679 1 2024-03-04T16:04:56.591 INFO:tasks.workunit.client.1.smithi152.stdout:3/978: rename d6/d14/d41/d49/d84/de3/f13a to d6/d60/d8e/dc0/db5/f14d 0 2024-03-04T16:04:56.595 INFO:tasks.workunit.client.1.smithi152.stdout:3/979: rename d6/d14/d41/f114 to d6/d60/d8e/dc0/db5/de7/f14e 0 2024-03-04T16:04:56.597 INFO:tasks.workunit.client.1.smithi152.stdout:3/980: rmdir d6/db1 39 2024-03-04T16:04:56.605 INFO:tasks.workunit.client.1.smithi152.stdout:3/981: rename d6/d14/d58/de4/l14c to d6/d10/d2d/d6e/d13e/l14f 0 2024-03-04T16:04:56.606 INFO:tasks.workunit.client.1.smithi152.stdout:3/982: chown d6/d14/d41/d49/d84/de2 3 1 2024-03-04T16:04:56.613 INFO:tasks.workunit.client.1.smithi152.stdout:3/983: getdents d6/d60/d8e/dc0/da1/de0 0 2024-03-04T16:04:56.620 INFO:tasks.workunit.client.1.smithi152.stdout:3/984: truncate d6/d60/d8e/dc0/da1/fc2 4116293 0 2024-03-04T16:04:56.622 INFO:tasks.workunit.client.1.smithi152.stdout:3/985: symlink d6/d14/d41/ddf/l150 0 2024-03-04T16:04:56.628 INFO:tasks.workunit.client.1.smithi152.stdout:3/986: truncate d6/d14/d41/d49/d84/de2/de9/d104/f138 361100 0 2024-03-04T16:04:56.630 INFO:tasks.workunit.client.1.smithi152.stdout:3/987: symlink d6/d14/d58/d105/l151 0 2024-03-04T16:04:56.634 INFO:tasks.workunit.client.1.smithi152.stdout:3/988: creat d6/d14/f152 x:0 0 0 2024-03-04T16:04:56.636 INFO:tasks.workunit.client.1.smithi152.stdout:3/989: write d6/d60/d8e/dc0/db5/de7/f14e [234595,86588] 0 2024-03-04T16:04:56.638 INFO:tasks.workunit.client.1.smithi152.stdout:3/990: mknod d6/d60/d8e/dc0/db5/de7/c153 0 2024-03-04T16:04:56.644 INFO:tasks.workunit.client.1.smithi152.stdout:3/991: rmdir d6/d60/d8e/dc0 39 2024-03-04T16:04:56.645 INFO:tasks.workunit.client.1.smithi152.stdout:3/992: symlink d6/d60/d8a/l154 0 2024-03-04T16:04:56.647 INFO:tasks.workunit.client.1.smithi152.stdout:3/993: mknod d6/d14/d58/d105/c155 0 2024-03-04T16:04:56.651 INFO:tasks.workunit.client.1.smithi152.stdout:3/994: truncate d6/d14/d41/d49/d84/de2/de9/d104/f138 882818 0 2024-03-04T16:04:56.655 INFO:tasks.workunit.client.1.smithi152.stdout:3/995: rename d6/d14/d93/db7/cd0 to d6/d14/d41/d49/c156 0 2024-03-04T16:04:56.657 INFO:tasks.workunit.client.1.smithi152.stdout:3/996: fsync d6/d60/d8e/dc0/da1/f117 0 2024-03-04T16:04:56.668 INFO:tasks.workunit.client.1.smithi152.stdout:3/997: dwrite d6/d14/dd7/f149 [0,4194304] 0 2024-03-04T16:04:56.689 INFO:tasks.workunit.client.1.smithi152.stdout:3/998: dread d6/d14/dd7/fdb [0,4194304] 0 2024-03-04T16:04:56.695 INFO:tasks.workunit.client.1.smithi152.stdout:3/999: dwrite d6/d5c/f13c [0,4194304] 0 2024-03-04T16:04:56.702 INFO:tasks.workunit.client.1.smithi152.stdout:0/934: sync 2024-03-04T16:04:56.702 INFO:tasks.workunit.client.1.smithi152.stdout:9/986: sync 2024-03-04T16:04:56.702 INFO:tasks.workunit.client.1.smithi152.stdout:7/900: sync 2024-03-04T16:04:56.702 INFO:tasks.workunit.client.1.smithi152.stdout:5/912: sync 2024-03-04T16:04:56.702 INFO:tasks.workunit.client.1.smithi152.stdout:2/949: sync 2024-03-04T16:04:56.703 INFO:tasks.workunit.client.1.smithi152.stdout:7/901: chown d14/d21/d31 112286 1 2024-03-04T16:04:56.704 INFO:tasks.workunit.client.1.smithi152.stdout:7/902: chown d14/d21/d47/d40/l99 210460 1 2024-03-04T16:04:56.706 INFO:tasks.workunit.client.1.smithi152.stdout:2/950: fdatasync da/d5f/f68 0 2024-03-04T16:04:56.708 INFO:tasks.workunit.client.1.smithi152.stdout:2/951: chown da/df/d2e/d51/l118 2 1 2024-03-04T16:04:56.709 INFO:tasks.workunit.client.1.smithi152.stdout:9/987: dwrite d2/d13/dad/dd2/d111/f6a [0,4194304] 0 2024-03-04T16:04:56.711 INFO:tasks.workunit.client.1.smithi152.stdout:5/913: write d6/d8/f1d [719164,113177] 0 2024-03-04T16:04:56.712 INFO:tasks.workunit.client.1.smithi152.stdout:5/914: chown d6/d8/d1f/d3d/d6f/f126 3645 1 2024-03-04T16:04:56.715 INFO:tasks.workunit.client.1.smithi152.stdout:2/952: chown da/df/d2e/d51/d11c/d81/cef 237791 1 2024-03-04T16:04:56.715 INFO:tasks.workunit.client.1.smithi152.stdout:2/953: dread - da/d5f/dbc/dd5/f148 zero size 2024-03-04T16:04:56.716 INFO:tasks.workunit.client.1.smithi152.stdout:2/954: chown da/df/d2e/d51/d11c/d47/d4f/d11a/d8e 40962304 1 2024-03-04T16:04:56.716 INFO:tasks.workunit.client.1.smithi152.stdout:9/988: rename d2/d13/d2e/c62 to d2/d13/d23/d42/db1/c145 0 2024-03-04T16:04:56.721 INFO:tasks.workunit.client.1.smithi152.stdout:0/935: creat d9/d1b/d54/d69/d6c/f132 x:0 0 0 2024-03-04T16:04:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:56 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:56 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:56.725 INFO:tasks.workunit.client.1.smithi152.stdout:7/903: truncate f7 1696403 0 2024-03-04T16:04:56.725 INFO:tasks.workunit.client.1.smithi152.stdout:7/904: stat d14/f42 0 2024-03-04T16:04:56.725 INFO:tasks.workunit.client.1.smithi152.stdout:7/905: stat d14/f42 0 2024-03-04T16:04:56.726 INFO:tasks.workunit.client.1.smithi152.stdout:2/955: creat da/df/d2e/d51/d11c/d80/d82/da6/f14a x:0 0 0 2024-03-04T16:04:56.727 INFO:tasks.workunit.client.1.smithi152.stdout:2/956: chown da/df/d2e/d51/d11c/d47/d4f 2504 1 2024-03-04T16:04:56.728 INFO:tasks.workunit.client.1.smithi152.stdout:9/989: write d2/d13/d23/fa0 [4821255,40899] 0 2024-03-04T16:04:56.729 INFO:tasks.workunit.client.1.smithi152.stdout:0/936: creat d9/d1b/d54/d80/da8/d120/f133 x:0 0 0 2024-03-04T16:04:56.730 INFO:tasks.workunit.client.1.smithi152.stdout:9/990: read d2/d13/d96/fe7 [234809,26461] 0 2024-03-04T16:04:56.731 INFO:tasks.workunit.client.1.smithi152.stdout:0/937: write d9/d1a/f79 [1474831,49330] 0 2024-03-04T16:04:56.732 INFO:tasks.workunit.client.1.smithi152.stdout:5/915: link d6/d8/d1f/d3d/f4c d6/d8/d5a/d64/d76/d9f/f12b 0 2024-03-04T16:04:56.734 INFO:tasks.workunit.client.1.smithi152.stdout:5/916: read d6/d8/d5a/d90/daf/fbf [837342,4277] 0 2024-03-04T16:04:56.740 INFO:tasks.workunit.client.1.smithi152.stdout:9/991: creat d2/d13/d23/d71/d102/f146 x:0 0 0 2024-03-04T16:04:56.742 INFO:tasks.workunit.client.1.smithi152.stdout:0/938: creat d9/d2c/d2e/d7f/f134 x:0 0 0 2024-03-04T16:04:56.751 INFO:tasks.workunit.client.1.smithi152.stdout:9/992: read d2/d13/f93 [243929,42626] 0 2024-03-04T16:04:56.752 INFO:tasks.workunit.client.1.smithi152.stdout:2/957: write da/df/fb7 [711604,53440] 0 2024-03-04T16:04:56.755 INFO:tasks.workunit.client.1.smithi152.stdout:7/906: dwrite d14/d21/f30 [0,4194304] 0 2024-03-04T16:04:56.756 INFO:tasks.workunit.client.1.smithi152.stdout:7/907: dread - d14/d9d/fdd zero size 2024-03-04T16:04:56.757 INFO:tasks.workunit.client.1.smithi152.stdout:9/993: mknod d2/d13/d23/d66/d7d/c147 0 2024-03-04T16:04:56.757 INFO:tasks.workunit.client.1.smithi152.stdout:5/917: creat d6/d8/d5a/d64/d76/f12c x:0 0 0 2024-03-04T16:04:56.763 INFO:tasks.workunit.client.1.smithi152.stdout:2/958: link da/d5f/d85/f13d da/d2c/f14b 0 2024-03-04T16:04:56.765 INFO:tasks.workunit.client.1.smithi152.stdout:7/908: dwrite d14/d21/f25 [0,4194304] 0 2024-03-04T16:04:56.772 INFO:tasks.workunit.client.1.smithi152.stdout:2/959: mknod da/df/d2e/d51/d8d/c14c 0 2024-03-04T16:04:56.774 INFO:tasks.workunit.client.1.smithi152.stdout:5/918: dwrite d6/d8/d1f/d3d/d6f/dbb/f121 [0,4194304] 0 2024-03-04T16:04:56.775 INFO:tasks.workunit.client.1.smithi152.stdout:9/994: dwrite d2/f67 [0,4194304] 0 2024-03-04T16:04:56.776 INFO:tasks.workunit.client.1.smithi152.stdout:5/919: chown d6/d8/d1f/f44 180301805 1 2024-03-04T16:04:56.780 INFO:tasks.workunit.client.1.smithi152.stdout:2/960: mkdir da/d2c/ddd/d14d 0 2024-03-04T16:04:56.781 INFO:tasks.workunit.client.1.smithi152.stdout:2/961: chown da/df/d2e/l3b 206872 1 2024-03-04T16:04:56.789 INFO:tasks.workunit.client.1.smithi152.stdout:5/920: getdents d6/d8/d1f/d3d/d6f/dbb 0 2024-03-04T16:04:56.797 INFO:tasks.workunit.client.1.smithi152.stdout:7/909: dwrite d14/d21/d47/d36/da7/fb1 [0,4194304] 0 2024-03-04T16:04:56.800 INFO:tasks.workunit.client.1.smithi152.stdout:9/995: write d2/d13/dad/dd2/d111/fc0 [285078,8877] 0 2024-03-04T16:04:56.801 INFO:tasks.workunit.client.1.smithi152.stdout:7/910: truncate d14/d21/d47/d40/f4c 277448 0 2024-03-04T16:04:56.802 INFO:tasks.workunit.client.1.smithi152.stdout:7/911: chown d14/c2d 18360993 1 2024-03-04T16:04:56.808 INFO:tasks.workunit.client.1.smithi152.stdout:2/962: dwrite da/d2c/d3f/f86 [0,4194304] 0 2024-03-04T16:04:56.808 INFO:tasks.workunit.client.1.smithi152.stdout:7/912: creat d14/d21/d47/f109 x:0 0 0 2024-03-04T16:04:56.812 INFO:tasks.workunit.client.1.smithi152.stdout:5/921: dwrite d6/d8/d1f/d3d/d6f/d70/f71 [4194304,4194304] 0 2024-03-04T16:04:56.812 INFO:tasks.workunit.client.1.smithi152.stdout:7/913: mkdir d14/db9/dfd/d10a 0 2024-03-04T16:04:56.813 INFO:tasks.workunit.client.1.smithi152.stdout:9/996: dread d2/d13/f142 [0,4194304] 0 2024-03-04T16:04:56.815 INFO:tasks.workunit.client.1.smithi152.stdout:9/997: chown d2/d13/dad/l115 456998503 1 2024-03-04T16:04:56.816 INFO:tasks.workunit.client.1.smithi152.stdout:2/963: symlink da/df/d2e/d51/d8d/d9a/df7/l14e 0 2024-03-04T16:04:56.819 INFO:tasks.workunit.client.1.smithi152.stdout:9/998: fsync d2/d13/dad/dd2/d111/f95 0 2024-03-04T16:04:56.819 INFO:tasks.workunit.client.1.smithi152.stdout:9/999: chown d2/d13/d35/d10c/c137 450 1 2024-03-04T16:04:56.820 INFO:tasks.workunit.client.1.smithi152.stdout:7/914: creat d14/db9/dc9/d83/d105/f10b x:0 0 0 2024-03-04T16:04:56.821 INFO:tasks.workunit.client.1.smithi152.stdout:2/964: fdatasync da/df/d2e/fa1 0 2024-03-04T16:04:56.821 INFO:tasks.workunit.client.1.smithi152.stdout:7/915: chown d14/c2d 242520528 1 2024-03-04T16:04:56.823 INFO:tasks.workunit.client.1.smithi152.stdout:5/922: mkdir d6/d8/d5a/d64/d76/d9f/d12d 0 2024-03-04T16:04:56.825 INFO:tasks.workunit.client.1.smithi152.stdout:2/965: creat da/d2c/f14f x:0 0 0 2024-03-04T16:04:56.827 INFO:tasks.workunit.client.1.smithi152.stdout:5/923: chown d6/d8/d5a/d64/d76/f7d 800 1 2024-03-04T16:04:56.837 INFO:tasks.workunit.client.1.smithi152.stdout:5/924: creat d6/d8/d5a/d8b/f12e x:0 0 0 2024-03-04T16:04:56.838 INFO:tasks.workunit.client.1.smithi152.stdout:7/916: link d14/d21/d47/d40/l59 d14/d21/d47/d36/da7/dbc/de6/l10c 0 2024-03-04T16:04:56.839 INFO:tasks.workunit.client.1.smithi152.stdout:7/917: truncate d14/d21/d31/fee 280733 0 2024-03-04T16:04:56.842 INFO:tasks.workunit.client.1.smithi152.stdout:5/925: creat d6/d8/d1f/d3d/d6f/d70/d98/de7/f12f x:0 0 0 2024-03-04T16:04:56.846 INFO:tasks.workunit.client.1.smithi152.stdout:7/918: mkdir d14/d21/d47/d36/da7/dbc/de6/df7/d10d 0 2024-03-04T16:04:56.846 INFO:tasks.workunit.client.1.smithi152.stdout:7/919: chown d14/d21/d47/d36/c38 4993724 1 2024-03-04T16:04:56.852 INFO:tasks.workunit.client.1.smithi152.stdout:2/966: dread da/f77 [0,4194304] 0 2024-03-04T16:04:56.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:56 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:56.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:56 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:56.863 INFO:tasks.workunit.client.1.smithi152.stdout:5/926: dwrite d6/d50/f59 [0,4194304] 0 2024-03-04T16:04:56.865 INFO:tasks.workunit.client.1.smithi152.stdout:7/920: dwrite d14/db9/dc9/f63 [8388608,4194304] 0 2024-03-04T16:04:56.869 INFO:tasks.workunit.client.1.smithi152.stdout:7/921: write d14/f2c [234867,97819] 0 2024-03-04T16:04:56.869 INFO:tasks.workunit.client.1.smithi152.stdout:5/927: stat d6/d8/d1f/d3d/d6f/d70/d98/f127 0 2024-03-04T16:04:56.869 INFO:tasks.workunit.client.1.smithi152.stdout:2/967: dwrite da/df/d2e/d51/fb4 [0,4194304] 0 2024-03-04T16:04:56.872 INFO:tasks.workunit.client.1.smithi152.stdout:7/922: readlink d14/d71/lf3 0 2024-03-04T16:04:56.878 INFO:tasks.workunit.client.1.smithi152.stdout:5/928: rmdir d6/d8/d1f 39 2024-03-04T16:04:56.879 INFO:tasks.workunit.client.1.smithi152.stdout:7/923: mknod d14/d9d/c10e 0 2024-03-04T16:04:56.883 INFO:tasks.workunit.client.1.smithi152.stdout:2/968: truncate da/df/d2e/d51/d11c/d81/fce 1740453 0 2024-03-04T16:04:56.887 INFO:tasks.workunit.client.1.smithi152.stdout:2/969: creat da/df/d2e/d51/d11c/d80/d82/da6/f150 x:0 0 0 2024-03-04T16:04:56.889 INFO:tasks.workunit.client.1.smithi152.stdout:2/970: truncate da/df/d2e/d51/d11c/d80/fd3 4734777 0 2024-03-04T16:04:56.890 INFO:tasks.workunit.client.1.smithi152.stdout:5/929: dread d6/d8/d5a/d64/fd0 [0,4194304] 0 2024-03-04T16:04:56.890 INFO:tasks.workunit.client.1.smithi152.stdout:5/930: stat d6/d8/d5a/d90/l11b 0 2024-03-04T16:04:56.892 INFO:tasks.workunit.client.1.smithi152.stdout:7/924: getdents d14/db9/dc9/d83/d105 0 2024-03-04T16:04:56.898 INFO:tasks.workunit.client.1.smithi152.stdout:7/925: creat d14/d21/d31/dde/f10f x:0 0 0 2024-03-04T16:04:56.903 INFO:tasks.workunit.client.1.smithi152.stdout:4/921: sync 2024-03-04T16:04:56.903 INFO:tasks.workunit.client.1.smithi152.stdout:0/939: sync 2024-03-04T16:04:56.907 INFO:tasks.workunit.client.1.smithi152.stdout:2/971: dwrite da/f16 [0,4194304] 0 2024-03-04T16:04:56.907 INFO:tasks.workunit.client.1.smithi152.stdout:5/931: dwrite d6/d8/ff [0,4194304] 0 2024-03-04T16:04:56.907 INFO:tasks.workunit.client.1.smithi152.stdout:7/926: truncate d14/d69/ffa 562776 0 2024-03-04T16:04:56.909 INFO:tasks.workunit.client.1.smithi152.stdout:5/932: write d6/d50/fca [1055307,77644] 0 2024-03-04T16:04:56.914 INFO:tasks.workunit.client.1.smithi152.stdout:5/933: chown d6/d8/d5a/d64 455 1 2024-03-04T16:04:56.922 INFO:tasks.workunit.client.1.smithi152.stdout:7/927: truncate d14/d21/d47/d36/f73 70518 0 2024-03-04T16:04:56.926 INFO:tasks.workunit.client.1.smithi152.stdout:7/928: unlink d14/db9/dc9/f63 0 2024-03-04T16:04:56.928 INFO:tasks.workunit.client.1.smithi152.stdout:0/940: dwrite d9/d2c/d2e/fa0 [0,4194304] 0 2024-03-04T16:04:56.930 INFO:tasks.workunit.client.1.smithi152.stdout:2/972: getdents da/df/d2e/d51/d8d 0 2024-03-04T16:04:56.934 INFO:tasks.workunit.client.1.smithi152.stdout:7/929: mkdir d14/d69/d110 0 2024-03-04T16:04:56.940 INFO:tasks.workunit.client.1.smithi152.stdout:2/973: dread - da/df/d2e/d51/d11c/d47/f4d zero size 2024-03-04T16:04:56.941 INFO:tasks.workunit.client.1.smithi152.stdout:4/922: write d8/d9/d24/fd2 [481237,64545] 0 2024-03-04T16:04:56.941 INFO:tasks.workunit.client.1.smithi152.stdout:0/941: write d9/d1b/d54/d80/f9e [27614,86024] 0 2024-03-04T16:04:56.941 INFO:tasks.workunit.client.1.smithi152.stdout:4/923: dread - d8/d58/ded/f118 zero size 2024-03-04T16:04:56.941 INFO:tasks.workunit.client.1.smithi152.stdout:4/924: stat d8/f3b 0 2024-03-04T16:04:56.942 INFO:tasks.workunit.client.1.smithi152.stdout:7/930: mknod d14/d21/d47/c111 0 2024-03-04T16:04:56.945 INFO:tasks.workunit.client.1.smithi152.stdout:2/974: mknod da/df/d2e/d51/d11c/d80/d82/da6/c151 0 2024-03-04T16:04:56.948 INFO:tasks.workunit.client.1.smithi152.stdout:7/931: creat d14/db9/f112 x:0 0 0 2024-03-04T16:04:56.950 INFO:tasks.workunit.client.1.smithi152.stdout:5/934: dwrite d6/d8/d1f/d3d/d6f/f95 [4194304,4194304] 0 2024-03-04T16:04:56.951 INFO:tasks.workunit.client.1.smithi152.stdout:2/975: symlink da/df/d2e/d51/d8d/d9a/d116/l152 0 2024-03-04T16:04:56.953 INFO:tasks.workunit.client.1.smithi152.stdout:5/935: write d6/d8/d1f/d3d/d6f/f126 [868713,26110] 0 2024-03-04T16:04:56.953 INFO:tasks.workunit.client.1.smithi152.stdout:4/925: symlink d8/d2f/d7c/l13f 0 2024-03-04T16:04:56.954 INFO:tasks.workunit.client.1.smithi152.stdout:4/926: chown d8/d2f/ce2 4807 1 2024-03-04T16:04:56.955 INFO:tasks.workunit.client.1.smithi152.stdout:7/932: creat d14/d21/d47/d36/dcf/f113 x:0 0 0 2024-03-04T16:04:56.956 INFO:tasks.workunit.client.1.smithi152.stdout:7/933: stat d14/d71/f98 0 2024-03-04T16:04:56.957 INFO:tasks.workunit.client.1.smithi152.stdout:0/942: creat d9/d1b/d3c/d84/f135 x:0 0 0 2024-03-04T16:04:56.961 INFO:tasks.workunit.client.1.smithi152.stdout:2/976: rename da/la0 to da/df/d2e/dc6/l153 0 2024-03-04T16:04:56.964 INFO:tasks.workunit.client.1.smithi152.stdout:5/936: symlink d6/d8/d5a/d64/d76/d9f/dc7/l130 0 2024-03-04T16:04:56.965 INFO:tasks.workunit.client.1.smithi152.stdout:4/927: stat d8/d9/d7a/f8f 0 2024-03-04T16:04:56.966 INFO:tasks.workunit.client.1.smithi152.stdout:4/928: write d8/d9/f9e [185882,90893] 0 2024-03-04T16:04:56.967 INFO:tasks.workunit.client.1.smithi152.stdout:2/977: dwrite da/df/d58/d6e/f89 [0,4194304] 0 2024-03-04T16:04:56.968 INFO:tasks.workunit.client.1.smithi152.stdout:4/929: readlink d8/d58/laa 0 2024-03-04T16:04:56.969 INFO:tasks.workunit.client.1.smithi152.stdout:2/978: write da/df/d2e/f90 [3998248,21923] 0 2024-03-04T16:04:56.970 INFO:tasks.workunit.client.1.smithi152.stdout:7/934: symlink d14/db9/l114 0 2024-03-04T16:04:56.972 INFO:tasks.workunit.client.1.smithi152.stdout:7/935: fdatasync d14/d21/d31/fee 0 2024-03-04T16:04:56.976 INFO:tasks.workunit.client.1.smithi152.stdout:4/930: stat d8/d58/c12e 0 2024-03-04T16:04:56.979 INFO:tasks.workunit.client.1.smithi152.stdout:2/979: truncate da/df/d2e/f10e 758124 0 2024-03-04T16:04:56.982 INFO:tasks.workunit.client.1.smithi152.stdout:4/931: truncate d8/d78/fb7 3287350 0 2024-03-04T16:04:56.984 INFO:tasks.workunit.client.1.smithi152.stdout:2/980: creat da/df/d2e/d51/d8d/d9a/df7/f154 x:0 0 0 2024-03-04T16:04:56.988 INFO:tasks.workunit.client.1.smithi152.stdout:0/943: link d9/d1b/d3c/d84/d56/d5b/cae d9/d2c/d2e/d7f/d9d/de8/df4/c136 0 2024-03-04T16:04:56.990 INFO:tasks.workunit.client.1.smithi152.stdout:2/981: rename da/df/d58/da2/fb3 to da/df/d2e/d51/d11c/d47/d73/f155 0 2024-03-04T16:04:56.992 INFO:tasks.workunit.client.1.smithi152.stdout:0/944: mknod d9/d1b/d3c/d84/d56/d5b/d7a/c137 0 2024-03-04T16:04:56.994 INFO:tasks.workunit.client.1.smithi152.stdout:5/937: dread d6/d8/d5a/d8b/fb1 [0,4194304] 0 2024-03-04T16:04:56.994 INFO:tasks.workunit.client.1.smithi152.stdout:2/982: creat da/df/d58/da2/db5/f156 x:0 0 0 2024-03-04T16:04:56.999 INFO:tasks.workunit.client.1.smithi152.stdout:5/938: creat d6/d8/d1f/d3d/d6f/dbb/de3/d108/f131 x:0 0 0 2024-03-04T16:04:57.000 INFO:tasks.workunit.client.1.smithi152.stdout:7/936: write d14/d71/f7d [307255,5762] 0 2024-03-04T16:04:57.001 INFO:tasks.workunit.client.1.smithi152.stdout:0/945: rmdir d9/d1b/d54/d69/d98/d10f 0 2024-03-04T16:04:57.006 INFO:tasks.workunit.client.1.smithi152.stdout:4/932: write d8/d2f/f4a [1326113,129737] 0 2024-03-04T16:04:57.009 INFO:tasks.workunit.client.1.smithi152.stdout:7/937: mkdir d14/d21/d31/dde/d115 0 2024-03-04T16:04:57.010 INFO:tasks.workunit.client.1.smithi152.stdout:7/938: dread - d14/d21/d47/d36/da7/dbc/de6/ffc zero size 2024-03-04T16:04:57.012 INFO:tasks.workunit.client.1.smithi152.stdout:5/939: creat d6/d8/d1f/d3d/d6f/d70/dfb/f132 x:0 0 0 2024-03-04T16:04:57.019 INFO:tasks.workunit.client.1.smithi152.stdout:2/983: getdents da/df/d2e/d51/d11c/d47/dcf 0 2024-03-04T16:04:57.020 INFO:tasks.workunit.client.1.smithi152.stdout:4/933: dwrite d8/d9/da2/f108 [0,4194304] 0 2024-03-04T16:04:57.027 INFO:tasks.workunit.client.1.smithi152.stdout:7/939: dwrite d14/f6a [0,4194304] 0 2024-03-04T16:04:57.032 INFO:tasks.workunit.client.1.smithi152.stdout:5/940: mknod d6/d8/d1f/d3d/da4/c133 0 2024-03-04T16:04:57.033 INFO:tasks.workunit.client.1.smithi152.stdout:0/946: getdents d9/d2c/d2e/d94/dac 0 2024-03-04T16:04:57.037 INFO:tasks.workunit.client.1.smithi152.stdout:7/940: symlink d14/db9/dc9/d83/de5/l116 0 2024-03-04T16:04:57.040 INFO:tasks.workunit.client.1.smithi152.stdout:4/934: dread d8/d9/d24/d26/f2a [0,4194304] 0 2024-03-04T16:04:57.044 INFO:tasks.workunit.client.1.smithi152.stdout:5/941: dwrite d6/f111 [0,4194304] 0 2024-03-04T16:04:57.044 INFO:tasks.workunit.client.1.smithi152.stdout:7/941: chown d14/db9/dc9/d83/ld0 2047343408 1 2024-03-04T16:04:57.045 INFO:tasks.workunit.client.1.smithi152.stdout:0/947: dwrite d9/d2c/d2e/d7f/d9d/fcd [0,4194304] 0 2024-03-04T16:04:57.049 INFO:tasks.workunit.client.1.smithi152.stdout:4/935: symlink d8/d2f/d6f/d90/d121/l140 0 2024-03-04T16:04:57.059 INFO:tasks.workunit.client.1.smithi152.stdout:2/984: truncate da/df/fae 3486102 0 2024-03-04T16:04:57.063 INFO:tasks.workunit.client.1.smithi152.stdout:7/942: mkdir d14/d21/d48/d117 0 2024-03-04T16:04:57.064 INFO:tasks.workunit.client.1.smithi152.stdout:0/948: symlink d9/d92/l138 0 2024-03-04T16:04:57.067 INFO:tasks.workunit.client.1.smithi152.stdout:7/943: dread d14/d71/f98 [0,4194304] 0 2024-03-04T16:04:57.071 INFO:tasks.workunit.client.1.smithi152.stdout:7/944: write d14/db9/dc9/f58 [2897246,105152] 0 2024-03-04T16:04:57.073 INFO:tasks.workunit.client.1.smithi152.stdout:0/949: dwrite d9/d2c/d2e/fa1 [0,4194304] 0 2024-03-04T16:04:57.075 INFO:tasks.workunit.client.1.smithi152.stdout:0/950: dread - d9/d1a/db4/ff6 zero size 2024-03-04T16:04:57.076 INFO:tasks.workunit.client.1.smithi152.stdout:7/945: dread d14/d21/d31/fee [0,4194304] 0 2024-03-04T16:04:57.079 INFO:tasks.workunit.client.1.smithi152.stdout:4/936: getdents d8/d9/d24 0 2024-03-04T16:04:57.085 INFO:tasks.workunit.client.1.smithi152.stdout:0/951: fsync d9/d2c/d2e/d7f/d9d/fbd 0 2024-03-04T16:04:57.085 INFO:tasks.workunit.client.1.smithi152.stdout:5/942: dwrite d6/d50/fb3 [0,4194304] 0 2024-03-04T16:04:57.096 INFO:tasks.workunit.client.1.smithi152.stdout:7/946: fdatasync d14/d69/d8e/fa5 0 2024-03-04T16:04:57.098 INFO:tasks.workunit.client.1.smithi152.stdout:4/937: dread d8/f37 [0,4194304] 0 2024-03-04T16:04:57.098 INFO:tasks.workunit.client.1.smithi152.stdout:4/938: readlink d8/d9/d24/d26/d42/l6e 0 2024-03-04T16:04:57.105 INFO:tasks.workunit.client.1.smithi152.stdout:0/952: mknod d9/d2c/d2e/d7f/d9d/da3/c139 0 2024-03-04T16:04:57.106 INFO:tasks.workunit.client.1.smithi152.stdout:2/985: dwrite da/df/d2e/d51/d11c/d80/dd0/fd7 [0,4194304] 0 2024-03-04T16:04:57.108 INFO:tasks.workunit.client.1.smithi152.stdout:2/986: chown da/d2c/d3f/f72 482330349 1 2024-03-04T16:04:57.109 INFO:tasks.workunit.client.1.smithi152.stdout:4/939: unlink d8/d9/d24/d26/l31 0 2024-03-04T16:04:57.114 INFO:tasks.workunit.client.1.smithi152.stdout:0/953: read - d9/d1b/d3c/d84/d56/d5b/ff3 zero size 2024-03-04T16:04:57.114 INFO:tasks.workunit.client.1.smithi152.stdout:0/954: stat d9/d1a/d7c/f112 0 2024-03-04T16:04:57.115 INFO:tasks.workunit.client.1.smithi152.stdout:7/947: getdents d14/db9/dfd/d10a 0 2024-03-04T16:04:57.121 INFO:tasks.workunit.client.1.smithi152.stdout:7/948: symlink d14/d21/d47/d36/da7/dbc/l118 0 2024-03-04T16:04:57.122 INFO:tasks.workunit.client.1.smithi152.stdout:0/955: dread d9/d1b/d3c/d84/d56/d5b/d7a/f8f [0,4194304] 0 2024-03-04T16:04:57.124 INFO:tasks.workunit.client.1.smithi152.stdout:2/987: rename da/df/d35/d92/f124 to da/df/d58/f157 0 2024-03-04T16:04:57.126 INFO:tasks.workunit.client.1.smithi152.stdout:5/943: dwrite d6/d8/f15 [0,4194304] 0 2024-03-04T16:04:57.126 INFO:tasks.workunit.client.1.smithi152.stdout:2/988: stat da/df/d2e/d51/d11c/d80/d82/da8/ff6 0 2024-03-04T16:04:57.126 INFO:tasks.workunit.client.1.smithi152.stdout:4/940: creat d8/f141 x:0 0 0 2024-03-04T16:04:57.129 INFO:tasks.workunit.client.1.smithi152.stdout:2/989: write da/d5f/dbc/fcc [4499546,58632] 0 2024-03-04T16:04:57.133 INFO:tasks.workunit.client.1.smithi152.stdout:2/990: dread - da/df/d2e/f8f zero size 2024-03-04T16:04:57.136 INFO:tasks.workunit.client.1.smithi152.stdout:2/991: write da/d2c/d3f/f86 [64682,92964] 0 2024-03-04T16:04:57.137 INFO:tasks.workunit.client.1.smithi152.stdout:7/949: mkdir d14/d21/d47/d36/da7/dbc/de6/df7/d119 0 2024-03-04T16:04:57.137 INFO:tasks.workunit.client.1.smithi152.stdout:4/941: dwrite d8/d4e/d8b/f13c [0,4194304] 0 2024-03-04T16:04:57.139 INFO:tasks.workunit.client.1.smithi152.stdout:4/942: write d8/d2f/d6f/f73 [2582451,128865] 0 2024-03-04T16:04:57.150 INFO:tasks.workunit.client.1.smithi152.stdout:7/950: mkdir d14/d21/d47/d40/d6e/d11a 0 2024-03-04T16:04:57.150 INFO:tasks.workunit.client.1.smithi152.stdout:4/943: creat d8/d9/dc1/f142 x:0 0 0 2024-03-04T16:04:57.155 INFO:tasks.workunit.client.1.smithi152.stdout:4/944: dread d8/f37 [0,4194304] 0 2024-03-04T16:04:57.157 INFO:tasks.workunit.client.1.smithi152.stdout:7/951: readlink d14/d21/d31/dde/le1 0 2024-03-04T16:04:57.158 INFO:tasks.workunit.client.1.smithi152.stdout:0/956: getdents d9/d1a/db4 0 2024-03-04T16:04:57.166 INFO:tasks.workunit.client.1.smithi152.stdout:2/992: getdents da/d5f/db0/d114 0 2024-03-04T16:04:57.166 INFO:tasks.workunit.client.1.smithi152.stdout:5/944: dread d6/d8/d5a/d64/d76/d9f/db0/fe5 [0,4194304] 0 2024-03-04T16:04:57.167 INFO:tasks.workunit.client.1.smithi152.stdout:5/945: write d6/d8/d5a/f105 [383301,28903] 0 2024-03-04T16:04:57.169 INFO:tasks.workunit.client.1.smithi152.stdout:0/957: creat d9/d1b/d54/d80/da8/d12e/f13a x:0 0 0 2024-03-04T16:04:57.169 INFO:tasks.workunit.client.1.smithi152.stdout:7/952: truncate d14/d21/f49 291188 0 2024-03-04T16:04:57.169 INFO:tasks.workunit.client.1.smithi152.stdout:5/946: write d6/d8/d1f/d3d/d6f/d70/f71 [950595,7130] 0 2024-03-04T16:04:57.171 INFO:tasks.workunit.client.1.smithi152.stdout:4/945: rmdir d8/d2f/d7c/d87/d12f 0 2024-03-04T16:04:57.171 INFO:tasks.workunit.client.1.smithi152.stdout:5/947: read - d6/d8/d1f/d3d/d6f/d70/d98/de7/f12f zero size 2024-03-04T16:04:57.173 INFO:tasks.workunit.client.1.smithi152.stdout:4/946: write d8/d58/ded/f11f [4359657,66127] 0 2024-03-04T16:04:57.173 INFO:tasks.workunit.client.1.smithi152.stdout:2/993: rename da/d2c/d3f/f72 to da/df/d58/d6e/d11f/f158 0 2024-03-04T16:04:57.173 INFO:tasks.workunit.client.1.smithi152.stdout:4/947: dread - d8/d4e/f103 zero size 2024-03-04T16:04:57.174 INFO:tasks.workunit.client.1.smithi152.stdout:4/948: chown d8/d9/da2 30940 1 2024-03-04T16:04:57.175 INFO:tasks.workunit.client.1.smithi152.stdout:0/958: mknod d9/d1a/db4/c13b 0 2024-03-04T16:04:57.176 INFO:tasks.workunit.client.1.smithi152.stdout:7/953: symlink d14/db9/dc9/d83/de5/l11b 0 2024-03-04T16:04:57.181 INFO:tasks.workunit.client.1.smithi152.stdout:4/949: dwrite d8/d9/da2/f101 [0,4194304] 0 2024-03-04T16:04:57.187 INFO:tasks.workunit.client.1.smithi152.stdout:0/959: truncate d9/d1b/d54/d80/f97 3351706 0 2024-03-04T16:04:57.189 INFO:tasks.workunit.client.1.smithi152.stdout:2/994: rename da/df/d2e/d51/d11c/d80 to da/df/d2e/d159 0 2024-03-04T16:04:57.193 INFO:tasks.workunit.client.1.smithi152.stdout:0/960: unlink d9/d2c/d2e/f110 0 2024-03-04T16:04:57.196 INFO:tasks.workunit.client.1.smithi152.stdout:2/995: dwrite da/df/d2e/f30 [4194304,4194304] 0 2024-03-04T16:04:57.197 INFO:tasks.workunit.client.1.smithi152.stdout:2/996: chown da/fc 800249109 1 2024-03-04T16:04:57.199 INFO:tasks.workunit.client.1.smithi152.stdout:0/961: creat d9/d2c/f13c x:0 0 0 2024-03-04T16:04:57.200 INFO:tasks.workunit.client.1.smithi152.stdout:0/962: chown d9/d1b/d54/d69/f9a 1712004 1 2024-03-04T16:04:57.201 INFO:tasks.workunit.client.1.smithi152.stdout:0/963: write d9/d2c/d2e/f32 [3150274,19658] 0 2024-03-04T16:04:57.203 INFO:tasks.workunit.client.1.smithi152.stdout:7/954: rename d14/db9/dc9/d83/db7/df6 to d14/d11c 0 2024-03-04T16:04:57.209 INFO:tasks.workunit.client.1.smithi152.stdout:0/964: symlink d9/d2c/d2e/l13d 0 2024-03-04T16:04:57.211 INFO:tasks.workunit.client.1.smithi152.stdout:7/955: creat d14/f11d x:0 0 0 2024-03-04T16:04:57.212 INFO:tasks.workunit.client.1.smithi152.stdout:0/965: rmdir d9/d1a/d7c 39 2024-03-04T16:04:57.218 INFO:tasks.workunit.client.1.smithi152.stdout:0/966: mkdir d9/d1b/d54/d80/da8/d12e/d13e 0 2024-03-04T16:04:57.223 INFO:tasks.workunit.client.1.smithi152.stdout:0/967: mknod d9/d1b/d54/d80/c13f 0 2024-03-04T16:04:57.227 INFO:tasks.workunit.client.1.smithi152.stdout:0/968: unlink d9/d2c/d2e/d7f/d9d/fcd 0 2024-03-04T16:04:57.228 INFO:tasks.workunit.client.1.smithi152.stdout:0/969: fdatasync d9/d2c/d2e/d7f/d9d/da3/faf 0 2024-03-04T16:04:57.233 INFO:tasks.workunit.client.1.smithi152.stdout:0/970: creat d9/d1a/d7c/f140 x:0 0 0 2024-03-04T16:04:57.243 INFO:tasks.workunit.client.1.smithi152.stdout:5/948: dwrite d6/d8/d1f/d88/fcf [0,4194304] 0 2024-03-04T16:04:57.249 INFO:tasks.workunit.client.1.smithi152.stdout:4/950: write d8/d2f/f47 [5003183,102835] 0 2024-03-04T16:04:57.428 INFO:tasks.workunit.client.1.smithi152.stdout:7/956: write d14/d21/d47/d40/fc4 [551389,80772] 0 2024-03-04T16:04:57.444 INFO:tasks.workunit.client.1.smithi152.stdout:4/951: creat d8/f143 x:0 0 0 2024-03-04T16:04:57.446 INFO:tasks.workunit.client.1.smithi152.stdout:4/952: creat d8/d2f/d6f/d84/f144 x:0 0 0 2024-03-04T16:04:57.449 INFO:tasks.workunit.client.1.smithi152.stdout:0/971: dwrite d9/d1b/f1d [0,4194304] 0 2024-03-04T16:04:57.452 INFO:tasks.workunit.client.1.smithi152.stdout:4/953: creat d8/d9/d24/da0/f145 x:0 0 0 2024-03-04T16:04:57.453 INFO:tasks.workunit.client.1.smithi152.stdout:4/954: write d8/d9/f9e [1012921,14617] 0 2024-03-04T16:04:57.456 INFO:tasks.workunit.client.1.smithi152.stdout:0/972: getdents d9/d2c/d2e/d94/dac 0 2024-03-04T16:04:57.460 INFO:tasks.workunit.client.1.smithi152.stdout:0/973: dread - d9/d1b/d54/d80/da8/f113 zero size 2024-03-04T16:04:57.464 INFO:tasks.workunit.client.1.smithi152.stdout:4/955: dwrite d8/d9/da2/f101 [4194304,4194304] 0 2024-03-04T16:04:57.469 INFO:tasks.workunit.client.1.smithi152.stdout:4/956: mknod d8/d89/db3/c146 0 2024-03-04T16:04:57.470 INFO:tasks.workunit.client.1.smithi152.stdout:5/949: dwrite d6/d8/f86 [0,4194304] 0 2024-03-04T16:04:57.474 INFO:tasks.workunit.client.1.smithi152.stdout:5/950: truncate d6/d50/fc8 5773805 0 2024-03-04T16:04:57.479 INFO:tasks.workunit.client.1.smithi152.stdout:4/957: unlink d8/l17 0 2024-03-04T16:04:57.483 INFO:tasks.workunit.client.1.smithi152.stdout:5/951: unlink d6/d8/d1f/d88/fcf 0 2024-03-04T16:04:57.487 INFO:tasks.workunit.client.1.smithi152.stdout:4/958: unlink d8/d2f/d6f/d84/fdd 0 2024-03-04T16:04:57.492 INFO:tasks.workunit.client.1.smithi152.stdout:5/952: creat d6/d8/d1f/d3d/d6f/d70/dc4/f134 x:0 0 0 2024-03-04T16:04:57.493 INFO:tasks.workunit.client.1.smithi152.stdout:0/974: dread d9/d1a/d7c/faa [0,4194304] 0 2024-03-04T16:04:57.495 INFO:tasks.workunit.client.1.smithi152.stdout:0/975: write d9/d2c/f13c [204027,52015] 0 2024-03-04T16:04:57.498 INFO:tasks.workunit.client.1.smithi152.stdout:5/953: mkdir d6/d8/d5a/d90/dd6/d135 0 2024-03-04T16:04:57.503 INFO:tasks.workunit.client.1.smithi152.stdout:0/976: symlink d9/d2c/d2e/d7f/l141 0 2024-03-04T16:04:57.506 INFO:tasks.workunit.client.1.smithi152.stdout:4/959: rmdir d8/d3d/d105/d135 0 2024-03-04T16:04:57.508 INFO:tasks.workunit.client.1.smithi152.stdout:5/954: fsync d6/d8/d1f/d3d/d6f/fdf 0 2024-03-04T16:04:57.514 INFO:tasks.workunit.client.1.smithi152.stdout:5/955: stat d6/d8/d1f/d3d/d6f/d70/ce8 0 2024-03-04T16:04:57.515 INFO:tasks.workunit.client.1.smithi152.stdout:2/997: sync 2024-03-04T16:04:57.519 INFO:tasks.workunit.client.1.smithi152.stdout:2/998: symlink da/df/d2e/d51/d8d/d9a/d116/l15a 0 2024-03-04T16:04:57.525 INFO:tasks.workunit.client.1.smithi152.stdout:2/999: creat da/df/d35/f15b x:0 0 0 2024-03-04T16:04:57.557 INFO:tasks.workunit.client.1.smithi152.stdout:4/960: truncate d8/d4e/d8b/fa9 403392 0 2024-03-04T16:04:57.558 INFO:tasks.workunit.client.1.smithi152.stdout:0/977: creat d9/d1b/d54/f142 x:0 0 0 2024-03-04T16:04:57.559 INFO:tasks.workunit.client.1.smithi152.stdout:4/961: write d8/d58/ded/f11f [1468485,128775] 0 2024-03-04T16:04:57.563 INFO:tasks.workunit.client.1.smithi152.stdout:4/962: creat d8/d2f/d130/f147 x:0 0 0 2024-03-04T16:04:57.564 INFO:tasks.workunit.client.1.smithi152.stdout:4/963: write d8/d58/ded/f11f [2294617,68172] 0 2024-03-04T16:04:57.614 INFO:tasks.workunit.client.1.smithi152.stdout:4/964: creat d8/d4e/f148 x:0 0 0 2024-03-04T16:04:57.615 INFO:tasks.workunit.client.1.smithi152.stdout:4/965: write d8/d9/d7a/f114 [656974,52832] 0 2024-03-04T16:04:57.618 INFO:tasks.workunit.client.1.smithi152.stdout:4/966: rename d8/d2f/d6f/deb/def/f112 to d8/d2f/d6f/deb/f149 0 2024-03-04T16:04:57.624 INFO:tasks.workunit.client.1.smithi152.stdout:4/967: link d8/d4e/dba/dcc/c138 d8/d9/d24/d26/d42/d115/c14a 0 2024-03-04T16:04:57.625 INFO:tasks.workunit.client.1.smithi152.stdout:0/978: dwrite d9/d1b/f35 [0,4194304] 0 2024-03-04T16:04:57.633 INFO:tasks.workunit.client.1.smithi152.stdout:0/979: creat d9/d1b/f143 x:0 0 0 2024-03-04T16:04:57.636 INFO:tasks.workunit.client.1.smithi152.stdout:0/980: mknod d9/d1b/d54/d69/d6c/c144 0 2024-03-04T16:04:57.637 INFO:tasks.workunit.client.1.smithi152.stdout:4/968: dread d8/d2f/f4a [0,4194304] 0 2024-03-04T16:04:57.639 INFO:tasks.workunit.client.1.smithi152.stdout:5/956: write d6/d50/fc8 [6318915,72222] 0 2024-03-04T16:04:57.640 INFO:tasks.workunit.client.1.smithi152.stdout:7/957: mknod d14/d21/d47/c11e 0 2024-03-04T16:04:57.641 INFO:tasks.workunit.client.1.smithi152.stdout:4/969: creat d8/d9/d24/da0/f14b x:0 0 0 2024-03-04T16:04:57.643 INFO:tasks.workunit.client.1.smithi152.stdout:0/981: dwrite d9/f49 [0,4194304] 0 2024-03-04T16:04:57.650 INFO:tasks.workunit.client.1.smithi152.stdout:4/970: write d8/f37 [4474334,65030] 0 2024-03-04T16:04:57.655 INFO:tasks.workunit.client.1.smithi152.stdout:0/982: mknod d9/d1b/d3c/d84/d56/d5b/d7a/c145 0 2024-03-04T16:04:57.657 INFO:tasks.workunit.client.1.smithi152.stdout:0/983: readlink d9/d1b/d3c/d84/l5e 0 2024-03-04T16:04:57.657 INFO:tasks.workunit.client.1.smithi152.stdout:4/971: dwrite d8/d2f/d7c/fb1 [0,4194304] 0 2024-03-04T16:04:57.664 INFO:tasks.workunit.client.1.smithi152.stdout:7/958: write d14/db9/dc9/fe4 [5220883,47163] 0 2024-03-04T16:04:57.670 INFO:tasks.workunit.client.1.smithi152.stdout:0/984: mkdir d9/d92/d146 0 2024-03-04T16:04:57.671 INFO:tasks.workunit.client.1.smithi152.stdout:0/985: readlink d9/d1b/d3c/d84/d56/d5b/l82 0 2024-03-04T16:04:57.672 INFO:tasks.workunit.client.1.smithi152.stdout:7/959: unlink d14/d21/d47/d40/d6e/fc8 0 2024-03-04T16:04:57.675 INFO:tasks.workunit.client.1.smithi152.stdout:4/972: write d8/f30 [1620483,113315] 0 2024-03-04T16:04:57.680 INFO:tasks.workunit.client.1.smithi152.stdout:5/957: dwrite d6/d8/d1f/d3d/d6f/dbb/de3/fb9 [0,4194304] 0 2024-03-04T16:04:57.681 INFO:tasks.workunit.client.1.smithi152.stdout:0/986: dwrite d9/d1b/d54/d69/d95/fb5 [4194304,4194304] 0 2024-03-04T16:04:57.685 INFO:tasks.workunit.client.1.smithi152.stdout:0/987: dread - d9/d1b/d3c/d84/fce zero size 2024-03-04T16:04:57.685 INFO:tasks.workunit.client.1.smithi152.stdout:4/973: dread d8/d2f/d6f/d84/fa4 [0,4194304] 0 2024-03-04T16:04:57.689 INFO:tasks.workunit.client.1.smithi152.stdout:5/958: mkdir d6/d8/d5a/d90/daf/d136 0 2024-03-04T16:04:57.696 INFO:tasks.workunit.client.1.smithi152.stdout:4/974: rename d8/d9/d79/f107 to d8/d2f/d7c/d87/f14c 0 2024-03-04T16:04:57.698 INFO:tasks.workunit.client.1.smithi152.stdout:4/975: truncate d8/d9/dc1/fdf 627600 0 2024-03-04T16:04:57.708 INFO:tasks.workunit.client.1.smithi152.stdout:5/959: mkdir d6/d8/d1f/d3d/da4/d137 0 2024-03-04T16:04:57.708 INFO:tasks.workunit.client.1.smithi152.stdout:4/976: link d8/d89/db3/d104/l12b d8/d89/l14d 0 2024-03-04T16:04:57.713 INFO:tasks.workunit.client.1.smithi152.stdout:4/977: rename d8/d2f/ce2 to d8/d9/da2/dd3/de6/c14e 0 2024-03-04T16:04:57.715 INFO:tasks.workunit.client.1.smithi152.stdout:4/978: write d8/d9/da2/f10f [4628717,116629] 0 2024-03-04T16:04:57.717 INFO:tasks.workunit.client.1.smithi152.stdout:4/979: stat d8/d2f/d6f/d90/fa5 0 2024-03-04T16:04:57.718 INFO:tasks.workunit.client.1.smithi152.stdout:7/960: write d14/db9/dc9/d83/f9c [3517605,38556] 0 2024-03-04T16:04:57.721 INFO:tasks.workunit.client.1.smithi152.stdout:0/988: dwrite d9/d1b/f6d [4194304,4194304] 0 2024-03-04T16:04:57.721 INFO:tasks.workunit.client.1.smithi152.stdout:7/961: truncate d14/d71/fb6 924640 0 2024-03-04T16:04:57.724 INFO:tasks.workunit.client.1.smithi152.stdout:7/962: read d14/d21/d47/d36/f82 [2423713,116047] 0 2024-03-04T16:04:57.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:57 smithi116 ceph-mon[29120]: pgmap v10: 65 pgs: 65 active+clean; 2.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 42 MiB/s rd, 100 MiB/s wr, 261 op/s 2024-03-04T16:04:57.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:57 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:57.729 INFO:tasks.workunit.client.1.smithi152.stdout:7/963: fdatasync d14/d21/d47/d36/da7/dbc/de6/ffc 0 2024-03-04T16:04:57.729 INFO:tasks.workunit.client.1.smithi152.stdout:5/960: dread d6/d50/fca [0,4194304] 0 2024-03-04T16:04:57.732 INFO:tasks.workunit.client.1.smithi152.stdout:4/980: creat d8/d9/f14f x:0 0 0 2024-03-04T16:04:57.733 INFO:tasks.workunit.client.1.smithi152.stdout:7/964: mkdir d14/d71/d11f 0 2024-03-04T16:04:57.734 INFO:tasks.workunit.client.1.smithi152.stdout:5/961: symlink d6/d8/d1f/d3d/d6f/d70/d98/l138 0 2024-03-04T16:04:57.736 INFO:tasks.workunit.client.1.smithi152.stdout:0/989: write d9/d1b/d54/d80/f97 [3346277,98795] 0 2024-03-04T16:04:57.737 INFO:tasks.workunit.client.1.smithi152.stdout:7/965: creat d14/d21/d47/d36/da7/dbc/de6/f120 x:0 0 0 2024-03-04T16:04:57.738 INFO:tasks.workunit.client.1.smithi152.stdout:5/962: mkdir d6/d8/d1f/d88/d139 0 2024-03-04T16:04:57.740 INFO:tasks.workunit.client.1.smithi152.stdout:5/963: write d6/d8/d1f/d3d/d6f/d70/d98/f127 [470985,8434] 0 2024-03-04T16:04:57.740 INFO:tasks.workunit.client.1.smithi152.stdout:4/981: write d8/d3d/f4c [1106822,4761] 0 2024-03-04T16:04:57.744 INFO:tasks.workunit.client.1.smithi152.stdout:5/964: truncate d6/d8/d1f/d3d/dc0/f10f 285102 0 2024-03-04T16:04:57.744 INFO:tasks.workunit.client.1.smithi152.stdout:7/966: mkdir d14/d121 0 2024-03-04T16:04:57.745 INFO:tasks.workunit.client.1.smithi152.stdout:7/967: write d14/d69/d8e/f9f [1488888,21407] 0 2024-03-04T16:04:57.746 INFO:tasks.workunit.client.1.smithi152.stdout:7/968: stat d14/d21/d31/lb5 0 2024-03-04T16:04:57.746 INFO:tasks.workunit.client.1.smithi152.stdout:4/982: creat d8/d2f/d7c/d87/df0/f150 x:0 0 0 2024-03-04T16:04:57.748 INFO:tasks.workunit.client.1.smithi152.stdout:0/990: rename d9/d2c/d2e/d7f/ce9 to d9/c147 0 2024-03-04T16:04:57.753 INFO:tasks.workunit.client.1.smithi152.stdout:7/969: dwrite d14/d71/f7d [0,4194304] 0 2024-03-04T16:04:57.753 INFO:tasks.workunit.client.1.smithi152.stdout:4/983: unlink d8/d2f/d7c/d87/df0/f13a 0 2024-03-04T16:04:57.754 INFO:tasks.workunit.client.1.smithi152.stdout:7/970: read f7 [3059435,118160] 0 2024-03-04T16:04:57.764 INFO:tasks.workunit.client.1.smithi152.stdout:7/971: rename d14/d21/d31/l96 to d14/d21/d47/d36/da7/l122 0 2024-03-04T16:04:57.765 INFO:tasks.workunit.client.1.smithi152.stdout:5/965: write d6/d8/d1f/d3d/d6f/d70/fe9 [1252216,68553] 0 2024-03-04T16:04:57.767 INFO:tasks.workunit.client.1.smithi152.stdout:0/991: write d9/d1b/d54/d80/da8/f103 [701774,24684] 0 2024-03-04T16:04:57.771 INFO:tasks.workunit.client.1.smithi152.stdout:0/992: stat d9/d1b/d3c/d84/dde/l118 0 2024-03-04T16:04:57.774 INFO:tasks.workunit.client.1.smithi152.stdout:7/972: dwrite d14/d21/f90 [0,4194304] 0 2024-03-04T16:04:57.775 INFO:tasks.workunit.client.1.smithi152.stdout:5/966: symlink d6/d8/d1f/d3d/d6f/d70/dc4/d128/l13a 0 2024-03-04T16:04:57.782 INFO:tasks.workunit.client.1.smithi152.stdout:4/984: truncate d8/d2f/d6f/d90/fbd 2308151 0 2024-03-04T16:04:57.784 INFO:tasks.workunit.client.1.smithi152.stdout:7/973: rename d14/c5f to d14/d21/d47/d36/da7/c123 0 2024-03-04T16:04:57.786 INFO:tasks.workunit.client.1.smithi152.stdout:7/974: chown d14/d21/d31/c9a 20570312 1 2024-03-04T16:04:57.788 INFO:tasks.workunit.client.1.smithi152.stdout:7/975: unlink d14/d21/d47/d32/c3c 0 2024-03-04T16:04:57.789 INFO:tasks.workunit.client.1.smithi152.stdout:0/993: dread d9/d1b/d3c/d84/fcb [0,4194304] 0 2024-03-04T16:04:57.791 INFO:tasks.workunit.client.1.smithi152.stdout:0/994: write d9/d1b/d54/d80/db2/fb9 [3713783,26891] 0 2024-03-04T16:04:57.792 INFO:tasks.workunit.client.1.smithi152.stdout:7/976: rename d14/d21/d47/d40/caa to d14/d21/d47/d40/d6e/c124 0 2024-03-04T16:04:57.794 INFO:tasks.workunit.client.1.smithi152.stdout:0/995: chown d9/l18 346941324 1 2024-03-04T16:04:57.795 INFO:tasks.workunit.client.1.smithi152.stdout:5/967: dread d6/f39 [0,4194304] 0 2024-03-04T16:04:57.796 INFO:tasks.workunit.client.1.smithi152.stdout:7/977: creat d14/d21/d48/f125 x:0 0 0 2024-03-04T16:04:57.797 INFO:tasks.workunit.client.1.smithi152.stdout:7/978: chown d14/d71/cba 12657 1 2024-03-04T16:04:57.800 INFO:tasks.workunit.client.1.smithi152.stdout:5/968: symlink d6/d8/d1f/d3d/d6f/dbb/l13b 0 2024-03-04T16:04:57.806 INFO:tasks.workunit.client.1.smithi152.stdout:4/985: write d8/d9/d24/f40 [4948720,97069] 0 2024-03-04T16:04:57.813 INFO:tasks.workunit.client.1.smithi152.stdout:0/996: link d9/d1a/db4/ld5 d9/d2c/d2e/d7f/d9d/da3/l148 0 2024-03-04T16:04:57.815 INFO:tasks.workunit.client.1.smithi152.stdout:4/986: rename d8/f139 to d8/d9/d79/f151 0 2024-03-04T16:04:57.817 INFO:tasks.workunit.client.1.smithi152.stdout:5/969: dread d6/d8/f2c [0,4194304] 0 2024-03-04T16:04:57.818 INFO:tasks.workunit.client.1.smithi152.stdout:7/979: symlink d14/db9/dc9/d83/l126 0 2024-03-04T16:04:57.818 INFO:tasks.workunit.client.1.smithi152.stdout:5/970: stat d6/d8/d1f/d3d/d6f/f95 0 2024-03-04T16:04:57.824 INFO:tasks.workunit.client.1.smithi152.stdout:5/971: mknod d6/d8/d5a/d90/daf/d136/c13c 0 2024-03-04T16:04:57.825 INFO:tasks.workunit.client.1.smithi152.stdout:0/997: getdents d9/d2c/d2e/d94 0 2024-03-04T16:04:57.826 INFO:tasks.workunit.client.1.smithi152.stdout:0/998: readlink d9/d1a/l86 0 2024-03-04T16:04:57.826 INFO:tasks.workunit.client.1.smithi152.stdout:5/972: readlink d6/d8/d1f/d3d/l125 0 2024-03-04T16:04:57.830 INFO:tasks.workunit.client.1.smithi152.stdout:0/999: mknod d9/d2c/d2e/d7f/d9d/de8/c149 0 2024-03-04T16:04:57.832 INFO:tasks.workunit.client.1.smithi152.stdout:4/987: dread d8/d2f/d7c/fce [0,4194304] 0 2024-03-04T16:04:57.832 INFO:tasks.workunit.client.1.smithi152.stdout:4/988: stat d8/d9/d24/l91 0 2024-03-04T16:04:57.834 INFO:tasks.workunit.client.1.smithi152.stdout:5/973: rename d6/d8/d5a/d64/d76/d9f/cd4 to d6/d8/d1f/d3d/d6f/dbb/de3/c13d 0 2024-03-04T16:04:57.836 INFO:tasks.workunit.client.1.smithi152.stdout:4/989: creat d8/d89/db3/f152 x:0 0 0 2024-03-04T16:04:57.837 INFO:tasks.workunit.client.1.smithi152.stdout:4/990: write d8/f37 [3275991,24553] 0 2024-03-04T16:04:57.838 INFO:tasks.workunit.client.1.smithi152.stdout:7/980: dread d14/d21/d31/f57 [0,4194304] 0 2024-03-04T16:04:57.841 INFO:tasks.workunit.client.1.smithi152.stdout:5/974: symlink d6/d8/d5a/d64/d76/l13e 0 2024-03-04T16:04:57.846 INFO:tasks.workunit.client.1.smithi152.stdout:5/975: truncate d6/d8/d5a/f87 1072580 0 2024-03-04T16:04:57.847 INFO:tasks.workunit.client.1.smithi152.stdout:5/976: write d6/d8/d5a/d64/d76/d9f/f12 [3363074,92769] 0 2024-03-04T16:04:57.852 INFO:tasks.workunit.client.1.smithi152.stdout:4/991: write d8/d9/fae [3809671,41398] 0 2024-03-04T16:04:57.852 INFO:tasks.workunit.client.1.smithi152.stdout:4/992: fdatasync d8/d4e/f148 0 2024-03-04T16:04:57.854 INFO:tasks.workunit.client.1.smithi152.stdout:7/981: write d14/d21/d31/f57 [3576545,22021] 0 2024-03-04T16:04:57.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:57 smithi152 ceph-mon[36876]: pgmap v10: 65 pgs: 65 active+clean; 2.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 42 MiB/s rd, 100 MiB/s wr, 261 op/s 2024-03-04T16:04:57.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:57 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:57.857 INFO:tasks.workunit.client.1.smithi152.stdout:5/977: creat d6/f13f x:0 0 0 2024-03-04T16:04:57.859 INFO:tasks.workunit.client.1.smithi152.stdout:7/982: mknod d14/d11c/c127 0 2024-03-04T16:04:57.864 INFO:tasks.workunit.client.1.smithi152.stdout:7/983: read d14/d21/d47/d36/f68 [2021573,28093] 0 2024-03-04T16:04:57.866 INFO:tasks.workunit.client.1.smithi152.stdout:5/978: rename d6/d8/d1f/d3d/d61/fe6 to d6/d8/d1f/d3d/d6f/dbb/de3/f140 0 2024-03-04T16:04:57.868 INFO:tasks.workunit.client.1.smithi152.stdout:7/984: rename d14/d21/d47/d36/lbd to d14/db9/dfd/l128 0 2024-03-04T16:04:57.869 INFO:tasks.workunit.client.1.smithi152.stdout:7/985: stat d14/db9/dc9/l53 0 2024-03-04T16:04:57.870 INFO:tasks.workunit.client.1.smithi152.stdout:5/979: unlink d6/d8/d5a/d64/d76/d9f/l45 0 2024-03-04T16:04:57.875 INFO:tasks.workunit.client.1.smithi152.stdout:5/980: getdents d6 0 2024-03-04T16:04:57.878 INFO:tasks.workunit.client.1.smithi152.stdout:5/981: rename d6/d8/d5a/d90/l11b to d6/d8/d1f/d88/l141 0 2024-03-04T16:04:57.883 INFO:tasks.workunit.client.1.smithi152.stdout:5/982: dread d6/d8/d1f/d3d/d6f/dbb/de3/fb9 [0,4194304] 0 2024-03-04T16:04:57.885 INFO:tasks.workunit.client.1.smithi152.stdout:5/983: write d6/d8/d1f/d88/fac [695435,85615] 0 2024-03-04T16:04:57.888 INFO:tasks.workunit.client.1.smithi152.stdout:5/984: chown d6/d8/d5a/d64/d76/d9f/db0/dd8/f122 98 1 2024-03-04T16:04:57.890 INFO:tasks.workunit.client.1.smithi152.stdout:5/985: rmdir d6/d8/d5a/d90 39 2024-03-04T16:04:57.897 INFO:tasks.workunit.client.1.smithi152.stdout:5/986: rename d6/d8/c1e to d6/d8/d5a/d64/d124/c142 0 2024-03-04T16:04:57.899 INFO:tasks.workunit.client.1.smithi152.stdout:5/987: symlink d6/d8/d1f/d3d/da4/l143 0 2024-03-04T16:04:57.900 INFO:tasks.workunit.client.1.smithi152.stdout:5/988: stat d6/d8/d5a/d64/d76/d9f/dc7/dcb/dd1/feb 0 2024-03-04T16:04:57.905 INFO:tasks.workunit.client.1.smithi152.stdout:5/989: creat d6/d8/d1f/d3d/d6f/f144 x:0 0 0 2024-03-04T16:04:57.911 INFO:tasks.workunit.client.1.smithi152.stdout:5/990: dwrite d6/d8/d5a/d64/d76/d9f/db0/dd8/f115 [0,4194304] 0 2024-03-04T16:04:57.913 INFO:tasks.workunit.client.1.smithi152.stdout:5/991: write d6/d8/f86 [5108482,23518] 0 2024-03-04T16:04:57.924 INFO:tasks.workunit.client.1.smithi152.stdout:5/992: symlink d6/d8/d1f/d3d/d6f/d70/dc4/l145 0 2024-03-04T16:04:57.931 INFO:tasks.workunit.client.1.smithi152.stdout:5/993: dread d6/d8/d1f/d3d/d6f/d70/fe9 [0,4194304] 0 2024-03-04T16:04:57.941 INFO:tasks.workunit.client.1.smithi152.stdout:5/994: getdents d6/d8/d5a/d64/d76 0 2024-03-04T16:04:57.956 INFO:tasks.workunit.client.1.smithi152.stdout:5/995: dwrite d6/d8/d1f/f3b [0,4194304] 0 2024-03-04T16:04:57.959 INFO:tasks.workunit.client.1.smithi152.stdout:5/996: creat d6/d8/d5a/d90/dd6/f146 x:0 0 0 2024-03-04T16:04:57.960 INFO:tasks.workunit.client.1.smithi152.stdout:5/997: chown d6/d8/d1f/l36 15632954 1 2024-03-04T16:04:57.964 INFO:tasks.workunit.client.1.smithi152.stdout:5/998: symlink d6/d8/d5a/d64/l147 0 2024-03-04T16:04:57.983 INFO:tasks.workunit.client.1.smithi152.stdout:5/999: dread d6/d8/d1f/d3d/d6f/d70/d98/fb7 [0,4194304] 0 2024-03-04T16:04:58.333 INFO:tasks.workunit.client.1.smithi152.stdout:7/986: sync 2024-03-04T16:04:58.334 INFO:tasks.workunit.client.1.smithi152.stdout:4/993: sync 2024-03-04T16:04:58.341 INFO:tasks.workunit.client.1.smithi152.stdout:7/987: write d14/d69/ffa [1010484,26217] 0 2024-03-04T16:04:58.342 INFO:tasks.workunit.client.1.smithi152.stdout:7/988: readlink d14/d21/d47/d36/da7/dbc/l118 0 2024-03-04T16:04:58.344 INFO:tasks.workunit.client.1.smithi152.stdout:4/994: getdents d8/d9/d24/da0 0 2024-03-04T16:04:58.349 INFO:tasks.workunit.client.1.smithi152.stdout:7/989: dwrite d14/db9/dc9/f7e [0,4194304] 0 2024-03-04T16:04:58.355 INFO:tasks.workunit.client.1.smithi152.stdout:4/995: write d8/d3d/f5f [525310,59531] 0 2024-03-04T16:04:58.356 INFO:tasks.workunit.client.1.smithi152.stdout:7/990: dread f4 [0,4194304] 0 2024-03-04T16:04:58.357 INFO:tasks.workunit.client.1.smithi152.stdout:4/996: write d8/d9/d79/f85 [1541278,66242] 0 2024-03-04T16:04:58.361 INFO:tasks.workunit.client.1.smithi152.stdout:7/991: creat d14/d21/d47/d36/dcf/f129 x:0 0 0 2024-03-04T16:04:58.361 INFO:tasks.workunit.client.1.smithi152.stdout:4/997: creat d8/d3d/f153 x:0 0 0 2024-03-04T16:04:58.364 INFO:tasks.workunit.client.1.smithi152.stdout:4/998: rename d8/d9/da2/dd3 to d8/d2f/d6f/deb/def/d154 0 2024-03-04T16:04:58.365 INFO:tasks.workunit.client.1.smithi152.stdout:7/992: creat d14/d21/df5/f12a x:0 0 0 2024-03-04T16:04:58.371 INFO:tasks.workunit.client.1.smithi152.stdout:4/999: dwrite d8/d4e/f103 [0,4194304] 0 2024-03-04T16:04:58.382 INFO:tasks.workunit.client.1.smithi152.stdout:7/993: dwrite d14/db9/dc9/d83/ff1 [0,4194304] 0 2024-03-04T16:04:58.385 INFO:tasks.workunit.client.1.smithi152.stdout:7/994: write d14/db9/dc9/d83/d105/f10b [999061,127684] 0 2024-03-04T16:04:58.388 INFO:tasks.workunit.client.1.smithi152.stdout:7/995: unlink d14/d21/d47/f91 0 2024-03-04T16:04:58.390 INFO:tasks.workunit.client.1.smithi152.stdout:7/996: write d14/d21/d47/d36/f74 [740540,8558] 0 2024-03-04T16:04:58.393 INFO:tasks.workunit.client.1.smithi152.stdout:7/997: rename d14/d21/d48/cb3 to d14/d21/d48/c12b 0 2024-03-04T16:04:58.402 INFO:tasks.workunit.client.1.smithi152.stdout:7/998: write d14/d21/d47/d36/fcc [513621,80347] 0 2024-03-04T16:04:58.405 INFO:tasks.workunit.client.1.smithi152.stdout:7/999: creat d14/d21/d47/d32/f12c x:0 0 0 2024-03-04T16:04:58.409 INFO:tasks.workunit.client.1.smithi152.stderr:+ rm -rf -- ./tmp.a3eCHG0wLq 2024-03-04T16:04:58.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:58 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:58.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:58 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:58.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:04:58 smithi152 ceph-mon[36876]: pgmap v11: 65 pgs: 65 active+clean; 2.3 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 51 MiB/s rd, 122 MiB/s wr, 338 op/s 2024-03-04T16:04:58.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:58 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:58.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:58 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:04:58.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:04:58 smithi116 ceph-mon[29120]: pgmap v11: 65 pgs: 65 active+clean; 2.3 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 51 MiB/s rd, 122 MiB/s wr, 338 op/s 2024-03-04T16:05:00.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:00 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:00.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:00 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:00.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:00 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:00.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:00 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:05:00.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:00 smithi116 ceph-mon[29120]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T16:05:00.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:00 smithi116 ceph-mon[29120]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T16:05:00.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:00 smithi116 ceph-mon[29120]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:05:00.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:00 smithi116 ceph-mon[29120]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:05:00.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:00 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:00.403 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:00 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:00.403 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:00 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:00.404 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:00 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:05:00.404 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:00 smithi152 ceph-mon[36876]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T16:05:00.404 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:00 smithi152 ceph-mon[36876]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T16:05:00.404 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:00 smithi152 ceph-mon[36876]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:05:00.404 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:00 smithi152 ceph-mon[36876]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:05:01.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:05:01.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:05:01.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:05:01.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:05:01.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: pgmap v12: 65 pgs: 65 active+clean; 2.3 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 51 MiB/s rd, 124 MiB/s wr, 306 op/s 2024-03-04T16:05:01.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:01.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:01.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:01.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:01.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:01.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:01 smithi116 ceph-mon[29120]: Reconfiguring prometheus.smithi116 (dependencies changed)... 2024-03-04T16:05:01.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:05:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:05:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:05:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:05:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: pgmap v12: 65 pgs: 65 active+clean; 2.3 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 51 MiB/s rd, 124 MiB/s wr, 306 op/s 2024-03-04T16:05:01.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:01.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:01 smithi152 ceph-mon[36876]: Reconfiguring prometheus.smithi116 (dependencies changed)... 2024-03-04T16:05:02.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:02 smithi152 ceph-mon[36876]: Reconfiguring daemon prometheus.smithi116 on smithi116 2024-03-04T16:05:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:02 smithi116 ceph-mon[29120]: Reconfiguring daemon prometheus.smithi116 on smithi116 2024-03-04T16:05:03.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:03 smithi116 ceph-mon[29120]: pgmap v13: 65 pgs: 65 active+clean; 2.3 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 37 MiB/s rd, 90 MiB/s wr, 248 op/s 2024-03-04T16:05:03.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:03 smithi152 ceph-mon[36876]: pgmap v13: 65 pgs: 65 active+clean; 2.3 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 37 MiB/s rd, 90 MiB/s wr, 248 op/s 2024-03-04T16:05:05.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:05 smithi116 ceph-mon[29120]: pgmap v14: 65 pgs: 65 active+clean; 2.1 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 37 MiB/s rd, 92 MiB/s wr, 358 op/s 2024-03-04T16:05:05.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:05 smithi152 ceph-mon[36876]: pgmap v14: 65 pgs: 65 active+clean; 2.1 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 37 MiB/s rd, 92 MiB/s wr, 358 op/s 2024-03-04T16:05:07.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:07 smithi152 ceph-mon[36876]: pgmap v15: 65 pgs: 65 active+clean; 2.1 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 19 MiB/s rd, 51 MiB/s wr, 244 op/s 2024-03-04T16:05:07.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:07 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:07.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:07 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:07.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:07 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:05:07.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:07 smithi152 ceph-mon[36876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:05:07.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:07 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:07 smithi116 ceph-mon[29120]: pgmap v15: 65 pgs: 65 active+clean; 2.1 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 19 MiB/s rd, 51 MiB/s wr, 244 op/s 2024-03-04T16:05:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:07 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:07 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:07 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:05:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:07 smithi116 ceph-mon[29120]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:05:07.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:07 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:08.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:08 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.aawieo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:05:08.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:08 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:05:08.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:08 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:08 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.aawieo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:05:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:08 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:05:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:08 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:09.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:09 smithi152 ceph-mon[36876]: pgmap v16: 65 pgs: 65 active+clean; 1.8 GiB data, 5.9 GiB used, 530 GiB / 536 GiB avail; 19 MiB/s rd, 51 MiB/s wr, 294 op/s 2024-03-04T16:05:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:09 smithi152 ceph-mon[36876]: Upgrade: Updating mgr.smithi152.aawieo 2024-03-04T16:05:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:09 smithi152 ceph-mon[36876]: Deploying daemon mgr.smithi152.aawieo on smithi152 2024-03-04T16:05:09.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:09 smithi116 ceph-mon[29120]: pgmap v16: 65 pgs: 65 active+clean; 1.8 GiB data, 5.9 GiB used, 530 GiB / 536 GiB avail; 19 MiB/s rd, 51 MiB/s wr, 294 op/s 2024-03-04T16:05:09.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:09 smithi116 ceph-mon[29120]: Upgrade: Updating mgr.smithi152.aawieo 2024-03-04T16:05:09.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:09 smithi116 ceph-mon[29120]: Deploying daemon mgr.smithi152.aawieo on smithi152 2024-03-04T16:05:11.520 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:11 smithi152 ceph-mon[36876]: pgmap v17: 65 pgs: 65 active+clean; 1.5 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 9.9 MiB/s rd, 29 MiB/s wr, 293 op/s 2024-03-04T16:05:11.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:11 smithi116 ceph-mon[29120]: pgmap v17: 65 pgs: 65 active+clean; 1.5 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 9.9 MiB/s rd, 29 MiB/s wr, 293 op/s 2024-03-04T16:05:12.566 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:12 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:12.708 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:12 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:13.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:13 smithi152 ceph-mon[36876]: pgmap v18: 65 pgs: 65 active+clean; 1.5 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 29 KiB/s rd, 2.6 MiB/s wr, 235 op/s 2024-03-04T16:05:13.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:13 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:13.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:13 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:13 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:13.634 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:13 smithi116 ceph-mon[29120]: pgmap v18: 65 pgs: 65 active+clean; 1.5 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 29 KiB/s rd, 2.6 MiB/s wr, 235 op/s 2024-03-04T16:05:13.634 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:13 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:13.634 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:13 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:13.634 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:13 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:15.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:15 smithi152 ceph-mon[36876]: pgmap v19: 65 pgs: 65 active+clean; 1.0 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 43 KiB/s rd, 3.3 MiB/s wr, 358 op/s 2024-03-04T16:05:15.640 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:15 smithi116 ceph-mon[29120]: pgmap v19: 65 pgs: 65 active+clean; 1.0 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 43 KiB/s rd, 3.3 MiB/s wr, 358 op/s 2024-03-04T16:05:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.878+0000 7f3187650700 1 -- 172.21.15.116:0/3190314411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180100bb0 msgr2=0x7f3180100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.878+0000 7f3187650700 1 --2- 172.21.15.116:0/3190314411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180100bb0 0x7f3180100fd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f317c00b0a0 tx=0x7f317c00b3b0 comp rx=0 tx=0).stop 2024-03-04T16:05:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.878+0000 7f3187650700 1 -- 172.21.15.116:0/3190314411 shutdown_connections 2024-03-04T16:05:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.878+0000 7f3187650700 1 --2- 172.21.15.116:0/3190314411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3180101db0 0x7f3180102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.878+0000 7f3187650700 1 --2- 172.21.15.116:0/3190314411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180100bb0 0x7f3180100fd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.878+0000 7f3187650700 1 -- 172.21.15.116:0/3190314411 >> 172.21.15.116:0/3190314411 conn(0x7f31800fc110 msgr2=0x7f31800fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.878+0000 7f3187650700 1 -- 172.21.15.116:0/3190314411 shutdown_connections 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.878+0000 7f3187650700 1 -- 172.21.15.116:0/3190314411 wait complete. 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3187650700 1 Processor -- start 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3187650700 1 -- start start 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3187650700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3180100bb0 0x7f3180198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3187650700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101db0 0x7f3180198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3187650700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31801990f0 con 0x7f3180101db0 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3187650700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3180199230 con 0x7f3180100bb0 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f31853ec700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3180100bb0 0x7f3180198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3184beb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101db0 0x7f3180198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f31853ec700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3180100bb0 0x7f3180198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:51046/0 (socket says 172.21.15.116:51046) 2024-03-04T16:05:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f31853ec700 1 -- 172.21.15.116:0/247935833 learned_addr learned my addr 172.21.15.116:0/247935833 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3184beb700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101db0 0x7f3180198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38704/0 (socket says 172.21.15.116:38704) 2024-03-04T16:05:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3184beb700 1 -- 172.21.15.116:0/247935833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3180100bb0 msgr2=0x7f3180198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3184beb700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3180100bb0 0x7f3180198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.879+0000 7f3184beb700 1 -- 172.21.15.116:0/247935833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c009d00 con 0x7f3180101db0 2024-03-04T16:05:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.880+0000 7f3184beb700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101db0 0x7f3180198ad0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f317000d6d0 tx=0x7f317000d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.880+0000 7f31767fc700 1 -- 172.21.15.116:0/247935833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31700041f0 con 0x7f3180101db0 2024-03-04T16:05:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.880+0000 7f31767fc700 1 -- 172.21.15.116:0/247935833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3170004ce0 con 0x7f3180101db0 2024-03-04T16:05:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.880+0000 7f3187650700 1 -- 172.21.15.116:0/247935833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318019dce0 con 0x7f3180101db0 2024-03-04T16:05:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.880+0000 7f31767fc700 1 -- 172.21.15.116:0/247935833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f317000f690 con 0x7f3180101db0 2024-03-04T16:05:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.880+0000 7f3187650700 1 -- 172.21.15.116:0/247935833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318019e230 con 0x7f3180101db0 2024-03-04T16:05:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.881+0000 7f3187650700 1 -- 172.21.15.116:0/247935833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3180105e90 con 0x7f3180101db0 2024-03-04T16:05:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.882+0000 7f31767fc700 1 -- 172.21.15.116:0/247935833 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 30) v1 ==== 91440+0+0 (secure 0 0 0) 0x7f3170004350 con 0x7f3180101db0 2024-03-04T16:05:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.882+0000 7f31767fc700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f316c06d660 0x7f316c06fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.882+0000 7f31767fc700 1 -- 172.21.15.116:0/247935833 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f317008cd30 con 0x7f3180101db0 2024-03-04T16:05:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.882+0000 7f31853ec700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f316c06d660 0x7f316c06fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.883+0000 7f31853ec700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f316c06d660 0x7f316c06fb20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f317c009480 tx=0x7f317c0060c0 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:15.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:15.887+0000 7f31767fc700 1 -- 172.21.15.116:0/247935833 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f317005a320 con 0x7f3180101db0 2024-03-04T16:05:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.110+0000 7f3187650700 1 -- 172.21.15.116:0/247935833 --> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f31800611d0 con 0x7f316c06d660 2024-03-04T16:05:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.111+0000 7f31767fc700 1 -- 172.21.15.116:0/247935833 <== mgr.14738 v2:172.21.15.116:6800/2966163948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+328 (secure 0 0 0) 0x7f31800611d0 con 0x7f316c06d660 2024-03-04T16:05:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.113+0000 7f3187650700 1 -- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f316c06d660 msgr2=0x7f316c06fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.113+0000 7f3187650700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f316c06d660 0x7f316c06fb20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f317c009480 tx=0x7f317c0060c0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.113+0000 7f3187650700 1 -- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101db0 msgr2=0x7f3180198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.113+0000 7f3187650700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101db0 0x7f3180198ad0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f317000d6d0 tx=0x7f317000d9e0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.113+0000 7f3187650700 1 -- 172.21.15.116:0/247935833 shutdown_connections 2024-03-04T16:05:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.113+0000 7f3187650700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f316c06d660 0x7f316c06fb20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.113+0000 7f3187650700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3180100bb0 0x7f3180198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.113+0000 7f3187650700 1 --2- 172.21.15.116:0/247935833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101db0 0x7f3180198ad0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.114+0000 7f3187650700 1 -- 172.21.15.116:0/247935833 >> 172.21.15.116:0/247935833 conn(0x7f31800fc110 msgr2=0x7f3180102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.114+0000 7f3187650700 1 -- 172.21.15.116:0/247935833 shutdown_connections 2024-03-04T16:05:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.114+0000 7f3187650700 1 -- 172.21.15.116:0/247935833 wait complete. 2024-03-04T16:05:16.125 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:05:16.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.223+0000 7fa081fca700 1 -- 172.21.15.116:0/3401551351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa07c0ffad0 msgr2=0x7fa07c0ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.223+0000 7fa081fca700 1 --2- 172.21.15.116:0/3401551351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa07c0ffad0 0x7fa07c0ffef0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009910 tx=0x7fa06c009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.223+0000 7fa081fca700 1 -- 172.21.15.116:0/3401551351 shutdown_connections 2024-03-04T16:05:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.223+0000 7fa081fca700 1 --2- 172.21.15.116:0/3401551351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa07c100cd0 0x7fa07c101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.223+0000 7fa081fca700 1 --2- 172.21.15.116:0/3401551351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa07c0ffad0 0x7fa07c0ffef0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.223+0000 7fa081fca700 1 -- 172.21.15.116:0/3401551351 >> 172.21.15.116:0/3401551351 conn(0x7fa07c0fb050 msgr2=0x7fa07c0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.223+0000 7fa081fca700 1 -- 172.21.15.116:0/3401551351 shutdown_connections 2024-03-04T16:05:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.223+0000 7fa081fca700 1 -- 172.21.15.116:0/3401551351 wait complete. 2024-03-04T16:05:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.224+0000 7fa081fca700 1 Processor -- start 2024-03-04T16:05:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.224+0000 7fa081fca700 1 -- start start 2024-03-04T16:05:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.224+0000 7fa081fca700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa07c0ffad0 0x7fa07c0698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.224+0000 7fa081fca700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa07c100cd0 0x7fa07c19b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.224+0000 7fa081fca700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c19b6e0 con 0x7fa07c100cd0 2024-03-04T16:05:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.224+0000 7fa081fca700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c19b820 con 0x7fa07c0ffad0 2024-03-04T16:05:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa07affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa07c100cd0 0x7fa07c19b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa07affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa07c100cd0 0x7fa07c19b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38720/0 (socket says 172.21.15.116:38720) 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa07affd700 1 -- 172.21.15.116:0/2579612611 learned_addr learned my addr 172.21.15.116:0/2579612611 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa07b7fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa07c0ffad0 0x7fa07c0698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa07affd700 1 -- 172.21.15.116:0/2579612611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa07c0ffad0 msgr2=0x7fa07c0698f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa07affd700 1 --2- 172.21.15.116:0/2579612611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa07c0ffad0 0x7fa07c0698f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa07affd700 1 -- 172.21.15.116:0/2579612611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c0095f0 con 0x7fa07c100cd0 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa07affd700 1 --2- 172.21.15.116:0/2579612611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa07c100cd0 0x7fa07c19b110 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa07000e920 tx=0x7fa07000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa078ff9700 1 -- 172.21.15.116:0/2579612611 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa07000c9d0 con 0x7fa07c100cd0 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa078ff9700 1 -- 172.21.15.116:0/2579612611 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa070010430 con 0x7fa07c100cd0 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa078ff9700 1 -- 172.21.15.116:0/2579612611 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa070018650 con 0x7fa07c100cd0 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa081fca700 1 -- 172.21.15.116:0/2579612611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07c19bb00 con 0x7fa07c100cd0 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.225+0000 7fa081fca700 1 -- 172.21.15.116:0/2579612611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07c19c050 con 0x7fa07c100cd0 2024-03-04T16:05:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.226+0000 7fa081fca700 1 -- 172.21.15.116:0/2579612611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07c108630 con 0x7fa07c100cd0 2024-03-04T16:05:16.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.227+0000 7fa078ff9700 1 -- 172.21.15.116:0/2579612611 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 30) v1 ==== 91440+0+0 (secure 0 0 0) 0x7fa0700105a0 con 0x7fa07c100cd0 2024-03-04T16:05:16.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.227+0000 7fa078ff9700 1 --2- 172.21.15.116:0/2579612611 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fa06406d660 0x7fa06406fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:16.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.227+0000 7fa078ff9700 1 -- 172.21.15.116:0/2579612611 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fa070014070 con 0x7fa07c100cd0 2024-03-04T16:05:16.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.227+0000 7fa07b7fe700 1 --2- 172.21.15.116:0/2579612611 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fa06406d660 0x7fa06406fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:16.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.228+0000 7fa07b7fe700 1 --2- 172.21.15.116:0/2579612611 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fa06406d660 0x7fa06406fb20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00b6f0 tx=0x7fa06c005940 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:16.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.232+0000 7fa078ff9700 1 -- 172.21.15.116:0/2579612611 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fa070052a70 con 0x7fa07c100cd0 2024-03-04T16:05:16.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.483+0000 7fa081fca700 1 -- 172.21.15.116:0/2579612611 --> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa07c105620 con 0x7fa06406d660 2024-03-04T16:05:16.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.485+0000 7fa078ff9700 1 -- 172.21.15.116:0/2579612611 <== mgr.14738 v2:172.21.15.116:6800/2966163948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+328 (secure 0 0 0) 0x7fa07c105620 con 0x7fa06406d660 2024-03-04T16:05:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 -- 172.21.15.116:0/2579612611 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fa06406d660 msgr2=0x7fa06406fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 --2- 172.21.15.116:0/2579612611 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fa06406d660 0x7fa06406fb20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00b6f0 tx=0x7fa06c005940 comp rx=0 tx=0).stop 2024-03-04T16:05:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 -- 172.21.15.116:0/2579612611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa07c100cd0 msgr2=0x7fa07c19b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 --2- 172.21.15.116:0/2579612611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa07c100cd0 0x7fa07c19b110 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa07000e920 tx=0x7fa07000ec30 comp rx=0 tx=0).stop 2024-03-04T16:05:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 -- 172.21.15.116:0/2579612611 shutdown_connections 2024-03-04T16:05:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 --2- 172.21.15.116:0/2579612611 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fa06406d660 0x7fa06406fb20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 --2- 172.21.15.116:0/2579612611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa07c0ffad0 0x7fa07c0698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 --2- 172.21.15.116:0/2579612611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa07c100cd0 0x7fa07c19b110 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 -- 172.21.15.116:0/2579612611 >> 172.21.15.116:0/2579612611 conn(0x7fa07c0fb050 msgr2=0x7fa07c103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 -- 172.21.15.116:0/2579612611 shutdown_connections 2024-03-04T16:05:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.488+0000 7fa081fca700 1 -- 172.21.15.116:0/2579612611 wait complete. 2024-03-04T16:05:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.637+0000 7feed26de700 1 -- 172.21.15.116:0/376209358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feecc101dd0 msgr2=0x7feecc104260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.637+0000 7feed26de700 1 --2- 172.21.15.116:0/376209358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feecc101dd0 0x7feecc104260 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7feec0009910 tx=0x7feec0009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.638+0000 7feed26de700 1 -- 172.21.15.116:0/376209358 shutdown_connections 2024-03-04T16:05:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.638+0000 7feed26de700 1 --2- 172.21.15.116:0/376209358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feecc101dd0 0x7feecc104260 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.638+0000 7feed26de700 1 --2- 172.21.15.116:0/376209358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feecc0ff460 0x7feecc101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.638+0000 7feed26de700 1 -- 172.21.15.116:0/376209358 >> 172.21.15.116:0/376209358 conn(0x7feecc0fb080 msgr2=0x7feecc0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.638+0000 7feed26de700 1 -- 172.21.15.116:0/376209358 shutdown_connections 2024-03-04T16:05:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.638+0000 7feed26de700 1 -- 172.21.15.116:0/376209358 wait complete. 2024-03-04T16:05:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.638+0000 7feed26de700 1 Processor -- start 2024-03-04T16:05:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.638+0000 7feed26de700 1 -- start start 2024-03-04T16:05:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.639+0000 7feed26de700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feecc0ff460 0x7feecc19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.639+0000 7feed26de700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feecc101dd0 0x7feecc19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.639+0000 7feed26de700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feecc19d500 con 0x7feecc0ff460 2024-03-04T16:05:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.639+0000 7feed26de700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feecc19d640 con 0x7feecc101dd0 2024-03-04T16:05:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.639+0000 7feecbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feecc0ff460 0x7feecc19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.639+0000 7feecbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feecc0ff460 0x7feecc19c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38736/0 (socket says 172.21.15.116:38736) 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.639+0000 7feecbfff700 1 -- 172.21.15.116:0/3621579944 learned_addr learned my addr 172.21.15.116:0/3621579944 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.639+0000 7feecb7fe700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feecc101dd0 0x7feecc19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.639+0000 7feecbfff700 1 -- 172.21.15.116:0/3621579944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feecc101dd0 msgr2=0x7feecc19cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.640+0000 7feecbfff700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feecc101dd0 0x7feecc19cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.640+0000 7feecbfff700 1 -- 172.21.15.116:0/3621579944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec00095f0 con 0x7feecc0ff460 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.640+0000 7feecb7fe700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feecc101dd0 0x7feecc19cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.640+0000 7feecbfff700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feecc0ff460 0x7feecc19c9a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feec400ba50 tx=0x7feec400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.640+0000 7feec97fa700 1 -- 172.21.15.116:0/3621579944 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feec400ca00 con 0x7feecc0ff460 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.640+0000 7feed26de700 1 -- 172.21.15.116:0/3621579944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feecc1a20f0 con 0x7feecc0ff460 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.640+0000 7feec97fa700 1 -- 172.21.15.116:0/3621579944 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feec400cb60 con 0x7feecc0ff460 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.640+0000 7feed26de700 1 -- 172.21.15.116:0/3621579944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feecc1a26c0 con 0x7feecc0ff460 2024-03-04T16:05:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.640+0000 7feec97fa700 1 -- 172.21.15.116:0/3621579944 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feec401d580 con 0x7feecc0ff460 2024-03-04T16:05:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.641+0000 7feed26de700 1 -- 172.21.15.116:0/3621579944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb8005320 con 0x7feecc0ff460 2024-03-04T16:05:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.642+0000 7feec97fa700 1 -- 172.21.15.116:0/3621579944 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 30) v1 ==== 91440+0+0 (secure 0 0 0) 0x7feec400ccd0 con 0x7feecc0ff460 2024-03-04T16:05:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.642+0000 7feec97fa700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7feeb406d660 0x7feeb406fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.642+0000 7feec97fa700 1 -- 172.21.15.116:0/3621579944 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7feec4006e40 con 0x7feecc0ff460 2024-03-04T16:05:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.642+0000 7feecb7fe700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7feeb406d660 0x7feeb406fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.643+0000 7feecb7fe700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7feeb406d660 0x7feeb406fb20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7feec00059d0 tx=0x7feec0005920 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:16.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.647+0000 7feec97fa700 1 -- 172.21.15.116:0/3621579944 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7feec4059d50 con 0x7feecc0ff460 2024-03-04T16:05:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.869+0000 7feed26de700 1 -- 172.21.15.116:0/3621579944 --> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7feeb8000bf0 con 0x7feeb406d660 2024-03-04T16:05:16.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.881+0000 7feec97fa700 1 -- 172.21.15.116:0/3621579944 <== mgr.14738 v2:172.21.15.116:6800/2966163948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7feeb8000bf0 con 0x7feeb406d660 2024-03-04T16:05:16.882 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:05:16.882 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (8m) 26s ago 9m 25.3M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:05:16.882 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (10m) 26s ago 10m 7658k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:05:16.882 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (8m) 0s ago 8m 7692k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:05:16.882 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (10m) 26s ago 10m 6665k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:05:16.882 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (8m) 0s ago 8m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:05:16.882 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (8m) 26s ago 9m 86.8M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:05:16.882 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (4m) 26s ago 4m 17.7M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (4m) 26s ago 4m 13.8M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (4m) 0s ago 4m 15.8M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (4m) 0s ago 4m 209M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (46s) 26s ago 11m 477M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (4s) 0s ago 8m 66.7M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (11m) 26s ago 11m 51.0M 2048M 18.2.1 5be31c24972a 60e0bf118c4d 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (8m) 0s ago 8m 44.0M 2048M 18.2.1 5be31c24972a 52450d235516 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (9m) 26s ago 9m 15.2M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (8m) 0s ago 8m 16.6M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (7m) 26s ago 7m 193M 1216M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (7m) 26s ago 7m 204M 1216M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (6m) 26s ago 6m 177M 1216M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (6m) 0s ago 6m 229M 2240M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:05:16.883 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (6m) 0s ago 6m 217M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:05:16.884 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (5m) 0s ago 5m 217M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:05:16.884 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 starting - - - - 2024-03-04T16:05:16.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 -- 172.21.15.116:0/3621579944 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7feeb406d660 msgr2=0x7feeb406fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7feeb406d660 0x7feeb406fb20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7feec00059d0 tx=0x7feec0005920 comp rx=0 tx=0).stop 2024-03-04T16:05:16.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 -- 172.21.15.116:0/3621579944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feecc0ff460 msgr2=0x7feecc19c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feecc0ff460 0x7feecc19c9a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feec400ba50 tx=0x7feec400be10 comp rx=0 tx=0).stop 2024-03-04T16:05:16.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 -- 172.21.15.116:0/3621579944 shutdown_connections 2024-03-04T16:05:16.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feecc0ff460 0x7feecc19c9a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7feeb406d660 0x7feeb406fb20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 --2- 172.21.15.116:0/3621579944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feecc101dd0 0x7feecc19cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 -- 172.21.15.116:0/3621579944 >> 172.21.15.116:0/3621579944 conn(0x7feecc0fb080 msgr2=0x7feecc0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:16.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 -- 172.21.15.116:0/3621579944 shutdown_connections 2024-03-04T16:05:16.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.883+0000 7feed26de700 1 -- 172.21.15.116:0/3621579944 wait complete. 2024-03-04T16:05:16.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.992+0000 7f683d17b700 1 -- 172.21.15.116:0/3536703877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6838104010 msgr2=0x7f6838106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.992+0000 7f683d17b700 1 --2- 172.21.15.116:0/3536703877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6838104010 0x7f6838106400 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f682c009910 tx=0x7f682c009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:16.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.992+0000 7f683d17b700 1 -- 172.21.15.116:0/3536703877 shutdown_connections 2024-03-04T16:05:16.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.992+0000 7f683d17b700 1 --2- 172.21.15.116:0/3536703877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6838104010 0x7f6838106400 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.992+0000 7f683d17b700 1 --2- 172.21.15.116:0/3536703877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68381016e0 0x7f6838103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.992+0000 7f683d17b700 1 -- 172.21.15.116:0/3536703877 >> 172.21.15.116:0/3536703877 conn(0x7f68380fb010 msgr2=0x7f68380fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:16.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.992+0000 7f683d17b700 1 -- 172.21.15.116:0/3536703877 shutdown_connections 2024-03-04T16:05:16.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.992+0000 7f683d17b700 1 -- 172.21.15.116:0/3536703877 wait complete. 2024-03-04T16:05:16.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683d17b700 1 Processor -- start 2024-03-04T16:05:16.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683d17b700 1 -- start start 2024-03-04T16:05:16.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683d17b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68381016e0 0x7f6838196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683d17b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6838104010 0x7f6838196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683d17b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6838196ea0 con 0x7f6838104010 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683d17b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6838196fe0 con 0x7f68381016e0 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6838104010 0x7f6838196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6838104010 0x7f6838196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38762/0 (socket says 172.21.15.116:38762) 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683659c700 1 -- 172.21.15.116:0/2051654837 learned_addr learned my addr 172.21.15.116:0/2051654837 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f6836d9d700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68381016e0 0x7f6838196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683659c700 1 -- 172.21.15.116:0/2051654837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68381016e0 msgr2=0x7f6838196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683659c700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68381016e0 0x7f6838196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.993+0000 7f683659c700 1 -- 172.21.15.116:0/2051654837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c0095f0 con 0x7f6838104010 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.994+0000 7f683659c700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6838104010 0x7f6838196880 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f682c000c00 tx=0x7f682c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.994+0000 7f681ffff700 1 -- 172.21.15.116:0/2051654837 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f682c01d070 con 0x7f6838104010 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.994+0000 7f681ffff700 1 -- 172.21.15.116:0/2051654837 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f682c005dc0 con 0x7f6838104010 2024-03-04T16:05:16.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.994+0000 7f683d17b700 1 -- 172.21.15.116:0/2051654837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683819ba30 con 0x7f6838104010 2024-03-04T16:05:16.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.994+0000 7f681ffff700 1 -- 172.21.15.116:0/2051654837 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f682c0226a0 con 0x7f6838104010 2024-03-04T16:05:16.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.994+0000 7f683d17b700 1 -- 172.21.15.116:0/2051654837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683819bf20 con 0x7f6838104010 2024-03-04T16:05:16.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.995+0000 7f683d17b700 1 -- 172.21.15.116:0/2051654837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6838190530 con 0x7f6838104010 2024-03-04T16:05:16.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.995+0000 7f681ffff700 1 -- 172.21.15.116:0/2051654837 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 30) v1 ==== 91440+0+0 (secure 0 0 0) 0x7f682c0038f0 con 0x7f6838104010 2024-03-04T16:05:16.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.996+0000 7f681ffff700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f682006d660 0x7f682006fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:16.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.996+0000 7f681ffff700 1 -- 172.21.15.116:0/2051654837 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f682c08e710 con 0x7f6838104010 2024-03-04T16:05:16.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.996+0000 7f6836d9d700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f682006d660 0x7f682006fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:16.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:16.996+0000 7f6836d9d700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f682006d660 0x7f682006fb20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6828009c60 tx=0x7f6828009260 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:17.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.000+0000 7f681ffff700 1 -- 172.21.15.116:0/2051654837 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f682c05bd00 con 0x7f6838104010 2024-03-04T16:05:17.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.325+0000 7f683d17b700 1 -- 172.21.15.116:0/2051654837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6838109800 con 0x7f6838104010 2024-03-04T16:05:17.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.326+0000 7f681ffff700 1 -- 172.21.15.116:0/2051654837 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f682c003680 con 0x7f6838104010 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:05:17.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:05:17.328 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:05:17.328 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-03-04T16:05:17.328 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:05:17.328 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:05:17.328 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:05:17.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 -- 172.21.15.116:0/2051654837 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f682006d660 msgr2=0x7f682006fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:17.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f682006d660 0x7f682006fb20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6828009c60 tx=0x7f6828009260 comp rx=0 tx=0).stop 2024-03-04T16:05:17.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 -- 172.21.15.116:0/2051654837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6838104010 msgr2=0x7f6838196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:17.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6838104010 0x7f6838196880 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f682c000c00 tx=0x7f682c00f690 comp rx=0 tx=0).stop 2024-03-04T16:05:17.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 -- 172.21.15.116:0/2051654837 shutdown_connections 2024-03-04T16:05:17.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f682006d660 0x7f682006fb20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68381016e0 0x7f6838196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 --2- 172.21.15.116:0/2051654837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6838104010 0x7f6838196880 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 -- 172.21.15.116:0/2051654837 >> 172.21.15.116:0/2051654837 conn(0x7f68380fb010 msgr2=0x7f68380fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:17.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 -- 172.21.15.116:0/2051654837 shutdown_connections 2024-03-04T16:05:17.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.328+0000 7f683d17b700 1 -- 172.21.15.116:0/2051654837 wait complete. 2024-03-04T16:05:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.452+0000 7f110e05d700 1 -- 172.21.15.116:0/2638704784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1108101db0 msgr2=0x7f1108102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.452+0000 7f110e05d700 1 --2- 172.21.15.116:0/2638704784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1108101db0 0x7f1108102210 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009910 tx=0x7f10fc009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.453+0000 7f110e05d700 1 -- 172.21.15.116:0/2638704784 shutdown_connections 2024-03-04T16:05:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.453+0000 7f110e05d700 1 --2- 172.21.15.116:0/2638704784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1108101db0 0x7f1108102210 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.453+0000 7f110e05d700 1 --2- 172.21.15.116:0/2638704784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1108100bb0 0x7f1108100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.453+0000 7f110e05d700 1 -- 172.21.15.116:0/2638704784 >> 172.21.15.116:0/2638704784 conn(0x7f11080fc110 msgr2=0x7f11080fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.453+0000 7f110e05d700 1 -- 172.21.15.116:0/2638704784 shutdown_connections 2024-03-04T16:05:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.453+0000 7f110e05d700 1 -- 172.21.15.116:0/2638704784 wait complete. 2024-03-04T16:05:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.453+0000 7f110e05d700 1 Processor -- start 2024-03-04T16:05:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.454+0000 7f110e05d700 1 -- start start 2024-03-04T16:05:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.454+0000 7f110e05d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1108100bb0 0x7f1108198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.454+0000 7f110e05d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1108101db0 0x7f1108198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.454+0000 7f110e05d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11081990b0 con 0x7f1108100bb0 2024-03-04T16:05:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.454+0000 7f110e05d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11081991f0 con 0x7f1108101db0 2024-03-04T16:05:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.454+0000 7f11077fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1108100bb0 0x7f1108198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.454+0000 7f11077fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1108100bb0 0x7f1108198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38790/0 (socket says 172.21.15.116:38790) 2024-03-04T16:05:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.454+0000 7f11077fe700 1 -- 172.21.15.116:0/2630838398 learned_addr learned my addr 172.21.15.116:0/2630838398 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.455+0000 7f11077fe700 1 -- 172.21.15.116:0/2630838398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1108101db0 msgr2=0x7f1108198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.455+0000 7f11077fe700 1 --2- 172.21.15.116:0/2630838398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1108101db0 0x7f1108198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.455+0000 7f11077fe700 1 -- 172.21.15.116:0/2630838398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc0095f0 con 0x7f1108100bb0 2024-03-04T16:05:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.455+0000 7f11077fe700 1 --2- 172.21.15.116:0/2630838398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1108100bb0 0x7f1108198550 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f10f8007e60 tx=0x7f10f800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.458+0000 7f1104ff9700 1 -- 172.21.15.116:0/2630838398 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10f8004a00 con 0x7f1108100bb0 2024-03-04T16:05:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.458+0000 7f1104ff9700 1 -- 172.21.15.116:0/2630838398 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f10f801a070 con 0x7f1108100bb0 2024-03-04T16:05:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.458+0000 7f1104ff9700 1 -- 172.21.15.116:0/2630838398 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10f800b710 con 0x7f1108100bb0 2024-03-04T16:05:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.458+0000 7f110e05d700 1 -- 172.21.15.116:0/2630838398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110818ede0 con 0x7f1108100bb0 2024-03-04T16:05:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.458+0000 7f110e05d700 1 -- 172.21.15.116:0/2630838398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110818f330 con 0x7f1108100bb0 2024-03-04T16:05:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.459+0000 7f110e05d700 1 -- 172.21.15.116:0/2630838398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f4005320 con 0x7f1108100bb0 2024-03-04T16:05:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.460+0000 7f1104ff9700 1 -- 172.21.15.116:0/2630838398 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 30) v1 ==== 91440+0+0 (secure 0 0 0) 0x7f10f8005020 con 0x7f1108100bb0 2024-03-04T16:05:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.461+0000 7f1104ff9700 1 --2- 172.21.15.116:0/2630838398 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f10f006d660 0x7f10f006fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.461+0000 7f1104ff9700 1 -- 172.21.15.116:0/2630838398 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f10f801c070 con 0x7f1108100bb0 2024-03-04T16:05:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.461+0000 7f1106ffd700 1 --2- 172.21.15.116:0/2630838398 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f10f006d660 0x7f10f006fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.462+0000 7f1106ffd700 1 --2- 172.21.15.116:0/2630838398 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f10f006d660 0x7f10f006fb20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f10fc00b5c0 tx=0x7f10fc005fd0 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:17.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.467+0000 7f1104ff9700 1 -- 172.21.15.116:0/2630838398 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f10f8062970 con 0x7f1108100bb0 2024-03-04T16:05:17.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:17 smithi152 ceph-mon[36876]: pgmap v20: 65 pgs: 65 active+clean; 1.0 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 28 KiB/s rd, 1.2 MiB/s wr, 248 op/s 2024-03-04T16:05:17.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:17 smithi152 ceph-mon[36876]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:17.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:17 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:17.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:17 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:17.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:17 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2051654837' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:17.644 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:17 smithi116 ceph-mon[29120]: pgmap v20: 65 pgs: 65 active+clean; 1.0 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 28 KiB/s rd, 1.2 MiB/s wr, 248 op/s 2024-03-04T16:05:17.644 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:17 smithi116 ceph-mon[29120]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:17.644 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:17 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:17.644 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:17 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:17.644 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:17 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2051654837' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.775+0000 7f110e05d700 1 -- 172.21.15.116:0/2630838398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f10f4005cc0 con 0x7f1108100bb0 2024-03-04T16:05:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.776+0000 7f1104ff9700 1 -- 172.21.15.116:0/2630838398 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1831 (secure 0 0 0) 0x7f10f8062500 con 0x7f1108100bb0 2024-03-04T16:05:17.777 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:05:17.777 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:05:17.777 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:05:17.777 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:05:17.777 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:05:17.777 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:05:17.778 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:05:17.779 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:05:17.780 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:05:17.780 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:05:17.780 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:17.780 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:17.780 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.778+0000 7f110e05d700 1 -- 172.21.15.116:0/2630838398 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f10f006d660 msgr2=0x7f10f006fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.778+0000 7f110e05d700 1 --2- 172.21.15.116:0/2630838398 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f10f006d660 0x7f10f006fb20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f10fc00b5c0 tx=0x7f10fc005fd0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.778+0000 7f110e05d700 1 -- 172.21.15.116:0/2630838398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1108100bb0 msgr2=0x7f1108198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.778+0000 7f110e05d700 1 --2- 172.21.15.116:0/2630838398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1108100bb0 0x7f1108198550 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f10f8007e60 tx=0x7f10f800dbe0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.778+0000 7f110e05d700 1 -- 172.21.15.116:0/2630838398 shutdown_connections 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.778+0000 7f110e05d700 1 --2- 172.21.15.116:0/2630838398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1108100bb0 0x7f1108198550 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.778+0000 7f110e05d700 1 --2- 172.21.15.116:0/2630838398 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f10f006d660 0x7f10f006fb20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.778+0000 7f110e05d700 1 --2- 172.21.15.116:0/2630838398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1108101db0 0x7f1108198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.778+0000 7f110e05d700 1 -- 172.21.15.116:0/2630838398 >> 172.21.15.116:0/2630838398 conn(0x7f11080fc110 msgr2=0x7f1108102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.779+0000 7f110e05d700 1 -- 172.21.15.116:0/2630838398 shutdown_connections 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.779+0000 7f110e05d700 1 -- 172.21.15.116:0/2630838398 wait complete. 2024-03-04T16:05:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:05:17.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.890+0000 7fc5654cb700 1 -- 172.21.15.116:0/499444949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560101680 msgr2=0x7fc560103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:17.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.890+0000 7fc5654cb700 1 --2- 172.21.15.116:0/499444949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560101680 0x7fc560103a70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc550009910 tx=0x7fc550009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:17.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.890+0000 7fc5654cb700 1 -- 172.21.15.116:0/499444949 shutdown_connections 2024-03-04T16:05:17.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.890+0000 7fc5654cb700 1 --2- 172.21.15.116:0/499444949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc560103fb0 0x7fc5601063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.890+0000 7fc5654cb700 1 --2- 172.21.15.116:0/499444949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560101680 0x7fc560103a70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.890+0000 7fc5654cb700 1 -- 172.21.15.116:0/499444949 >> 172.21.15.116:0/499444949 conn(0x7fc5600fb050 msgr2=0x7fc5600fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:17.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.890+0000 7fc5654cb700 1 -- 172.21.15.116:0/499444949 shutdown_connections 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.890+0000 7fc5654cb700 1 -- 172.21.15.116:0/499444949 wait complete. 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc5654cb700 1 Processor -- start 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc5654cb700 1 -- start start 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc5654cb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560101680 0x7fc560196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc5654cb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc560103fb0 0x7fc560196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc5654cb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc560196e80 con 0x7fc560101680 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc5654cb700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc560196fc0 con 0x7fc560103fb0 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc55effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560101680 0x7fc560196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc55effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560101680 0x7fc560196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38816/0 (socket says 172.21.15.116:38816) 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc55effd700 1 -- 172.21.15.116:0/3549054473 learned_addr learned my addr 172.21.15.116:0/3549054473 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc55e7fc700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc560103fb0 0x7fc560196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:17.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc55effd700 1 -- 172.21.15.116:0/3549054473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc560103fb0 msgr2=0x7fc560196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:17.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc55effd700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc560103fb0 0x7fc560196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:17.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.891+0000 7fc55effd700 1 -- 172.21.15.116:0/3549054473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5500095f0 con 0x7fc560101680 2024-03-04T16:05:17.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.892+0000 7fc55effd700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560101680 0x7fc560196320 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc55000bd70 tx=0x7fc55000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:17.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.892+0000 7fc547fff700 1 -- 172.21.15.116:0/3549054473 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc55001d070 con 0x7fc560101680 2024-03-04T16:05:17.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.892+0000 7fc547fff700 1 -- 172.21.15.116:0/3549054473 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc550022470 con 0x7fc560101680 2024-03-04T16:05:17.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.892+0000 7fc5654cb700 1 -- 172.21.15.116:0/3549054473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56019ba10 con 0x7fc560101680 2024-03-04T16:05:17.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.892+0000 7fc547fff700 1 -- 172.21.15.116:0/3549054473 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc55000f670 con 0x7fc560101680 2024-03-04T16:05:17.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.892+0000 7fc5654cb700 1 -- 172.21.15.116:0/3549054473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56019bf00 con 0x7fc560101680 2024-03-04T16:05:17.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.893+0000 7fc5654cb700 1 -- 172.21.15.116:0/3549054473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc560190520 con 0x7fc560101680 2024-03-04T16:05:17.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.893+0000 7fc547fff700 1 -- 172.21.15.116:0/3549054473 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 30) v1 ==== 91440+0+0 (secure 0 0 0) 0x7fc550022a50 con 0x7fc560101680 2024-03-04T16:05:17.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.894+0000 7fc547fff700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fc54806d660 0x7fc54806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:17.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.894+0000 7fc547fff700 1 -- 172.21.15.116:0/3549054473 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fc55008e990 con 0x7fc560101680 2024-03-04T16:05:17.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.894+0000 7fc55e7fc700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fc54806d660 0x7fc54806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:17.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.894+0000 7fc55e7fc700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fc54806d660 0x7fc54806fb20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc55400bbd0 tx=0x7fc55400b5b0 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:17.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:17.898+0000 7fc547fff700 1 -- 172.21.15.116:0/3549054473 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fc55005c150 con 0x7fc560101680 2024-03-04T16:05:18.082 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-04T16:05:18.082 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-04T16:05:18.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.143+0000 7fc5654cb700 1 -- 172.21.15.116:0/3549054473 --> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5600611d0 con 0x7fc54806d660 2024-03-04T16:05:18.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.144+0000 7fc547fff700 1 -- 172.21.15.116:0/3549054473 <== mgr.14738 v2:172.21.15.116:6800/2966163948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fc5600611d0 con 0x7fc54806d660 2024-03-04T16:05:18.145 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:05:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:05:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:05:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:05:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:05:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:05:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:05:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "2/23 daemons upgraded", 2024-03-04T16:05:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mgr daemons", 2024-03-04T16:05:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:05:18.146 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:05:18.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 -- 172.21.15.116:0/3549054473 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fc54806d660 msgr2=0x7fc54806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:18.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fc54806d660 0x7fc54806fb20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc55400bbd0 tx=0x7fc55400b5b0 comp rx=0 tx=0).stop 2024-03-04T16:05:18.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 -- 172.21.15.116:0/3549054473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560101680 msgr2=0x7fc560196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:18.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560101680 0x7fc560196320 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc55000bd70 tx=0x7fc55000be50 comp rx=0 tx=0).stop 2024-03-04T16:05:18.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 -- 172.21.15.116:0/3549054473 shutdown_connections 2024-03-04T16:05:18.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560101680 0x7fc560196320 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:18.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7fc54806d660 0x7fc54806fb20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:18.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 --2- 172.21.15.116:0/3549054473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc560103fb0 0x7fc560196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:18.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 -- 172.21.15.116:0/3549054473 >> 172.21.15.116:0/3549054473 conn(0x7fc5600fb050 msgr2=0x7fc5600fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:18.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 -- 172.21.15.116:0/3549054473 shutdown_connections 2024-03-04T16:05:18.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.146+0000 7fc5654cb700 1 -- 172.21.15.116:0/3549054473 wait complete. 2024-03-04T16:05:18.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.255+0000 7f08a901c700 1 -- 172.21.15.116:0/2871238577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08a40ffae0 msgr2=0x7f08a40fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:18.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.255+0000 7f08a901c700 1 --2- 172.21.15.116:0/2871238577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08a40ffae0 0x7f08a40fff00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0894009910 tx=0x7f0894009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.255+0000 7f08a901c700 1 -- 172.21.15.116:0/2871238577 shutdown_connections 2024-03-04T16:05:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.255+0000 7f08a901c700 1 --2- 172.21.15.116:0/2871238577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08a4100ce0 0x7f08a4101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.255+0000 7f08a901c700 1 --2- 172.21.15.116:0/2871238577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08a40ffae0 0x7f08a40fff00 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.255+0000 7f08a901c700 1 -- 172.21.15.116:0/2871238577 >> 172.21.15.116:0/2871238577 conn(0x7f08a40fb080 msgr2=0x7f08a40fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.255+0000 7f08a901c700 1 -- 172.21.15.116:0/2871238577 shutdown_connections 2024-03-04T16:05:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.255+0000 7f08a901c700 1 -- 172.21.15.116:0/2871238577 wait complete. 2024-03-04T16:05:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.256+0000 7f08a901c700 1 Processor -- start 2024-03-04T16:05:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.256+0000 7f08a901c700 1 -- start start 2024-03-04T16:05:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.256+0000 7f08a901c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08a40ffae0 0x7f08a4069890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.256+0000 7f08a901c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08a4100ce0 0x7f08a419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.256+0000 7f08a901c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a419b6a0 con 0x7f08a4100ce0 2024-03-04T16:05:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.256+0000 7f08a901c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a419b7e0 con 0x7f08a40ffae0 2024-03-04T16:05:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.256+0000 7f08a259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08a4100ce0 0x7f08a419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.256+0000 7f08a259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08a4100ce0 0x7f08a419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38844/0 (socket says 172.21.15.116:38844) 2024-03-04T16:05:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.256+0000 7f08a259c700 1 -- 172.21.15.116:0/3830373093 learned_addr learned my addr 172.21.15.116:0/3830373093 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.256+0000 7f08a2d9d700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08a40ffae0 0x7f08a4069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.257+0000 7f08a2d9d700 1 -- 172.21.15.116:0/3830373093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08a4100ce0 msgr2=0x7f08a419b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.257+0000 7f08a2d9d700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08a4100ce0 0x7f08a419b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.257+0000 7f08a2d9d700 1 -- 172.21.15.116:0/3830373093 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08940095f0 con 0x7f08a40ffae0 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.257+0000 7f08a259c700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08a4100ce0 0x7f08a419b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.257+0000 7f08a2d9d700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08a40ffae0 0x7f08a4069890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0894006010 tx=0x7f089400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.257+0000 7f088bfff700 1 -- 172.21.15.116:0/3830373093 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f089401d070 con 0x7f08a40ffae0 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.257+0000 7f08a901c700 1 -- 172.21.15.116:0/3830373093 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a419ba60 con 0x7f08a40ffae0 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.257+0000 7f08a901c700 1 -- 172.21.15.116:0/3830373093 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a419bf50 con 0x7f08a40ffae0 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.257+0000 7f088bfff700 1 -- 172.21.15.116:0/3830373093 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0894005dc0 con 0x7f08a40ffae0 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.257+0000 7f088bfff700 1 -- 172.21.15.116:0/3830373093 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0894017400 con 0x7f08a40ffae0 2024-03-04T16:05:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.258+0000 7f08a901c700 1 -- 172.21.15.116:0/3830373093 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a4108640 con 0x7f08a40ffae0 2024-03-04T16:05:18.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.259+0000 7f088bfff700 1 -- 172.21.15.116:0/3830373093 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 30) v1 ==== 91440+0+0 (secure 0 0 0) 0x7f0894003810 con 0x7f08a40ffae0 2024-03-04T16:05:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.259+0000 7f088bfff700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f088c06d660 0x7f088c06fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.259+0000 7f088bfff700 1 -- 172.21.15.116:0/3830373093 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f089408e870 con 0x7f08a40ffae0 2024-03-04T16:05:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.259+0000 7f08a259c700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f088c06d660 0x7f088c06fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.260+0000 7f08a259c700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f088c06d660 0x7f088c06fb20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f08a4069e10 tx=0x7f0898009260 comp rx=0 tx=0).ready entity=mgr.14738 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:18.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.267+0000 7f088bfff700 1 -- 172.21.15.116:0/3830373093 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f089405be40 con 0x7f08a40ffae0 2024-03-04T16:05:18.469 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:18 smithi116 ceph-mon[29120]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:18.469 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:18 smithi116 ceph-mon[29120]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:18.469 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:18 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/2630838398' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:05:18.469 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:18 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:18.469 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:18 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:18.502 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:18 smithi152 ceph-mon[36876]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:18.502 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:18 smithi152 ceph-mon[36876]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:18.503 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:18 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2630838398' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:05:18.503 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:18 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:18.503 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:18 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:18.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.640+0000 7f08a901c700 1 -- 172.21.15.116:0/3830373093 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f08a4108830 con 0x7f08a40ffae0 2024-03-04T16:05:18.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.641+0000 7f088bfff700 1 -- 172.21.15.116:0/3830373093 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f089405b9d0 con 0x7f08a40ffae0 2024-03-04T16:05:18.642 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:05:18.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 -- 172.21.15.116:0/3830373093 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f088c06d660 msgr2=0x7f088c06fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:18.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f088c06d660 0x7f088c06fb20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f08a4069e10 tx=0x7f0898009260 comp rx=0 tx=0).stop 2024-03-04T16:05:18.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 -- 172.21.15.116:0/3830373093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08a40ffae0 msgr2=0x7f08a4069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:18.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08a40ffae0 0x7f08a4069890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0894006010 tx=0x7f089400f690 comp rx=0 tx=0).stop 2024-03-04T16:05:18.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 -- 172.21.15.116:0/3830373093 shutdown_connections 2024-03-04T16:05:18.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f088c06d660 0x7f088c06fb20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:18.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08a40ffae0 0x7f08a4069890 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:18.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 --2- 172.21.15.116:0/3830373093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08a4100ce0 0x7f08a419b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:18.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 -- 172.21.15.116:0/3830373093 >> 172.21.15.116:0/3830373093 conn(0x7f08a40fb080 msgr2=0x7f08a4103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:18.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 -- 172.21.15.116:0/3830373093 shutdown_connections 2024-03-04T16:05:18.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:18.643+0000 7f08a901c700 1 -- 172.21.15.116:0/3830373093 wait complete. 2024-03-04T16:05:19.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:19 smithi116 ceph-mon[29120]: pgmap v21: 65 pgs: 65 active+clean; 830 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 41 KiB/s rd, 1.4 MiB/s wr, 295 op/s 2024-03-04T16:05:19.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:19 smithi116 ceph-mon[29120]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:19.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:19 smithi116 ceph-mon[29120]: from='client.? 172.21.15.116:0/3830373093' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:05:19.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:19 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:19.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:19 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:19.819 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:19 smithi152 ceph-mon[36876]: pgmap v21: 65 pgs: 65 active+clean; 830 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 41 KiB/s rd, 1.4 MiB/s wr, 295 op/s 2024-03-04T16:05:19.819 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:19 smithi152 ceph-mon[36876]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:19.819 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:19 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/3830373093' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:05:19.819 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:19 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:19.819 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:19 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:21.600 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:21 smithi152 ceph-mon[36876]: pgmap v22: 65 pgs: 65 active+clean; 577 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 27 KiB/s rd, 1.6 MiB/s wr, 305 op/s 2024-03-04T16:05:21.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:21 smithi116 ceph-mon[29120]: pgmap v22: 65 pgs: 65 active+clean; 577 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 27 KiB/s rd, 1.6 MiB/s wr, 305 op/s 2024-03-04T16:05:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:22 smithi116 ceph-mon[29120]: Standby manager daemon smithi152.aawieo restarted 2024-03-04T16:05:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:22 smithi116 ceph-mon[29120]: Standby manager daemon smithi152.aawieo started 2024-03-04T16:05:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:22 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3245689484' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/crt"}]: dispatch 2024-03-04T16:05:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:22 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3245689484' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:05:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:22 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3245689484' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/key"}]: dispatch 2024-03-04T16:05:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:22 smithi116 ceph-mon[29120]: from='mgr.? 172.21.15.152:0/3245689484' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:05:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:22 smithi152 ceph-mon[36876]: Standby manager daemon smithi152.aawieo restarted 2024-03-04T16:05:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:22 smithi152 ceph-mon[36876]: Standby manager daemon smithi152.aawieo started 2024-03-04T16:05:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:22 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3245689484' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/crt"}]: dispatch 2024-03-04T16:05:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:22 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3245689484' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:05:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:22 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3245689484' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/key"}]: dispatch 2024-03-04T16:05:22.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:22 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/3245689484' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:05:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:23 smithi116 ceph-mon[29120]: pgmap v23: 65 pgs: 65 active+clean; 577 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 27 KiB/s rd, 1.1 MiB/s wr, 228 op/s 2024-03-04T16:05:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:23 smithi116 ceph-mon[29120]: mgrmap e31: smithi116.jwdhop(active, since 40s), standbys: smithi152.aawieo 2024-03-04T16:05:23.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:23 smithi152 ceph-mon[36876]: pgmap v23: 65 pgs: 65 active+clean; 577 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 27 KiB/s rd, 1.1 MiB/s wr, 228 op/s 2024-03-04T16:05:23.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:23 smithi152 ceph-mon[36876]: mgrmap e31: smithi116.jwdhop(active, since 40s), standbys: smithi152.aawieo 2024-03-04T16:05:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:25 smithi116 ceph-mon[29120]: pgmap v24: 65 pgs: 65 active+clean; 278 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 41 KiB/s rd, 1.6 MiB/s wr, 359 op/s 2024-03-04T16:05:25.809 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:25 smithi152 ceph-mon[36876]: pgmap v24: 65 pgs: 65 active+clean; 278 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 41 KiB/s rd, 1.6 MiB/s wr, 359 op/s 2024-03-04T16:05:27.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:27 smithi116 ceph-mon[29120]: pgmap v25: 65 pgs: 65 active+clean; 278 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 27 KiB/s rd, 1.0 MiB/s wr, 237 op/s 2024-03-04T16:05:27.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:27 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:27.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:27 smithi152 ceph-mon[36876]: pgmap v25: 65 pgs: 65 active+clean; 278 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 27 KiB/s rd, 1.0 MiB/s wr, 237 op/s 2024-03-04T16:05:27.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:27 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:29.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:29 smithi116 ceph-mon[29120]: pgmap v26: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 43 KiB/s rd, 1.0 MiB/s wr, 300 op/s 2024-03-04T16:05:29.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:29 smithi152 ceph-mon[36876]: pgmap v26: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 43 KiB/s rd, 1.0 MiB/s wr, 300 op/s 2024-03-04T16:05:31.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:31 smithi152 ceph-mon[36876]: pgmap v27: 65 pgs: 65 active+clean; 276 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 1.4 MiB/s wr, 335 op/s 2024-03-04T16:05:31.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:31 smithi116 ceph-mon[29120]: pgmap v27: 65 pgs: 65 active+clean; 276 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 1.4 MiB/s wr, 335 op/s 2024-03-04T16:05:33.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:33 smithi116 ceph-mon[29120]: pgmap v28: 65 pgs: 65 active+clean; 276 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 1.1 MiB/s wr, 276 op/s 2024-03-04T16:05:33.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:33 smithi152 ceph-mon[36876]: pgmap v28: 65 pgs: 65 active+clean; 276 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 1.1 MiB/s wr, 276 op/s 2024-03-04T16:05:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:35 smithi116 ceph-mon[29120]: pgmap v29: 65 pgs: 65 active+clean; 277 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44 KiB/s rd, 1.6 MiB/s wr, 442 op/s 2024-03-04T16:05:35.826 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:35 smithi152 ceph-mon[36876]: pgmap v29: 65 pgs: 65 active+clean; 277 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44 KiB/s rd, 1.6 MiB/s wr, 442 op/s 2024-03-04T16:05:37.548 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:37 smithi152 ceph-mon[36876]: pgmap v30: 65 pgs: 65 active+clean; 277 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 1.1 MiB/s wr, 311 op/s 2024-03-04T16:05:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:37 smithi116 ceph-mon[29120]: pgmap v30: 65 pgs: 65 active+clean; 277 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 1.1 MiB/s wr, 311 op/s 2024-03-04T16:05:38.619 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:38.619 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:38.619 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:38.619 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: Upgrade: Setting container_image for all mgr 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi116.jwdhop"}]: dispatch 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi116.jwdhop"}]': finished 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi152.aawieo"}]: dispatch 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi152.aawieo"}]': finished 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:05:38.620 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:38 smithi116 ceph-mon[29120]: pgmap v31: 65 pgs: 65 active+clean; 279 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44 KiB/s rd, 1.4 MiB/s wr, 374 op/s 2024-03-04T16:05:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: Upgrade: Setting container_image for all mgr 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi116.jwdhop"}]: dispatch 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi116.jwdhop"}]': finished 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi152.aawieo"}]: dispatch 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi152.aawieo"}]': finished 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:05:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:38 smithi152 ceph-mon[36876]: pgmap v31: 65 pgs: 65 active+clean; 279 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44 KiB/s rd, 1.4 MiB/s wr, 374 op/s 2024-03-04T16:05:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:40 smithi116 ceph-mon[29120]: Upgrade: Updating mon.smithi116 2024-03-04T16:05:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:40 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:40 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:05:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:40 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:05:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:40 smithi116 ceph-mon[29120]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:40.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:40 smithi116 ceph-mon[29120]: Deploying daemon mon.smithi116 on smithi116 2024-03-04T16:05:40.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:40 smithi116 ceph-mon[29120]: pgmap v32: 65 pgs: 65 active+clean; 281 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 28 KiB/s rd, 1.6 MiB/s wr, 364 op/s 2024-03-04T16:05:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:40 smithi152 ceph-mon[36876]: Upgrade: Updating mon.smithi116 2024-03-04T16:05:40.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:40 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:40.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:40 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:05:40.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:40 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:05:40.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:40 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:40.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:40 smithi152 ceph-mon[36876]: Deploying daemon mon.smithi116 on smithi116 2024-03-04T16:05:40.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:40 smithi152 ceph-mon[36876]: pgmap v32: 65 pgs: 65 active+clean; 281 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 28 KiB/s rd, 1.6 MiB/s wr, 364 op/s 2024-03-04T16:05:41.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:41 smithi116 systemd[1]: Stopping Ceph mon.smithi116 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:05:42.357 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:42 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116[29116]: 2024-03-04T16:05:42.090+0000 7fa13b32c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi116 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:05:42.357 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:42 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116[29116]: 2024-03-04T16:05:42.090+0000 7fa13b32c700 -1 mon.smithi116@0(leader) e2 *** Got Signal Terminated *** 2024-03-04T16:05:42.357 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:42 smithi116 podman[93976]: 2024-03-04 16:05:42.132339403 +0000 UTC m=+0.445544930 container died 60e0bf118c4de686f5559d34e2b95dda8785a9e2b515c8e2b2dfd708a3c11169 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2) 2024-03-04T16:05:42.891 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:42 smithi116 podman[93976]: 2024-03-04 16:05:42.614278955 +0000 UTC m=+0.927484479 container cleanup 60e0bf118c4de686f5559d34e2b95dda8785a9e2b515c8e2b2dfd708a3c11169 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-04T16:05:42.891 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:42 smithi116 bash[93976]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116 2024-03-04T16:05:43.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:42 smithi116 podman[94001]: 2024-03-04 16:05:42.891187838 +0000 UTC m=+0.756134318 container remove 60e0bf118c4de686f5559d34e2b95dda8785a9e2b515c8e2b2dfd708a3c11169 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-03-04T16:05:43.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi116.service: Deactivated successfully. 2024-03-04T16:05:43.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 systemd[1]: Stopped Ceph mon.smithi116 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:05:43.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi116.service: Consumed 11.822s CPU time. 2024-03-04T16:05:43.575 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 systemd[1]: Starting Ceph mon.smithi116 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:05:43.575 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 podman[94144]: 2024-03-04 16:05:43.419831872 +0000 UTC m=+0.022934894 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:05:43.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 podman[94144]: 2024-03-04 16:05:43.575571184 +0000 UTC m=+0.178674188 container create 4ecbaf85d8a2575b809ea886bbfd181d6aac4fe12b6de9e19635f8eae67005b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux ) 2024-03-04T16:05:43.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 podman[94144]: 2024-03-04 16:05:43.81310735 +0000 UTC m=+0.416210384 container init 4ecbaf85d8a2575b809ea886bbfd181d6aac4fe12b6de9e19635f8eae67005b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-03-04T16:05:43.841 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 podman[94144]: 2024-03-04 16:05:43.816547241 +0000 UTC m=+0.419650256 container start 4ecbaf85d8a2575b809ea886bbfd181d6aac4fe12b6de9e19635f8eae67005b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-04T16:05:44.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T16:05:44.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 2 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: pidfile_write: ignore empty --pid-file 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: load: jerasure load: lrc 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: RocksDB version: 7.9.2 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Git sha 0 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: DB SUMMARY 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: DB Session ID: 8JZRMROGS0AFRERL1LKW 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: CURRENT file: CURRENT 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1509 Bytes 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi116/store.db dir, Total Num: 1, files: 000032.sst 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi116/store.db: 000030.log size: 1524759 ; 2024-03-04T16:05:44.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.error_if_exists: 0 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.create_if_missing: 0 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.paranoid_checks: 1 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.env: 0x55ca2213bb20 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.info_log: 0x55ca241bb780 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.statistics: (nil) 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.use_fsync: 0 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_log_file_size: 0 2024-03-04T16:05:44.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.allow_fallocate: 1 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.use_direct_reads: 0 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.db_log_dir: 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.wal_dir: 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T16:05:44.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.write_buffer_manager: 0x55ca2418f400 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T16:05:44.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.unordered_write: 0 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.row_cache: None 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.wal_filter: None 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.two_write_queues: 0 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.wal_compression: 0 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.atomic_flush: 0 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T16:05:44.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T16:05:44.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.log_readahead_size: 0 2024-03-04T16:05:44.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T16:05:44.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T16:05:44.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T16:05:44.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T16:05:44.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T16:05:44.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T16:05:44.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T16:05:44.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_background_jobs: 2 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_background_compactions: -1 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_subcompactions: 1 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_open_files: -1 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T16:05:44.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_background_flushes: -1 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Compression algorithms supported: 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: kZSTD supported: 0 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: kXpressCompression supported: 0 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: kZlibCompression supported: 1 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: kSnappyCompression supported: 1 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: kLZ4Compression supported: 1 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: kBZip2Compression supported: 0 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi116/store.db/MANIFEST-000015 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.merge_operator: 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_filter: None 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ca241bb8a0) 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_type: 0 2024-03-04T16:05:44.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: data_block_index_type: 0 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_shortening: 1 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: checksum: 4 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: no_block_cache: 0 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache: 0x55ca241c1090 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_options: 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: capacity : 536870912 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: num_shard_bits : 4 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: strict_capacity_limit : 0 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_compressed: (nil) 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: persistent_cache: (nil) 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_size: 4096 2024-03-04T16:05:44.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_size_deviation: 10 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_restart_interval: 16 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_block_restart_interval: 1 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: metadata_block_size: 4096 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: partition_filters: 0 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: use_delta_encoding: 1 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: filter_policy: bloomfilter 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: whole_key_filtering: 1 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: verify_compression: 0 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: format_version: 5 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: enable_index_compression: 1 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_align: 0 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: max_auto_readahead_size: 262144 2024-03-04T16:05:44.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: prepopulate_block_cache: 0 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: initial_auto_readahead_size: 8192 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compression: NoCompression 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.num_levels: 7 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T16:05:44.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T16:05:44.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T16:05:44.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.table_properties_collectors: 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.inplace_update_support: 0 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.bloom_locality: 0 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.max_successive_merges: 0 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T16:05:44.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.ttl: 2592000 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.enable_blob_files: false 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.min_blob_size: 0 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T16:05:44.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi116/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12089, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 46a12290-7bb5-4234-a8dd-39d260c12269 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568343847582, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568343853315, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 1355388, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12091, "largest_seqno": 12953, "table_properties": {"data_size": 1350990, "index_size": 2488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 901, "raw_key_size": 8163, "raw_average_key_size": 22, "raw_value_size": 1343178, "raw_average_value_size": 3783, "num_data_blocks": 114, "num_entries": 355, "num_filter_entries": 355, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709568343, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "46a12290-7bb5-4234-a8dd-39d260c12269", "db_session_id": "8JZRMROGS0AFRERL1LKW", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568343853440, "job": 1, "event": "recovery_finished"} 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi116/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ca242b2000 2024-03-04T16:05:44.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: DB pointer 0x55ca2429e000 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** DB Stats ** 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T16:05:44.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: L0 1/0 1.29 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 390.4 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: L6 1/0 5.52 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Sum 2/0 6.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 390.4 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 390.4 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 390.4 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-04T16:05:44.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative compaction: 0.00 GB write, 158.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval compaction: 0.00 GB write, 158.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Block cache BinnedLRUCache@0x55ca241c1090#2 capacity: 512.00 MB usage: 3.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.03 KB,0.000196695%) IndexBlock(1,2.58 KB,0.000491738%) Misc(1,0.00 KB,0%) 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: starting mon.smithi116 rank 0 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi116 fsid 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: mon.smithi116@-1(???) e2 preinit fsid 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: mon.smithi116@-1(???).mds e14 new map 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: mon.smithi116@-1(???).mds e14 print_map 2024-03-04T16:05:44.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: e14 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: legacy client fscid: 1 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Filesystem 'cephfs' (1) 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: fs_name cephfs 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: epoch 14 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: tableserver 0 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: root 0 2024-03-04T16:05:44.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: session_timeout 60 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: session_autoclose 300 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: max_file_size 1099511627776 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: max_xattr_size 65536 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: required_client_features {} 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: last_failure 0 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: last_failure_osd_epoch 47 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: max_mds 1 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: in 0 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: up {0=24247} 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: failed 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: damaged 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: stopped 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: data_pools [3] 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: metadata_pool 2 2024-03-04T16:05:44.249 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: inline_data disabled 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: balancer 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: bal_rank_mask -1 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: standby_count_wanted 1 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: [mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Standby daemons: 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: [mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: [mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: [mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: mon.smithi116@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: mon.smithi116@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: mon.smithi116@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-04T16:05:44.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: mon.smithi116@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-04T16:05:44.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 ceph-mon[94164]: mon.smithi116@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-03-04T16:05:44.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 bash[94144]: 4ecbaf85d8a2575b809ea886bbfd181d6aac4fe12b6de9e19635f8eae67005b9 2024-03-04T16:05:44.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:43 smithi116 systemd[1]: Started Ceph mon.smithi116 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:05:45.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: pgmap v34: 65 pgs: 65 active+clean; 283 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44 KiB/s rd, 1.6 MiB/s wr, 458 op/s 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: mon.smithi116 calling monitor election 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: mon.smithi116 is new leader, mons smithi116,smithi152 in quorum (ranks 0,1) 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: monmap e2: 2 mons at {smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],smithi152=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 3 up:standby 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: osdmap e49: 6 total, 6 up, 6 in 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: mgrmap e31: smithi116.jwdhop(active, since 63s), standbys: smithi152.aawieo 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: overall HEALTH_OK 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: from='mgr.14738 ' entity='' 2024-03-04T16:05:45.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:45 smithi152 ceph-mon[36876]: mgrmap e32: smithi116.jwdhop(active, since 63s), standbys: smithi152.aawieo 2024-03-04T16:05:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T16:05:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: from='mgr.14738 172.21.15.116:0/460264221' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T16:05:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: pgmap v34: 65 pgs: 65 active+clean; 283 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44 KiB/s rd, 1.6 MiB/s wr, 458 op/s 2024-03-04T16:05:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: mon.smithi116 calling monitor election 2024-03-04T16:05:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: mon.smithi116 is new leader, mons smithi116,smithi152 in quorum (ranks 0,1) 2024-03-04T16:05:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: monmap e2: 2 mons at {smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],smithi152=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:05:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 3 up:standby 2024-03-04T16:05:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: osdmap e49: 6 total, 6 up, 6 in 2024-03-04T16:05:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: mgrmap e31: smithi116.jwdhop(active, since 63s), standbys: smithi152.aawieo 2024-03-04T16:05:45.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: overall HEALTH_OK 2024-03-04T16:05:45.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: from='mgr.14738 ' entity='' 2024-03-04T16:05:45.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:45 smithi116 ceph-mon[94164]: mgrmap e32: smithi116.jwdhop(active, since 63s), standbys: smithi152.aawieo 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.760+0000 7f6b3cb50700 1 -- 172.21.15.116:0/3422790705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b380feed0 msgr2=0x7f6b380ff2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.760+0000 7f6b3cb50700 1 --2- 172.21.15.116:0/3422790705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b380feed0 0x7f6b380ff2f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009910 tx=0x7f6b28009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.760+0000 7f6b3cb50700 1 -- 172.21.15.116:0/3422790705 shutdown_connections 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.760+0000 7f6b3cb50700 1 --2- 172.21.15.116:0/3422790705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b38101eb0 0x7f6b38102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.760+0000 7f6b3cb50700 1 --2- 172.21.15.116:0/3422790705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b380feed0 0x7f6b380ff2f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.760+0000 7f6b3cb50700 1 -- 172.21.15.116:0/3422790705 >> 172.21.15.116:0/3422790705 conn(0x7f6b380fa810 msgr2=0x7f6b380fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.760+0000 7f6b3cb50700 1 -- 172.21.15.116:0/3422790705 shutdown_connections 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.760+0000 7f6b3cb50700 1 -- 172.21.15.116:0/3422790705 wait complete. 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.761+0000 7f6b3cb50700 1 Processor -- start 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.761+0000 7f6b3cb50700 1 -- start start 2024-03-04T16:05:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.761+0000 7f6b3cb50700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b380feed0 0x7f6b38196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.761+0000 7f6b3cb50700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b38101eb0 0x7f6b38196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.761+0000 7f6b3cb50700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b38196fa0 con 0x7f6b380feed0 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.761+0000 7f6b3cb50700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b381970e0 con 0x7f6b38101eb0 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b3659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b380feed0 0x7f6b38196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b35d9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b38101eb0 0x7f6b38196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b35d9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b38101eb0 0x7f6b38196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41328/0 (socket says 172.21.15.116:41328) 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b3659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b380feed0 0x7f6b38196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33078/0 (socket says 172.21.15.116:33078) 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b35d9b700 1 -- 172.21.15.116:0/2557382364 learned_addr learned my addr 172.21.15.116:0/2557382364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b3659c700 1 -- 172.21.15.116:0/2557382364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b38101eb0 msgr2=0x7f6b38196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b3659c700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b38101eb0 0x7f6b38196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b3659c700 1 -- 172.21.15.116:0/2557382364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b280095f0 con 0x7f6b380feed0 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b3659c700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b380feed0 0x7f6b38196440 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6b28006010 tx=0x7f6b2800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b2801d070 con 0x7f6b380feed0 2024-03-04T16:05:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6b28005640 con 0x7f6b380feed0 2024-03-04T16:05:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b3cb50700 1 -- 172.21.15.116:0/2557382364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3819bb30 con 0x7f6b380feed0 2024-03-04T16:05:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.762+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b28022470 con 0x7f6b380feed0 2024-03-04T16:05:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.763+0000 7f6b3cb50700 1 -- 172.21.15.116:0/2557382364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3819bff0 con 0x7f6b380feed0 2024-03-04T16:05:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.764+0000 7f6b3cb50700 1 -- 172.21.15.116:0/2557382364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b38109770 con 0x7f6b380feed0 2024-03-04T16:05:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.764+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 32) v1 ==== 91587+0+0 (secure 0 0 0) 0x7f6b28005160 con 0x7f6b380feed0 2024-03-04T16:05:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.765+0000 7f6b277fe700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 0x7f6b2006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.765+0000 7f6b35d9b700 1 -- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 msgr2=0x7f6b2006fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2966163948 2024-03-04T16:05:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.765+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f6b2808e9f0 con 0x7f6b380feed0 2024-03-04T16:05:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.765+0000 7f6b35d9b700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 0x7f6b2006fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T16:05:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.770+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6b28090e60 con 0x7f6b380feed0 2024-03-04T16:05:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.966+0000 7f6b35d9b700 1 -- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 msgr2=0x7f6b2006fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2966163948 2024-03-04T16:05:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.966+0000 7f6b35d9b700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 0x7f6b2006fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T16:05:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:48.993+0000 7f6b3cb50700 1 -- 172.21.15.116:0/2557382364 --> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b38106800 con 0x7f6b2006d9e0 2024-03-04T16:05:49.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:49.367+0000 7f6b35d9b700 1 -- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 msgr2=0x7f6b2006fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2966163948 2024-03-04T16:05:49.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:49.367+0000 7f6b35d9b700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 0x7f6b2006fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-04T16:05:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:50.168+0000 7f6b35d9b700 1 -- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 msgr2=0x7f6b2006fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2966163948 2024-03-04T16:05:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:50.168+0000 7f6b35d9b700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 0x7f6b2006fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-04T16:05:51.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:51.769+0000 7f6b35d9b700 1 -- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 msgr2=0x7f6b2006fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2966163948 2024-03-04T16:05:51.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:51.769+0000 7f6b35d9b700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 0x7f6b2006fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-04T16:05:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:53.001+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 33) v1 ==== 91108+0+0 (secure 0 0 0) 0x7f6b2808cd50 con 0x7f6b380feed0 2024-03-04T16:05:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:53.001+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 msgr2=0x7f6b2006fea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T16:05:53.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:53.001+0000 7f6b277fe700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 0x7f6b2006fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:53.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: Active manager daemon smithi116.jwdhop restarted 2024-03-04T16:05:53.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: Activating manager daemon smithi116.jwdhop 2024-03-04T16:05:53.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: osdmap e50: 6 total, 6 up, 6 in 2024-03-04T16:05:53.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: mgrmap e33: smithi116.jwdhop(active, starting, since 0.00969702s), standbys: smithi152.aawieo 2024-03-04T16:05:53.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T16:05:53.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi116.jwdhop", "id": "smithi116.jwdhop"}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi152.aawieo", "id": "smithi152.aawieo"}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:05:53.251 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:05:53.252 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:05:53.252 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:53 smithi116 ceph-mon[94164]: Manager daemon smithi116.jwdhop is now available 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: Active manager daemon smithi116.jwdhop restarted 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: Activating manager daemon smithi116.jwdhop 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: osdmap e50: 6 total, 6 up, 6 in 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: mgrmap e33: smithi116.jwdhop(active, starting, since 0.00969702s), standbys: smithi152.aawieo 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:05:53.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi116.jwdhop", "id": "smithi116.jwdhop"}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr metadata", "who": "smithi152.aawieo", "id": "smithi152.aawieo"}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:05:53.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:53 smithi152 ceph-mon[36876]: Manager daemon smithi116.jwdhop is now available 2024-03-04T16:05:54.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.001+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 34) v1 ==== 91235+0+0 (secure 0 0 0) 0x7f6b2808cd50 con 0x7f6b380feed0 2024-03-04T16:05:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.001+0000 7f6b277fe700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6b200369b0 0x7f6b20077650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.002+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b38106800 con 0x7f6b200369b0 2024-03-04T16:05:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.002+0000 7f6b35d9b700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6b200369b0 0x7f6b20077650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.003+0000 7f6b35d9b700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6b200369b0 0x7f6b20077650 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009940 tx=0x7f6b2c009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.006+0000 7f6b277fe700 1 -- 172.21.15.116:0/2557382364 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f6b38106800 con 0x7f6b200369b0 2024-03-04T16:05:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 -- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6b200369b0 msgr2=0x7f6b20077650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6b200369b0 0x7f6b20077650 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009940 tx=0x7f6b2c009260 comp rx=0 tx=0).stop 2024-03-04T16:05:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 -- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b380feed0 msgr2=0x7f6b38196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b380feed0 0x7f6b38196440 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6b28006010 tx=0x7f6b2800fa00 comp rx=0 tx=0).stop 2024-03-04T16:05:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 -- 172.21.15.116:0/2557382364 shutdown_connections 2024-03-04T16:05:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6b200369b0 0x7f6b20077650 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b380feed0 0x7f6b38196440 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.116:6800/2966163948,v1:172.21.15.116:6801/2966163948] conn(0x7f6b2006d9e0 0x7f6b2006fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 --2- 172.21.15.116:0/2557382364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b38101eb0 0x7f6b38196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 -- 172.21.15.116:0/2557382364 >> 172.21.15.116:0/2557382364 conn(0x7f6b380fa810 msgr2=0x7f6b381050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.010+0000 7f6b3cb50700 1 -- 172.21.15.116:0/2557382364 shutdown_connections 2024-03-04T16:05:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.011+0000 7f6b3cb50700 1 -- 172.21.15.116:0/2557382364 wait complete. 2024-03-04T16:05:54.025 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:05:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.165+0000 7f5d04cd7700 1 -- 172.21.15.116:0/1594119989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cfc004a00 con 0x7f5d000ffab0 2024-03-04T16:05:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.165+0000 7f5d07f3d700 1 -- 172.21.15.116:0/1594119989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d000ffab0 msgr2=0x7f5d000ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.165+0000 7f5d07f3d700 1 --2- 172.21.15.116:0/1594119989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d000ffab0 0x7f5d000ffed0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc009910 tx=0x7f5cfc009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.165+0000 7f5d07f3d700 1 -- 172.21.15.116:0/1594119989 shutdown_connections 2024-03-04T16:05:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.165+0000 7f5d07f3d700 1 --2- 172.21.15.116:0/1594119989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00100cb0 0x7f5d00101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.165+0000 7f5d07f3d700 1 --2- 172.21.15.116:0/1594119989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d000ffab0 0x7f5d000ffed0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.165+0000 7f5d07f3d700 1 -- 172.21.15.116:0/1594119989 >> 172.21.15.116:0/1594119989 conn(0x7f5d000fb030 msgr2=0x7f5d000fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.166+0000 7f5d07f3d700 1 -- 172.21.15.116:0/1594119989 shutdown_connections 2024-03-04T16:05:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.166+0000 7f5d07f3d700 1 -- 172.21.15.116:0/1594119989 wait complete. 2024-03-04T16:05:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.167+0000 7f5d07f3d700 1 Processor -- start 2024-03-04T16:05:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.167+0000 7f5d07f3d700 1 -- start start 2024-03-04T16:05:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.168+0000 7f5d07f3d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d000ffab0 0x7f5d0010f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.168+0000 7f5d07f3d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00100cb0 0x7f5d0010f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.168+0000 7f5d07f3d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0010fe70 con 0x7f5d000ffab0 2024-03-04T16:05:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.168+0000 7f5d07f3d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0010ffb0 con 0x7f5d00100cb0 2024-03-04T16:05:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.168+0000 7f5d054d8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00100cb0 0x7f5d0010f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.168+0000 7f5d054d8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00100cb0 0x7f5d0010f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41440/0 (socket says 172.21.15.116:41440) 2024-03-04T16:05:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.168+0000 7f5d054d8700 1 -- 172.21.15.116:0/3471730666 learned_addr learned my addr 172.21.15.116:0/3471730666 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.168+0000 7f5d05cd9700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d000ffab0 0x7f5d0010f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.168+0000 7f5d05cd9700 1 -- 172.21.15.116:0/3471730666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00100cb0 msgr2=0x7f5d0010f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.168+0000 7f5d05cd9700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00100cb0 0x7f5d0010f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.169+0000 7f5d05cd9700 1 -- 172.21.15.116:0/3471730666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cfc0095f0 con 0x7f5d000ffab0 2024-03-04T16:05:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.169+0000 7f5d05cd9700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d000ffab0 0x7f5d0010f3f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc000c00 tx=0x7f5cfc004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.169+0000 7f5cf6ffd700 1 -- 172.21.15.116:0/3471730666 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cfc01d070 con 0x7f5d000ffab0 2024-03-04T16:05:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.169+0000 7f5d07f3d700 1 -- 172.21.15.116:0/3471730666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d001aa840 con 0x7f5d000ffab0 2024-03-04T16:05:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.169+0000 7f5d07f3d700 1 -- 172.21.15.116:0/3471730666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d001aad80 con 0x7f5d000ffab0 2024-03-04T16:05:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.169+0000 7f5cf6ffd700 1 -- 172.21.15.116:0/3471730666 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5cfc022470 con 0x7f5d000ffab0 2024-03-04T16:05:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.170+0000 7f5cf6ffd700 1 -- 172.21.15.116:0/3471730666 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cfc01d070 con 0x7f5d000ffab0 2024-03-04T16:05:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.171+0000 7f5d07f3d700 1 -- 172.21.15.116:0/3471730666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d0004ea90 con 0x7f5d000ffab0 2024-03-04T16:05:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.171+0000 7f5cf6ffd700 1 -- 172.21.15.116:0/3471730666 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 34) v1 ==== 91235+0+0 (secure 0 0 0) 0x7f5cfc00f460 con 0x7f5d000ffab0 2024-03-04T16:05:54.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.172+0000 7f5cf6ffd700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5cec06d670 0x7f5cec06fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:54.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.172+0000 7f5cf6ffd700 1 -- 172.21.15.116:0/3471730666 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5cfc064250 con 0x7f5d000ffab0 2024-03-04T16:05:54.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.172+0000 7f5d054d8700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5cec06d670 0x7f5cec06fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:54.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.174+0000 7f5d054d8700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5cec06d670 0x7f5cec06fb30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0005950 tx=0x7f5cf000b500 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:54.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.179+0000 7f5cf6ffd700 1 -- 172.21.15.116:0/3471730666 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5cfc058770 con 0x7f5d000ffab0 2024-03-04T16:05:54.319 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:54 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:54.319 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:54 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:54.319 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:54 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:54.319 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:54 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:05:54.319 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:54 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/trash_purge_schedule"}]: dispatch 2024-03-04T16:05:54.319 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:54 smithi116 ceph-mon[94164]: mgrmap e34: smithi116.jwdhop(active, since 1.01099s), standbys: smithi152.aawieo 2024-03-04T16:05:54.331 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:54 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:54.331 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:54 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:54.331 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:54 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:54.331 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:54 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:05:54.331 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:54 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.jwdhop/trash_purge_schedule"}]: dispatch 2024-03-04T16:05:54.331 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:54 smithi152 ceph-mon[36876]: mgrmap e34: smithi116.jwdhop(active, since 1.01099s), standbys: smithi152.aawieo 2024-03-04T16:05:54.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.416+0000 7f5d07f3d700 1 -- 172.21.15.116:0/3471730666 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d00105600 con 0x7f5cec06d670 2024-03-04T16:05:54.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.417+0000 7f5cf6ffd700 1 -- 172.21.15.116:0/3471730666 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f5d00105600 con 0x7f5cec06d670 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.419+0000 7f5d07f3d700 1 -- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5cec06d670 msgr2=0x7f5cec06fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.419+0000 7f5d07f3d700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5cec06d670 0x7f5cec06fb30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0005950 tx=0x7f5cf000b500 comp rx=0 tx=0).stop 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.419+0000 7f5d07f3d700 1 -- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d000ffab0 msgr2=0x7f5d0010f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.419+0000 7f5d07f3d700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d000ffab0 0x7f5d0010f3f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc000c00 tx=0x7f5cfc004fa0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.420+0000 7f5d07f3d700 1 -- 172.21.15.116:0/3471730666 shutdown_connections 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.420+0000 7f5d07f3d700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5cec06d670 0x7f5cec06fb30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.420+0000 7f5d07f3d700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d000ffab0 0x7f5d0010f3f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.420+0000 7f5d07f3d700 1 --2- 172.21.15.116:0/3471730666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00100cb0 0x7f5d0010f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.420+0000 7f5d07f3d700 1 -- 172.21.15.116:0/3471730666 >> 172.21.15.116:0/3471730666 conn(0x7f5d000fb030 msgr2=0x7f5d00103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.420+0000 7f5d07f3d700 1 -- 172.21.15.116:0/3471730666 shutdown_connections 2024-03-04T16:05:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.420+0000 7f5d07f3d700 1 -- 172.21.15.116:0/3471730666 wait complete. 2024-03-04T16:05:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.531+0000 7f0338bd4700 1 -- 172.21.15.116:0/84336614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03340ffab0 msgr2=0x7f03340ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.531+0000 7f0338bd4700 1 --2- 172.21.15.116:0/84336614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03340ffab0 0x7f03340ffed0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0324009910 tx=0x7f0324009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.531+0000 7f0338bd4700 1 -- 172.21.15.116:0/84336614 shutdown_connections 2024-03-04T16:05:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.531+0000 7f0338bd4700 1 --2- 172.21.15.116:0/84336614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0334100cb0 0x7f0334101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.531+0000 7f0338bd4700 1 --2- 172.21.15.116:0/84336614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03340ffab0 0x7f03340ffed0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.532+0000 7f0338bd4700 1 -- 172.21.15.116:0/84336614 >> 172.21.15.116:0/84336614 conn(0x7f03340fb030 msgr2=0x7f03340fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.532+0000 7f0338bd4700 1 -- 172.21.15.116:0/84336614 shutdown_connections 2024-03-04T16:05:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.532+0000 7f0338bd4700 1 -- 172.21.15.116:0/84336614 wait complete. 2024-03-04T16:05:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.532+0000 7f0338bd4700 1 Processor -- start 2024-03-04T16:05:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.532+0000 7f0338bd4700 1 -- start start 2024-03-04T16:05:54.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.532+0000 7f0338bd4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03340ffab0 0x7f0334196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:54.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.532+0000 7f0338bd4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0334100cb0 0x7f0334196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:54.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.532+0000 7f0338bd4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0334196e80 con 0x7f03340ffab0 2024-03-04T16:05:54.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.532+0000 7f0338bd4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0334196fc0 con 0x7f0334100cb0 2024-03-04T16:05:54.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f033259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03340ffab0 0x7f0334196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:54.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f033259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03340ffab0 0x7f0334196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33210/0 (socket says 172.21.15.116:33210) 2024-03-04T16:05:54.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f033259c700 1 -- 172.21.15.116:0/1848288115 learned_addr learned my addr 172.21.15.116:0/1848288115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:54.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f0331d9b700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0334100cb0 0x7f0334196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:54.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f033259c700 1 -- 172.21.15.116:0/1848288115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0334100cb0 msgr2=0x7f0334196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f033259c700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0334100cb0 0x7f0334196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f033259c700 1 -- 172.21.15.116:0/1848288115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03240095f0 con 0x7f03340ffab0 2024-03-04T16:05:54.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f033259c700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03340ffab0 0x7f0334196320 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f032400bda0 tx=0x7f032400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:54.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f03237fe700 1 -- 172.21.15.116:0/1848288115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f032401d070 con 0x7f03340ffab0 2024-03-04T16:05:54.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f03237fe700 1 -- 172.21.15.116:0/1848288115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f032400f460 con 0x7f03340ffab0 2024-03-04T16:05:54.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f0338bd4700 1 -- 172.21.15.116:0/1848288115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033419ba10 con 0x7f03340ffab0 2024-03-04T16:05:54.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f03237fe700 1 -- 172.21.15.116:0/1848288115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0324021640 con 0x7f03340ffab0 2024-03-04T16:05:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.533+0000 7f0338bd4700 1 -- 172.21.15.116:0/1848288115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033419bf00 con 0x7f03340ffab0 2024-03-04T16:05:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.534+0000 7f0338bd4700 1 -- 172.21.15.116:0/1848288115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0334108610 con 0x7f03340ffab0 2024-03-04T16:05:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.535+0000 7f03237fe700 1 -- 172.21.15.116:0/1848288115 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 34) v1 ==== 91235+0+0 (secure 0 0 0) 0x7f032400f5d0 con 0x7f03340ffab0 2024-03-04T16:05:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.535+0000 7f03237fe700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f031c06d5c0 0x7f031c06fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.535+0000 7f03237fe700 1 -- 172.21.15.116:0/1848288115 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f032408e220 con 0x7f03340ffab0 2024-03-04T16:05:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.535+0000 7f0331d9b700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f031c06d5c0 0x7f031c06fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:54.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.536+0000 7f0331d9b700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f031c06d5c0 0x7f031c06fa80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0328007c60 tx=0x7f03280073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:54.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.540+0000 7f03237fe700 1 -- 172.21.15.116:0/1848288115 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0324057d00 con 0x7f03340ffab0 2024-03-04T16:05:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.758+0000 7f0338bd4700 1 -- 172.21.15.116:0/1848288115 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f033419c650 con 0x7f031c06d5c0 2024-03-04T16:05:54.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.769+0000 7f03237fe700 1 -- 172.21.15.116:0/1848288115 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f033419c650 con 0x7f031c06d5c0 2024-03-04T16:05:54.770 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:05:54.770 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (8m) 35s ago 10m 25.4M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:05:54.770 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (10m) 35s ago 10m 7696k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:05:54.770 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (9m) 38s ago 9m 7692k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:05:54.770 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (10m) 35s ago 10m 6665k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:05:54.770 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (9m) 38s ago 9m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:05:54.770 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (8m) 35s ago 10m 88.1M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (4m) 35s ago 4m 18.0M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (4m) 35s ago 4m 14.1M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (4m) 38s ago 4m 15.8M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (4m) 38s ago 4m 209M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (84s) 35s ago 11m 493M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (42s) 38s ago 9m 66.7M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (12m) 35s ago 12m 52.4M 2048M 18.2.1 5be31c24972a 60e0bf118c4d 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (9m) 38s ago 9m 44.0M 2048M 18.2.1 5be31c24972a 52450d235516 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (10m) 35s ago 10m 15.5M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (9m) 38s ago 9m 16.6M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (8m) 35s ago 8m 205M 1216M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (8m) 35s ago 8m 207M 1216M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (7m) 35s ago 7m 170M 1216M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:05:54.771 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (7m) 38s ago 7m 229M 2240M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:05:54.772 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (6m) 38s ago 6m 217M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:05:54.772 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (6m) 38s ago 6m 217M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:05:54.772 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (48s) 35s ago 9m 39.9M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:05:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 -- 172.21.15.116:0/1848288115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f031c06d5c0 msgr2=0x7f031c06fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f031c06d5c0 0x7f031c06fa80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0328007c60 tx=0x7f03280073d0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 -- 172.21.15.116:0/1848288115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03340ffab0 msgr2=0x7f0334196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03340ffab0 0x7f0334196320 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f032400bda0 tx=0x7f032400bdd0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 -- 172.21.15.116:0/1848288115 shutdown_connections 2024-03-04T16:05:54.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f031c06d5c0 0x7f031c06fa80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03340ffab0 0x7f0334196320 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 --2- 172.21.15.116:0/1848288115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0334100cb0 0x7f0334196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 -- 172.21.15.116:0/1848288115 >> 172.21.15.116:0/1848288115 conn(0x7f03340fb030 msgr2=0x7f03340683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:54.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 -- 172.21.15.116:0/1848288115 shutdown_connections 2024-03-04T16:05:54.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.771+0000 7f0338bd4700 1 -- 172.21.15.116:0/1848288115 wait complete. 2024-03-04T16:05:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.878+0000 7fdf36cd5700 1 -- 172.21.15.116:0/4005395601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30101eb0 msgr2=0x7fdf30102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.878+0000 7fdf36cd5700 1 --2- 172.21.15.116:0/4005395601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30101eb0 0x7fdf30102310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009910 tx=0x7fdf1c009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.878+0000 7fdf36cd5700 1 -- 172.21.15.116:0/4005395601 shutdown_connections 2024-03-04T16:05:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.878+0000 7fdf36cd5700 1 --2- 172.21.15.116:0/4005395601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30101eb0 0x7fdf30102310 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.878+0000 7fdf36cd5700 1 --2- 172.21.15.116:0/4005395601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf300feed0 0x7fdf300ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.878+0000 7fdf36cd5700 1 -- 172.21.15.116:0/4005395601 >> 172.21.15.116:0/4005395601 conn(0x7fdf300fa810 msgr2=0x7fdf300fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.878+0000 7fdf36cd5700 1 -- 172.21.15.116:0/4005395601 shutdown_connections 2024-03-04T16:05:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.878+0000 7fdf36cd5700 1 -- 172.21.15.116:0/4005395601 wait complete. 2024-03-04T16:05:54.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.879+0000 7fdf36cd5700 1 Processor -- start 2024-03-04T16:05:54.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.879+0000 7fdf36cd5700 1 -- start start 2024-03-04T16:05:54.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.879+0000 7fdf36cd5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf300feed0 0x7fdf301963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:54.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.879+0000 7fdf36cd5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30101eb0 0x7fdf30196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:54.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.879+0000 7fdf36cd5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf30196f50 con 0x7fdf300feed0 2024-03-04T16:05:54.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.879+0000 7fdf36cd5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf30197090 con 0x7fdf30101eb0 2024-03-04T16:05:54.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.879+0000 7fdf2ffff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30101eb0 0x7fdf30196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:54.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.879+0000 7fdf2ffff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30101eb0 0x7fdf30196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:45608/0 (socket says 172.21.15.116:45608) 2024-03-04T16:05:54.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.879+0000 7fdf2ffff700 1 -- 172.21.15.116:0/2840984500 learned_addr learned my addr 172.21.15.116:0/2840984500 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:54.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.879+0000 7fdf34a71700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf300feed0 0x7fdf301963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:54.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.880+0000 7fdf2ffff700 1 -- 172.21.15.116:0/2840984500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf300feed0 msgr2=0x7fdf301963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:54.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.880+0000 7fdf2ffff700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf300feed0 0x7fdf301963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:54.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.880+0000 7fdf2ffff700 1 -- 172.21.15.116:0/2840984500 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c0095f0 con 0x7fdf30101eb0 2024-03-04T16:05:54.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.880+0000 7fdf2ffff700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30101eb0 0x7fdf30196930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00fa00 tx=0x7fdf1c00fa30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:54.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.880+0000 7fdf2dffb700 1 -- 172.21.15.116:0/2840984500 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf1c01d070 con 0x7fdf30101eb0 2024-03-04T16:05:54.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.880+0000 7fdf2dffb700 1 -- 172.21.15.116:0/2840984500 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdf1c00fe90 con 0x7fdf30101eb0 2024-03-04T16:05:54.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.880+0000 7fdf2dffb700 1 -- 172.21.15.116:0/2840984500 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf1c0226a0 con 0x7fdf30101eb0 2024-03-04T16:05:54.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.880+0000 7fdf36cd5700 1 -- 172.21.15.116:0/2840984500 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3019bae0 con 0x7fdf30101eb0 2024-03-04T16:05:54.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.880+0000 7fdf36cd5700 1 -- 172.21.15.116:0/2840984500 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3019bfd0 con 0x7fdf30101eb0 2024-03-04T16:05:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.882+0000 7fdf36cd5700 1 -- 172.21.15.116:0/2840984500 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf30109770 con 0x7fdf30101eb0 2024-03-04T16:05:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.882+0000 7fdf2dffb700 1 -- 172.21.15.116:0/2840984500 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 34) v1 ==== 91227+0+0 (secure 0 0 0) 0x7fdf1c003680 con 0x7fdf30101eb0 2024-03-04T16:05:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.882+0000 7fdf2dffb700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdf2006d5c0 0x7fdf2006fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.882+0000 7fdf2dffb700 1 -- 172.21.15.116:0/2840984500 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdf1c08e540 con 0x7fdf30101eb0 2024-03-04T16:05:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.882+0000 7fdf34a71700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdf2006d5c0 0x7fdf2006fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.883+0000 7fdf34a71700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdf2006d5c0 0x7fdf2006fa80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdf30068b70 tx=0x7fdf24009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:54.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:54.890+0000 7fdf2dffb700 1 -- 172.21.15.116:0/2840984500 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdf1c057f60 con 0x7fdf30101eb0 2024-03-04T16:05:55.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.056+0000 7fdf2dffb700 1 -- 172.21.15.116:0/2840984500 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mgrmap(e 35) v1 ==== 91523+0+0 (secure 0 0 0) 0x7fdf1c003930 con 0x7fdf30101eb0 2024-03-04T16:05:55.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: [04/Mar/2024:16:05:53] ENGINE Bus STARTING 2024-03-04T16:05:55.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: [04/Mar/2024:16:05:53] ENGINE Serving on https://172.21.15.116:7150 2024-03-04T16:05:55.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: [04/Mar/2024:16:05:53] ENGINE Serving on http://172.21.15.116:8765 2024-03-04T16:05:55.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: [04/Mar/2024:16:05:53] ENGINE Bus STARTED 2024-03-04T16:05:55.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:55.220 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: pgmap v3: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:05:55.220 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: Standby manager daemon smithi152.aawieo restarted 2024-03-04T16:05:55.220 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: Standby manager daemon smithi152.aawieo started 2024-03-04T16:05:55.220 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: from='mgr.? 172.21.15.152:0/271913108' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/crt"}]: dispatch 2024-03-04T16:05:55.220 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: from='mgr.? 172.21.15.152:0/271913108' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:05:55.220 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: from='mgr.? 172.21.15.152:0/271913108' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/key"}]: dispatch 2024-03-04T16:05:55.220 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:55 smithi116 ceph-mon[94164]: from='mgr.? 172.21.15.152:0/271913108' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:05:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.241+0000 7fdf36cd5700 1 -- 172.21.15.116:0/2840984500 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdf3019c640 con 0x7fdf30101eb0 2024-03-04T16:05:55.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.243+0000 7fdf2dffb700 1 -- 172.21.15.116:0/2840984500 <== mon.1 v2:172.21.15.152:3300/0 8 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+779 (secure 0 0 0) 0x7fdf1c003e60 con 0x7fdf30101eb0 2024-03-04T16:05:55.244 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:05:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:05:55.244 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 11, 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:05:55.245 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:05:55.246 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:05:55.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.247+0000 7fdf36cd5700 1 -- 172.21.15.116:0/2840984500 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdf2006d5c0 msgr2=0x7fdf2006fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:55.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.247+0000 7fdf36cd5700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdf2006d5c0 0x7fdf2006fa80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdf30068b70 tx=0x7fdf24009260 comp rx=0 tx=0).stop 2024-03-04T16:05:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.247+0000 7fdf36cd5700 1 -- 172.21.15.116:0/2840984500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30101eb0 msgr2=0x7fdf30196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.247+0000 7fdf36cd5700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30101eb0 0x7fdf30196930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00fa00 tx=0x7fdf1c00fa30 comp rx=0 tx=0).stop 2024-03-04T16:05:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.250+0000 7fdf36cd5700 1 -- 172.21.15.116:0/2840984500 shutdown_connections 2024-03-04T16:05:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.250+0000 7fdf36cd5700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdf2006d5c0 0x7fdf2006fa80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.250+0000 7fdf36cd5700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf300feed0 0x7fdf301963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.250+0000 7fdf36cd5700 1 --2- 172.21.15.116:0/2840984500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30101eb0 0x7fdf30196930 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.250+0000 7fdf36cd5700 1 -- 172.21.15.116:0/2840984500 >> 172.21.15.116:0/2840984500 conn(0x7fdf300fa810 msgr2=0x7fdf301050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.254+0000 7fdf36cd5700 1 -- 172.21.15.116:0/2840984500 shutdown_connections 2024-03-04T16:05:55.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.258+0000 7fdf36cd5700 1 -- 172.21.15.116:0/2840984500 wait complete. 2024-03-04T16:05:55.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: [04/Mar/2024:16:05:53] ENGINE Bus STARTING 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: [04/Mar/2024:16:05:53] ENGINE Serving on https://172.21.15.116:7150 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: [04/Mar/2024:16:05:53] ENGINE Serving on http://172.21.15.116:8765 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: [04/Mar/2024:16:05:53] ENGINE Bus STARTED 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: pgmap v3: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: Standby manager daemon smithi152.aawieo restarted 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: Standby manager daemon smithi152.aawieo started 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/271913108' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/crt"}]: dispatch 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/271913108' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/271913108' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.aawieo/key"}]: dispatch 2024-03-04T16:05:55.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:55 smithi152 ceph-mon[36876]: from='mgr.? 172.21.15.152:0/271913108' entity='mgr.smithi152.aawieo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:05:55.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.387+0000 7f69b83de700 1 -- 172.21.15.116:0/322657262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101640 msgr2=0x7f69b0103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:55.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.387+0000 7f69b83de700 1 --2- 172.21.15.116:0/322657262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101640 0x7f69b0103a30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009910 tx=0x7f69ac009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:55.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 -- 172.21.15.116:0/322657262 shutdown_connections 2024-03-04T16:05:55.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 --2- 172.21.15.116:0/322657262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69b0103f70 0x7f69b0106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 --2- 172.21.15.116:0/322657262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101640 0x7f69b0103a30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 -- 172.21.15.116:0/322657262 >> 172.21.15.116:0/322657262 conn(0x7f69b00fb030 msgr2=0x7f69b00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:55.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 -- 172.21.15.116:0/322657262 shutdown_connections 2024-03-04T16:05:55.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 -- 172.21.15.116:0/322657262 wait complete. 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 Processor -- start 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 -- start start 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101640 0x7f69b0100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69b0103f70 0x7f69b00ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b00ffb30 con 0x7f69b0101640 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b83de700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b00ffc70 con 0x7f69b0103f70 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b617a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101640 0x7f69b0100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b617a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101640 0x7f69b0100fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39546/0 (socket says 172.21.15.116:39546) 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.388+0000 7f69b617a700 1 -- 172.21.15.116:0/1954448636 learned_addr learned my addr 172.21.15.116:0/1954448636 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.389+0000 7f69b5979700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69b0103f70 0x7f69b00ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.389+0000 7f69b617a700 1 -- 172.21.15.116:0/1954448636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69b0103f70 msgr2=0x7f69b00ff5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.389+0000 7f69b617a700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69b0103f70 0x7f69b00ff5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.389+0000 7f69b617a700 1 -- 172.21.15.116:0/1954448636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac0095f0 con 0x7f69b0101640 2024-03-04T16:05:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.389+0000 7f69b617a700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101640 0x7f69b0100fa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0092e0 tx=0x7f69ac00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:55.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.389+0000 7f69a77fe700 1 -- 172.21.15.116:0/1954448636 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69ac01d070 con 0x7f69b0101640 2024-03-04T16:05:55.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.389+0000 7f69a77fe700 1 -- 172.21.15.116:0/1954448636 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f69ac005dc0 con 0x7f69b0101640 2024-03-04T16:05:55.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.389+0000 7f69a77fe700 1 -- 172.21.15.116:0/1954448636 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69ac017630 con 0x7f69b0101640 2024-03-04T16:05:55.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.389+0000 7f69b83de700 1 -- 172.21.15.116:0/1954448636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b00ffef0 con 0x7f69b0101640 2024-03-04T16:05:55.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.389+0000 7f69b83de700 1 -- 172.21.15.116:0/1954448636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b0100320 con 0x7f69b0101640 2024-03-04T16:05:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.390+0000 7f69b83de700 1 -- 172.21.15.116:0/1954448636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b0190520 con 0x7f69b0101640 2024-03-04T16:05:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.390+0000 7f69a77fe700 1 -- 172.21.15.116:0/1954448636 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 35) v1 ==== 91531+0+0 (secure 0 0 0) 0x7f69ac003680 con 0x7f69b0101640 2024-03-04T16:05:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.391+0000 7f69a77fe700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f699c06d940 0x7f699c06fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.391+0000 7f69a77fe700 1 -- 172.21.15.116:0/1954448636 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f69ac08ea00 con 0x7f69b0101640 2024-03-04T16:05:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.391+0000 7f69b5979700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f699c06d940 0x7f699c06fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.391+0000 7f69b5979700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f699c06d940 0x7f699c06fe00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f69a0009c60 tx=0x7f69a0009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.395+0000 7f69a77fe700 1 -- 172.21.15.116:0/1954448636 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f69ac05bc70 con 0x7f69b0101640 2024-03-04T16:05:55.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.648+0000 7f69b83de700 1 -- 172.21.15.116:0/1954448636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f69b002cf00 con 0x7f69b0101640 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.649+0000 7f69a77fe700 1 -- 172.21.15.116:0/1954448636 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1852 (secure 0 0 0) 0x7f69ac05b800 con 0x7f69b0101640 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:05:55.650 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:05:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:05:55.652 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:05:55.653 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:55.653 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:55.653 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:05:55.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 -- 172.21.15.116:0/1954448636 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f699c06d940 msgr2=0x7f699c06fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f699c06d940 0x7f699c06fe00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f69a0009c60 tx=0x7f69a0009260 comp rx=0 tx=0).stop 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 -- 172.21.15.116:0/1954448636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101640 msgr2=0x7f69b0100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101640 0x7f69b0100fa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0092e0 tx=0x7f69ac00f690 comp rx=0 tx=0).stop 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 -- 172.21.15.116:0/1954448636 shutdown_connections 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f699c06d940 0x7f699c06fe00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101640 0x7f69b0100fa0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 --2- 172.21.15.116:0/1954448636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69b0103f70 0x7f69b00ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 -- 172.21.15.116:0/1954448636 >> 172.21.15.116:0/1954448636 conn(0x7f69b00fb030 msgr2=0x7f69b00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 -- 172.21.15.116:0/1954448636 shutdown_connections 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.651+0000 7f69b83de700 1 -- 172.21.15.116:0/1954448636 wait complete. 2024-03-04T16:05:55.654 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:05:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.758+0000 7fad7810b700 1 -- 172.21.15.116:0/311634872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad700fee80 msgr2=0x7fad700ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.758+0000 7fad7810b700 1 --2- 172.21.15.116:0/311634872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad700fee80 0x7fad700ff2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009910 tx=0x7fad6c009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.758+0000 7fad7810b700 1 -- 172.21.15.116:0/311634872 shutdown_connections 2024-03-04T16:05:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.758+0000 7fad7810b700 1 --2- 172.21.15.116:0/311634872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad70101e60 0x7fad701022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.758+0000 7fad7810b700 1 --2- 172.21.15.116:0/311634872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad700fee80 0x7fad700ff2a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.758+0000 7fad7810b700 1 -- 172.21.15.116:0/311634872 >> 172.21.15.116:0/311634872 conn(0x7fad700fa7c0 msgr2=0x7fad700fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.758+0000 7fad7810b700 1 -- 172.21.15.116:0/311634872 shutdown_connections 2024-03-04T16:05:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.758+0000 7fad7810b700 1 -- 172.21.15.116:0/311634872 wait complete. 2024-03-04T16:05:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad7810b700 1 Processor -- start 2024-03-04T16:05:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad7810b700 1 -- start start 2024-03-04T16:05:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad7810b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad700fee80 0x7fad701963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad7810b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad70101e60 0x7fad701968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad7810b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad70196f00 con 0x7fad70101e60 2024-03-04T16:05:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad7810b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad70197040 con 0x7fad700fee80 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad756a6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad70101e60 0x7fad701968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad756a6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad70101e60 0x7fad701968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39568/0 (socket says 172.21.15.116:39568) 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad756a6700 1 -- 172.21.15.116:0/2858592818 learned_addr learned my addr 172.21.15.116:0/2858592818 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad75ea7700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad700fee80 0x7fad701963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad756a6700 1 -- 172.21.15.116:0/2858592818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad700fee80 msgr2=0x7fad701963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad756a6700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad700fee80 0x7fad701963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad756a6700 1 -- 172.21.15.116:0/2858592818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c0095f0 con 0x7fad70101e60 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.759+0000 7fad756a6700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad70101e60 0x7fad701968e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fad6000e920 tx=0x7fad6000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.760+0000 7fad66ffd700 1 -- 172.21.15.116:0/2858592818 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad6000c9d0 con 0x7fad70101e60 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.760+0000 7fad7810b700 1 -- 172.21.15.116:0/2858592818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7019baf0 con 0x7fad70101e60 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.760+0000 7fad66ffd700 1 -- 172.21.15.116:0/2858592818 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fad60010430 con 0x7fad70101e60 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.760+0000 7fad66ffd700 1 -- 172.21.15.116:0/2858592818 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad60018650 con 0x7fad70101e60 2024-03-04T16:05:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.760+0000 7fad7810b700 1 -- 172.21.15.116:0/2858592818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7019c040 con 0x7fad70101e60 2024-03-04T16:05:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.761+0000 7fad7810b700 1 -- 172.21.15.116:0/2858592818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad70109720 con 0x7fad70101e60 2024-03-04T16:05:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.762+0000 7fad66ffd700 1 -- 172.21.15.116:0/2858592818 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 35) v1 ==== 91531+0+0 (secure 0 0 0) 0x7fad600105a0 con 0x7fad70101e60 2024-03-04T16:05:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.762+0000 7fad66ffd700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad5c06d940 0x7fad5c06fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.762+0000 7fad66ffd700 1 -- 172.21.15.116:0/2858592818 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fad60014070 con 0x7fad70101e60 2024-03-04T16:05:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.762+0000 7fad75ea7700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad5c06d940 0x7fad5c06fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.763+0000 7fad75ea7700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad5c06d940 0x7fad5c06fe00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00b6f0 tx=0x7fad6c005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:55.767+0000 7fad66ffd700 1 -- 172.21.15.116:0/2858592818 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fad60052d20 con 0x7fad70101e60 2024-03-04T16:05:56.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.007+0000 7fad7810b700 1 -- 172.21.15.116:0/2858592818 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fad701067b0 con 0x7fad5c06d940 2024-03-04T16:05:56.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.008+0000 7fad66ffd700 1 -- 172.21.15.116:0/2858592818 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7fad701067b0 con 0x7fad5c06d940 2024-03-04T16:05:56.009 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:05:56.009 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:05:56.009 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:05:56.009 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:05:56.009 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:05:56.009 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:05:56.009 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:05:56.009 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "2/23 daemons upgraded", 2024-03-04T16:05:56.010 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2024-03-04T16:05:56.010 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:05:56.010 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:05:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 -- 172.21.15.116:0/2858592818 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad5c06d940 msgr2=0x7fad5c06fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad5c06d940 0x7fad5c06fe00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00b6f0 tx=0x7fad6c005940 comp rx=0 tx=0).stop 2024-03-04T16:05:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 -- 172.21.15.116:0/2858592818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad70101e60 msgr2=0x7fad701968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad70101e60 0x7fad701968e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fad6000e920 tx=0x7fad6000ec30 comp rx=0 tx=0).stop 2024-03-04T16:05:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 -- 172.21.15.116:0/2858592818 shutdown_connections 2024-03-04T16:05:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad5c06d940 0x7fad5c06fe00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad700fee80 0x7fad701963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 --2- 172.21.15.116:0/2858592818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad70101e60 0x7fad701968e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:56.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 -- 172.21.15.116:0/2858592818 >> 172.21.15.116:0/2858592818 conn(0x7fad700fa7c0 msgr2=0x7fad70105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:56.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 -- 172.21.15.116:0/2858592818 shutdown_connections 2024-03-04T16:05:56.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.010+0000 7fad7810b700 1 -- 172.21.15.116:0/2858592818 wait complete. 2024-03-04T16:05:56.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.148+0000 7f3174098700 1 -- 172.21.15.116:0/2870992533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316c101da0 msgr2=0x7f316c104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:56.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.148+0000 7f3174098700 1 --2- 172.21.15.116:0/2870992533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316c101da0 0x7f316c104230 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f315c009910 tx=0x7f315c009c20 comp rx=0 tx=0).stop 2024-03-04T16:05:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.149+0000 7f3174098700 1 -- 172.21.15.116:0/2870992533 shutdown_connections 2024-03-04T16:05:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.149+0000 7f3174098700 1 --2- 172.21.15.116:0/2870992533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316c101da0 0x7f316c104230 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.149+0000 7f3174098700 1 --2- 172.21.15.116:0/2870992533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f316c0ff430 0x7f316c101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.149+0000 7f3174098700 1 -- 172.21.15.116:0/2870992533 >> 172.21.15.116:0/2870992533 conn(0x7f316c0fb030 msgr2=0x7f316c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.149+0000 7f3174098700 1 -- 172.21.15.116:0/2870992533 shutdown_connections 2024-03-04T16:05:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.150+0000 7f3174098700 1 -- 172.21.15.116:0/2870992533 wait complete. 2024-03-04T16:05:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.150+0000 7f3174098700 1 Processor -- start 2024-03-04T16:05:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.150+0000 7f3174098700 1 -- start start 2024-03-04T16:05:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.151+0000 7f3174098700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f316c0ff430 0x7f316c19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.151+0000 7f3174098700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316c101da0 0x7f316c19ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.151+0000 7f3174098700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c19d470 con 0x7f316c101da0 2024-03-04T16:05:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.151+0000 7f3174098700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c19d5b0 con 0x7f316c0ff430 2024-03-04T16:05:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.151+0000 7f3171633700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316c101da0 0x7f316c19ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.151+0000 7f3171633700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316c101da0 0x7f316c19ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39594/0 (socket says 172.21.15.116:39594) 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.151+0000 7f3171633700 1 -- 172.21.15.116:0/726977019 learned_addr learned my addr 172.21.15.116:0/726977019 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.152+0000 7f3171633700 1 -- 172.21.15.116:0/726977019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f316c0ff430 msgr2=0x7f316c19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.152+0000 7f3171633700 1 --2- 172.21.15.116:0/726977019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f316c0ff430 0x7f316c19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.152+0000 7f3171633700 1 -- 172.21.15.116:0/726977019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c0095f0 con 0x7f316c101da0 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.152+0000 7f3171633700 1 --2- 172.21.15.116:0/726977019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316c101da0 0x7f316c19ce50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f315c000c00 tx=0x7f315c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.152+0000 7f3162ffd700 1 -- 172.21.15.116:0/726977019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f315c01d070 con 0x7f316c101da0 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.152+0000 7f3162ffd700 1 -- 172.21.15.116:0/726977019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f315c005640 con 0x7f316c101da0 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.152+0000 7f3162ffd700 1 -- 172.21.15.116:0/726977019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f315c0226a0 con 0x7f316c101da0 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.153+0000 7f3174098700 1 -- 172.21.15.116:0/726977019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316c1a2000 con 0x7f316c101da0 2024-03-04T16:05:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.153+0000 7f3174098700 1 -- 172.21.15.116:0/726977019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316c1a24f0 con 0x7f316c101da0 2024-03-04T16:05:56.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.154+0000 7f3160ff9700 1 -- 172.21.15.116:0/726977019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31540052f0 con 0x7f316c101da0 2024-03-04T16:05:56.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.156+0000 7f3162ffd700 1 -- 172.21.15.116:0/726977019 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 35) v1 ==== 91531+0+0 (secure 0 0 0) 0x7f315c003910 con 0x7f316c101da0 2024-03-04T16:05:56.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.156+0000 7f3162ffd700 1 --2- 172.21.15.116:0/726977019 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f315806d9d0 0x7f315806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:05:56.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.157+0000 7f3162ffd700 1 -- 172.21.15.116:0/726977019 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f315c08ed40 con 0x7f316c101da0 2024-03-04T16:05:56.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.157+0000 7f3171e34700 1 --2- 172.21.15.116:0/726977019 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f315806d9d0 0x7f315806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:05:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.158+0000 7f3171e34700 1 --2- 172.21.15.116:0/726977019 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f315806d9d0 0x7f315806fe90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3168005950 tx=0x7f31680058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:05:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.163+0000 7f3162ffd700 1 -- 172.21.15.116:0/726977019 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f315c05c140 con 0x7f316c101da0 2024-03-04T16:05:56.268 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:56 smithi116 ceph-mon[94164]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:56.268 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:56 smithi116 ceph-mon[94164]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:56.268 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:56 smithi116 ceph-mon[94164]: pgmap v4: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:05:56.268 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:56 smithi116 ceph-mon[94164]: mgrmap e35: smithi116.jwdhop(active, since 2s), standbys: smithi152.aawieo 2024-03-04T16:05:56.268 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:56 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2840984500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:56.268 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:56 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1954448636' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:05:56.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:56 smithi152 ceph-mon[36876]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:56.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:56 smithi152 ceph-mon[36876]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:56.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:56 smithi152 ceph-mon[36876]: pgmap v4: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:05:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:56 smithi152 ceph-mon[36876]: mgrmap e35: smithi116.jwdhop(active, since 2s), standbys: smithi152.aawieo 2024-03-04T16:05:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:56 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/2840984500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:56 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/1954448636' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:05:56.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.470+0000 7f3160ff9700 1 -- 172.21.15.116:0/726977019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3154005160 con 0x7f316c101da0 2024-03-04T16:05:56.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.471+0000 7f3162ffd700 1 -- 172.21.15.116:0/726977019 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f315c05bcd0 con 0x7f316c101da0 2024-03-04T16:05:56.472 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:05:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 -- 172.21.15.116:0/726977019 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f315806d9d0 msgr2=0x7f315806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 --2- 172.21.15.116:0/726977019 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f315806d9d0 0x7f315806fe90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3168005950 tx=0x7f31680058e0 comp rx=0 tx=0).stop 2024-03-04T16:05:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 -- 172.21.15.116:0/726977019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316c101da0 msgr2=0x7f316c19ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:05:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 --2- 172.21.15.116:0/726977019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316c101da0 0x7f316c19ce50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f315c000c00 tx=0x7f315c00f740 comp rx=0 tx=0).stop 2024-03-04T16:05:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 -- 172.21.15.116:0/726977019 shutdown_connections 2024-03-04T16:05:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 --2- 172.21.15.116:0/726977019 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f315806d9d0 0x7f315806fe90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 --2- 172.21.15.116:0/726977019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f316c0ff430 0x7f316c19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 --2- 172.21.15.116:0/726977019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316c101da0 0x7f316c19ce50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:05:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 -- 172.21.15.116:0/726977019 >> 172.21.15.116:0/726977019 conn(0x7f316c0fb030 msgr2=0x7f316c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:05:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 -- 172.21.15.116:0/726977019 shutdown_connections 2024-03-04T16:05:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:05:56.473+0000 7f3160ff9700 1 -- 172.21.15.116:0/726977019 wait complete. 2024-03-04T16:05:57.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:57 smithi116 ceph-mon[94164]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:57.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:57 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/726977019' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:05:57.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:57 smithi152 ceph-mon[36876]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:57.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:57 smithi152 ceph-mon[36876]: from='client.? 172.21.15.116:0/726977019' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:05:58.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:58 smithi152 ceph-mon[36876]: pgmap v5: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:05:58.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:58 smithi152 ceph-mon[36876]: mgrmap e36: smithi116.jwdhop(active, since 4s), standbys: smithi152.aawieo 2024-03-04T16:05:58.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:58 smithi116 ceph-mon[94164]: pgmap v5: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:05:58.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:58 smithi116 ceph-mon[94164]: mgrmap e36: smithi116.jwdhop(active, since 4s), standbys: smithi152.aawieo 2024-03-04T16:05:59.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:59 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:59.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:59 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:05:59.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:05:59 smithi116 ceph-mon[94164]: pgmap v6: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 66 KiB/s rd, 1.1 MiB/s wr, 286 op/s 2024-03-04T16:06:00.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:59 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:00.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:59 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:00.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:05:59 smithi152 ceph-mon[36876]: pgmap v6: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 66 KiB/s rd, 1.1 MiB/s wr, 286 op/s 2024-03-04T16:06:01.084 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:00 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:01.084 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:00 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:01.084 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:00 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:01.084 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:00 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:01.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:00 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:01.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:00 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:01.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:00 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:01.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:00 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:02 smithi116 ceph-mon[94164]: pgmap v7: 65 pgs: 65 active+clean; 253 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 47 KiB/s rd, 822 KiB/s wr, 383 op/s 2024-03-04T16:06:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:02 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:02 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:02.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:02 smithi152 ceph-mon[36876]: pgmap v7: 65 pgs: 65 active+clean; 253 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 47 KiB/s rd, 822 KiB/s wr, 383 op/s 2024-03-04T16:06:02.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:02 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:02.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:02 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:04 smithi116 ceph-mon[94164]: pgmap v8: 65 pgs: 65 active+clean; 253 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 37 KiB/s rd, 639 KiB/s wr, 298 op/s 2024-03-04T16:06:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:04 smithi152 ceph-mon[36876]: pgmap v8: 65 pgs: 65 active+clean; 253 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 37 KiB/s rd, 639 KiB/s wr, 298 op/s 2024-03-04T16:06:06.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:06 smithi116 ceph-mon[94164]: pgmap v9: 65 pgs: 65 active+clean; 242 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 43 KiB/s rd, 1.0 MiB/s wr, 392 op/s 2024-03-04T16:06:06.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:06 smithi152 ceph-mon[36876]: pgmap v9: 65 pgs: 65 active+clean; 242 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 43 KiB/s rd, 1.0 MiB/s wr, 392 op/s 2024-03-04T16:06:08.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:08 smithi152 ceph-mon[36876]: pgmap v10: 65 pgs: 65 active+clean; 242 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 39 KiB/s rd, 977 KiB/s wr, 359 op/s 2024-03-04T16:06:08.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:08 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:08.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:08 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:08.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:08 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:08.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:08 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:06:08.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:08 smithi152 ceph-mon[36876]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T16:06:08.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:08 smithi152 ceph-mon[36876]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T16:06:08.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:08 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:08 smithi116 ceph-mon[94164]: pgmap v10: 65 pgs: 65 active+clean; 242 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 39 KiB/s rd, 977 KiB/s wr, 359 op/s 2024-03-04T16:06:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:06:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:08 smithi116 ceph-mon[94164]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T16:06:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:08 smithi116 ceph-mon[94164]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T16:06:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:09.397 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:06:09.397 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:06:09.397 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:06:09.397 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:06:09.397 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:06:09.397 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:06:09.397 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:09.398 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:09.398 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:09.398 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:09.398 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:09.398 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:09.398 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:09.398 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:09 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:06:09.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:06:09.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:06:09.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:06:09.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:06:09.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: Updating smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:06:09.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: Updating smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:06:09.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:09.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:09.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:09.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:09.729 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:09.729 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:09.729 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:09.729 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:06:09.837 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-04T16:06:09.837 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-04T16:06:10.103 DEBUG:teuthology.parallel:result is None 2024-03-04T16:06:10.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:10 smithi116 ceph-mon[94164]: pgmap v11: 65 pgs: 65 active+clean; 238 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 52 KiB/s rd, 1.5 MiB/s wr, 477 op/s 2024-03-04T16:06:10.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:10.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:06:10.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:06:10.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:10 smithi152 ceph-mon[36876]: pgmap v11: 65 pgs: 65 active+clean; 238 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 52 KiB/s rd, 1.5 MiB/s wr, 477 op/s 2024-03-04T16:06:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:10 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:10 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:06:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:10 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:06:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:10 smithi152 ceph-mon[36876]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:11.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:11 smithi152 ceph-mon[36876]: Upgrade: Updating mon.smithi152 2024-03-04T16:06:11.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:11 smithi152 ceph-mon[36876]: Deploying daemon mon.smithi152 on smithi152 2024-03-04T16:06:11.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:11 smithi116 ceph-mon[94164]: Upgrade: Updating mon.smithi152 2024-03-04T16:06:11.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:11 smithi116 ceph-mon[94164]: Deploying daemon mon.smithi152 on smithi152 2024-03-04T16:06:12.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:12 smithi152 ceph-mon[36876]: pgmap v12: 65 pgs: 65 active+clean; 237 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 25 KiB/s rd, 1.0 MiB/s wr, 379 op/s 2024-03-04T16:06:12.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:12 smithi152 systemd[1]: Stopping Ceph mon.smithi152 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:06:12.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:12 smithi116 ceph-mon[94164]: pgmap v12: 65 pgs: 65 active+clean; 237 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 25 KiB/s rd, 1.0 MiB/s wr, 379 op/s 2024-03-04T16:06:13.008 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152[36865]: 2024-03-04T16:06:12.690+0000 7f07f014d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi152 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:06:13.009 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152[36865]: 2024-03-04T16:06:12.690+0000 7f07f014d700 -1 mon.smithi152@1(peon) e2 *** Got Signal Terminated *** 2024-03-04T16:06:13.009 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:12 smithi152 podman[82688]: 2024-03-04 16:06:12.73171433 +0000 UTC m=+0.170632661 container died 52450d235516a1021efcfe190da941dc12f95d275bbccfd48bb4ceeb8582f145 (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux ) 2024-03-04T16:06:13.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:13 smithi152 podman[82688]: 2024-03-04 16:06:13.241947533 +0000 UTC m=+0.680865864 container cleanup 52450d235516a1021efcfe190da941dc12f95d275bbccfd48bb4ceeb8582f145 (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) 2024-03-04T16:06:13.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:13 smithi152 bash[82688]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152 2024-03-04T16:06:13.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:13 smithi152 podman[82702]: 2024-03-04 16:06:13.485286857 +0000 UTC m=+0.755486342 container remove 52450d235516a1021efcfe190da941dc12f95d275bbccfd48bb4ceeb8582f145 (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux ) 2024-03-04T16:06:13.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:13 smithi152 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi152.service: Deactivated successfully. 2024-03-04T16:06:13.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:13 smithi152 systemd[1]: Stopped Ceph mon.smithi152 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:06:13.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:13 smithi152 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi152.service: Consumed 6.767s CPU time. 2024-03-04T16:06:14.186 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:13 smithi152 systemd[1]: Starting Ceph mon.smithi152 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:06:14.186 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 podman[82826]: 2024-03-04 16:06:14.04300503 +0000 UTC m=+0.023649479 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:06:14.471 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 podman[82826]: 2024-03-04 16:06:14.186646039 +0000 UTC m=+0.167290494 container create 9129fd088663105002dc11995866534db45077524cf445bac5cc1f76c1f3ae8f (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:06:14.471 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 podman[82826]: 2024-03-04 16:06:14.423368077 +0000 UTC m=+0.404012536 container init 9129fd088663105002dc11995866534db45077524cf445bac5cc1f76c1f3ae8f (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-04T16:06:14.471 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 podman[82826]: 2024-03-04 16:06:14.433694797 +0000 UTC m=+0.414339249 container start 9129fd088663105002dc11995866534db45077524cf445bac5cc1f76c1f3ae8f (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-03-04T16:06:14.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T16:06:14.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 2 2024-03-04T16:06:14.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: pidfile_write: ignore empty --pid-file 2024-03-04T16:06:14.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: load: jerasure load: lrc 2024-03-04T16:06:14.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: RocksDB version: 7.9.2 2024-03-04T16:06:14.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Git sha 0 2024-03-04T16:06:14.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: DB SUMMARY 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: DB Session ID: YDEBG9BD29TWHU29IASF 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: CURRENT file: CURRENT 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1403 Bytes 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi152/store.db dir, Total Num: 1, files: 000027.sst 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi152/store.db: 000025.log size: 4932508 ; 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.error_if_exists: 0 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.create_if_missing: 0 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.paranoid_checks: 1 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T16:06:14.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.env: 0x55f1774cab20 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.info_log: 0x55f177edd780 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.statistics: (nil) 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.use_fsync: 0 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_log_file_size: 0 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T16:06:14.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T16:06:14.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.allow_fallocate: 1 2024-03-04T16:06:14.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T16:06:14.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T16:06:14.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.use_direct_reads: 0 2024-03-04T16:06:14.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T16:06:14.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T16:06:14.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.db_log_dir: 2024-03-04T16:06:14.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.wal_dir: 2024-03-04T16:06:14.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T16:06:14.863 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T16:06:14.863 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T16:06:14.863 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T16:06:14.863 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T16:06:14.863 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T16:06:14.863 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.write_buffer_manager: 0x55f177eb1400 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.unordered_write: 0 2024-03-04T16:06:14.864 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.row_cache: None 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.wal_filter: None 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.two_write_queues: 0 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.wal_compression: 0 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.atomic_flush: 0 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T16:06:14.865 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.log_readahead_size: 0 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_background_jobs: 2 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_background_compactions: -1 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_subcompactions: 1 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T16:06:14.866 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_open_files: -1 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_background_flushes: -1 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Compression algorithms supported: 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T16:06:14.867 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: kZSTD supported: 0 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: kXpressCompression supported: 0 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: kZlibCompression supported: 1 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: kSnappyCompression supported: 1 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: kLZ4Compression supported: 1 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: kBZip2Compression supported: 0 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi152/store.db/MANIFEST-000010 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.merge_operator: 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_filter: None 2024-03-04T16:06:14.868 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f177edd8a0) 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: cache_index_and_filter_blocks: 1 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: pin_top_level_index_and_filter: 1 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: index_type: 0 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: data_block_index_type: 0 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: index_shortening: 1 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: checksum: 4 2024-03-04T16:06:14.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: no_block_cache: 0 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_cache: 0x55f177ee3090 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_cache_name: BinnedLRUCache 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_cache_options: 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: capacity : 536870912 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: num_shard_bits : 4 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: strict_capacity_limit : 0 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: high_pri_pool_ratio: 0.000 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_cache_compressed: (nil) 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: persistent_cache: (nil) 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_size: 4096 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_size_deviation: 10 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_restart_interval: 16 2024-03-04T16:06:14.870 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: index_block_restart_interval: 1 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: metadata_block_size: 4096 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: partition_filters: 0 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: use_delta_encoding: 1 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: filter_policy: bloomfilter 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: whole_key_filtering: 1 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: verify_compression: 0 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: read_amp_bytes_per_bit: 0 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: format_version: 5 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: enable_index_compression: 1 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_align: 0 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: max_auto_readahead_size: 262144 2024-03-04T16:06:14.871 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: prepopulate_block_cache: 0 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: initial_auto_readahead_size: 8192 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compression: NoCompression 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.num_levels: 7 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T16:06:14.872 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T16:06:14.873 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T16:06:14.874 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T16:06:14.875 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.table_properties_collectors: 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.inplace_update_support: 0 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T16:06:14.876 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.bloom_locality: 0 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.max_successive_merges: 0 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.ttl: 2592000 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.enable_blob_files: false 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.min_blob_size: 0 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T16:06:14.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi152/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12095, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 86a50f4e-246f-40fd-ae89-affd8bab5816 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568374478563, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568374490795, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3011089, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12101, "largest_seqno": 13584, "table_properties": {"data_size": 3004576, "index_size": 4022, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1477, "raw_key_size": 13320, "raw_average_key_size": 23, "raw_value_size": 2992015, "raw_average_value_size": 5176, "num_data_blocks": 184, "num_entries": 578, "num_filter_entries": 578, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709568374, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "86a50f4e-246f-40fd-ae89-affd8bab5816", "db_session_id": "YDEBG9BD29TWHU29IASF", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568374491033, "job": 1, "event": "recovery_finished"} 2024-03-04T16:06:14.878 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi152/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f177fd4000 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: DB pointer 0x55f177fc0000 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** DB Stats ** 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T16:06:14.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** Compaction Stats [default] ** 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: L0 1/0 2.87 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 609.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: L6 1/0 5.52 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Sum 2/0 8.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 609.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 609.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** Compaction Stats [default] ** 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T16:06:14.880 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 609.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Cumulative compaction: 0.00 GB write, 153.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Interval compaction: 0.00 GB write, 153.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T16:06:14.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Block cache BinnedLRUCache@0x55f177ee3090#2 capacity: 512.00 MB usage: 5.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.53 KB,0.000292063%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: starting mon.smithi152 rank 1 at public addrs [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] at bind addrs [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi152 fsid 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: mon.smithi152@-1(???) e2 preinit fsid 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: mon.smithi152@-1(???).mds e14 new map 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: mon.smithi152@-1(???).mds e14 print_map 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: e14 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: legacy client fscid: 1 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Filesystem 'cephfs' (1) 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: fs_name cephfs 2024-03-04T16:06:14.882 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: epoch 14 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: tableserver 0 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: root 0 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: session_timeout 60 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: session_autoclose 300 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: max_file_size 1099511627776 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: max_xattr_size 65536 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: required_client_features {} 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: last_failure 0 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: last_failure_osd_epoch 47 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: max_mds 1 2024-03-04T16:06:14.883 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: in 0 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: up {0=24247} 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: failed 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: damaged 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: stopped 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: data_pools [3] 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: metadata_pool 2 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: inline_data disabled 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: balancer 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: bal_rank_mask -1 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: standby_count_wanted 1 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: [mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-03-04T16:06:14.884 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Standby daemons: 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: [mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: [mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: [mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: mon.smithi152@-1(???).osd e50 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: mon.smithi152@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: mon.smithi152@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: mon.smithi152@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 ceph-mon[82845]: mon.smithi152@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 bash[82826]: 9129fd088663105002dc11995866534db45077524cf445bac5cc1f76c1f3ae8f 2024-03-04T16:06:14.885 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:14 smithi152 systemd[1]: Started Ceph mon.smithi152 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:06:15.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T16:06:15.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: mon.smithi116 calling monitor election 2024-03-04T16:06:15.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: mon.smithi152 calling monitor election 2024-03-04T16:06:15.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: mon.smithi116 is new leader, mons smithi116,smithi152 in quorum (ranks 0,1) 2024-03-04T16:06:15.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T16:06:15.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: monmap e3: 2 mons at {smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],smithi152=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:06:15.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 3 up:standby 2024-03-04T16:06:15.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: osdmap e50: 6 total, 6 up, 6 in 2024-03-04T16:06:15.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: mgrmap e36: smithi116.jwdhop(active, since 21s), standbys: smithi152.aawieo 2024-03-04T16:06:15.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: overall HEALTH_OK 2024-03-04T16:06:15.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:15.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:15.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:15.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: pgmap v14: 65 pgs: 65 active+clean; 228 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 38 KiB/s rd, 1.5 MiB/s wr, 424 op/s 2024-03-04T16:06:15.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:15.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:16.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: mon.smithi116 calling monitor election 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: mon.smithi152 calling monitor election 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: mon.smithi116 is new leader, mons smithi116,smithi152 in quorum (ranks 0,1) 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: monmap e3: 2 mons at {smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],smithi152=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 3 up:standby 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: osdmap e50: 6 total, 6 up, 6 in 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: mgrmap e36: smithi116.jwdhop(active, since 21s), standbys: smithi152.aawieo 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: overall HEALTH_OK 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:16.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: pgmap v14: 65 pgs: 65 active+clean; 228 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 38 KiB/s rd, 1.5 MiB/s wr, 424 op/s 2024-03-04T16:06:16.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:16.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:18.231 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:18 smithi152 ceph-mon[82845]: pgmap v15: 65 pgs: 65 active+clean; 228 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 26 KiB/s rd, 1009 KiB/s wr, 288 op/s 2024-03-04T16:06:18.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:18 smithi116 ceph-mon[94164]: pgmap v15: 65 pgs: 65 active+clean; 228 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 26 KiB/s rd, 1009 KiB/s wr, 288 op/s 2024-03-04T16:06:20.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:19 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:20.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:19 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:20.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:19 smithi116 ceph-mon[94164]: pgmap v16: 65 pgs: 65 active+clean; 227 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 37 KiB/s rd, 1.4 MiB/s wr, 393 op/s 2024-03-04T16:06:20.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:19 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:20.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:19 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:20.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:19 smithi152 ceph-mon[82845]: pgmap v16: 65 pgs: 65 active+clean; 227 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 37 KiB/s rd, 1.4 MiB/s wr, 393 op/s 2024-03-04T16:06:20.675 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-04T16:06:20.675 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-03-04T16:06:20.966 DEBUG:teuthology.parallel:result is None 2024-03-04T16:06:20.967 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-04T16:06:21.003 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-04T16:06:21.003 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-03-04T16:06:21.040 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-03-04T16:06:21.040 DEBUG:teuthology.parallel:result is None 2024-03-04T16:06:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:21 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:21 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:21.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:21 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:21 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:22.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:22 smithi152 ceph-mon[82845]: pgmap v17: 65 pgs: 65 active+clean; 222 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 895 KiB/s wr, 292 op/s 2024-03-04T16:06:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:22 smithi116 ceph-mon[94164]: pgmap v17: 65 pgs: 65 active+clean; 222 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 895 KiB/s wr, 292 op/s 2024-03-04T16:06:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:23.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:24.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:24 smithi152 ceph-mon[82845]: pgmap v18: 65 pgs: 65 active+clean; 222 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 895 KiB/s wr, 270 op/s 2024-03-04T16:06:24.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:24 smithi116 ceph-mon[94164]: pgmap v18: 65 pgs: 65 active+clean; 222 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 895 KiB/s wr, 270 op/s 2024-03-04T16:06:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:06:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: Reconfiguring mon.smithi116 (monmap changed)... 2024-03-04T16:06:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:06:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:06:25.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:25.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: Reconfiguring daemon mon.smithi116 on smithi116 2024-03-04T16:06:25.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:25 smithi116 ceph-mon[94164]: pgmap v19: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 34 KiB/s rd, 1.1 MiB/s wr, 363 op/s 2024-03-04T16:06:25.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:25.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:25.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:25.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:06:25.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:25.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: Reconfiguring mon.smithi116 (monmap changed)... 2024-03-04T16:06:25.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:06:25.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:06:25.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:25.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: Reconfiguring daemon mon.smithi116 on smithi116 2024-03-04T16:06:25.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:25 smithi152 ceph-mon[82845]: pgmap v19: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 34 KiB/s rd, 1.1 MiB/s wr, 363 op/s 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.583+0000 7f1a266c8700 1 -- 172.21.15.116:0/2006806576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a20101070 msgr2=0x7f1a20101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.583+0000 7f1a266c8700 1 --2- 172.21.15.116:0/2006806576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a20101070 0x7f1a20101490 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009910 tx=0x7f1a08009c20 comp rx=0 tx=0).stop 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.583+0000 7f1a266c8700 1 -- 172.21.15.116:0/2006806576 shutdown_connections 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.583+0000 7f1a266c8700 1 --2- 172.21.15.116:0/2006806576 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a200ffcd0 0x7f1a20100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.583+0000 7f1a266c8700 1 --2- 172.21.15.116:0/2006806576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a20101070 0x7f1a20101490 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.583+0000 7f1a266c8700 1 -- 172.21.15.116:0/2006806576 >> 172.21.15.116:0/2006806576 conn(0x7f1a200fa7c0 msgr2=0x7f1a200fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.583+0000 7f1a266c8700 1 -- 172.21.15.116:0/2006806576 shutdown_connections 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.583+0000 7f1a266c8700 1 -- 172.21.15.116:0/2006806576 wait complete. 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.584+0000 7f1a266c8700 1 Processor -- start 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.584+0000 7f1a266c8700 1 -- start start 2024-03-04T16:06:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.584+0000 7f1a266c8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a200ffcd0 0x7f1a201963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.584+0000 7f1a266c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a20101070 0x7f1a20196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.584+0000 7f1a266c8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a20196f40 con 0x7f1a20101070 2024-03-04T16:06:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.584+0000 7f1a266c8700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a20197080 con 0x7f1a200ffcd0 2024-03-04T16:06:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.584+0000 7f1a1f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a20101070 0x7f1a20196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.584+0000 7f1a1f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a20101070 0x7f1a20196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41080/0 (socket says 172.21.15.116:41080) 2024-03-04T16:06:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.584+0000 7f1a1f7fe700 1 -- 172.21.15.116:0/1949828189 learned_addr learned my addr 172.21.15.116:0/1949828189 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a1ffff700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a200ffcd0 0x7f1a201963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a1f7fe700 1 -- 172.21.15.116:0/1949828189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a200ffcd0 msgr2=0x7f1a201963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a1f7fe700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a200ffcd0 0x7f1a201963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a1f7fe700 1 -- 172.21.15.116:0/1949828189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a080095f0 con 0x7f1a20101070 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a1ffff700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a200ffcd0 0x7f1a201963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a1f7fe700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a20101070 0x7f1a20196920 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1a1000b4b0 tx=0x7f1a1000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a1d7fa700 1 -- 172.21.15.116:0/1949828189 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a10011800 con 0x7f1a20101070 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a266c8700 1 -- 172.21.15.116:0/1949828189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2019bb30 con 0x7f1a20101070 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a1d7fa700 1 -- 172.21.15.116:0/1949828189 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1a10011e10 con 0x7f1a20101070 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a1d7fa700 1 -- 172.21.15.116:0/1949828189 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a1000f2e0 con 0x7f1a20101070 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.585+0000 7f1a266c8700 1 -- 172.21.15.116:0/1949828189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2019c100 con 0x7f1a20101070 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.586+0000 7f1a266c8700 1 -- 172.21.15.116:0/1949828189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a20108690 con 0x7f1a20101070 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.587+0000 7f1a1d7fa700 1 -- 172.21.15.116:0/1949828189 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7f1a1000f510 con 0x7f1a20101070 2024-03-04T16:06:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.587+0000 7f1a1d7fa700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1a0c06d9e0 0x7f1a0c06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.587+0000 7f1a1d7fa700 1 -- 172.21.15.116:0/1949828189 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1a1008d360 con 0x7f1a20101070 2024-03-04T16:06:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.587+0000 7f1a1ffff700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1a0c06d9e0 0x7f1a0c06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.587+0000 7f1a1ffff700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1a0c06d9e0 0x7f1a0c06fea0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1a08005b40 tx=0x7f1a08005ab0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.596+0000 7f1a1d7fa700 1 -- 172.21.15.116:0/1949828189 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1a1005a4e0 con 0x7f1a20101070 2024-03-04T16:06:26.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.818+0000 7f1a266c8700 1 -- 172.21.15.116:0/1949828189 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a200ff2b0 con 0x7f1a0c06d9e0 2024-03-04T16:06:26.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.819+0000 7f1a1d7fa700 1 -- 172.21.15.116:0/1949828189 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f1a200ff2b0 con 0x7f1a0c06d9e0 2024-03-04T16:06:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 -- 172.21.15.116:0/1949828189 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1a0c06d9e0 msgr2=0x7f1a0c06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1a0c06d9e0 0x7f1a0c06fea0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1a08005b40 tx=0x7f1a08005ab0 comp rx=0 tx=0).stop 2024-03-04T16:06:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 -- 172.21.15.116:0/1949828189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a20101070 msgr2=0x7f1a20196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a20101070 0x7f1a20196920 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1a1000b4b0 tx=0x7f1a1000b870 comp rx=0 tx=0).stop 2024-03-04T16:06:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 -- 172.21.15.116:0/1949828189 shutdown_connections 2024-03-04T16:06:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1a0c06d9e0 0x7f1a0c06fea0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a200ffcd0 0x7f1a201963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 --2- 172.21.15.116:0/1949828189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a20101070 0x7f1a20196920 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 -- 172.21.15.116:0/1949828189 >> 172.21.15.116:0/1949828189 conn(0x7f1a200fa7c0 msgr2=0x7f1a200feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 -- 172.21.15.116:0/1949828189 shutdown_connections 2024-03-04T16:06:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.822+0000 7f1a266c8700 1 -- 172.21.15.116:0/1949828189 wait complete. 2024-03-04T16:06:26.834 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:06:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3431839084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc8100c00 msgr2=0x7fcdc8101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 --2- 172.21.15.116:0/3431839084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc8100c00 0x7fcdc8101020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8009910 tx=0x7fcdb8009c20 comp rx=0 tx=0).stop 2024-03-04T16:06:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3431839084 shutdown_connections 2024-03-04T16:06:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 --2- 172.21.15.116:0/3431839084 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdc8101e00 0x7fcdc8102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 --2- 172.21.15.116:0/3431839084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc8100c00 0x7fcdc8101020 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3431839084 >> 172.21.15.116:0/3431839084 conn(0x7fcdc80fc180 msgr2=0x7fcdc80fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3431839084 shutdown_connections 2024-03-04T16:06:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3431839084 wait complete. 2024-03-04T16:06:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 Processor -- start 2024-03-04T16:06:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 -- start start 2024-03-04T16:06:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.932+0000 7fcdcdff1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc8100c00 0x7fcdc81963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdcdff1700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdc8101e00 0x7fcdc8196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdcdff1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc8196f40 con 0x7fcdc8100c00 2024-03-04T16:06:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdcdff1700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc8197080 con 0x7fcdc8101e00 2024-03-04T16:06:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc8100c00 0x7fcdc81963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc8100c00 0x7fcdc81963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41100/0 (socket says 172.21.15.116:41100) 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc77fe700 1 -- 172.21.15.116:0/3645659614 learned_addr learned my addr 172.21.15.116:0/3645659614 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc6ffd700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdc8101e00 0x7fcdc8196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc77fe700 1 -- 172.21.15.116:0/3645659614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdc8101e00 msgr2=0x7fcdc8196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc77fe700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdc8101e00 0x7fcdc8196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc77fe700 1 -- 172.21.15.116:0/3645659614 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb80095f0 con 0x7fcdc8100c00 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc77fe700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc8100c00 0x7fcdc81963e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcdb800bda0 tx=0x7fcdb800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc4ff9700 1 -- 172.21.15.116:0/3645659614 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb801d070 con 0x7fcdc8100c00 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc4ff9700 1 -- 172.21.15.116:0/3645659614 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcdb8022470 con 0x7fcdc8100c00 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdc4ff9700 1 -- 172.21.15.116:0/3645659614 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb800f670 con 0x7fcdc8100c00 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3645659614 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc819bad0 con 0x7fcdc8100c00 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.933+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3645659614 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc819c040 con 0x7fcdc8100c00 2024-03-04T16:06:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.934+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3645659614 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc81097e0 con 0x7fcdc8100c00 2024-03-04T16:06:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.935+0000 7fcdc4ff9700 1 -- 172.21.15.116:0/3645659614 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7fcdb8004af0 con 0x7fcdc8100c00 2024-03-04T16:06:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.935+0000 7fcdc4ff9700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcdb006d9e0 0x7fcdb006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.935+0000 7fcdc6ffd700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcdb006d9e0 0x7fcdb006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.935+0000 7fcdc4ff9700 1 -- 172.21.15.116:0/3645659614 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcdb808ed00 con 0x7fcdc8100c00 2024-03-04T16:06:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.936+0000 7fcdc6ffd700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcdb006d9e0 0x7fcdb006fea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009a10 tx=0x7fcdbc009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:26.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:26.940+0000 7fcdc4ff9700 1 -- 172.21.15.116:0/3645659614 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcdb8092030 con 0x7fcdc8100c00 2024-03-04T16:06:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.156+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3645659614 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdc8106750 con 0x7fcdb006d9e0 2024-03-04T16:06:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.157+0000 7fcdc4ff9700 1 -- 172.21.15.116:0/3645659614 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fcdc8106750 con 0x7fcdb006d9e0 2024-03-04T16:06:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3645659614 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcdb006d9e0 msgr2=0x7fcdb006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcdb006d9e0 0x7fcdb006fea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009a10 tx=0x7fcdbc009190 comp rx=0 tx=0).stop 2024-03-04T16:06:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3645659614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc8100c00 msgr2=0x7fcdc81963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc8100c00 0x7fcdc81963e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcdb800bda0 tx=0x7fcdb800bdd0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3645659614 shutdown_connections 2024-03-04T16:06:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcdb006d9e0 0x7fcdb006fea0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc8100c00 0x7fcdc81963e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 --2- 172.21.15.116:0/3645659614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdc8101e00 0x7fcdc8196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3645659614 >> 172.21.15.116:0/3645659614 conn(0x7fcdc80fc180 msgr2=0x7fcdc8105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3645659614 shutdown_connections 2024-03-04T16:06:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.159+0000 7fcdcdff1700 1 -- 172.21.15.116:0/3645659614 wait complete. 2024-03-04T16:06:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.265+0000 7f2aff2b3700 1 -- 172.21.15.116:0/591907635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2af80ffab0 msgr2=0x7f2af80ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.265+0000 7f2aff2b3700 1 --2- 172.21.15.116:0/591907635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2af80ffab0 0x7f2af80ffed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2af000b0a0 tx=0x7f2af000b3b0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.265+0000 7f2aff2b3700 1 -- 172.21.15.116:0/591907635 shutdown_connections 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.265+0000 7f2aff2b3700 1 --2- 172.21.15.116:0/591907635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2af8100cb0 0x7f2af8101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.265+0000 7f2aff2b3700 1 --2- 172.21.15.116:0/591907635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2af80ffab0 0x7f2af80ffed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.265+0000 7f2aff2b3700 1 -- 172.21.15.116:0/591907635 >> 172.21.15.116:0/591907635 conn(0x7f2af80fb030 msgr2=0x7f2af80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.265+0000 7f2aff2b3700 1 -- 172.21.15.116:0/591907635 shutdown_connections 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.265+0000 7f2aff2b3700 1 -- 172.21.15.116:0/591907635 wait complete. 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.265+0000 7f2aff2b3700 1 Processor -- start 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2aff2b3700 1 -- start start 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2aff2b3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2af80ffab0 0x7f2af8196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2aff2b3700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2af8100cb0 0x7f2af81968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2aff2b3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af8196ec0 con 0x7f2af80ffab0 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2aff2b3700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af8197000 con 0x7f2af8100cb0 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2afd04f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2af80ffab0 0x7f2af8196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2afd04f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2af80ffab0 0x7f2af8196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41128/0 (socket says 172.21.15.116:41128) 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2afd04f700 1 -- 172.21.15.116:0/3169360560 learned_addr learned my addr 172.21.15.116:0/3169360560 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2afc84e700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2af8100cb0 0x7f2af81968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2afd04f700 1 -- 172.21.15.116:0/3169360560 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2af8100cb0 msgr2=0x7f2af81968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2afd04f700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2af8100cb0 0x7f2af81968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2afd04f700 1 -- 172.21.15.116:0/3169360560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af0009d00 con 0x7f2af80ffab0 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2afd04f700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2af80ffab0 0x7f2af8196360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2af00077b0 tx=0x7f2af0009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2aee7fc700 1 -- 172.21.15.116:0/3169360560 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2af000e040 con 0x7f2af80ffab0 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2aee7fc700 1 -- 172.21.15.116:0/3169360560 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2af00044f0 con 0x7f2af80ffab0 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2aff2b3700 1 -- 172.21.15.116:0/3169360560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af819ba50 con 0x7f2af80ffab0 2024-03-04T16:06:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2aee7fc700 1 -- 172.21.15.116:0/3169360560 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2af001d9e0 con 0x7f2af80ffab0 2024-03-04T16:06:27.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.266+0000 7f2aff2b3700 1 -- 172.21.15.116:0/3169360560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af819bfc0 con 0x7f2af80ffab0 2024-03-04T16:06:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.267+0000 7f2aff2b3700 1 -- 172.21.15.116:0/3169360560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af8108610 con 0x7f2af80ffab0 2024-03-04T16:06:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.268+0000 7f2aee7fc700 1 -- 172.21.15.116:0/3169360560 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7f2af0004040 con 0x7f2af80ffab0 2024-03-04T16:06:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.268+0000 7f2aee7fc700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2ae406d9e0 0x7f2ae406fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.268+0000 7f2aee7fc700 1 -- 172.21.15.116:0/3169360560 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2af008f160 con 0x7f2af80ffab0 2024-03-04T16:06:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.268+0000 7f2afc84e700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2ae406d9e0 0x7f2ae406fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.269+0000 7f2afc84e700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2ae406d9e0 0x7f2ae406fea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8005950 tx=0x7f2ae80058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.273+0000 7f2aee7fc700 1 -- 172.21.15.116:0/3169360560 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2af005c2f0 con 0x7f2af80ffab0 2024-03-04T16:06:27.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.483+0000 7f2aff2b3700 1 -- 172.21.15.116:0/3169360560 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2af8068b20 con 0x7f2ae406d9e0 2024-03-04T16:06:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.495+0000 7f2aee7fc700 1 -- 172.21.15.116:0/3169360560 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f2af8068b20 con 0x7f2ae406d9e0 2024-03-04T16:06:27.496 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:27 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:27.496 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:27 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:27.496 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:27 smithi116 ceph-mon[94164]: Reconfiguring mgr.smithi116.jwdhop (monmap changed)... 2024-03-04T16:06:27.496 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:27 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.jwdhop", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:06:27.496 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:27 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:06:27.496 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:27 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:27.496 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:27 smithi116 ceph-mon[94164]: Reconfiguring daemon mgr.smithi116.jwdhop on smithi116 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (9m) 27s ago 10m 25.4M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (11m) 27s ago 11m 7750k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (10m) 8s ago 10m 7772k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (11m) 27s ago 11m 6669k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (10m) 8s ago 10m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (9m) 27s ago 10m 86.9M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (5m) 27s ago 5m 18.4M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (5m) 27s ago 5m 14.9M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (5m) 8s ago 5m 16.4M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (5m) 8s ago 5m 153M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:06:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (117s) 27s ago 12m 473M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (75s) 8s ago 9m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (44s) 27s ago 12m 28.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (13s) 8s ago 9m 22.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (11m) 27s ago 11m 15.9M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (10m) 8s ago 10m 16.5M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (9m) 27s ago 9m 213M 1216M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (8m) 27s ago 8m 223M 1216M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (8m) 27s ago 8m 183M 1216M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (7m) 8s ago 7m 267M 2240M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (7m) 8s ago 7m 236M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (6m) 8s ago 6m 228M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:06:27.498 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (81s) 27s ago 10m 47.1M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:06:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 -- 172.21.15.116:0/3169360560 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2ae406d9e0 msgr2=0x7f2ae406fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2ae406d9e0 0x7f2ae406fea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8005950 tx=0x7f2ae80058e0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 -- 172.21.15.116:0/3169360560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2af80ffab0 msgr2=0x7f2af8196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2af80ffab0 0x7f2af8196360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2af00077b0 tx=0x7f2af0009810 comp rx=0 tx=0).stop 2024-03-04T16:06:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 -- 172.21.15.116:0/3169360560 shutdown_connections 2024-03-04T16:06:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2ae406d9e0 0x7f2ae406fea0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2af80ffab0 0x7f2af8196360 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 --2- 172.21.15.116:0/3169360560 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2af8100cb0 0x7f2af81968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 -- 172.21.15.116:0/3169360560 >> 172.21.15.116:0/3169360560 conn(0x7f2af80fb030 msgr2=0x7f2af8068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 -- 172.21.15.116:0/3169360560 shutdown_connections 2024-03-04T16:06:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.497+0000 7f2aff2b3700 1 -- 172.21.15.116:0/3169360560 wait complete. 2024-03-04T16:06:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.600+0000 7f44f30e0700 1 -- 172.21.15.116:0/2239937747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ec100cd0 msgr2=0x7f44ec101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.600+0000 7f44f30e0700 1 --2- 172.21.15.116:0/2239937747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ec100cd0 0x7f44ec101130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009910 tx=0x7f44e0009c20 comp rx=0 tx=0).stop 2024-03-04T16:06:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.600+0000 7f44f30e0700 1 -- 172.21.15.116:0/2239937747 shutdown_connections 2024-03-04T16:06:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.600+0000 7f44f30e0700 1 --2- 172.21.15.116:0/2239937747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ec100cd0 0x7f44ec101130 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.600+0000 7f44f30e0700 1 --2- 172.21.15.116:0/2239937747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44ec0ffad0 0x7f44ec0ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.600+0000 7f44f30e0700 1 -- 172.21.15.116:0/2239937747 >> 172.21.15.116:0/2239937747 conn(0x7f44ec0fb050 msgr2=0x7f44ec0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.600+0000 7f44f30e0700 1 -- 172.21.15.116:0/2239937747 shutdown_connections 2024-03-04T16:06:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.600+0000 7f44f30e0700 1 -- 172.21.15.116:0/2239937747 wait complete. 2024-03-04T16:06:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f30e0700 1 Processor -- start 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f30e0700 1 -- start start 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f30e0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ec0ffad0 0x7f44ec0698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f30e0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44ec1993c0 0x7f44ec19b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f30e0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec19bd80 con 0x7f44ec0ffad0 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f30e0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec19bec0 con 0x7f44ec1993c0 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f0e7c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ec0ffad0 0x7f44ec0698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f0e7c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ec0ffad0 0x7f44ec0698f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41154/0 (socket says 172.21.15.116:41154) 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f0e7c700 1 -- 172.21.15.116:0/3714104260 learned_addr learned my addr 172.21.15.116:0/3714104260 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44ebfff700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44ec1993c0 0x7f44ec19b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f0e7c700 1 -- 172.21.15.116:0/3714104260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44ec1993c0 msgr2=0x7f44ec19b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f0e7c700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44ec1993c0 0x7f44ec19b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f0e7c700 1 -- 172.21.15.116:0/3714104260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e00095f0 con 0x7f44ec0ffad0 2024-03-04T16:06:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.601+0000 7f44f0e7c700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ec0ffad0 0x7f44ec0698f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f44e400b9c0 tx=0x7f44e400bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.602+0000 7f44e9ffb700 1 -- 172.21.15.116:0/3714104260 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44e400c980 con 0x7f44ec0ffad0 2024-03-04T16:06:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.602+0000 7f44e9ffb700 1 -- 172.21.15.116:0/3714104260 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f44e4014440 con 0x7f44ec0ffad0 2024-03-04T16:06:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.602+0000 7f44e9ffb700 1 -- 172.21.15.116:0/3714104260 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44e4012570 con 0x7f44ec0ffad0 2024-03-04T16:06:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.602+0000 7f44f30e0700 1 -- 172.21.15.116:0/3714104260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ec19c1a0 con 0x7f44ec0ffad0 2024-03-04T16:06:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.602+0000 7f44f30e0700 1 -- 172.21.15.116:0/3714104260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ec19c740 con 0x7f44ec0ffad0 2024-03-04T16:06:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.603+0000 7f44f30e0700 1 -- 172.21.15.116:0/3714104260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ec1086d0 con 0x7f44ec0ffad0 2024-03-04T16:06:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.603+0000 7f44e9ffb700 1 -- 172.21.15.116:0/3714104260 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7f44e40145b0 con 0x7f44ec0ffad0 2024-03-04T16:06:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.603+0000 7f44e9ffb700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f44d406d9e0 0x7f44d406fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.604+0000 7f44e9ffb700 1 -- 172.21.15.116:0/3714104260 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f44e4006e40 con 0x7f44ec0ffad0 2024-03-04T16:06:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.604+0000 7f44ebfff700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f44d406d9e0 0x7f44d406fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.604+0000 7f44ebfff700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f44d406d9e0 0x7f44d406fea0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f44e0005090 tx=0x7f44e0005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:27 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:27 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:27 smithi152 ceph-mon[82845]: Reconfiguring mgr.smithi116.jwdhop (monmap changed)... 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:27 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.jwdhop", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:27 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:27 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:27 smithi152 ceph-mon[82845]: Reconfiguring daemon mgr.smithi116.jwdhop on smithi116 2024-03-04T16:06:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.608+0000 7f44e9ffb700 1 -- 172.21.15.116:0/3714104260 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f44e405a030 con 0x7f44ec0ffad0 2024-03-04T16:06:27.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.887+0000 7f44f30e0700 1 -- 172.21.15.116:0/3714104260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f44ec1088c0 con 0x7f44ec0ffad0 2024-03-04T16:06:27.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.887+0000 7f44e9ffb700 1 -- 172.21.15.116:0/3714104260 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f44e4059bc0 con 0x7f44ec0ffad0 2024-03-04T16:06:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-03-04T16:06:27.889 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:06:27.890 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:06:27.890 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:06:27.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.889+0000 7f44f30e0700 1 -- 172.21.15.116:0/3714104260 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f44d406d9e0 msgr2=0x7f44d406fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:27.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.889+0000 7f44f30e0700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f44d406d9e0 0x7f44d406fea0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f44e0005090 tx=0x7f44e0005960 comp rx=0 tx=0).stop 2024-03-04T16:06:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.889+0000 7f44f30e0700 1 -- 172.21.15.116:0/3714104260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ec0ffad0 msgr2=0x7f44ec0698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.889+0000 7f44f30e0700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ec0ffad0 0x7f44ec0698f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f44e400b9c0 tx=0x7f44e400bd80 comp rx=0 tx=0).stop 2024-03-04T16:06:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.890+0000 7f44f30e0700 1 -- 172.21.15.116:0/3714104260 shutdown_connections 2024-03-04T16:06:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.890+0000 7f44f30e0700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f44d406d9e0 0x7f44d406fea0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.890+0000 7f44f30e0700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ec0ffad0 0x7f44ec0698f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.890+0000 7f44f30e0700 1 --2- 172.21.15.116:0/3714104260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44ec1993c0 0x7f44ec19b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.890+0000 7f44f30e0700 1 -- 172.21.15.116:0/3714104260 >> 172.21.15.116:0/3714104260 conn(0x7f44ec0fb050 msgr2=0x7f44ec103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.890+0000 7f44f30e0700 1 -- 172.21.15.116:0/3714104260 shutdown_connections 2024-03-04T16:06:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.890+0000 7f44f30e0700 1 -- 172.21.15.116:0/3714104260 wait complete. 2024-03-04T16:06:28.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.999+0000 7fcbbe166700 1 -- 172.21.15.116:0/3158603805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbb8100cb0 msgr2=0x7fcbb8101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:28.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.999+0000 7fcbbe166700 1 --2- 172.21.15.116:0/3158603805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbb8100cb0 0x7fcbb8101110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009910 tx=0x7fcbac009c20 comp rx=0 tx=0).stop 2024-03-04T16:06:28.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.999+0000 7fcbbe166700 1 -- 172.21.15.116:0/3158603805 shutdown_connections 2024-03-04T16:06:28.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.999+0000 7fcbbe166700 1 --2- 172.21.15.116:0/3158603805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbb8100cb0 0x7fcbb8101110 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.999+0000 7fcbbe166700 1 --2- 172.21.15.116:0/3158603805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbb80ffab0 0x7fcbb80ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.999+0000 7fcbbe166700 1 -- 172.21.15.116:0/3158603805 >> 172.21.15.116:0/3158603805 conn(0x7fcbb80fb050 msgr2=0x7fcbb80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.999+0000 7fcbbe166700 1 -- 172.21.15.116:0/3158603805 shutdown_connections 2024-03-04T16:06:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.999+0000 7fcbbe166700 1 -- 172.21.15.116:0/3158603805 wait complete. 2024-03-04T16:06:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:27.999+0000 7fcbbe166700 1 Processor -- start 2024-03-04T16:06:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbbe166700 1 -- start start 2024-03-04T16:06:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbbe166700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbb80ffab0 0x7fcbb80699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbbe166700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbb8100cb0 0x7fcbb819b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbbe166700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb819b670 con 0x7fcbb8100cb0 2024-03-04T16:06:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbbe166700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb819b7e0 con 0x7fcbb80ffab0 2024-03-04T16:06:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbb6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbb8100cb0 0x7fcbb819b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbb6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbb8100cb0 0x7fcbb819b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41176/0 (socket says 172.21.15.116:41176) 2024-03-04T16:06:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbb6ffd700 1 -- 172.21.15.116:0/2444581792 learned_addr learned my addr 172.21.15.116:0/2444581792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:06:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbb77fe700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbb80ffab0 0x7fcbb80699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbb6ffd700 1 -- 172.21.15.116:0/2444581792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbb80ffab0 msgr2=0x7fcbb80699b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbb6ffd700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbb80ffab0 0x7fcbb80699b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbb6ffd700 1 -- 172.21.15.116:0/2444581792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0095f0 con 0x7fcbb8100cb0 2024-03-04T16:06:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.000+0000 7fcbb6ffd700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbb8100cb0 0x7fcbb819b0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcbac006010 tx=0x7fcbac00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.002+0000 7fcbb4ff9700 1 -- 172.21.15.116:0/2444581792 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbac01d070 con 0x7fcbb8100cb0 2024-03-04T16:06:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.002+0000 7fcbbe166700 1 -- 172.21.15.116:0/2444581792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb819ba60 con 0x7fcbb8100cb0 2024-03-04T16:06:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.002+0000 7fcbbe166700 1 -- 172.21.15.116:0/2444581792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb819bf50 con 0x7fcbb8100cb0 2024-03-04T16:06:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.002+0000 7fcbb4ff9700 1 -- 172.21.15.116:0/2444581792 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcbac00f460 con 0x7fcbb8100cb0 2024-03-04T16:06:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.002+0000 7fcbb4ff9700 1 -- 172.21.15.116:0/2444581792 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbac017630 con 0x7fcbb8100cb0 2024-03-04T16:06:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.003+0000 7fcbbe166700 1 -- 172.21.15.116:0/2444581792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba4005320 con 0x7fcbb8100cb0 2024-03-04T16:06:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.004+0000 7fcbb4ff9700 1 -- 172.21.15.116:0/2444581792 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7fcbac017790 con 0x7fcbb8100cb0 2024-03-04T16:06:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.004+0000 7fcbb4ff9700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcba006d9e0 0x7fcba006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.004+0000 7fcbb4ff9700 1 -- 172.21.15.116:0/2444581792 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcbac08e840 con 0x7fcbb8100cb0 2024-03-04T16:06:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.004+0000 7fcbb77fe700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcba006d9e0 0x7fcba006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.005+0000 7fcbb77fe700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcba006d9e0 0x7fcba006fea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8100b10 tx=0x7fcba800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.009+0000 7fcbb4ff9700 1 -- 172.21.15.116:0/2444581792 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcbac05bba0 con 0x7fcbb8100cb0 2024-03-04T16:06:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.254+0000 7fcbbe166700 1 -- 172.21.15.116:0/2444581792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcba4005cc0 con 0x7fcbb8100cb0 2024-03-04T16:06:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.254+0000 7fcbb4ff9700 1 -- 172.21.15.116:0/2444581792 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1852 (secure 0 0 0) 0x7fcbac05b730 con 0x7fcbb8100cb0 2024-03-04T16:06:28.255 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:06:28.256 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:06:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:06:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:06:28.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 -- 172.21.15.116:0/2444581792 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcba006d9e0 msgr2=0x7fcba006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:28.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcba006d9e0 0x7fcba006fea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8100b10 tx=0x7fcba800b410 comp rx=0 tx=0).stop 2024-03-04T16:06:28.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 -- 172.21.15.116:0/2444581792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbb8100cb0 msgr2=0x7fcbb819b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:28.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbb8100cb0 0x7fcbb819b0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcbac006010 tx=0x7fcbac00bd70 comp rx=0 tx=0).stop 2024-03-04T16:06:28.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 -- 172.21.15.116:0/2444581792 shutdown_connections 2024-03-04T16:06:28.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcba006d9e0 0x7fcba006fea0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbb80ffab0 0x7fcbb80699b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 --2- 172.21.15.116:0/2444581792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbb8100cb0 0x7fcbb819b0a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 -- 172.21.15.116:0/2444581792 >> 172.21.15.116:0/2444581792 conn(0x7fcbb80fb050 msgr2=0x7fcbb8103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 -- 172.21.15.116:0/2444581792 shutdown_connections 2024-03-04T16:06:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.257+0000 7fcbbe166700 1 -- 172.21.15.116:0/2444581792 wait complete. 2024-03-04T16:06:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:06:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.367+0000 7f1969cfb700 1 -- 172.21.15.116:0/1506072518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1964103f70 msgr2=0x7f1964106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.367+0000 7f1969cfb700 1 --2- 172.21.15.116:0/1506072518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1964103f70 0x7f1964106360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1954009910 tx=0x7f1954009c20 comp rx=0 tx=0).stop 2024-03-04T16:06:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.367+0000 7f1969cfb700 1 -- 172.21.15.116:0/1506072518 shutdown_connections 2024-03-04T16:06:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.367+0000 7f1969cfb700 1 --2- 172.21.15.116:0/1506072518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1964103f70 0x7f1964106360 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.367+0000 7f1969cfb700 1 --2- 172.21.15.116:0/1506072518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1964101640 0x7f1964103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.367+0000 7f1969cfb700 1 -- 172.21.15.116:0/1506072518 >> 172.21.15.116:0/1506072518 conn(0x7f19640fb050 msgr2=0x7f19640fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.367+0000 7f1969cfb700 1 -- 172.21.15.116:0/1506072518 shutdown_connections 2024-03-04T16:06:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.367+0000 7f1969cfb700 1 -- 172.21.15.116:0/1506072518 wait complete. 2024-03-04T16:06:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f1969cfb700 1 Processor -- start 2024-03-04T16:06:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f1969cfb700 1 -- start start 2024-03-04T16:06:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f1969cfb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1964101640 0x7f1964196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f1969cfb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1964103f70 0x7f1964196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f1969cfb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1964196e80 con 0x7f1964101640 2024-03-04T16:06:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f1969cfb700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1964196fc0 con 0x7f1964103f70 2024-03-04T16:06:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f19637fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1964101640 0x7f1964196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f19637fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1964101640 0x7f1964196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41182/0 (socket says 172.21.15.116:41182) 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f19637fe700 1 -- 172.21.15.116:0/4170134337 learned_addr learned my addr 172.21.15.116:0/4170134337 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f1962ffd700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1964103f70 0x7f1964196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f19637fe700 1 -- 172.21.15.116:0/4170134337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1964103f70 msgr2=0x7f1964196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f19637fe700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1964103f70 0x7f1964196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.368+0000 7f19637fe700 1 -- 172.21.15.116:0/4170134337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19540095f0 con 0x7f1964101640 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.369+0000 7f19637fe700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1964101640 0x7f1964196320 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f195800c8a0 tx=0x7f195800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.369+0000 7f1960ff9700 1 -- 172.21.15.116:0/4170134337 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f195800cea0 con 0x7f1964101640 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.369+0000 7f1969cfb700 1 -- 172.21.15.116:0/4170134337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196419ba70 con 0x7f1964101640 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.369+0000 7f1960ff9700 1 -- 172.21.15.116:0/4170134337 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1958004830 con 0x7f1964101640 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.369+0000 7f1960ff9700 1 -- 172.21.15.116:0/4170134337 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19580055b0 con 0x7f1964101640 2024-03-04T16:06:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.369+0000 7f1969cfb700 1 -- 172.21.15.116:0/4170134337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196419bfc0 con 0x7f1964101640 2024-03-04T16:06:28.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.370+0000 7f1969cfb700 1 -- 172.21.15.116:0/4170134337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1964190520 con 0x7f1964101640 2024-03-04T16:06:28.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.370+0000 7f1960ff9700 1 -- 172.21.15.116:0/4170134337 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7f1958007ad0 con 0x7f1964101640 2024-03-04T16:06:28.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.371+0000 7f1960ff9700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f194c06d9e0 0x7f194c06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.371+0000 7f1960ff9700 1 -- 172.21.15.116:0/4170134337 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f195808d480 con 0x7f1964101640 2024-03-04T16:06:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.371+0000 7f1962ffd700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f194c06d9e0 0x7f194c06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.371+0000 7f1962ffd700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f194c06d9e0 0x7f194c06fea0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1954006010 tx=0x7f195400b580 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:28.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.376+0000 7f1960ff9700 1 -- 172.21.15.116:0/4170134337 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f195805a590 con 0x7f1964101640 2024-03-04T16:06:28.593 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:28.593 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: pgmap v20: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 613 KiB/s wr, 214 op/s 2024-03-04T16:06:28.593 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:28.593 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3714104260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:28.593 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:28.593 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:28.593 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:06:28.593 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:06:28.594 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-04T16:06:28.594 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi116"}]: dispatch 2024-03-04T16:06:28.594 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:28.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.601+0000 7f1969cfb700 1 -- 172.21.15.116:0/4170134337 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f19640611d0 con 0x7f194c06d9e0 2024-03-04T16:06:28.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.602+0000 7f1960ff9700 1 -- 172.21.15.116:0/4170134337 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f19640611d0 con 0x7f194c06d9e0 2024-03-04T16:06:28.603 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:06:28.603 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:06:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:06:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:06:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:06:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:06:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:06:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:06:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "4/23 daemons upgraded", 2024-03-04T16:06:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mon daemons", 2024-03-04T16:06:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:06:28.604 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.604+0000 7f1969cfb700 1 -- 172.21.15.116:0/4170134337 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f194c06d9e0 msgr2=0x7f194c06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.604+0000 7f1969cfb700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f194c06d9e0 0x7f194c06fea0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1954006010 tx=0x7f195400b580 comp rx=0 tx=0).stop 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.604+0000 7f1969cfb700 1 -- 172.21.15.116:0/4170134337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1964101640 msgr2=0x7f1964196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.604+0000 7f1969cfb700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1964101640 0x7f1964196320 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f195800c8a0 tx=0x7f195800cc60 comp rx=0 tx=0).stop 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.605+0000 7f1969cfb700 1 -- 172.21.15.116:0/4170134337 shutdown_connections 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.605+0000 7f1969cfb700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f194c06d9e0 0x7f194c06fea0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.605+0000 7f1969cfb700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1964101640 0x7f1964196320 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.605+0000 7f1969cfb700 1 --2- 172.21.15.116:0/4170134337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1964103f70 0x7f1964196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.605+0000 7f1969cfb700 1 -- 172.21.15.116:0/4170134337 >> 172.21.15.116:0/4170134337 conn(0x7f19640fb050 msgr2=0x7f19640fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.605+0000 7f1969cfb700 1 -- 172.21.15.116:0/4170134337 shutdown_connections 2024-03-04T16:06:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.605+0000 7f1969cfb700 1 -- 172.21.15.116:0/4170134337 wait complete. 2024-03-04T16:06:28.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:28.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: pgmap v20: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 613 KiB/s wr, 214 op/s 2024-03-04T16:06:28.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3714104260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:06:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:06:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-04T16:06:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi116"}]: dispatch 2024-03-04T16:06:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:28.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.710+0000 7fcfa6011700 1 -- 172.21.15.116:0/1184580246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa0100cf0 msgr2=0x7fcfa0101150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:28.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.710+0000 7fcfa6011700 1 --2- 172.21.15.116:0/1184580246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa0100cf0 0x7fcfa0101150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009910 tx=0x7fcf88009c20 comp rx=0 tx=0).stop 2024-03-04T16:06:28.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.710+0000 7fcfa6011700 1 -- 172.21.15.116:0/1184580246 shutdown_connections 2024-03-04T16:06:28.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.710+0000 7fcfa6011700 1 --2- 172.21.15.116:0/1184580246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa0100cf0 0x7fcfa0101150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.710+0000 7fcfa6011700 1 --2- 172.21.15.116:0/1184580246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa00ffaf0 0x7fcfa00fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.710+0000 7fcfa6011700 1 -- 172.21.15.116:0/1184580246 >> 172.21.15.116:0/1184580246 conn(0x7fcfa00fb070 msgr2=0x7fcfa00fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:28.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.710+0000 7fcfa6011700 1 -- 172.21.15.116:0/1184580246 shutdown_connections 2024-03-04T16:06:28.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.710+0000 7fcfa6011700 1 -- 172.21.15.116:0/1184580246 wait complete. 2024-03-04T16:06:28.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.710+0000 7fcfa6011700 1 Processor -- start 2024-03-04T16:06:28.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.710+0000 7fcfa6011700 1 -- start start 2024-03-04T16:06:28.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcfa6011700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa00ffaf0 0x7fcfa00698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:28.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcfa6011700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa0100cf0 0x7fcfa019b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:28.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcfa6011700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa019b6d0 con 0x7fcfa0100cf0 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcfa6011700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa019b810 con 0x7fcfa00ffaf0 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf96ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa0100cf0 0x7fcfa019b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf96ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa0100cf0 0x7fcfa019b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41202/0 (socket says 172.21.15.116:41202) 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf96ffd700 1 -- 172.21.15.116:0/3627306908 learned_addr learned my addr 172.21.15.116:0/3627306908 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf9f7fe700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa00ffaf0 0x7fcfa00698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf96ffd700 1 -- 172.21.15.116:0/3627306908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa00ffaf0 msgr2=0x7fcfa00698f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf96ffd700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa00ffaf0 0x7fcfa00698f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf96ffd700 1 -- 172.21.15.116:0/3627306908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf880095f0 con 0x7fcfa0100cf0 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf9f7fe700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa00ffaf0 0x7fcfa00698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf96ffd700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa0100cf0 0x7fcfa019b100 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcf88006010 tx=0x7fcf8800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf9d7fa700 1 -- 172.21.15.116:0/3627306908 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf8801d070 con 0x7fcfa0100cf0 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcfa6011700 1 -- 172.21.15.116:0/3627306908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa019ba90 con 0x7fcfa0100cf0 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf9d7fa700 1 -- 172.21.15.116:0/3627306908 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcf8800f460 con 0x7fcfa0100cf0 2024-03-04T16:06:28.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.711+0000 7fcf9d7fa700 1 -- 172.21.15.116:0/3627306908 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf88021640 con 0x7fcfa0100cf0 2024-03-04T16:06:28.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.712+0000 7fcfa6011700 1 -- 172.21.15.116:0/3627306908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa019bf80 con 0x7fcfa0100cf0 2024-03-04T16:06:28.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.713+0000 7fcfa6011700 1 -- 172.21.15.116:0/3627306908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa0108650 con 0x7fcfa0100cf0 2024-03-04T16:06:28.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.713+0000 7fcf9d7fa700 1 -- 172.21.15.116:0/3627306908 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7fcf88004b20 con 0x7fcfa0100cf0 2024-03-04T16:06:28.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.713+0000 7fcf9d7fa700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcf8c06d9e0 0x7fcf8c06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:28.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.713+0000 7fcf9f7fe700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcf8c06d9e0 0x7fcf8c06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:28.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.713+0000 7fcf9d7fa700 1 -- 172.21.15.116:0/3627306908 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcf8808ec00 con 0x7fcfa0100cf0 2024-03-04T16:06:28.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.714+0000 7fcf9f7fe700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcf8c06d9e0 0x7fcf8c06fea0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0100b50 tx=0x7fcf90005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:28.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:28.718+0000 7fcf9d7fa700 1 -- 172.21.15.116:0/3627306908 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcf88094020 con 0x7fcfa0100cf0 2024-03-04T16:06:29.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.000+0000 7fcfa6011700 1 -- 172.21.15.116:0/3627306908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcfa019c680 con 0x7fcfa0100cf0 2024-03-04T16:06:29.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.001+0000 7fcf9d7fa700 1 -- 172.21.15.116:0/3627306908 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fcf8805bd10 con 0x7fcfa0100cf0 2024-03-04T16:06:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 -- 172.21.15.116:0/3627306908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcf8c06d9e0 msgr2=0x7fcf8c06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcf8c06d9e0 0x7fcf8c06fea0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0100b50 tx=0x7fcf90005f90 comp rx=0 tx=0).stop 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 -- 172.21.15.116:0/3627306908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa0100cf0 msgr2=0x7fcfa019b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa0100cf0 0x7fcfa019b100 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcf88006010 tx=0x7fcf8800be20 comp rx=0 tx=0).stop 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 -- 172.21.15.116:0/3627306908 shutdown_connections 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcf8c06d9e0 0x7fcf8c06fea0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa00ffaf0 0x7fcfa00698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 --2- 172.21.15.116:0/3627306908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa0100cf0 0x7fcfa019b100 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 -- 172.21.15.116:0/3627306908 >> 172.21.15.116:0/3627306908 conn(0x7fcfa00fb070 msgr2=0x7fcfa0103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 -- 172.21.15.116:0/3627306908 shutdown_connections 2024-03-04T16:06:29.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:29.003+0000 7fcfa6011700 1 -- 172.21.15.116:0/3627306908 wait complete. 2024-03-04T16:06:29.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:29 smithi152 ceph-mon[82845]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:29.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:29 smithi152 ceph-mon[82845]: Reconfiguring ceph-exporter.smithi116 (monmap changed)... 2024-03-04T16:06:29.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:29 smithi152 ceph-mon[82845]: Unable to update caps for client.ceph-exporter.smithi116 2024-03-04T16:06:29.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:29 smithi152 ceph-mon[82845]: Reconfiguring daemon ceph-exporter.smithi116 on smithi116 2024-03-04T16:06:29.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:29 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2444581792' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:06:29.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:29 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3627306908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:06:29.678 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:29 smithi116 ceph-mon[94164]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:29.678 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:29 smithi116 ceph-mon[94164]: Reconfiguring ceph-exporter.smithi116 (monmap changed)... 2024-03-04T16:06:29.678 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:29 smithi116 ceph-mon[94164]: Unable to update caps for client.ceph-exporter.smithi116 2024-03-04T16:06:29.679 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:29 smithi116 ceph-mon[94164]: Reconfiguring daemon ceph-exporter.smithi116 on smithi116 2024-03-04T16:06:29.679 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:29 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2444581792' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:06:29.679 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:29 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3627306908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:06:30.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:30 smithi116 ceph-mon[94164]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:30.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:30 smithi116 ceph-mon[94164]: pgmap v21: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 630 KiB/s wr, 215 op/s 2024-03-04T16:06:30.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:30 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:30.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:30 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:30.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:30 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T16:06:30.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:30 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:30.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:30 smithi152 ceph-mon[82845]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:30.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:30 smithi152 ceph-mon[82845]: pgmap v21: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 630 KiB/s wr, 215 op/s 2024-03-04T16:06:30.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:30 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:30.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:30 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:30.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:30 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T16:06:30.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:30 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:31.595 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:31 smithi116 ceph-mon[94164]: Reconfiguring crash.smithi116 (monmap changed)... 2024-03-04T16:06:31.595 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:31 smithi116 ceph-mon[94164]: Reconfiguring daemon crash.smithi116 on smithi116 2024-03-04T16:06:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:31 smithi152 ceph-mon[82845]: Reconfiguring crash.smithi116 (monmap changed)... 2024-03-04T16:06:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:31 smithi152 ceph-mon[82845]: Reconfiguring daemon crash.smithi116 on smithi116 2024-03-04T16:06:32.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:32 smithi116 ceph-mon[94164]: pgmap v22: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 10 KiB/s rd, 208 KiB/s wr, 109 op/s 2024-03-04T16:06:32.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:32.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:32.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T16:06:32.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:32 smithi152 ceph-mon[82845]: pgmap v22: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 10 KiB/s rd, 208 KiB/s wr, 109 op/s 2024-03-04T16:06:32.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:32.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:32.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T16:06:32.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:33.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:33 smithi152 ceph-mon[82845]: Reconfiguring osd.0 (monmap changed)... 2024-03-04T16:06:33.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:33 smithi152 ceph-mon[82845]: Reconfiguring daemon osd.0 on smithi116 2024-03-04T16:06:33.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:33 smithi116 ceph-mon[94164]: Reconfiguring osd.0 (monmap changed)... 2024-03-04T16:06:33.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:33 smithi116 ceph-mon[94164]: Reconfiguring daemon osd.0 on smithi116 2024-03-04T16:06:34.289 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:34 smithi116 ceph-mon[94164]: pgmap v23: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 10 KiB/s rd, 208 KiB/s wr, 93 op/s 2024-03-04T16:06:34.289 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:34 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:34.289 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:34 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:34.289 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:34 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T16:06:34.289 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:34 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:34.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:34 smithi152 ceph-mon[82845]: pgmap v23: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 10 KiB/s rd, 208 KiB/s wr, 93 op/s 2024-03-04T16:06:34.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:34 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:34 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:34 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T16:06:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:34 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:35 smithi116 ceph-mon[94164]: Reconfiguring osd.1 (monmap changed)... 2024-03-04T16:06:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:35 smithi116 ceph-mon[94164]: Reconfiguring daemon osd.1 on smithi116 2024-03-04T16:06:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:06:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:35.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:35 smithi152 ceph-mon[82845]: Reconfiguring osd.1 (monmap changed)... 2024-03-04T16:06:35.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:35 smithi152 ceph-mon[82845]: Reconfiguring daemon osd.1 on smithi116 2024-03-04T16:06:35.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:06:35.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:36.453 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:36 smithi116 ceph-mon[94164]: pgmap v24: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 10 KiB/s rd, 208 KiB/s wr, 93 op/s 2024-03-04T16:06:36.454 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:36 smithi116 ceph-mon[94164]: Reconfiguring osd.2 (monmap changed)... 2024-03-04T16:06:36.454 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:36 smithi116 ceph-mon[94164]: Reconfiguring daemon osd.2 on smithi116 2024-03-04T16:06:36.525 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:36 smithi152 ceph-mon[82845]: pgmap v24: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 10 KiB/s rd, 208 KiB/s wr, 93 op/s 2024-03-04T16:06:36.525 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:36 smithi152 ceph-mon[82845]: Reconfiguring osd.2 (monmap changed)... 2024-03-04T16:06:36.525 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:36 smithi152 ceph-mon[82845]: Reconfiguring daemon osd.2 on smithi116 2024-03-04T16:06:38.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:38.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:38.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:37 smithi152 ceph-mon[82845]: Reconfiguring mds.cephfs.smithi116.pjrmzv (monmap changed)... 2024-03-04T16:06:38.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.pjrmzv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:06:38.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:38.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:37 smithi152 ceph-mon[82845]: Reconfiguring daemon mds.cephfs.smithi116.pjrmzv on smithi116 2024-03-04T16:06:38.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:37 smithi152 ceph-mon[82845]: pgmap v25: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 85 B/s rd, 17 KiB/s wr, 0 op/s 2024-03-04T16:06:38.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:38.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:38.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:37 smithi116 ceph-mon[94164]: Reconfiguring mds.cephfs.smithi116.pjrmzv (monmap changed)... 2024-03-04T16:06:38.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.pjrmzv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:06:38.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:38.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:37 smithi116 ceph-mon[94164]: Reconfiguring daemon mds.cephfs.smithi116.pjrmzv on smithi116 2024-03-04T16:06:38.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:37 smithi116 ceph-mon[94164]: pgmap v25: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 85 B/s rd, 17 KiB/s wr, 0 op/s 2024-03-04T16:06:39.218 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:39.218 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:39.218 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:39.218 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.gkenwy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:06:39.218 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:39.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:39.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:39.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:39.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.gkenwy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:06:39.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:40.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:39 smithi116 ceph-mon[94164]: Reconfiguring mds.cephfs.smithi116.gkenwy (monmap changed)... 2024-03-04T16:06:40.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:39 smithi116 ceph-mon[94164]: Reconfiguring daemon mds.cephfs.smithi116.gkenwy on smithi116 2024-03-04T16:06:40.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:39 smithi116 ceph-mon[94164]: pgmap v26: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 85 B/s rd, 17 KiB/s wr, 0 op/s 2024-03-04T16:06:40.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:39 smithi152 ceph-mon[82845]: Reconfiguring mds.cephfs.smithi116.gkenwy (monmap changed)... 2024-03-04T16:06:40.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:39 smithi152 ceph-mon[82845]: Reconfiguring daemon mds.cephfs.smithi116.gkenwy on smithi116 2024-03-04T16:06:40.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:39 smithi152 ceph-mon[82845]: pgmap v26: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 85 B/s rd, 17 KiB/s wr, 0 op/s 2024-03-04T16:06:41.127 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:41.127 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:41.127 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:41 smithi116 ceph-mon[94164]: Reconfiguring ceph-exporter.smithi152 (monmap changed)... 2024-03-04T16:06:41.127 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:06:41.127 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:06:41.127 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-04T16:06:41.127 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:41 smithi116 ceph-mon[94164]: Unable to update caps for client.ceph-exporter.smithi152 2024-03-04T16:06:41.127 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi152"}]: dispatch 2024-03-04T16:06:41.127 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:41.127 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:41 smithi116 ceph-mon[94164]: Reconfiguring daemon ceph-exporter.smithi152 on smithi152 2024-03-04T16:06:41.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:41.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:41.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:41 smithi152 ceph-mon[82845]: Reconfiguring ceph-exporter.smithi152 (monmap changed)... 2024-03-04T16:06:41.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:06:41.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:06:41.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-04T16:06:41.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:41 smithi152 ceph-mon[82845]: Unable to update caps for client.ceph-exporter.smithi152 2024-03-04T16:06:41.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi152"}]: dispatch 2024-03-04T16:06:41.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:41.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:41 smithi152 ceph-mon[82845]: Reconfiguring daemon ceph-exporter.smithi152 on smithi152 2024-03-04T16:06:42.397 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:42 smithi116 ceph-mon[94164]: pgmap v27: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-04T16:06:42.397 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:42.397 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:42.397 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T16:06:42.397 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:42.401 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:42 smithi152 ceph-mon[82845]: pgmap v27: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-04T16:06:42.401 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:42.402 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:42.402 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T16:06:42.402 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:43.310 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:43 smithi152 ceph-mon[82845]: Reconfiguring crash.smithi152 (monmap changed)... 2024-03-04T16:06:43.310 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:43 smithi152 ceph-mon[82845]: Reconfiguring daemon crash.smithi152 on smithi152 2024-03-04T16:06:43.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:43 smithi116 ceph-mon[94164]: Reconfiguring crash.smithi152 (monmap changed)... 2024-03-04T16:06:43.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:43 smithi116 ceph-mon[94164]: Reconfiguring daemon crash.smithi152 on smithi152 2024-03-04T16:06:44.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:44 smithi152 ceph-mon[82845]: pgmap v28: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-04T16:06:44.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:44 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:44.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:44 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:44.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:44 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.aawieo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:06:44.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:44 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:06:44.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:44 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:44 smithi116 ceph-mon[94164]: pgmap v28: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-04T16:06:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:44 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:44 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:44 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.aawieo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:06:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:44 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:06:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:44 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:45.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:45 smithi152 ceph-mon[82845]: Reconfiguring mgr.smithi152.aawieo (monmap changed)... 2024-03-04T16:06:45.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:45 smithi152 ceph-mon[82845]: Reconfiguring daemon mgr.smithi152.aawieo on smithi152 2024-03-04T16:06:45.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:45 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:45.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:45 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:45.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:45 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:06:45.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:45 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:06:45.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:45 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:45 smithi116 ceph-mon[94164]: Reconfiguring mgr.smithi152.aawieo (monmap changed)... 2024-03-04T16:06:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:45 smithi116 ceph-mon[94164]: Reconfiguring daemon mgr.smithi152.aawieo on smithi152 2024-03-04T16:06:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:45 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:45.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:45 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:45.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:45 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:06:45.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:45 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:06:45.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:45 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:46.600 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:46 smithi152 ceph-mon[82845]: Reconfiguring mon.smithi152 (monmap changed)... 2024-03-04T16:06:46.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:46 smithi152 ceph-mon[82845]: Reconfiguring daemon mon.smithi152 on smithi152 2024-03-04T16:06:46.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:46 smithi152 ceph-mon[82845]: pgmap v29: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-04T16:06:46.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:46 smithi116 ceph-mon[94164]: Reconfiguring mon.smithi152 (monmap changed)... 2024-03-04T16:06:46.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:46 smithi116 ceph-mon[94164]: Reconfiguring daemon mon.smithi152 on smithi152 2024-03-04T16:06:46.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:46 smithi116 ceph-mon[94164]: pgmap v29: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-04T16:06:47.768 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:47 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:47.768 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:47 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:47.768 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:47 smithi152 ceph-mon[82845]: Reconfiguring osd.3 (monmap changed)... 2024-03-04T16:06:47.768 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:47 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:06:47.769 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:47 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:47.769 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:47 smithi152 ceph-mon[82845]: Reconfiguring daemon osd.3 on smithi152 2024-03-04T16:06:47.769 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:47 smithi152 ceph-mon[82845]: pgmap v30: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:47.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:47 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:47.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:47 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:47.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:47 smithi116 ceph-mon[94164]: Reconfiguring osd.3 (monmap changed)... 2024-03-04T16:06:47.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:47 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:06:47.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:47 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:47.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:47 smithi116 ceph-mon[94164]: Reconfiguring daemon osd.3 on smithi152 2024-03-04T16:06:47.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:47 smithi116 ceph-mon[94164]: pgmap v30: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:49.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:49.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:49.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:06:49.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:50.634 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:50 smithi152 ceph-mon[82845]: Reconfiguring osd.4 (monmap changed)... 2024-03-04T16:06:50.634 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:50 smithi152 ceph-mon[82845]: Reconfiguring daemon osd.4 on smithi152 2024-03-04T16:06:50.634 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:50 smithi152 ceph-mon[82845]: pgmap v31: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:50.634 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:50 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:50.634 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:50 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:50.634 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:50 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:06:50.634 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:50 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:50 smithi116 ceph-mon[94164]: Reconfiguring osd.4 (monmap changed)... 2024-03-04T16:06:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:50 smithi116 ceph-mon[94164]: Reconfiguring daemon osd.4 on smithi152 2024-03-04T16:06:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:50 smithi116 ceph-mon[94164]: pgmap v31: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:50 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:50 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:50 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:06:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:50 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:51.655 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:51 smithi152 ceph-mon[82845]: Reconfiguring osd.5 (monmap changed)... 2024-03-04T16:06:51.655 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:51 smithi152 ceph-mon[82845]: Reconfiguring daemon osd.5 on smithi152 2024-03-04T16:06:51.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:51 smithi116 ceph-mon[94164]: Reconfiguring osd.5 (monmap changed)... 2024-03-04T16:06:51.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:51 smithi116 ceph-mon[94164]: Reconfiguring daemon osd.5 on smithi152 2024-03-04T16:06:52.546 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:52 smithi152 ceph-mon[82845]: pgmap v32: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:52.546 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:52.546 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:52.546 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:52 smithi152 ceph-mon[82845]: Reconfiguring mds.cephfs.smithi152.tlpshy (monmap changed)... 2024-03-04T16:06:52.547 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.tlpshy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:06:52.547 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:52.547 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:52 smithi152 ceph-mon[82845]: Reconfiguring daemon mds.cephfs.smithi152.tlpshy on smithi152 2024-03-04T16:06:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:52 smithi116 ceph-mon[94164]: pgmap v32: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:52 smithi116 ceph-mon[94164]: Reconfiguring mds.cephfs.smithi152.tlpshy (monmap changed)... 2024-03-04T16:06:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.tlpshy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:06:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:52 smithi116 ceph-mon[94164]: Reconfiguring daemon mds.cephfs.smithi152.tlpshy on smithi152 2024-03-04T16:06:53.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:53 smithi152 ceph-mon[82845]: pgmap v33: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:53 smithi152 ceph-mon[82845]: Reconfiguring mds.cephfs.smithi152.kqoybf (monmap changed)... 2024-03-04T16:06:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.kqoybf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:06:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:53 smithi152 ceph-mon[82845]: Reconfiguring daemon mds.cephfs.smithi152.kqoybf on smithi152 2024-03-04T16:06:53.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:53 smithi116 ceph-mon[94164]: pgmap v33: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:53.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:53.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:53.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:53.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:53 smithi116 ceph-mon[94164]: Reconfiguring mds.cephfs.smithi152.kqoybf (monmap changed)... 2024-03-04T16:06:53.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.kqoybf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:06:53.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:53.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:53 smithi116 ceph-mon[94164]: Reconfiguring daemon mds.cephfs.smithi152.kqoybf on smithi152 2024-03-04T16:06:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: pgmap v34: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: Upgrade: Setting container_image for all mon 2024-03-04T16:06:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi116"}]: dispatch 2024-03-04T16:06:56.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi116"}]': finished 2024-03-04T16:06:56.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi152"}]: dispatch 2024-03-04T16:06:56.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi152"}]': finished 2024-03-04T16:06:56.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:56.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: pgmap v34: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: Upgrade: Setting container_image for all mon 2024-03-04T16:06:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi116"}]: dispatch 2024-03-04T16:06:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi116"}]': finished 2024-03-04T16:06:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi152"}]: dispatch 2024-03-04T16:06:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi152"}]': finished 2024-03-04T16:06:57.846 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:57 smithi116 ceph-mon[94164]: Upgrade: Updating crash.smithi116 (1/2) 2024-03-04T16:06:57.846 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:57.846 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T16:06:57.846 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:57.846 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:57 smithi116 ceph-mon[94164]: Deploying daemon crash.smithi116 on smithi116 2024-03-04T16:06:57.846 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:06:57 smithi116 ceph-mon[94164]: pgmap v35: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:58.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:57 smithi152 ceph-mon[82845]: Upgrade: Updating crash.smithi116 (1/2) 2024-03-04T16:06:58.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:06:58.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T16:06:58.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:58.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:57 smithi152 ceph-mon[82845]: Deploying daemon crash.smithi116 on smithi116 2024-03-04T16:06:58.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:06:57 smithi152 ceph-mon[82845]: pgmap v35: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:06:59.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.119+0000 7f546bc06700 1 -- 172.21.15.116:0/1606739092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54641016a0 msgr2=0x7f5464103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:59.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.119+0000 7f546bc06700 1 --2- 172.21.15.116:0/1606739092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54641016a0 0x7f5464103a90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5460009910 tx=0x7f5460009c20 comp rx=0 tx=0).stop 2024-03-04T16:06:59.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.119+0000 7f546bc06700 1 -- 172.21.15.116:0/1606739092 shutdown_connections 2024-03-04T16:06:59.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.119+0000 7f546bc06700 1 --2- 172.21.15.116:0/1606739092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5464103fd0 0x7f54641063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.119+0000 7f546bc06700 1 --2- 172.21.15.116:0/1606739092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54641016a0 0x7f5464103a90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.119+0000 7f546bc06700 1 -- 172.21.15.116:0/1606739092 >> 172.21.15.116:0/1606739092 conn(0x7f54640fb050 msgr2=0x7f54640fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:59.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.119+0000 7f546bc06700 1 -- 172.21.15.116:0/1606739092 shutdown_connections 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.119+0000 7f546bc06700 1 -- 172.21.15.116:0/1606739092 wait complete. 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.119+0000 7f546bc06700 1 Processor -- start 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.119+0000 7f546bc06700 1 -- start start 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f546bc06700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54641016a0 0x7f5464196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f546bc06700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5464103fd0 0x7f5464196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f546bc06700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464196e80 con 0x7f5464103fd0 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f546bc06700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464196fc0 con 0x7f54641016a0 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f54691a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5464103fd0 0x7f5464196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f54691a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5464103fd0 0x7f5464196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34656/0 (socket says 172.21.15.116:34656) 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f54691a1700 1 -- 172.21.15.116:0/2109594715 learned_addr learned my addr 172.21.15.116:0/2109594715 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f54699a2700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54641016a0 0x7f5464196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f54691a1700 1 -- 172.21.15.116:0/2109594715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54641016a0 msgr2=0x7f5464196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f54691a1700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54641016a0 0x7f5464196320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f54691a1700 1 -- 172.21.15.116:0/2109594715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54600095f0 con 0x7f5464103fd0 2024-03-04T16:06:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f54691a1700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5464103fd0 0x7f5464196860 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f545400c8a0 tx=0x7f545400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f545affd700 1 -- 172.21.15.116:0/2109594715 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f545400cea0 con 0x7f5464103fd0 2024-03-04T16:06:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f546bc06700 1 -- 172.21.15.116:0/2109594715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546419ba70 con 0x7f5464103fd0 2024-03-04T16:06:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f545affd700 1 -- 172.21.15.116:0/2109594715 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5454004830 con 0x7f5464103fd0 2024-03-04T16:06:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f545affd700 1 -- 172.21.15.116:0/2109594715 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54540055b0 con 0x7f5464103fd0 2024-03-04T16:06:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.120+0000 7f546bc06700 1 -- 172.21.15.116:0/2109594715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546419bfc0 con 0x7f5464103fd0 2024-03-04T16:06:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.121+0000 7f546bc06700 1 -- 172.21.15.116:0/2109594715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5464190520 con 0x7f5464103fd0 2024-03-04T16:06:59.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.122+0000 7f545affd700 1 -- 172.21.15.116:0/2109594715 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7f5454007ad0 con 0x7f5464103fd0 2024-03-04T16:06:59.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.122+0000 7f545affd700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f545006d9e0 0x7f545006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:59.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.123+0000 7f54699a2700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f545006d9e0 0x7f545006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:59.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.123+0000 7f545affd700 1 -- 172.21.15.116:0/2109594715 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f545408d480 con 0x7f5464103fd0 2024-03-04T16:06:59.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.123+0000 7f54699a2700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f545006d9e0 0x7f545006fea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f54600059d0 tx=0x7f5460005920 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:59.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.127+0000 7f545affd700 1 -- 172.21.15.116:0/2109594715 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f545405a590 con 0x7f5464103fd0 2024-03-04T16:06:59.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.350+0000 7f546bc06700 1 -- 172.21.15.116:0/2109594715 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f54640611d0 con 0x7f545006d9e0 2024-03-04T16:06:59.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.352+0000 7f545affd700 1 -- 172.21.15.116:0/2109594715 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f54640611d0 con 0x7f545006d9e0 2024-03-04T16:06:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.354+0000 7f546bc06700 1 -- 172.21.15.116:0/2109594715 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f545006d9e0 msgr2=0x7f545006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.354+0000 7f546bc06700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f545006d9e0 0x7f545006fea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f54600059d0 tx=0x7f5460005920 comp rx=0 tx=0).stop 2024-03-04T16:06:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.354+0000 7f546bc06700 1 -- 172.21.15.116:0/2109594715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5464103fd0 msgr2=0x7f5464196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:59.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.354+0000 7f546bc06700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5464103fd0 0x7f5464196860 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f545400c8a0 tx=0x7f545400cc60 comp rx=0 tx=0).stop 2024-03-04T16:06:59.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.354+0000 7f546bc06700 1 -- 172.21.15.116:0/2109594715 shutdown_connections 2024-03-04T16:06:59.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.354+0000 7f546bc06700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f545006d9e0 0x7f545006fea0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.354+0000 7f546bc06700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54641016a0 0x7f5464196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.354+0000 7f546bc06700 1 --2- 172.21.15.116:0/2109594715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5464103fd0 0x7f5464196860 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.354+0000 7f546bc06700 1 -- 172.21.15.116:0/2109594715 >> 172.21.15.116:0/2109594715 conn(0x7f54640fb050 msgr2=0x7f54640fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:59.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.355+0000 7f546bc06700 1 -- 172.21.15.116:0/2109594715 shutdown_connections 2024-03-04T16:06:59.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.355+0000 7f546bc06700 1 -- 172.21.15.116:0/2109594715 wait complete. 2024-03-04T16:06:59.369 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:06:59.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.468+0000 7fcdba234700 1 -- 172.21.15.116:0/2488689329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb4101640 msgr2=0x7fcdb4103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.468+0000 7fcdba234700 1 --2- 172.21.15.116:0/2488689329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb4101640 0x7fcdb4103a30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009910 tx=0x7fcda4009c20 comp rx=0 tx=0).stop 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.468+0000 7fcdba234700 1 -- 172.21.15.116:0/2488689329 shutdown_connections 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.468+0000 7fcdba234700 1 --2- 172.21.15.116:0/2488689329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdb4103f70 0x7fcdb4106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.468+0000 7fcdba234700 1 --2- 172.21.15.116:0/2488689329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb4101640 0x7fcdb4103a30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.468+0000 7fcdba234700 1 -- 172.21.15.116:0/2488689329 >> 172.21.15.116:0/2488689329 conn(0x7fcdb40fb030 msgr2=0x7fcdb40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.468+0000 7fcdba234700 1 -- 172.21.15.116:0/2488689329 shutdown_connections 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.468+0000 7fcdba234700 1 -- 172.21.15.116:0/2488689329 wait complete. 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.468+0000 7fcdba234700 1 Processor -- start 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.468+0000 7fcdba234700 1 -- start start 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdba234700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdb4101640 0x7fcdb4100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdba234700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb4103f70 0x7fcdb40ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdba234700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb40ffb30 con 0x7fcdb4103f70 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdba234700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb40ffc70 con 0x7fcdb4101640 2024-03-04T16:06:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb4103f70 0x7fcdb40ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb4103f70 0x7fcdb40ff5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34678/0 (socket says 172.21.15.116:34678) 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb2ffd700 1 -- 172.21.15.116:0/4232105098 learned_addr learned my addr 172.21.15.116:0/4232105098 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb37fe700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdb4101640 0x7fcdb4100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb2ffd700 1 -- 172.21.15.116:0/4232105098 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdb4101640 msgr2=0x7fcdb4100fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb2ffd700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdb4101640 0x7fcdb4100fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb2ffd700 1 -- 172.21.15.116:0/4232105098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda40095f0 con 0x7fcdb4103f70 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb2ffd700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb4103f70 0x7fcdb40ff5f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcda800e920 tx=0x7fcda800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb0ff9700 1 -- 172.21.15.116:0/4232105098 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcda800c9d0 con 0x7fcdb4103f70 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdba234700 1 -- 172.21.15.116:0/4232105098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb40fff50 con 0x7fcdb4103f70 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb0ff9700 1 -- 172.21.15.116:0/4232105098 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcda8010430 con 0x7fcdb4103f70 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdb0ff9700 1 -- 172.21.15.116:0/4232105098 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcda8018650 con 0x7fcdb4103f70 2024-03-04T16:06:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.469+0000 7fcdba234700 1 -- 172.21.15.116:0/4232105098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb419e360 con 0x7fcdb4103f70 2024-03-04T16:06:59.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.470+0000 7fcdba234700 1 -- 172.21.15.116:0/4232105098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb4190520 con 0x7fcdb4103f70 2024-03-04T16:06:59.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.471+0000 7fcdb0ff9700 1 -- 172.21.15.116:0/4232105098 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7fcda80105a0 con 0x7fcdb4103f70 2024-03-04T16:06:59.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.471+0000 7fcdb0ff9700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcd9c06d9e0 0x7fcd9c06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:59.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.471+0000 7fcdb37fe700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcd9c06d9e0 0x7fcd9c06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:59.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.471+0000 7fcdb0ff9700 1 -- 172.21.15.116:0/4232105098 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcda8014070 con 0x7fcdb4103f70 2024-03-04T16:06:59.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.472+0000 7fcdb37fe700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcd9c06d9e0 0x7fcd9c06fea0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009fd0 tx=0x7fcda4005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:59.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.476+0000 7fcdb0ff9700 1 -- 172.21.15.116:0/4232105098 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcda8052d20 con 0x7fcdb4103f70 2024-03-04T16:06:59.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.706+0000 7fcdba234700 1 -- 172.21.15.116:0/4232105098 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdb40611d0 con 0x7fcd9c06d9e0 2024-03-04T16:06:59.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.707+0000 7fcdb0ff9700 1 -- 172.21.15.116:0/4232105098 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7fcdb40611d0 con 0x7fcd9c06d9e0 2024-03-04T16:06:59.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 -- 172.21.15.116:0/4232105098 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcd9c06d9e0 msgr2=0x7fcd9c06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:59.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcd9c06d9e0 0x7fcd9c06fea0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009fd0 tx=0x7fcda4005940 comp rx=0 tx=0).stop 2024-03-04T16:06:59.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 -- 172.21.15.116:0/4232105098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb4103f70 msgr2=0x7fcdb40ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:59.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb4103f70 0x7fcdb40ff5f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcda800e920 tx=0x7fcda800ec30 comp rx=0 tx=0).stop 2024-03-04T16:06:59.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 -- 172.21.15.116:0/4232105098 shutdown_connections 2024-03-04T16:06:59.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcd9c06d9e0 0x7fcd9c06fea0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcdb4101640 0x7fcdb4100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 --2- 172.21.15.116:0/4232105098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb4103f70 0x7fcdb40ff5f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 -- 172.21.15.116:0/4232105098 >> 172.21.15.116:0/4232105098 conn(0x7fcdb40fb030 msgr2=0x7fcdb40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:59.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 -- 172.21.15.116:0/4232105098 shutdown_connections 2024-03-04T16:06:59.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.710+0000 7fcdba234700 1 -- 172.21.15.116:0/4232105098 wait complete. 2024-03-04T16:06:59.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.825+0000 7fd2f63f1700 1 -- 172.21.15.116:0/2643092350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2f01016e0 msgr2=0x7fd2f0103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:59.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.825+0000 7fd2f63f1700 1 --2- 172.21.15.116:0/2643092350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2f01016e0 0x7fd2f0103ad0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0009910 tx=0x7fd2e0009c20 comp rx=0 tx=0).stop 2024-03-04T16:06:59.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.825+0000 7fd2f63f1700 1 -- 172.21.15.116:0/2643092350 shutdown_connections 2024-03-04T16:06:59.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.825+0000 7fd2f63f1700 1 --2- 172.21.15.116:0/2643092350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2f0104010 0x7fd2f0106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.825+0000 7fd2f63f1700 1 --2- 172.21.15.116:0/2643092350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2f01016e0 0x7fd2f0103ad0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.825+0000 7fd2f63f1700 1 -- 172.21.15.116:0/2643092350 >> 172.21.15.116:0/2643092350 conn(0x7fd2f00fb030 msgr2=0x7fd2f00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:06:59.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.825+0000 7fd2f63f1700 1 -- 172.21.15.116:0/2643092350 shutdown_connections 2024-03-04T16:06:59.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.825+0000 7fd2f63f1700 1 -- 172.21.15.116:0/2643092350 wait complete. 2024-03-04T16:06:59.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.825+0000 7fd2f63f1700 1 Processor -- start 2024-03-04T16:06:59.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2f63f1700 1 -- start start 2024-03-04T16:06:59.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2f63f1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2f01016e0 0x7fd2f0196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:59.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2f63f1700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2f0104010 0x7fd2f0196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:59.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2f63f1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f0196e80 con 0x7fd2f01016e0 2024-03-04T16:06:59.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2f63f1700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f0196fc0 con 0x7fd2f0104010 2024-03-04T16:06:59.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2effff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2f01016e0 0x7fd2f0196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:59.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2effff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2f01016e0 0x7fd2f0196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34702/0 (socket says 172.21.15.116:34702) 2024-03-04T16:06:59.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2effff700 1 -- 172.21.15.116:0/288843042 learned_addr learned my addr 172.21.15.116:0/288843042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:06:59.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2ef7fe700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2f0104010 0x7fd2f0196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:59.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2effff700 1 -- 172.21.15.116:0/288843042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2f0104010 msgr2=0x7fd2f0196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:06:59.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2effff700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2f0104010 0x7fd2f0196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:06:59.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2effff700 1 -- 172.21.15.116:0/288843042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e00095f0 con 0x7fd2f01016e0 2024-03-04T16:06:59.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.826+0000 7fd2effff700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2f01016e0 0x7fd2f0196320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0006010 tx=0x7fd2e000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:59.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.827+0000 7fd2ed7fa700 1 -- 172.21.15.116:0/288843042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2e001d070 con 0x7fd2f01016e0 2024-03-04T16:06:59.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.827+0000 7fd2ed7fa700 1 -- 172.21.15.116:0/288843042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd2e000f460 con 0x7fd2f01016e0 2024-03-04T16:06:59.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.827+0000 7fd2f63f1700 1 -- 172.21.15.116:0/288843042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f019ba10 con 0x7fd2f01016e0 2024-03-04T16:06:59.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.827+0000 7fd2ed7fa700 1 -- 172.21.15.116:0/288843042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2e0021640 con 0x7fd2f01016e0 2024-03-04T16:06:59.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.827+0000 7fd2f63f1700 1 -- 172.21.15.116:0/288843042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f019bf00 con 0x7fd2f01016e0 2024-03-04T16:06:59.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.828+0000 7fd2f63f1700 1 -- 172.21.15.116:0/288843042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f0190530 con 0x7fd2f01016e0 2024-03-04T16:06:59.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.828+0000 7fd2ed7fa700 1 -- 172.21.15.116:0/288843042 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7fd2e000fab0 con 0x7fd2f01016e0 2024-03-04T16:06:59.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.829+0000 7fd2ed7fa700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd2d806d9e0 0x7fd2d806fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:06:59.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.829+0000 7fd2ed7fa700 1 -- 172.21.15.116:0/288843042 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd2e008ea90 con 0x7fd2f01016e0 2024-03-04T16:06:59.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.829+0000 7fd2ef7fe700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd2d806d9e0 0x7fd2d806fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:06:59.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.829+0000 7fd2ef7fe700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd2d806d9e0 0x7fd2d806fea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4005950 tx=0x7fd2e400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:06:59.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:06:59.833+0000 7fd2ed7fa700 1 -- 172.21.15.116:0/288843042 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd2e005bba0 con 0x7fd2f01016e0 2024-03-04T16:07:00.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.047+0000 7fd2f63f1700 1 -- 172.21.15.116:0/288843042 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2f00611d0 con 0x7fd2d806d9e0 2024-03-04T16:07:00.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.058+0000 7fd2ed7fa700 1 -- 172.21.15.116:0/288843042 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fd2f00611d0 con 0x7fd2d806d9e0 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (10m) 60s ago 11m 25.4M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (11m) 60s ago 11m 7750k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (10m) 41s ago 10m 7772k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (11m) 60s ago 11m 6669k - 18.2.1 5be31c24972a 64f1052c51ad 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (10m) 41s ago 10m 6656k - 18.2.1 5be31c24972a 5a5ee5348c1c 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (9m) 60s ago 11m 86.9M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (5m) 60s ago 5m 18.4M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (5m) 60s ago 5m 14.9M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (5m) 41s ago 5m 16.4M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (5m) 41s ago 5m 153M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (2m) 60s ago 13m 473M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:07:00.059 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (107s) 41s ago 10m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (76s) 60s ago 13m 28.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (46s) 41s ago 10m 22.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (11m) 60s ago 11m 15.9M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (10m) 41s ago 10m 16.5M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (9m) 60s ago 9m 213M 1216M 18.2.1 5be31c24972a 946b46d01490 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (9m) 60s ago 9m 223M 1216M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (8m) 60s ago 8m 183M 1216M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (8m) 41s ago 8m 267M 2240M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (7m) 41s ago 7m 236M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (7m) 41s ago 7m 228M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:07:00.060 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (114s) 60s ago 11m 47.1M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:07:00.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.060+0000 7fd2f63f1700 1 -- 172.21.15.116:0/288843042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd2d806d9e0 msgr2=0x7fd2d806fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.060+0000 7fd2f63f1700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd2d806d9e0 0x7fd2d806fea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4005950 tx=0x7fd2e400b410 comp rx=0 tx=0).stop 2024-03-04T16:07:00.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.060+0000 7fd2f63f1700 1 -- 172.21.15.116:0/288843042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2f01016e0 msgr2=0x7fd2f0196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.060+0000 7fd2f63f1700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2f01016e0 0x7fd2f0196320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0006010 tx=0x7fd2e000bd70 comp rx=0 tx=0).stop 2024-03-04T16:07:00.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.060+0000 7fd2f63f1700 1 -- 172.21.15.116:0/288843042 shutdown_connections 2024-03-04T16:07:00.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.060+0000 7fd2f63f1700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd2d806d9e0 0x7fd2d806fea0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.060+0000 7fd2f63f1700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2f01016e0 0x7fd2f0196320 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.060+0000 7fd2f63f1700 1 --2- 172.21.15.116:0/288843042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2f0104010 0x7fd2f0196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.060+0000 7fd2f63f1700 1 -- 172.21.15.116:0/288843042 >> 172.21.15.116:0/288843042 conn(0x7fd2f00fb030 msgr2=0x7fd2f00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:00.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.060+0000 7fd2f63f1700 1 -- 172.21.15.116:0/288843042 shutdown_connections 2024-03-04T16:07:00.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.061+0000 7fd2f63f1700 1 -- 172.21.15.116:0/288843042 wait complete. 2024-03-04T16:07:00.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.168+0000 7f90ad480700 1 -- 172.21.15.116:0/3135251046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a80ffaf0 msgr2=0x7f90a80fff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.168+0000 7f90ad480700 1 --2- 172.21.15.116:0/3135251046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a80ffaf0 0x7f90a80fff10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f909c009910 tx=0x7f909c009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:00.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.168+0000 7f90ad480700 1 -- 172.21.15.116:0/3135251046 shutdown_connections 2024-03-04T16:07:00.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.168+0000 7f90ad480700 1 --2- 172.21.15.116:0/3135251046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90a8100cf0 0x7f90a8101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.168+0000 7f90ad480700 1 --2- 172.21.15.116:0/3135251046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a80ffaf0 0x7f90a80fff10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.168+0000 7f90ad480700 1 -- 172.21.15.116:0/3135251046 >> 172.21.15.116:0/3135251046 conn(0x7f90a80fb070 msgr2=0x7f90a80fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:00.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.168+0000 7f90ad480700 1 -- 172.21.15.116:0/3135251046 shutdown_connections 2024-03-04T16:07:00.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.168+0000 7f90ad480700 1 -- 172.21.15.116:0/3135251046 wait complete. 2024-03-04T16:07:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90ad480700 1 Processor -- start 2024-03-04T16:07:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90ad480700 1 -- start start 2024-03-04T16:07:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90ad480700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a80ffaf0 0x7f90a80698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90ad480700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90a8100cf0 0x7f90a819b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90ad480700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a819b6f0 con 0x7f90a80ffaf0 2024-03-04T16:07:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90ad480700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a819b830 con 0x7f90a8100cf0 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90a6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a80ffaf0 0x7f90a80698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90a6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a80ffaf0 0x7f90a80698f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34724/0 (socket says 172.21.15.116:34724) 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90a6ffd700 1 -- 172.21.15.116:0/422346540 learned_addr learned my addr 172.21.15.116:0/422346540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90a67fc700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90a8100cf0 0x7f90a819b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90a6ffd700 1 -- 172.21.15.116:0/422346540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90a8100cf0 msgr2=0x7f90a819b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90a6ffd700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90a8100cf0 0x7f90a819b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90a6ffd700 1 -- 172.21.15.116:0/422346540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c0095f0 con 0x7f90a80ffaf0 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.169+0000 7f90a67fc700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90a8100cf0 0x7f90a819b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.170+0000 7f90a6ffd700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a80ffaf0 0x7f90a80698f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f909c00bda0 tx=0x7f909c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.170+0000 7f908ffff700 1 -- 172.21.15.116:0/422346540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f909c01d070 con 0x7f90a80ffaf0 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.170+0000 7f908ffff700 1 -- 172.21.15.116:0/422346540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f909c00f460 con 0x7f90a80ffaf0 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.170+0000 7f90ad480700 1 -- 172.21.15.116:0/422346540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a819bab0 con 0x7f90a80ffaf0 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.170+0000 7f908ffff700 1 -- 172.21.15.116:0/422346540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f909c021640 con 0x7f90a80ffaf0 2024-03-04T16:07:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.170+0000 7f90ad480700 1 -- 172.21.15.116:0/422346540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a819bfa0 con 0x7f90a80ffaf0 2024-03-04T16:07:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.171+0000 7f90ad480700 1 -- 172.21.15.116:0/422346540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a8108650 con 0x7f90a80ffaf0 2024-03-04T16:07:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.172+0000 7f908ffff700 1 -- 172.21.15.116:0/422346540 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7f909c00f5d0 con 0x7f90a80ffaf0 2024-03-04T16:07:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.172+0000 7f908ffff700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f909006d9e0 0x7f909006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.172+0000 7f908ffff700 1 -- 172.21.15.116:0/422346540 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f909c08e9a0 con 0x7f90a80ffaf0 2024-03-04T16:07:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.172+0000 7f90a67fc700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f909006d9e0 0x7f909006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.173+0000 7f90a67fc700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f909006d9e0 0x7f909006fea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9098005d90 tx=0x7f9098005d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:00.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.178+0000 7f908ffff700 1 -- 172.21.15.116:0/422346540 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f909c05bab0 con 0x7f90a80ffaf0 2024-03-04T16:07:00.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:00 smithi116 ceph-mon[94164]: pgmap v36: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:00.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:00 smithi116 ceph-mon[94164]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:00.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:00 smithi152 ceph-mon[82845]: pgmap v36: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:00.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:00 smithi152 ceph-mon[82845]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.475+0000 7f90ad480700 1 -- 172.21.15.116:0/422346540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f90a819c6a0 con 0x7f90a80ffaf0 2024-03-04T16:07:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.476+0000 7f908ffff700 1 -- 172.21.15.116:0/422346540 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f909c05b640 con 0x7f90a80ffaf0 2024-03-04T16:07:00.477 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:07:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:07:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:07:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:07:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:07:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:07:00.478 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:07:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 -- 172.21.15.116:0/422346540 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f909006d9e0 msgr2=0x7f909006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f909006d9e0 0x7f909006fea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9098005d90 tx=0x7f9098005d20 comp rx=0 tx=0).stop 2024-03-04T16:07:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 -- 172.21.15.116:0/422346540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a80ffaf0 msgr2=0x7f90a80698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a80ffaf0 0x7f90a80698f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f909c00bda0 tx=0x7f909c00be80 comp rx=0 tx=0).stop 2024-03-04T16:07:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 -- 172.21.15.116:0/422346540 shutdown_connections 2024-03-04T16:07:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f909006d9e0 0x7f909006fea0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a80ffaf0 0x7f90a80698f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 --2- 172.21.15.116:0/422346540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90a8100cf0 0x7f90a819b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 -- 172.21.15.116:0/422346540 >> 172.21.15.116:0/422346540 conn(0x7f90a80fb070 msgr2=0x7f90a8103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 -- 172.21.15.116:0/422346540 shutdown_connections 2024-03-04T16:07:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.478+0000 7f90ad480700 1 -- 172.21.15.116:0/422346540 wait complete. 2024-03-04T16:07:00.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.583+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/3332647096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3f4100cc0 msgr2=0x7fd3f4101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.583+0000 7fd3f9bf6700 1 --2- 172.21.15.116:0/3332647096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3f4100cc0 0x7fd3f4101120 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8009910 tx=0x7fd3e8009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:00.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.583+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/3332647096 shutdown_connections 2024-03-04T16:07:00.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.583+0000 7fd3f9bf6700 1 --2- 172.21.15.116:0/3332647096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3f4100cc0 0x7fd3f4101120 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.583+0000 7fd3f9bf6700 1 --2- 172.21.15.116:0/3332647096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f40ffac0 0x7fd3f40ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.583+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/3332647096 >> 172.21.15.116:0/3332647096 conn(0x7fd3f40fb060 msgr2=0x7fd3f40fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:00.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.583+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/3332647096 shutdown_connections 2024-03-04T16:07:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.583+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/3332647096 wait complete. 2024-03-04T16:07:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.583+0000 7fd3f9bf6700 1 Processor -- start 2024-03-04T16:07:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.583+0000 7fd3f9bf6700 1 -- start start 2024-03-04T16:07:00.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f9bf6700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f40ffac0 0x7fd3f4069910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:00.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f9bf6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3f4100cc0 0x7fd3f419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:00.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f9bf6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f419b6a0 con 0x7fd3f4100cc0 2024-03-04T16:07:00.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f9bf6700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f419b7e0 con 0x7fd3f40ffac0 2024-03-04T16:07:00.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3f4100cc0 0x7fd3f419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:00.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3f4100cc0 0x7fd3f419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34756/0 (socket says 172.21.15.116:34756) 2024-03-04T16:07:00.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f2ffd700 1 -- 172.21.15.116:0/910024734 learned_addr learned my addr 172.21.15.116:0/910024734 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:00.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f37fe700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f40ffac0 0x7fd3f4069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:00.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f2ffd700 1 -- 172.21.15.116:0/910024734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f40ffac0 msgr2=0x7fd3f4069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f2ffd700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f40ffac0 0x7fd3f4069910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f2ffd700 1 -- 172.21.15.116:0/910024734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e80095f0 con 0x7fd3f4100cc0 2024-03-04T16:07:00.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f2ffd700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3f4100cc0 0x7fd3f419b0d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8000c00 tx=0x7fd3e800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:00.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f0ff9700 1 -- 172.21.15.116:0/910024734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3e801d070 con 0x7fd3f4100cc0 2024-03-04T16:07:00.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.584+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/910024734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f419ba60 con 0x7fd3f4100cc0 2024-03-04T16:07:00.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.585+0000 7fd3f0ff9700 1 -- 172.21.15.116:0/910024734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd3e8005dc0 con 0x7fd3f4100cc0 2024-03-04T16:07:00.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.585+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/910024734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f419bf50 con 0x7fd3f4100cc0 2024-03-04T16:07:00.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.585+0000 7fd3f0ff9700 1 -- 172.21.15.116:0/910024734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3e80226a0 con 0x7fd3f4100cc0 2024-03-04T16:07:00.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.585+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/910024734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f4108620 con 0x7fd3f4100cc0 2024-03-04T16:07:00.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.586+0000 7fd3f0ff9700 1 -- 172.21.15.116:0/910024734 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7fd3e800faf0 con 0x7fd3f4100cc0 2024-03-04T16:07:00.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.586+0000 7fd3f0ff9700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3dc06d9e0 0x7fd3dc06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:00.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.586+0000 7fd3f0ff9700 1 -- 172.21.15.116:0/910024734 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd3e808ebc0 con 0x7fd3f4100cc0 2024-03-04T16:07:00.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.586+0000 7fd3f37fe700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3dc06d9e0 0x7fd3dc06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:00.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.587+0000 7fd3f37fe700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3dc06d9e0 0x7fd3dc06fea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4100b20 tx=0x7fd3e4009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:00.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.591+0000 7fd3f0ff9700 1 -- 172.21.15.116:0/910024734 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd3e805bd50 con 0x7fd3f4100cc0 2024-03-04T16:07:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.831+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/910024734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd3f4108840 con 0x7fd3f4100cc0 2024-03-04T16:07:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.831+0000 7fd3f0ff9700 1 -- 172.21.15.116:0/910024734 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1852 (secure 0 0 0) 0x7fd3e805b8e0 con 0x7fd3f4100cc0 2024-03-04T16:07:00.832 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:07:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:07:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:07:00.835 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:07:00.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.833+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/910024734 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3dc06d9e0 msgr2=0x7fd3dc06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.834+0000 7fd3f9bf6700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3dc06d9e0 0x7fd3dc06fea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4100b20 tx=0x7fd3e4009260 comp rx=0 tx=0).stop 2024-03-04T16:07:00.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.834+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/910024734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3f4100cc0 msgr2=0x7fd3f419b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.834+0000 7fd3f9bf6700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3f4100cc0 0x7fd3f419b0d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8000c00 tx=0x7fd3e800f690 comp rx=0 tx=0).stop 2024-03-04T16:07:00.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.834+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/910024734 shutdown_connections 2024-03-04T16:07:00.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.834+0000 7fd3f9bf6700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3dc06d9e0 0x7fd3dc06fea0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.834+0000 7fd3f9bf6700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f40ffac0 0x7fd3f4069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.834+0000 7fd3f9bf6700 1 --2- 172.21.15.116:0/910024734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3f4100cc0 0x7fd3f419b0d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.834+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/910024734 >> 172.21.15.116:0/910024734 conn(0x7fd3f40fb060 msgr2=0x7fd3f4103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.834+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/910024734 shutdown_connections 2024-03-04T16:07:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.834+0000 7fd3f9bf6700 1 -- 172.21.15.116:0/910024734 wait complete. 2024-03-04T16:07:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:07:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.938+0000 7f65a3d0c700 1 -- 172.21.15.116:0/1411680338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f659c073790 msgr2=0x7f659c073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.938+0000 7f65a3d0c700 1 --2- 172.21.15.116:0/1411680338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f659c073790 0x7f659c073c10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f658c009910 tx=0x7f658c009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.938+0000 7f65a3d0c700 1 -- 172.21.15.116:0/1411680338 shutdown_connections 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.938+0000 7f65a3d0c700 1 --2- 172.21.15.116:0/1411680338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f659c073790 0x7f659c073c10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.938+0000 7f65a3d0c700 1 --2- 172.21.15.116:0/1411680338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f659c074d60 0x7f659c0731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.938+0000 7f65a3d0c700 1 -- 172.21.15.116:0/1411680338 >> 172.21.15.116:0/1411680338 conn(0x7f659c0fc180 msgr2=0x7f659c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.938+0000 7f65a3d0c700 1 -- 172.21.15.116:0/1411680338 shutdown_connections 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.938+0000 7f65a3d0c700 1 -- 172.21.15.116:0/1411680338 wait complete. 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.938+0000 7f65a3d0c700 1 Processor -- start 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.938+0000 7f65a3d0c700 1 -- start start 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a3d0c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f659c073790 0x7f659c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a3d0c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f659c074d60 0x7f659c198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a3d0c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c1990e0 con 0x7f659c073790 2024-03-04T16:07:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a3d0c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c199220 con 0x7f659c074d60 2024-03-04T16:07:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a1aa8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f659c073790 0x7f659c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a1aa8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f659c073790 0x7f659c198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34768/0 (socket says 172.21.15.116:34768) 2024-03-04T16:07:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a1aa8700 1 -- 172.21.15.116:0/781828890 learned_addr learned my addr 172.21.15.116:0/781828890 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a12a7700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f659c074d60 0x7f659c198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a1aa8700 1 -- 172.21.15.116:0/781828890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f659c074d60 msgr2=0x7f659c198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a1aa8700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f659c074d60 0x7f659c198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a1aa8700 1 -- 172.21.15.116:0/781828890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c0095f0 con 0x7f659c073790 2024-03-04T16:07:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a12a7700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f659c074d60 0x7f659c198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:07:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a1aa8700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f659c073790 0x7f659c198580 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6594007e90 tx=0x7f659400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f6592ffd700 1 -- 172.21.15.116:0/781828890 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f659400de60 con 0x7f659c073790 2024-03-04T16:07:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a3d0c700 1 -- 172.21.15.116:0/781828890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659c19dcd0 con 0x7f659c073790 2024-03-04T16:07:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f6592ffd700 1 -- 172.21.15.116:0/781828890 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f659400f040 con 0x7f659c073790 2024-03-04T16:07:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f6592ffd700 1 -- 172.21.15.116:0/781828890 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65940045b0 con 0x7f659c073790 2024-03-04T16:07:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.939+0000 7f65a3d0c700 1 -- 172.21.15.116:0/781828890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c19e1c0 con 0x7f659c073790 2024-03-04T16:07:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.940+0000 7f65a3d0c700 1 -- 172.21.15.116:0/781828890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659c10b950 con 0x7f659c073790 2024-03-04T16:07:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.941+0000 7f6592ffd700 1 -- 172.21.15.116:0/781828890 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7f6594004790 con 0x7f659c073790 2024-03-04T16:07:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.941+0000 7f6592ffd700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f658806d9e0 0x7f658806fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.941+0000 7f65a12a7700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f658806d9e0 0x7f658806fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.941+0000 7f6592ffd700 1 -- 172.21.15.116:0/781828890 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f659408d3b0 con 0x7f659c073790 2024-03-04T16:07:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.942+0000 7f65a12a7700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f658806d9e0 0x7f658806fea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f658c005b40 tx=0x7f658c005ab0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:00.946+0000 7f6592ffd700 1 -- 172.21.15.116:0/781828890 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f659405a4c0 con 0x7f659c073790 2024-03-04T16:07:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.165+0000 7f65a3d0c700 1 -- 172.21.15.116:0/781828890 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f659c108160 con 0x7f658806d9e0 2024-03-04T16:07:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.167+0000 7f6592ffd700 1 -- 172.21.15.116:0/781828890 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f659c108160 con 0x7f658806d9e0 2024-03-04T16:07:01.167 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "4/23 daemons upgraded", 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading crash daemons", 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:07:01.168 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:07:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.169+0000 7f65a3d0c700 1 -- 172.21.15.116:0/781828890 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f658806d9e0 msgr2=0x7f658806fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.169+0000 7f65a3d0c700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f658806d9e0 0x7f658806fea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f658c005b40 tx=0x7f658c005ab0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.169+0000 7f65a3d0c700 1 -- 172.21.15.116:0/781828890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f659c073790 msgr2=0x7f659c198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.169+0000 7f65a3d0c700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f659c073790 0x7f659c198580 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6594007e90 tx=0x7f659400d3b0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.169+0000 7f65a3d0c700 1 -- 172.21.15.116:0/781828890 shutdown_connections 2024-03-04T16:07:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.169+0000 7f65a3d0c700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f658806d9e0 0x7f658806fea0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.169+0000 7f65a3d0c700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f659c073790 0x7f659c198580 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.169+0000 7f65a3d0c700 1 --2- 172.21.15.116:0/781828890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f659c074d60 0x7f659c198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.169+0000 7f65a3d0c700 1 -- 172.21.15.116:0/781828890 >> 172.21.15.116:0/781828890 conn(0x7f659c0fc180 msgr2=0x7f659c106a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.170+0000 7f65a3d0c700 1 -- 172.21.15.116:0/781828890 shutdown_connections 2024-03-04T16:07:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.170+0000 7f65a3d0c700 1 -- 172.21.15.116:0/781828890 wait complete. 2024-03-04T16:07:01.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.277+0000 7fc251e33700 1 -- 172.21.15.116:0/350876043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c100cb0 msgr2=0x7fc24c101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:01.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.277+0000 7fc251e33700 1 --2- 172.21.15.116:0/350876043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c100cb0 0x7fc24c101110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc234009910 tx=0x7fc234009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:01.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.277+0000 7fc251e33700 1 -- 172.21.15.116:0/350876043 shutdown_connections 2024-03-04T16:07:01.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.277+0000 7fc251e33700 1 --2- 172.21.15.116:0/350876043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c100cb0 0x7fc24c101110 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.277+0000 7fc251e33700 1 --2- 172.21.15.116:0/350876043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc24c0ffab0 0x7fc24c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.277+0000 7fc251e33700 1 -- 172.21.15.116:0/350876043 >> 172.21.15.116:0/350876043 conn(0x7fc24c0fb030 msgr2=0x7fc24c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:01.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.277+0000 7fc251e33700 1 -- 172.21.15.116:0/350876043 shutdown_connections 2024-03-04T16:07:01.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.277+0000 7fc251e33700 1 -- 172.21.15.116:0/350876043 wait complete. 2024-03-04T16:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc251e33700 1 Processor -- start 2024-03-04T16:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc251e33700 1 -- start start 2024-03-04T16:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc251e33700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c0ffab0 0x7fc24c068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc251e33700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc24c100cb0 0x7fc24c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc251e33700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c196f30 con 0x7fc24c0ffab0 2024-03-04T16:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc251e33700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c19da00 con 0x7fc24c100cb0 2024-03-04T16:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc24b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c0ffab0 0x7fc24c068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc24b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c0ffab0 0x7fc24c068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34794/0 (socket says 172.21.15.116:34794) 2024-03-04T16:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc24b7fe700 1 -- 172.21.15.116:0/450702688 learned_addr learned my addr 172.21.15.116:0/450702688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc243fff700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc24c100cb0 0x7fc24c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc24b7fe700 1 -- 172.21.15.116:0/450702688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc24c100cb0 msgr2=0x7fc24c19d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc24b7fe700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc24c100cb0 0x7fc24c19d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.278+0000 7fc24b7fe700 1 -- 172.21.15.116:0/450702688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2340095f0 con 0x7fc24c0ffab0 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.279+0000 7fc24b7fe700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c0ffab0 0x7fc24c068f70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00d930 tx=0x7fc23c00dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.279+0000 7fc2497fa700 1 -- 172.21.15.116:0/450702688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc23c0041f0 con 0x7fc24c0ffab0 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.279+0000 7fc2497fa700 1 -- 172.21.15.116:0/450702688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc23c004350 con 0x7fc24c0ffab0 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.279+0000 7fc2497fa700 1 -- 172.21.15.116:0/450702688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc23c003d20 con 0x7fc24c0ffab0 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.279+0000 7fc251e33700 1 -- 172.21.15.116:0/450702688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24c19dcb0 con 0x7fc24c0ffab0 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.279+0000 7fc251e33700 1 -- 172.21.15.116:0/450702688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc24c19e200 con 0x7fc24c0ffab0 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.280+0000 7fc251e33700 1 -- 172.21.15.116:0/450702688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc24c108610 con 0x7fc24c0ffab0 2024-03-04T16:07:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.280+0000 7fc2497fa700 1 -- 172.21.15.116:0/450702688 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 36) v1 ==== 91587+0+0 (secure 0 0 0) 0x7fc23c009730 con 0x7fc24c0ffab0 2024-03-04T16:07:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.281+0000 7fc2497fa700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc23806d9e0 0x7fc23806fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.281+0000 7fc2497fa700 1 -- 172.21.15.116:0/450702688 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc23c08d570 con 0x7fc24c0ffab0 2024-03-04T16:07:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.281+0000 7fc243fff700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc23806d9e0 0x7fc23806fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.281+0000 7fc243fff700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc23806d9e0 0x7fc23806fea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0694f0 tx=0x7fc23400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.285+0000 7fc2497fa700 1 -- 172.21.15.116:0/450702688 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc23c05a700 con 0x7fc24c0ffab0 2024-03-04T16:07:01.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:01 smithi152 ceph-mon[82845]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:01.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:01 smithi152 ceph-mon[82845]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:01.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:01 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:01.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:01 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:01.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:01 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/422346540' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:01.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:01 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/910024734' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:07:01.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:01 smithi116 ceph-mon[94164]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:01.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:01 smithi116 ceph-mon[94164]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:01.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:01 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:01.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:01 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:01.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:01 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/422346540' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:01.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:01 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/910024734' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:07:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.576+0000 7fc251e33700 1 -- 172.21.15.116:0/450702688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc24c19e920 con 0x7fc24c0ffab0 2024-03-04T16:07:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.576+0000 7fc2497fa700 1 -- 172.21.15.116:0/450702688 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fc23c05a290 con 0x7fc24c0ffab0 2024-03-04T16:07:01.577 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:07:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 -- 172.21.15.116:0/450702688 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc23806d9e0 msgr2=0x7fc23806fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc23806d9e0 0x7fc23806fea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0694f0 tx=0x7fc23400b540 comp rx=0 tx=0).stop 2024-03-04T16:07:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 -- 172.21.15.116:0/450702688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c0ffab0 msgr2=0x7fc24c068f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c0ffab0 0x7fc24c068f70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00d930 tx=0x7fc23c00dc40 comp rx=0 tx=0).stop 2024-03-04T16:07:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 -- 172.21.15.116:0/450702688 shutdown_connections 2024-03-04T16:07:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc23806d9e0 0x7fc23806fea0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c0ffab0 0x7fc24c068f70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 --2- 172.21.15.116:0/450702688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc24c100cb0 0x7fc24c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 -- 172.21.15.116:0/450702688 >> 172.21.15.116:0/450702688 conn(0x7fc24c0fb030 msgr2=0x7fc24c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 -- 172.21.15.116:0/450702688 shutdown_connections 2024-03-04T16:07:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:01.578+0000 7fc251e33700 1 -- 172.21.15.116:0/450702688 wait complete. 2024-03-04T16:07:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:02 smithi116 ceph-mon[94164]: pgmap v37: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:02 smithi116 ceph-mon[94164]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:02 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/450702688' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:07:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:02 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:02 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T16:07:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:02 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:02.544 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:02 smithi152 ceph-mon[82845]: pgmap v37: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:02.544 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:02 smithi152 ceph-mon[82845]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:02.544 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:02 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/450702688' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:07:02.544 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:02 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:02.544 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:02 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-04T16:07:02.544 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:02 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:03.428 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:03 smithi152 ceph-mon[82845]: Upgrade: Updating crash.smithi152 (2/2) 2024-03-04T16:07:03.428 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:03 smithi152 ceph-mon[82845]: Deploying daemon crash.smithi152 on smithi152 2024-03-04T16:07:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:03 smithi116 ceph-mon[94164]: Upgrade: Updating crash.smithi152 (2/2) 2024-03-04T16:07:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:03 smithi116 ceph-mon[94164]: Deploying daemon crash.smithi152 on smithi152 2024-03-04T16:07:04.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:04 smithi152 ceph-mon[82845]: pgmap v38: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:04.943 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:04 smithi116 ceph-mon[94164]: pgmap v38: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:06.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:06 smithi116 ceph-mon[94164]: pgmap v39: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:06.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:06 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:06.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:06 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:06.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:06 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:06.567 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:06 smithi152 ceph-mon[82845]: pgmap v39: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:06.568 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:06 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:06.568 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:06 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:06.568 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:06 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:08.551 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:08 smithi116 ceph-mon[94164]: pgmap v40: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:08.551 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:08.578 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:08 smithi152 ceph-mon[82845]: pgmap v40: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:08.578 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:10.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:10.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:10.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:09 smithi152 ceph-mon[82845]: pgmap v41: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:10.120 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:10.120 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:10.120 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:09 smithi116 ceph-mon[94164]: pgmap v41: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:11.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:10 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:11.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:10 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:11.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:10 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:11.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:10 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:11.109 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:11.109 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:11.109 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:11.109 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:12.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:11 smithi152 ceph-mon[82845]: pgmap v42: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:12.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:12.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:12.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:11 smithi116 ceph-mon[94164]: pgmap v42: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:12.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:12.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:14.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:14 smithi152 ceph-mon[82845]: pgmap v43: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:14.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:14 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:14.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:14 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:14.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:14 smithi116 ceph-mon[94164]: pgmap v43: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:14.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:14 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:14.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:14 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: pgmap v44: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:07:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:16.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: Upgrade: Setting container_image for all crash 2024-03-04T16:07:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi116"}]: dispatch 2024-03-04T16:07:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi116"}]': finished 2024-03-04T16:07:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi152"}]: dispatch 2024-03-04T16:07:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi152"}]': finished 2024-03-04T16:07:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-04T16:07:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-04T16:07:16.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:16 smithi152 ceph-mon[82845]: Upgrade: osd.0 is safe to restart 2024-03-04T16:07:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: pgmap v44: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:07:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: Upgrade: Setting container_image for all crash 2024-03-04T16:07:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi116"}]: dispatch 2024-03-04T16:07:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi116"}]': finished 2024-03-04T16:07:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi152"}]: dispatch 2024-03-04T16:07:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi152"}]': finished 2024-03-04T16:07:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-04T16:07:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-04T16:07:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:16 smithi116 ceph-mon[94164]: Upgrade: osd.0 is safe to restart 2024-03-04T16:07:17.965 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:17 smithi116 ceph-mon[94164]: Upgrade: Updating osd.0 2024-03-04T16:07:17.966 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:17.966 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T16:07:17.966 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:17.966 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:17 smithi116 ceph-mon[94164]: Deploying daemon osd.0 on smithi116 2024-03-04T16:07:17.966 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:17 smithi116 ceph-mon[94164]: pgmap v45: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:18.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:17 smithi152 ceph-mon[82845]: Upgrade: Updating osd.0 2024-03-04T16:07:18.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:18.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T16:07:18.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:18.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:17 smithi152 ceph-mon[82845]: Deploying daemon osd.0 on smithi116 2024-03-04T16:07:18.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:17 smithi152 ceph-mon[82845]: pgmap v45: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:20.252 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:20 smithi116 ceph-mon[94164]: pgmap v46: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:20.253 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:19 smithi116 systemd[1]: Stopping Ceph osd.0 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:20.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:20 smithi152 ceph-mon[82845]: pgmap v46: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:20.725 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:20 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[48197]: 2024-03-04T16:07:20.250+0000 7fb9c3927700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:07:20.725 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:20 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[48197]: 2024-03-04T16:07:20.250+0000 7fb9c3927700 -1 osd.0 50 *** Got signal Terminated *** 2024-03-04T16:07:20.725 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:20 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[48197]: 2024-03-04T16:07:20.250+0000 7fb9c3927700 -1 osd.0 50 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:07:21.336 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:21 smithi116 ceph-mon[94164]: osd.0 marked itself down and dead 2024-03-04T16:07:21.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:21 smithi152 ceph-mon[82845]: osd.0 marked itself down and dead 2024-03-04T16:07:21.724 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:21 smithi116 podman[102359]: 2024-03-04 16:07:21.466491089 +0000 UTC m=+1.326921347 container died 946b46d0149064e91f4e338f658206723e0478f9a70bd9be81bb5874a670bc2a (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1) 2024-03-04T16:07:22.083 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:21 smithi116 podman[102359]: 2024-03-04 16:07:21.87668711 +0000 UTC m=+1.737117413 container cleanup 946b46d0149064e91f4e338f658206723e0478f9a70bd9be81bb5874a670bc2a (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-03-04T16:07:22.083 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:21 smithi116 bash[102359]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0 2024-03-04T16:07:22.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:22 smithi152 ceph-mon[82845]: pgmap v47: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:22.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:22 smithi152 ceph-mon[82845]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:07:22.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:22 smithi152 ceph-mon[82845]: osdmap e51: 6 total, 5 up, 6 in 2024-03-04T16:07:22.475 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:22 smithi116 podman[102378]: 2024-03-04 16:07:22.11115583 +0000 UTC m=+0.641869039 container remove 946b46d0149064e91f4e338f658206723e0478f9a70bd9be81bb5874a670bc2a (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-03-04T16:07:22.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:22 smithi116 ceph-mon[94164]: pgmap v47: 65 pgs: 65 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:22.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:22 smithi116 ceph-mon[94164]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:07:22.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:22 smithi116 ceph-mon[94164]: osdmap e51: 6 total, 5 up, 6 in 2024-03-04T16:07:22.831 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:22 smithi116 podman[102469]: 2024-03-04 16:07:22.440502533 +0000 UTC m=+0.017850055 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:07:22.831 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:22 smithi116 podman[102469]: 2024-03-04 16:07:22.570785272 +0000 UTC m=+0.148132828 container create c30e7672fc794fca931e217eba4653c8bfa7f2043377dfb71dbf0b45c8a2c316 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-04T16:07:22.831 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:22 smithi116 podman[102469]: 2024-03-04 16:07:22.764703354 +0000 UTC m=+0.342050868 container init c30e7672fc794fca931e217eba4653c8bfa7f2043377dfb71dbf0b45c8a2c316 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:07:22.831 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:22 smithi116 podman[102469]: 2024-03-04 16:07:22.767793293 +0000 UTC m=+0.345140814 container start c30e7672fc794fca931e217eba4653c8bfa7f2043377dfb71dbf0b45c8a2c316 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:07:22.831 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:22 smithi116 podman[102469]: 2024-03-04 16:07:22.831345123 +0000 UTC m=+0.408692650 container attach c30e7672fc794fca931e217eba4653c8bfa7f2043377dfb71dbf0b45c8a2c316 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-04T16:07:23.093 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:23 smithi116 podman[102469]: 2024-03-04 16:07:23.054411033 +0000 UTC m=+0.631758547 container died c30e7672fc794fca931e217eba4653c8bfa7f2043377dfb71dbf0b45c8a2c316 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222) 2024-03-04T16:07:23.344 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:23 smithi116 ceph-mon[94164]: osdmap e52: 6 total, 5 up, 6 in 2024-03-04T16:07:23.344 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:23.344 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:23.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:23 smithi152 ceph-mon[82845]: osdmap e52: 6 total, 5 up, 6 in 2024-03-04T16:07:23.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:23.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:23.975 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:23 smithi116 podman[102494]: 2024-03-04 16:07:23.618826061 +0000 UTC m=+0.559217741 container remove c30e7672fc794fca931e217eba4653c8bfa7f2043377dfb71dbf0b45c8a2c316 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-04T16:07:23.975 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:23 smithi116 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-04T16:07:23.975 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:23 smithi116 systemd[1]: Stopped Ceph osd.0 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:23.975 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:23 smithi116 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.0.service: Consumed 58.844s CPU time. 2024-03-04T16:07:23.975 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:23 smithi116 systemd[1]: Starting Ceph osd.0 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:24.331 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:24 smithi116 ceph-mon[94164]: pgmap v50: 65 pgs: 9 stale+active+clean, 56 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:24.331 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 podman[102597]: 2024-03-04 16:07:23.965188953 +0000 UTC m=+0.018069760 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:07:24.331 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 podman[102597]: 2024-03-04 16:07:24.103317272 +0000 UTC m=+0.156198074 container create cdd6871a838fc41f3d385cb2527754374ca9f0f238c162fe55a1e47732813a25 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-03-04T16:07:24.331 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 podman[102597]: 2024-03-04 16:07:24.331071874 +0000 UTC m=+0.383952693 container init cdd6871a838fc41f3d385cb2527754374ca9f0f238c162fe55a1e47732813a25 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-04T16:07:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:24 smithi152 ceph-mon[82845]: pgmap v50: 65 pgs: 9 stale+active+clean, 56 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:24.725 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 podman[102597]: 2024-03-04 16:07:24.334346783 +0000 UTC m=+0.387227592 container start cdd6871a838fc41f3d385cb2527754374ca9f0f238c162fe55a1e47732813a25 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-04T16:07:24.725 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 podman[102597]: 2024-03-04 16:07:24.431020427 +0000 UTC m=+0.483901239 container attach cdd6871a838fc41f3d385cb2527754374ca9f0f238c162fe55a1e47732813a25 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True) 2024-03-04T16:07:25.106 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate[102612]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T16:07:25.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate[102612]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-04T16:07:25.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 bash[102597]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T16:07:25.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 bash[102597]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-04T16:07:25.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 bash[102597]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-04T16:07:25.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 bash[102597]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-04T16:07:25.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 bash[102597]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-04T16:07:25.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 bash[102597]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T16:07:25.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 bash[102597]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-04T16:07:25.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate[102612]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-04T16:07:25.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate[102612]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-04T16:07:25.108 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate[102612]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-04T16:07:25.108 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate[102612]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T16:07:25.108 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate[102612]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-04T16:07:25.108 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:24 smithi116 podman[102597]: 2024-03-04 16:07:24.805656295 +0000 UTC m=+0.858537092 container died cdd6871a838fc41f3d385cb2527754374ca9f0f238c162fe55a1e47732813a25 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-04T16:07:25.359 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:25 smithi152 ceph-mon[82845]: mgrmap e37: smithi116.jwdhop(active, since 91s), standbys: smithi152.aawieo 2024-03-04T16:07:25.438 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:25 smithi116 ceph-mon[94164]: mgrmap e37: smithi116.jwdhop(active, since 91s), standbys: smithi152.aawieo 2024-03-04T16:07:25.725 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:25 smithi116 podman[102742]: 2024-03-04 16:07:25.438628655 +0000 UTC m=+0.626719529 container remove cdd6871a838fc41f3d385cb2527754374ca9f0f238c162fe55a1e47732813a25 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:07:25.725 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:25 smithi116 podman[102807]: 2024-03-04 16:07:25.55251145 +0000 UTC m=+0.017744051 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:07:25.725 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:25 smithi116 podman[102807]: 2024-03-04 16:07:25.660860244 +0000 UTC m=+0.126092833 container create 98c52b3190c9ebc68c49bafa084891627f7a06db09fd320d99fc907163652d0a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-04T16:07:26.107 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:25 smithi116 podman[102807]: 2024-03-04 16:07:25.888834792 +0000 UTC m=+0.354067382 container init 98c52b3190c9ebc68c49bafa084891627f7a06db09fd320d99fc907163652d0a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-04T16:07:26.108 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:25 smithi116 podman[102807]: 2024-03-04 16:07:25.891995222 +0000 UTC m=+0.357227814 container start 98c52b3190c9ebc68c49bafa084891627f7a06db09fd320d99fc907163652d0a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True) 2024-03-04T16:07:26.108 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:25 smithi116 ceph-osd[102826]: -- 172.21.15.116:0/1854234312 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a5ab86b680 con 0x55a5ab835000 2024-03-04T16:07:26.108 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:25 smithi116 bash[102807]: 98c52b3190c9ebc68c49bafa084891627f7a06db09fd320d99fc907163652d0a 2024-03-04T16:07:26.108 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:25 smithi116 systemd[1]: Started Ceph osd.0 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:26.441 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:26 smithi116 ceph-mon[94164]: pgmap v51: 65 pgs: 18 active+undersized+degraded, 16 active+undersized, 31 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 383 B/s wr, 0 op/s; 42/231 objects degraded (18.182%) 2024-03-04T16:07:26.441 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:26 smithi116 ceph-mon[94164]: Health check failed: Degraded data redundancy: 42/231 objects degraded (18.182%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:07:26.441 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:26 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:26.441 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:26 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:26.441 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:26 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:26 smithi152 ceph-mon[82845]: pgmap v51: 65 pgs: 18 active+undersized+degraded, 16 active+undersized, 31 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 383 B/s wr, 0 op/s; 42/231 objects degraded (18.182%) 2024-03-04T16:07:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:26 smithi152 ceph-mon[82845]: Health check failed: Degraded data redundancy: 42/231 objects degraded (18.182%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:07:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:26 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:26 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:26.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:26 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:26.725 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:26 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[102822]: 2024-03-04T16:07:26.440+0000 7f6d646ce7c0 -1 Falling back to public interface 2024-03-04T16:07:28.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:28 smithi116 ceph-mon[94164]: pgmap v52: 65 pgs: 18 active+undersized+degraded, 16 active+undersized, 31 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 383 B/s wr, 0 op/s; 42/231 objects degraded (18.182%) 2024-03-04T16:07:28.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:28 smithi152 ceph-mon[82845]: pgmap v52: 65 pgs: 18 active+undersized+degraded, 16 active+undersized, 31 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 383 B/s wr, 0 op/s; 42/231 objects degraded (18.182%) 2024-03-04T16:07:30.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:30 smithi116 ceph-mon[94164]: pgmap v53: 65 pgs: 18 active+undersized+degraded, 16 active+undersized, 31 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 383 B/s wr, 0 op/s; 42/231 objects degraded (18.182%) 2024-03-04T16:07:30.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:30 smithi152 ceph-mon[82845]: pgmap v53: 65 pgs: 18 active+undersized+degraded, 16 active+undersized, 31 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 383 B/s wr, 0 op/s; 42/231 objects degraded (18.182%) 2024-03-04T16:07:31.225 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:30 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[102822]: 2024-03-04T16:07:30.947+0000 7f6d646ce7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-04T16:07:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.689+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3791398211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefb80ffad0 msgr2=0x7fefb80ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.689+0000 7fefbd4cd700 1 --2- 172.21.15.116:0/3791398211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefb80ffad0 0x7fefb80ffef0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009910 tx=0x7fefa8009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.689+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3791398211 shutdown_connections 2024-03-04T16:07:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.689+0000 7fefbd4cd700 1 --2- 172.21.15.116:0/3791398211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefb8100cd0 0x7fefb8101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.689+0000 7fefbd4cd700 1 --2- 172.21.15.116:0/3791398211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefb80ffad0 0x7fefb80ffef0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.689+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3791398211 >> 172.21.15.116:0/3791398211 conn(0x7fefb80fb050 msgr2=0x7fefb80fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.689+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3791398211 shutdown_connections 2024-03-04T16:07:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.689+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3791398211 wait complete. 2024-03-04T16:07:31.692 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:31 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[102822]: 2024-03-04T16:07:31.425+0000 7f6d646ce7c0 -1 osd.0 50 log_to_monitors true 2024-03-04T16:07:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.690+0000 7fefbd4cd700 1 Processor -- start 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.690+0000 7fefbd4cd700 1 -- start start 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.690+0000 7fefbd4cd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefb80ffad0 0x7fefb81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.690+0000 7fefbd4cd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefb8100cd0 0x7fefb8198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.690+0000 7fefbd4cd700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb8199140 con 0x7fefb80ffad0 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.690+0000 7fefbd4cd700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb8199280 con 0x7fefb8100cd0 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.690+0000 7fefb6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefb80ffad0 0x7fefb81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.690+0000 7fefb6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefb80ffad0 0x7fefb81985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43024/0 (socket says 172.21.15.116:43024) 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.690+0000 7fefb6ffd700 1 -- 172.21.15.116:0/3227861887 learned_addr learned my addr 172.21.15.116:0/3227861887 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.690+0000 7fefb67fc700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefb8100cd0 0x7fefb8198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.691+0000 7fefb6ffd700 1 -- 172.21.15.116:0/3227861887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefb8100cd0 msgr2=0x7fefb8198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.691+0000 7fefb6ffd700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefb8100cd0 0x7fefb8198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.691+0000 7fefb6ffd700 1 -- 172.21.15.116:0/3227861887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa80095f0 con 0x7fefb80ffad0 2024-03-04T16:07:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.691+0000 7fefb6ffd700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefb80ffad0 0x7fefb81985e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fefa800bda0 tx=0x7fefa800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.691+0000 7fef9ffff700 1 -- 172.21.15.116:0/3227861887 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefa801d070 con 0x7fefb80ffad0 2024-03-04T16:07:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.691+0000 7fef9ffff700 1 -- 172.21.15.116:0/3227861887 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fefa800f460 con 0x7fefb80ffad0 2024-03-04T16:07:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.691+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3227861887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb819dcd0 con 0x7fefb80ffad0 2024-03-04T16:07:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.691+0000 7fef9ffff700 1 -- 172.21.15.116:0/3227861887 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefa8021620 con 0x7fefb80ffad0 2024-03-04T16:07:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.691+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3227861887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb819e1c0 con 0x7fefb80ffad0 2024-03-04T16:07:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.692+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3227861887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa4005320 con 0x7fefb80ffad0 2024-03-04T16:07:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.693+0000 7fef9ffff700 1 -- 172.21.15.116:0/3227861887 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fefa802b440 con 0x7fefb80ffad0 2024-03-04T16:07:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.693+0000 7fef9ffff700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefa006d8a0 0x7fefa006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.693+0000 7fef9ffff700 1 -- 172.21.15.116:0/3227861887 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fefa808e710 con 0x7fefb80ffad0 2024-03-04T16:07:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.693+0000 7fefb67fc700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefa006d8a0 0x7fefa006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.694+0000 7fefb67fc700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefa006d8a0 0x7fefa006fd60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fefac004630 tx=0x7fefac00b3c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.698+0000 7fef9ffff700 1 -- 172.21.15.116:0/3227861887 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fefa805bc70 con 0x7fefb80ffad0 2024-03-04T16:07:31.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.921+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3227861887 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fefa4000bf0 con 0x7fefa006d8a0 2024-03-04T16:07:31.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.922+0000 7fef9ffff700 1 -- 172.21.15.116:0/3227861887 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fefa4000bf0 con 0x7fefa006d8a0 2024-03-04T16:07:31.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.924+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3227861887 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefa006d8a0 msgr2=0x7fefa006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:31.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.924+0000 7fefbd4cd700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefa006d8a0 0x7fefa006fd60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fefac004630 tx=0x7fefac00b3c0 comp rx=0 tx=0).stop 2024-03-04T16:07:31.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.924+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3227861887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefb80ffad0 msgr2=0x7fefb81985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:31.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.924+0000 7fefbd4cd700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefb80ffad0 0x7fefb81985e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fefa800bda0 tx=0x7fefa800bdd0 comp rx=0 tx=0).stop 2024-03-04T16:07:31.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.924+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3227861887 shutdown_connections 2024-03-04T16:07:31.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.924+0000 7fefbd4cd700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefa006d8a0 0x7fefa006fd60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:31.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.924+0000 7fefbd4cd700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefb80ffad0 0x7fefb81985e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:31.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.924+0000 7fefbd4cd700 1 --2- 172.21.15.116:0/3227861887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefb8100cd0 0x7fefb8198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:31.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.924+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3227861887 >> 172.21.15.116:0/3227861887 conn(0x7fefb80fb050 msgr2=0x7fefb8068d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:31.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.925+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3227861887 shutdown_connections 2024-03-04T16:07:31.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:31.925+0000 7fefbd4cd700 1 -- 172.21.15.116:0/3227861887 wait complete. 2024-03-04T16:07:31.937 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:07:32.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.034+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/3374514122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f680ffab0 msgr2=0x7f4f680ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:32.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.034+0000 7f4f6e1cc700 1 --2- 172.21.15.116:0/3374514122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f680ffab0 0x7f4f680ffed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009910 tx=0x7f4f5c009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:32.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.034+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/3374514122 shutdown_connections 2024-03-04T16:07:32.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.034+0000 7f4f6e1cc700 1 --2- 172.21.15.116:0/3374514122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f68100cb0 0x7f4f68101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.034+0000 7f4f6e1cc700 1 --2- 172.21.15.116:0/3374514122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f680ffab0 0x7f4f680ffed0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.034+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/3374514122 >> 172.21.15.116:0/3374514122 conn(0x7f4f680fb050 msgr2=0x7f4f680fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:32.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.034+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/3374514122 shutdown_connections 2024-03-04T16:07:32.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.034+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/3374514122 wait complete. 2024-03-04T16:07:32.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.035+0000 7f4f6e1cc700 1 Processor -- start 2024-03-04T16:07:32.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.035+0000 7f4f6e1cc700 1 -- start start 2024-03-04T16:07:32.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.035+0000 7f4f6e1cc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f680ffab0 0x7f4f68196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:32.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.035+0000 7f4f6e1cc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f68100cb0 0x7f4f681968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:32.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.035+0000 7f4f6e1cc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68196ed0 con 0x7f4f680ffab0 2024-03-04T16:07:32.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.035+0000 7f4f6e1cc700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68197010 con 0x7f4f68100cb0 2024-03-04T16:07:32.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.035+0000 7f4f677fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f680ffab0 0x7f4f68196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:32.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f677fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f680ffab0 0x7f4f68196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43040/0 (socket says 172.21.15.116:43040) 2024-03-04T16:07:32.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f677fe700 1 -- 172.21.15.116:0/2825690142 learned_addr learned my addr 172.21.15.116:0/2825690142 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:32.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f66ffd700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f68100cb0 0x7f4f681968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:32.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f677fe700 1 -- 172.21.15.116:0/2825690142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f68100cb0 msgr2=0x7f4f681968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:32.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f677fe700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f68100cb0 0x7f4f681968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f677fe700 1 -- 172.21.15.116:0/2825690142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c0095f0 con 0x7f4f680ffab0 2024-03-04T16:07:32.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f66ffd700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f68100cb0 0x7f4f681968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:07:32.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f677fe700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f680ffab0 0x7f4f68196370 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00bda0 tx=0x7f4f5c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:32.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f64ff9700 1 -- 172.21.15.116:0/2825690142 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f5c01d070 con 0x7f4f680ffab0 2024-03-04T16:07:32.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f64ff9700 1 -- 172.21.15.116:0/2825690142 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4f5c00f460 con 0x7f4f680ffab0 2024-03-04T16:07:32.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/2825690142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6819ba60 con 0x7f4f680ffab0 2024-03-04T16:07:32.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f64ff9700 1 -- 172.21.15.116:0/2825690142 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f5c021640 con 0x7f4f680ffab0 2024-03-04T16:07:32.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.036+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/2825690142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6819bf50 con 0x7f4f680ffab0 2024-03-04T16:07:32.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.037+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/2825690142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f68108610 con 0x7f4f680ffab0 2024-03-04T16:07:32.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.038+0000 7f4f64ff9700 1 -- 172.21.15.116:0/2825690142 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f4f5c00fa00 con 0x7f4f680ffab0 2024-03-04T16:07:32.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.038+0000 7f4f64ff9700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4f5006d8a0 0x7f4f5006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:32.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.039+0000 7f4f64ff9700 1 -- 172.21.15.116:0/2825690142 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4f5c08e770 con 0x7f4f680ffab0 2024-03-04T16:07:32.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.039+0000 7f4f66ffd700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4f5006d8a0 0x7f4f5006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:32.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.039+0000 7f4f66ffd700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4f5006d8a0 0x7f4f5006fd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f58005fd0 tx=0x7f4f58005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:32.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.043+0000 7f4f64ff9700 1 -- 172.21.15.116:0/2825690142 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4f5c05baa0 con 0x7f4f680ffab0 2024-03-04T16:07:32.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.262+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/2825690142 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f68069390 con 0x7f4f5006d8a0 2024-03-04T16:07:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.263+0000 7f4f64ff9700 1 -- 172.21.15.116:0/2825690142 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f4f68069390 con 0x7f4f5006d8a0 2024-03-04T16:07:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.265+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/2825690142 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4f5006d8a0 msgr2=0x7f4f5006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.265+0000 7f4f6e1cc700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4f5006d8a0 0x7f4f5006fd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f58005fd0 tx=0x7f4f58005ee0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.265+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/2825690142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f680ffab0 msgr2=0x7f4f68196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.265+0000 7f4f6e1cc700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f680ffab0 0x7f4f68196370 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00bda0 tx=0x7f4f5c00be80 comp rx=0 tx=0).stop 2024-03-04T16:07:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.265+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/2825690142 shutdown_connections 2024-03-04T16:07:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.265+0000 7f4f6e1cc700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4f5006d8a0 0x7f4f5006fd60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.265+0000 7f4f6e1cc700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f680ffab0 0x7f4f68196370 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.265+0000 7f4f6e1cc700 1 --2- 172.21.15.116:0/2825690142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f68100cb0 0x7f4f681968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.265+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/2825690142 >> 172.21.15.116:0/2825690142 conn(0x7f4f680fb050 msgr2=0x7f4f68068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.266+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/2825690142 shutdown_connections 2024-03-04T16:07:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.266+0000 7f4f6e1cc700 1 -- 172.21.15.116:0/2825690142 wait complete. 2024-03-04T16:07:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.381+0000 7f29d1784700 1 -- 172.21.15.116:0/1233467509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc100cb0 msgr2=0x7f29cc101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.381+0000 7f29d1784700 1 --2- 172.21.15.116:0/1233467509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc100cb0 0x7f29cc101110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009910 tx=0x7f29c0009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.381+0000 7f29d1784700 1 -- 172.21.15.116:0/1233467509 shutdown_connections 2024-03-04T16:07:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.381+0000 7f29d1784700 1 --2- 172.21.15.116:0/1233467509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc100cb0 0x7f29cc101110 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.381+0000 7f29d1784700 1 --2- 172.21.15.116:0/1233467509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc0ffab0 0x7f29cc0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.381+0000 7f29d1784700 1 -- 172.21.15.116:0/1233467509 >> 172.21.15.116:0/1233467509 conn(0x7f29cc0fb050 msgr2=0x7f29cc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.381+0000 7f29d1784700 1 -- 172.21.15.116:0/1233467509 shutdown_connections 2024-03-04T16:07:32.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.381+0000 7f29d1784700 1 -- 172.21.15.116:0/1233467509 wait complete. 2024-03-04T16:07:32.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.381+0000 7f29d1784700 1 Processor -- start 2024-03-04T16:07:32.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.381+0000 7f29d1784700 1 -- start start 2024-03-04T16:07:32.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29d1784700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc0ffab0 0x7f29cc069820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:32.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29d1784700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc100cb0 0x7f29cc19b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29d1784700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc19b600 con 0x7f29cc100cb0 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29d1784700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc19b770 con 0x7f29cc0ffab0 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29ca7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc100cb0 0x7f29cc19b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29ca7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc100cb0 0x7f29cc19b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43052/0 (socket says 172.21.15.116:43052) 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29ca7fc700 1 -- 172.21.15.116:0/2145329351 learned_addr learned my addr 172.21.15.116:0/2145329351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29caffd700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc0ffab0 0x7f29cc069820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29ca7fc700 1 -- 172.21.15.116:0/2145329351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc0ffab0 msgr2=0x7f29cc069820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29ca7fc700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc0ffab0 0x7f29cc069820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29ca7fc700 1 -- 172.21.15.116:0/2145329351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c00095f0 con 0x7f29cc100cb0 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29ca7fc700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc100cb0 0x7f29cc19b030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29c0000c00 tx=0x7f29c000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.382+0000 7f29b3fff700 1 -- 172.21.15.116:0/2145329351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29c001d070 con 0x7f29cc100cb0 2024-03-04T16:07:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.383+0000 7f29b3fff700 1 -- 172.21.15.116:0/2145329351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f29c0005dc0 con 0x7f29cc100cb0 2024-03-04T16:07:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.383+0000 7f29d1784700 1 -- 172.21.15.116:0/2145329351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29cc19b9f0 con 0x7f29cc100cb0 2024-03-04T16:07:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.383+0000 7f29b3fff700 1 -- 172.21.15.116:0/2145329351 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29c00226a0 con 0x7f29cc100cb0 2024-03-04T16:07:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.383+0000 7f29d1784700 1 -- 172.21.15.116:0/2145329351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29cc19bee0 con 0x7f29cc100cb0 2024-03-04T16:07:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.384+0000 7f29d1784700 1 -- 172.21.15.116:0/2145329351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29cc108610 con 0x7f29cc100cb0 2024-03-04T16:07:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.384+0000 7f29b3fff700 1 -- 172.21.15.116:0/2145329351 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f29c00038f0 con 0x7f29cc100cb0 2024-03-04T16:07:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.385+0000 7f29b3fff700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f29b406d8a0 0x7f29b406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.385+0000 7f29b3fff700 1 -- 172.21.15.116:0/2145329351 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f29c008ea20 con 0x7f29cc100cb0 2024-03-04T16:07:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.385+0000 7f29caffd700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f29b406d8a0 0x7f29b406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.385+0000 7f29caffd700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f29b406d8a0 0x7f29b406fd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f29cc100b10 tx=0x7f29bc009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.389+0000 7f29b3fff700 1 -- 172.21.15.116:0/2145329351 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f29c005bd50 con 0x7f29cc100cb0 2024-03-04T16:07:32.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.602+0000 7f29d1784700 1 -- 172.21.15.116:0/2145329351 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f29cc105600 con 0x7f29b406d8a0 2024-03-04T16:07:32.603 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:07:32 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[102822]: 2024-03-04T16:07:32.329+0000 7f6d55b26700 -1 osd.0 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:07:32.604 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:32 smithi116 ceph-mon[94164]: pgmap v54: 65 pgs: 18 active+undersized+degraded, 16 active+undersized, 31 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 308 B/s wr, 0 op/s; 42/231 objects degraded (18.182%) 2024-03-04T16:07:32.604 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:32.604 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:32.604 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:32 smithi116 ceph-mon[94164]: from='osd.0 [v2:172.21.15.116:6802/2502615728,v1:172.21.15.116:6803/2502615728]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T16:07:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:32 smithi152 ceph-mon[82845]: pgmap v54: 65 pgs: 18 active+undersized+degraded, 16 active+undersized, 31 active+clean; 212 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 308 B/s wr, 0 op/s; 42/231 objects degraded (18.182%) 2024-03-04T16:07:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:32 smithi152 ceph-mon[82845]: from='osd.0 [v2:172.21.15.116:6802/2502615728,v1:172.21.15.116:6803/2502615728]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T16:07:32.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.621+0000 7f29b3fff700 1 -- 172.21.15.116:0/2145329351 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f29cc105600 con 0x7f29b406d8a0 2024-03-04T16:07:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:07:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (10m) 1s ago 12m 25.3M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:07:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (12m) 1s ago 12m 7876k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:07:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (11m) 23s ago 11m 7838k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:07:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (32s) 1s ago 12m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:07:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (27s) 23s ago 11m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:07:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (10m) 1s ago 11m 88.2M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:07:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (6m) 1s ago 6m 19.4M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:07:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (6m) 1s ago 6m 15.6M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:07:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (6m) 23s ago 6m 16.7M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (6m) 23s ago 6m 143M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (3m) 1s ago 13m 497M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (2m) 23s ago 11m 449M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (109s) 1s ago 13m 44.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (78s) 23s ago 10m 33.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (12m) 1s ago 12m 15.5M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (11m) 23s ago 11m 16.7M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (7s) 1s ago 10m 12.8M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (9m) 1s ago 9m 235M 1216M 18.2.1 5be31c24972a d8b733db1a46 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (9m) 1s ago 9m 196M 1216M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (8m) 23s ago 8m 268M 2240M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (8m) 23s ago 8m 238M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (7m) 23s ago 7m 228M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:07:32.624 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (2m) 1s ago 11m 62.8M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:07:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 -- 172.21.15.116:0/2145329351 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f29b406d8a0 msgr2=0x7f29b406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f29b406d8a0 0x7f29b406fd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f29cc100b10 tx=0x7f29bc009260 comp rx=0 tx=0).stop 2024-03-04T16:07:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 -- 172.21.15.116:0/2145329351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc100cb0 msgr2=0x7f29cc19b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc100cb0 0x7f29cc19b030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29c0000c00 tx=0x7f29c000f690 comp rx=0 tx=0).stop 2024-03-04T16:07:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 -- 172.21.15.116:0/2145329351 shutdown_connections 2024-03-04T16:07:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f29b406d8a0 0x7f29b406fd60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc0ffab0 0x7f29cc069820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 --2- 172.21.15.116:0/2145329351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc100cb0 0x7f29cc19b030 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 -- 172.21.15.116:0/2145329351 >> 172.21.15.116:0/2145329351 conn(0x7f29cc0fb050 msgr2=0x7f29cc103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 -- 172.21.15.116:0/2145329351 shutdown_connections 2024-03-04T16:07:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.624+0000 7f29d1784700 1 -- 172.21.15.116:0/2145329351 wait complete. 2024-03-04T16:07:32.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.730+0000 7facbd628700 1 -- 172.21.15.116:0/1336106837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb8100bd0 msgr2=0x7facb8100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:32.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.730+0000 7facbd628700 1 --2- 172.21.15.116:0/1336106837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb8100bd0 0x7facb8100ff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faca8009910 tx=0x7faca8009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:32.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.730+0000 7facbd628700 1 -- 172.21.15.116:0/1336106837 shutdown_connections 2024-03-04T16:07:32.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.730+0000 7facbd628700 1 --2- 172.21.15.116:0/1336106837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7facb8101dd0 0x7facb8102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.730+0000 7facbd628700 1 --2- 172.21.15.116:0/1336106837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb8100bd0 0x7facb8100ff0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.730+0000 7facbd628700 1 -- 172.21.15.116:0/1336106837 >> 172.21.15.116:0/1336106837 conn(0x7facb80fc170 msgr2=0x7facb80fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:32.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.730+0000 7facbd628700 1 -- 172.21.15.116:0/1336106837 shutdown_connections 2024-03-04T16:07:32.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.730+0000 7facbd628700 1 -- 172.21.15.116:0/1336106837 wait complete. 2024-03-04T16:07:32.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facbd628700 1 Processor -- start 2024-03-04T16:07:32.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facbd628700 1 -- start start 2024-03-04T16:07:32.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facbd628700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7facb8100bd0 0x7facb81963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:32.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facbd628700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb8101dd0 0x7facb81968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:32.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facbd628700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb8196f00 con 0x7facb8101dd0 2024-03-04T16:07:32.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facbd628700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb8197040 con 0x7facb8100bd0 2024-03-04T16:07:32.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facb67fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb8101dd0 0x7facb81968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:32.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facb67fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb8101dd0 0x7facb81968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43072/0 (socket says 172.21.15.116:43072) 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facb67fc700 1 -- 172.21.15.116:0/3226204457 learned_addr learned my addr 172.21.15.116:0/3226204457 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facb6ffd700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7facb8100bd0 0x7facb81963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facb67fc700 1 -- 172.21.15.116:0/3226204457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7facb8100bd0 msgr2=0x7facb81963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facb67fc700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7facb8100bd0 0x7facb81963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.731+0000 7facb67fc700 1 -- 172.21.15.116:0/3226204457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca80095f0 con 0x7facb8101dd0 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.732+0000 7facb67fc700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb8101dd0 0x7facb81968e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7facac00d900 tx=0x7facac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.732+0000 7fac9ffff700 1 -- 172.21.15.116:0/3226204457 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facac004a00 con 0x7facb8101dd0 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.732+0000 7fac9ffff700 1 -- 172.21.15.116:0/3226204457 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7facac0054d0 con 0x7facb8101dd0 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.732+0000 7facbd628700 1 -- 172.21.15.116:0/3226204457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb819baf0 con 0x7facb8101dd0 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.732+0000 7fac9ffff700 1 -- 172.21.15.116:0/3226204457 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facac009d60 con 0x7facb8101dd0 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.732+0000 7facbd628700 1 -- 172.21.15.116:0/3226204457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb819c040 con 0x7facb8101dd0 2024-03-04T16:07:32.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.733+0000 7facbd628700 1 -- 172.21.15.116:0/3226204457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb8109750 con 0x7facb8101dd0 2024-03-04T16:07:32.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.733+0000 7fac9ffff700 1 -- 172.21.15.116:0/3226204457 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7facac005020 con 0x7facb8101dd0 2024-03-04T16:07:32.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.734+0000 7fac9ffff700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faca006d8a0 0x7faca006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:32.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.734+0000 7facb6ffd700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faca006d8a0 0x7faca006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:32.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.734+0000 7fac9ffff700 1 -- 172.21.15.116:0/3226204457 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6500+0+0 (secure 0 0 0) 0x7facac08d4c0 con 0x7facb8101dd0 2024-03-04T16:07:32.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.735+0000 7facb6ffd700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faca006d8a0 0x7faca006fd60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faca80059d0 tx=0x7faca8005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:32.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:32.739+0000 7fac9ffff700 1 -- 172.21.15.116:0/3226204457 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7facac05a7d0 con 0x7facb8101dd0 2024-03-04T16:07:33.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.036+0000 7facbd628700 1 -- 172.21.15.116:0/3226204457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7facb8109940 con 0x7facb8101dd0 2024-03-04T16:07:33.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.037+0000 7fac9ffff700 1 -- 172.21.15.116:0/3226204457 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+687 (secure 0 0 0) 0x7facac05a360 con 0x7facb8101dd0 2024-03-04T16:07:33.038 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:07:33.038 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:07:33.038 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:07:33.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:07:33.038 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:07:33.038 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 9, 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:07:33.039 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:07:33.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 -- 172.21.15.116:0/3226204457 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faca006d8a0 msgr2=0x7faca006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faca006d8a0 0x7faca006fd60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faca80059d0 tx=0x7faca8005960 comp rx=0 tx=0).stop 2024-03-04T16:07:33.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 -- 172.21.15.116:0/3226204457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb8101dd0 msgr2=0x7facb81968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb8101dd0 0x7facb81968e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7facac00d900 tx=0x7facac00dcc0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 -- 172.21.15.116:0/3226204457 shutdown_connections 2024-03-04T16:07:33.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faca006d8a0 0x7faca006fd60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7facb8100bd0 0x7facb81963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 --2- 172.21.15.116:0/3226204457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb8101dd0 0x7facb81968e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 -- 172.21.15.116:0/3226204457 >> 172.21.15.116:0/3226204457 conn(0x7facb80fc170 msgr2=0x7facb8105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:33.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 -- 172.21.15.116:0/3226204457 shutdown_connections 2024-03-04T16:07:33.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.039+0000 7facbd628700 1 -- 172.21.15.116:0/3226204457 wait complete. 2024-03-04T16:07:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.144+0000 7fb9e1136700 1 -- 172.21.15.116:0/1707747623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc101dc0 msgr2=0x7fb9dc102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.144+0000 7fb9e1136700 1 --2- 172.21.15.116:0/1707747623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc101dc0 0x7fb9dc102220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009910 tx=0x7fb9d0009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.144+0000 7fb9e1136700 1 -- 172.21.15.116:0/1707747623 shutdown_connections 2024-03-04T16:07:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.144+0000 7fb9e1136700 1 --2- 172.21.15.116:0/1707747623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc101dc0 0x7fb9dc102220 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.144+0000 7fb9e1136700 1 --2- 172.21.15.116:0/1707747623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9dc100bc0 0x7fb9dc100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.144+0000 7fb9e1136700 1 -- 172.21.15.116:0/1707747623 >> 172.21.15.116:0/1707747623 conn(0x7fb9dc0fc140 msgr2=0x7fb9dc0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.144+0000 7fb9e1136700 1 -- 172.21.15.116:0/1707747623 shutdown_connections 2024-03-04T16:07:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.144+0000 7fb9e1136700 1 -- 172.21.15.116:0/1707747623 wait complete. 2024-03-04T16:07:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9e1136700 1 Processor -- start 2024-03-04T16:07:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9e1136700 1 -- start start 2024-03-04T16:07:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9e1136700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9dc100bc0 0x7fb9dc196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9e1136700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc101dc0 0x7fb9dc1968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9e1136700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc196ed0 con 0x7fb9dc101dc0 2024-03-04T16:07:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9e1136700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc197010 con 0x7fb9dc100bc0 2024-03-04T16:07:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9da59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc101dc0 0x7fb9dc1968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9da59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc101dc0 0x7fb9dc1968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43080/0 (socket says 172.21.15.116:43080) 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9da59c700 1 -- 172.21.15.116:0/2992532418 learned_addr learned my addr 172.21.15.116:0/2992532418 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9dad9d700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9dc100bc0 0x7fb9dc196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9da59c700 1 -- 172.21.15.116:0/2992532418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9dc100bc0 msgr2=0x7fb9dc196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9da59c700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9dc100bc0 0x7fb9dc196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.145+0000 7fb9da59c700 1 -- 172.21.15.116:0/2992532418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d00095f0 con 0x7fb9dc101dc0 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.146+0000 7fb9da59c700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc101dc0 0x7fb9dc1968b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0000c00 tx=0x7fb9d000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.146+0000 7fb9c3fff700 1 -- 172.21.15.116:0/2992532418 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9d001d070 con 0x7fb9dc101dc0 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.146+0000 7fb9e1136700 1 -- 172.21.15.116:0/2992532418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc19ba60 con 0x7fb9dc101dc0 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.146+0000 7fb9c3fff700 1 -- 172.21.15.116:0/2992532418 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb9d0005dc0 con 0x7fb9dc101dc0 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.146+0000 7fb9c3fff700 1 -- 172.21.15.116:0/2992532418 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9d00226a0 con 0x7fb9dc101dc0 2024-03-04T16:07:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.146+0000 7fb9e1136700 1 -- 172.21.15.116:0/2992532418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc19bf50 con 0x7fb9dc101dc0 2024-03-04T16:07:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.147+0000 7fb9e1136700 1 -- 172.21.15.116:0/2992532418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc109720 con 0x7fb9dc101dc0 2024-03-04T16:07:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.147+0000 7fb9c3fff700 1 -- 172.21.15.116:0/2992532418 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fb9d0003680 con 0x7fb9dc101dc0 2024-03-04T16:07:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.148+0000 7fb9c3fff700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb9c406d8a0 0x7fb9c406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.148+0000 7fb9c3fff700 1 -- 172.21.15.116:0/2992532418 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb9d008eb60 con 0x7fb9dc101dc0 2024-03-04T16:07:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.148+0000 7fb9dad9d700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb9c406d8a0 0x7fb9c406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.148+0000 7fb9dad9d700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb9c406d8a0 0x7fb9c406fd60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc101c20 tx=0x7fb9cc009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:33.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.152+0000 7fb9c3fff700 1 -- 172.21.15.116:0/2992532418 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb9d005be90 con 0x7fb9dc101dc0 2024-03-04T16:07:33.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.417+0000 7fb9e1136700 1 -- 172.21.15.116:0/2992532418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb9dc1098e0 con 0x7fb9dc101dc0 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.418+0000 7fb9c3fff700 1 -- 172.21.15.116:0/2992532418 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1852 (secure 0 0 0) 0x7fb9d0003930 con 0x7fb9dc101dc0 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:07:33.421 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:07:33.422 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:07:33.422 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:07:33.422 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:07:33.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 -- 172.21.15.116:0/2992532418 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb9c406d8a0 msgr2=0x7fb9c406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb9c406d8a0 0x7fb9c406fd60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc101c20 tx=0x7fb9cc009260 comp rx=0 tx=0).stop 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 -- 172.21.15.116:0/2992532418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc101dc0 msgr2=0x7fb9dc1968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc101dc0 0x7fb9dc1968b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0000c00 tx=0x7fb9d000f740 comp rx=0 tx=0).stop 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 -- 172.21.15.116:0/2992532418 shutdown_connections 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb9c406d8a0 0x7fb9c406fd60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9dc100bc0 0x7fb9dc196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 --2- 172.21.15.116:0/2992532418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc101dc0 0x7fb9dc1968b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 -- 172.21.15.116:0/2992532418 >> 172.21.15.116:0/2992532418 conn(0x7fb9dc0fc140 msgr2=0x7fb9dc104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 -- 172.21.15.116:0/2992532418 shutdown_connections 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.420+0000 7fb9e1136700 1 -- 172.21.15.116:0/2992532418 wait complete. 2024-03-04T16:07:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:07:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.528+0000 7fc086914700 1 -- 172.21.15.116:0/949963664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0800ffac0 msgr2=0x7fc0800ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.528+0000 7fc086914700 1 --2- 172.21.15.116:0/949963664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0800ffac0 0x7fc0800ffee0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc070009910 tx=0x7fc070009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.528+0000 7fc086914700 1 -- 172.21.15.116:0/949963664 shutdown_connections 2024-03-04T16:07:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.528+0000 7fc086914700 1 --2- 172.21.15.116:0/949963664 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc080100cc0 0x7fc080101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.528+0000 7fc086914700 1 --2- 172.21.15.116:0/949963664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0800ffac0 0x7fc0800ffee0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.528+0000 7fc086914700 1 -- 172.21.15.116:0/949963664 >> 172.21.15.116:0/949963664 conn(0x7fc0800fb060 msgr2=0x7fc0800fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.528+0000 7fc086914700 1 -- 172.21.15.116:0/949963664 shutdown_connections 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.528+0000 7fc086914700 1 -- 172.21.15.116:0/949963664 wait complete. 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.529+0000 7fc086914700 1 Processor -- start 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.529+0000 7fc086914700 1 -- start start 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.529+0000 7fc086914700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0800ffac0 0x7fc080069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.529+0000 7fc086914700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc080100cc0 0x7fc08019b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.529+0000 7fc086914700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08019b6c0 con 0x7fc080100cc0 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.529+0000 7fc086914700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08019b800 con 0x7fc0800ffac0 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.529+0000 7fc07f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc080100cc0 0x7fc08019b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.529+0000 7fc07f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc080100cc0 0x7fc08019b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43100/0 (socket says 172.21.15.116:43100) 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.529+0000 7fc07f7fe700 1 -- 172.21.15.116:0/3963584109 learned_addr learned my addr 172.21.15.116:0/3963584109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.529+0000 7fc07ffff700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0800ffac0 0x7fc080069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.530+0000 7fc07f7fe700 1 -- 172.21.15.116:0/3963584109 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0800ffac0 msgr2=0x7fc080069970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.530+0000 7fc07f7fe700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0800ffac0 0x7fc080069970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.530+0000 7fc07f7fe700 1 -- 172.21.15.116:0/3963584109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0700095f0 con 0x7fc080100cc0 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.530+0000 7fc07f7fe700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc080100cc0 0x7fc08019b0a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc07400d8d0 tx=0x7fc07400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.530+0000 7fc07d7fa700 1 -- 172.21.15.116:0/3963584109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0740098a0 con 0x7fc080100cc0 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.530+0000 7fc086914700 1 -- 172.21.15.116:0/3963584109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08019bae0 con 0x7fc080100cc0 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.530+0000 7fc07d7fa700 1 -- 172.21.15.116:0/3963584109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc074010460 con 0x7fc080100cc0 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.530+0000 7fc07d7fa700 1 -- 172.21.15.116:0/3963584109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc07400f5f0 con 0x7fc080100cc0 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.530+0000 7fc086914700 1 -- 172.21.15.116:0/3963584109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08019c030 con 0x7fc080100cc0 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.531+0000 7fc086914700 1 -- 172.21.15.116:0/3963584109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc080108620 con 0x7fc080100cc0 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.532+0000 7fc07d7fa700 1 -- 172.21.15.116:0/3963584109 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fc074009a00 con 0x7fc080100cc0 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.532+0000 7fc07d7fa700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc06806d8a0 0x7fc06806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.532+0000 7fc07d7fa700 1 -- 172.21.15.116:0/3963584109 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc07408d360 con 0x7fc080100cc0 2024-03-04T16:07:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.532+0000 7fc07ffff700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc06806d8a0 0x7fc06806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.532+0000 7fc07ffff700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc06806d8a0 0x7fc06806fd60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc070006010 tx=0x7fc07000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.536+0000 7fc07d7fa700 1 -- 172.21.15.116:0/3963584109 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc07405a690 con 0x7fc080100cc0 2024-03-04T16:07:33.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:33 smithi152 ceph-mon[82845]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:33.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:33 smithi152 ceph-mon[82845]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:33.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:33 smithi152 ceph-mon[82845]: from='osd.0 [v2:172.21.15.116:6802/2502615728,v1:172.21.15.116:6803/2502615728]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T16:07:33.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:33 smithi152 ceph-mon[82845]: osdmap e53: 6 total, 5 up, 6 in 2024-03-04T16:07:33.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:33 smithi152 ceph-mon[82845]: from='osd.0 [v2:172.21.15.116:6802/2502615728,v1:172.21.15.116:6803/2502615728]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T16:07:33.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:33 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:33.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:33 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:33.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:33 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3226204457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:33.646 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:33 smithi116 ceph-mon[94164]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:33.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:33 smithi116 ceph-mon[94164]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:33.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:33 smithi116 ceph-mon[94164]: from='osd.0 [v2:172.21.15.116:6802/2502615728,v1:172.21.15.116:6803/2502615728]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T16:07:33.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:33 smithi116 ceph-mon[94164]: osdmap e53: 6 total, 5 up, 6 in 2024-03-04T16:07:33.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:33 smithi116 ceph-mon[94164]: from='osd.0 [v2:172.21.15.116:6802/2502615728,v1:172.21.15.116:6803/2502615728]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T16:07:33.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:33 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:33.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:33 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:33.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:33 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3226204457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.756+0000 7fc086914700 1 -- 172.21.15.116:0/3963584109 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc080105610 con 0x7fc06806d8a0 2024-03-04T16:07:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.758+0000 7fc07d7fa700 1 -- 172.21.15.116:0/3963584109 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fc080105610 con 0x7fc06806d8a0 2024-03-04T16:07:33.758 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:07:33.758 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "7/23 daemons upgraded", 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:07:33.759 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 -- 172.21.15.116:0/3963584109 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc06806d8a0 msgr2=0x7fc06806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc06806d8a0 0x7fc06806fd60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc070006010 tx=0x7fc07000b540 comp rx=0 tx=0).stop 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 -- 172.21.15.116:0/3963584109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc080100cc0 msgr2=0x7fc08019b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc080100cc0 0x7fc08019b0a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc07400d8d0 tx=0x7fc07400dbe0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 -- 172.21.15.116:0/3963584109 shutdown_connections 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc06806d8a0 0x7fc06806fd60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0800ffac0 0x7fc080069970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 --2- 172.21.15.116:0/3963584109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc080100cc0 0x7fc08019b0a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 -- 172.21.15.116:0/3963584109 >> 172.21.15.116:0/3963584109 conn(0x7fc0800fb060 msgr2=0x7fc080103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 -- 172.21.15.116:0/3963584109 shutdown_connections 2024-03-04T16:07:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.760+0000 7fc086914700 1 -- 172.21.15.116:0/3963584109 wait complete. 2024-03-04T16:07:33.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.863+0000 7f42942ff700 1 -- 172.21.15.116:0/2080558899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428c103f70 msgr2=0x7f428c106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.863+0000 7f42942ff700 1 --2- 172.21.15.116:0/2080558899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428c103f70 0x7f428c106360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f427c009910 tx=0x7f427c009c20 comp rx=0 tx=0).stop 2024-03-04T16:07:33.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 -- 172.21.15.116:0/2080558899 shutdown_connections 2024-03-04T16:07:33.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 --2- 172.21.15.116:0/2080558899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428c103f70 0x7f428c106360 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 --2- 172.21.15.116:0/2080558899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f428c101640 0x7f428c103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 -- 172.21.15.116:0/2080558899 >> 172.21.15.116:0/2080558899 conn(0x7f428c0fb030 msgr2=0x7f428c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:33.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 -- 172.21.15.116:0/2080558899 shutdown_connections 2024-03-04T16:07:33.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 -- 172.21.15.116:0/2080558899 wait complete. 2024-03-04T16:07:33.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 Processor -- start 2024-03-04T16:07:33.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 -- start start 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f428c101640 0x7f428c196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428c103f70 0x7f428c196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c196e90 con 0x7f428c103f70 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f42942ff700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c196fd0 con 0x7f428c101640 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.864+0000 7f429189a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428c103f70 0x7f428c196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f429189a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428c103f70 0x7f428c196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43112/0 (socket says 172.21.15.116:43112) 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f429189a700 1 -- 172.21.15.116:0/3631409167 learned_addr learned my addr 172.21.15.116:0/3631409167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f429209b700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f428c101640 0x7f428c196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f429189a700 1 -- 172.21.15.116:0/3631409167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f428c101640 msgr2=0x7f428c196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f429189a700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f428c101640 0x7f428c196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f429189a700 1 -- 172.21.15.116:0/3631409167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c0095f0 con 0x7f428c103f70 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f429189a700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428c103f70 0x7f428c196870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f427c000c00 tx=0x7f427c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f42837fe700 1 -- 172.21.15.116:0/3631409167 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f427c01d070 con 0x7f428c103f70 2024-03-04T16:07:33.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f42837fe700 1 -- 172.21.15.116:0/3631409167 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f427c005dc0 con 0x7f428c103f70 2024-03-04T16:07:33.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f42942ff700 1 -- 172.21.15.116:0/3631409167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c19ba20 con 0x7f428c103f70 2024-03-04T16:07:33.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f42837fe700 1 -- 172.21.15.116:0/3631409167 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f427c0226a0 con 0x7f428c103f70 2024-03-04T16:07:33.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.865+0000 7f42942ff700 1 -- 172.21.15.116:0/3631409167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c19bf10 con 0x7f428c103f70 2024-03-04T16:07:33.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.866+0000 7f42942ff700 1 -- 172.21.15.116:0/3631409167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428c190520 con 0x7f428c103f70 2024-03-04T16:07:33.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.867+0000 7f42837fe700 1 -- 172.21.15.116:0/3631409167 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f427c0038f0 con 0x7f428c103f70 2024-03-04T16:07:33.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.867+0000 7f42837fe700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f427806d8a0 0x7f427806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:07:33.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.867+0000 7f42837fe700 1 -- 172.21.15.116:0/3631409167 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f427c08ea20 con 0x7f428c103f70 2024-03-04T16:07:33.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.867+0000 7f429209b700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f427806d8a0 0x7f427806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:07:33.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.867+0000 7f429209b700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f427806d8a0 0x7f427806fd60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4288009c60 tx=0x7f4288009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:07:33.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:33.872+0000 7f42837fe700 1 -- 172.21.15.116:0/3631409167 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f427c05bd50 con 0x7f428c103f70 2024-03-04T16:07:34.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.152+0000 7f42942ff700 1 -- 172.21.15.116:0/3631409167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f428c02cc70 con 0x7f428c103f70 2024-03-04T16:07:34.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.153+0000 7f42837fe700 1 -- 172.21.15.116:0/3631409167 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1182 (secure 0 0 0) 0x7f427c003680 con 0x7f428c103f70 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN Degraded data redundancy: 42/231 objects degraded (18.182%), 18 pgs degraded 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 42/231 objects degraded (18.182%), 18 pgs degraded 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 1.0 is active+undersized+degraded, acting [3,1] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.0 is active+undersized+degraded, acting [3,1] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1 is active+undersized+degraded, acting [2,1] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.4 is active+undersized+degraded, acting [1,4] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.5 is active+undersized+degraded, acting [3,4] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.8 is active+undersized+degraded, acting [3,5] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.9 is active+undersized+degraded, acting [1,4] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.c is active+undersized+degraded, acting [2,3] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.f is active+undersized+degraded, acting [4,5] 2024-03-04T16:07:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-03-04T16:07:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.13 is active+undersized+degraded, acting [4,2] 2024-03-04T16:07:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.15 is active+undersized+degraded, acting [1,3] 2024-03-04T16:07:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.19 is active+undersized+degraded, acting [4,2] 2024-03-04T16:07:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1d is active+undersized+degraded, acting [3,5] 2024-03-04T16:07:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1e is active+undersized+degraded, acting [2,5] 2024-03-04T16:07:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1f is active+undersized+degraded, acting [3,4] 2024-03-04T16:07:34.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 -- 172.21.15.116:0/3631409167 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f427806d8a0 msgr2=0x7f427806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f427806d8a0 0x7f427806fd60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4288009c60 tx=0x7f4288009260 comp rx=0 tx=0).stop 2024-03-04T16:07:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 -- 172.21.15.116:0/3631409167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428c103f70 msgr2=0x7f428c196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:07:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428c103f70 0x7f428c196870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f427c000c00 tx=0x7f427c00f690 comp rx=0 tx=0).stop 2024-03-04T16:07:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 -- 172.21.15.116:0/3631409167 shutdown_connections 2024-03-04T16:07:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f427806d8a0 0x7f427806fd60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f428c101640 0x7f428c196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 --2- 172.21.15.116:0/3631409167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428c103f70 0x7f428c196870 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:07:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 -- 172.21.15.116:0/3631409167 >> 172.21.15.116:0/3631409167 conn(0x7f428c0fb030 msgr2=0x7f428c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:07:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 -- 172.21.15.116:0/3631409167 shutdown_connections 2024-03-04T16:07:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:07:34.155+0000 7f42942ff700 1 -- 172.21.15.116:0/3631409167 wait complete. 2024-03-04T16:07:34.444 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:34 smithi116 ceph-mon[94164]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:34.444 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:34 smithi116 ceph-mon[94164]: pgmap v56: 65 pgs: 18 active+undersized+degraded, 16 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 307 B/s wr, 0 op/s; 42/231 objects degraded (18.182%) 2024-03-04T16:07:34.444 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:34 smithi116 ceph-mon[94164]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:07:34.444 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:34 smithi116 ceph-mon[94164]: osd.0 [v2:172.21.15.116:6802/2502615728,v1:172.21.15.116:6803/2502615728] boot 2024-03-04T16:07:34.444 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:34 smithi116 ceph-mon[94164]: osdmap e54: 6 total, 6 up, 6 in 2024-03-04T16:07:34.444 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:34 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:07:34.444 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:34 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2992532418' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:07:34.445 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:34 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3631409167' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:07:34.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:34 smithi152 ceph-mon[82845]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:34 smithi152 ceph-mon[82845]: pgmap v56: 65 pgs: 18 active+undersized+degraded, 16 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 307 B/s wr, 0 op/s; 42/231 objects degraded (18.182%) 2024-03-04T16:07:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:34 smithi152 ceph-mon[82845]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:07:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:34 smithi152 ceph-mon[82845]: osd.0 [v2:172.21.15.116:6802/2502615728,v1:172.21.15.116:6803/2502615728] boot 2024-03-04T16:07:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:34 smithi152 ceph-mon[82845]: osdmap e54: 6 total, 6 up, 6 in 2024-03-04T16:07:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:34 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:07:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:34 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2992532418' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:07:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:34 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3631409167' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:07:35.506 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:35 smithi116 ceph-mon[94164]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:35.514 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:35 smithi116 ceph-mon[94164]: osdmap e55: 6 total, 6 up, 6 in 2024-03-04T16:07:35.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:35 smithi152 ceph-mon[82845]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:35.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:35 smithi152 ceph-mon[82845]: osdmap e55: 6 total, 6 up, 6 in 2024-03-04T16:07:36.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:36 smithi116 ceph-mon[94164]: pgmap v59: 65 pgs: 25 peering, 3 active+undersized+degraded, 6 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7/231 objects degraded (3.030%) 2024-03-04T16:07:36.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:36 smithi116 ceph-mon[94164]: Health check update: Degraded data redundancy: 7/231 objects degraded (3.030%), 3 pgs degraded (PG_DEGRADED) 2024-03-04T16:07:36.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:36.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:36.465 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:36.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:07:36.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:36.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:36 smithi152 ceph-mon[82845]: pgmap v59: 65 pgs: 25 peering, 3 active+undersized+degraded, 6 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7/231 objects degraded (3.030%) 2024-03-04T16:07:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:36 smithi152 ceph-mon[82845]: Health check update: Degraded data redundancy: 7/231 objects degraded (3.030%), 3 pgs degraded (PG_DEGRADED) 2024-03-04T16:07:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:07:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:07:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:07:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:38 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:07:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:38 smithi152 ceph-mon[82845]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-03-04T16:07:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:38 smithi152 ceph-mon[82845]: pgmap v60: 65 pgs: 25 peering, 3 active+undersized+degraded, 6 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7/231 objects degraded (3.030%) 2024-03-04T16:07:38.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:38.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:38 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:07:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:38 smithi116 ceph-mon[94164]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-03-04T16:07:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:38 smithi116 ceph-mon[94164]: pgmap v60: 65 pgs: 25 peering, 3 active+undersized+degraded, 6 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7/231 objects degraded (3.030%) 2024-03-04T16:07:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:39 smithi116 ceph-mon[94164]: pgmap v61: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 41 B/s, 0 objects/s recovering 2024-03-04T16:07:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:39 smithi116 ceph-mon[94164]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/231 objects degraded (3.030%), 3 pgs degraded) 2024-03-04T16:07:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:39 smithi116 ceph-mon[94164]: Cluster is now healthy 2024-03-04T16:07:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:39 smithi152 ceph-mon[82845]: pgmap v61: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 41 B/s, 0 objects/s recovering 2024-03-04T16:07:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:39 smithi152 ceph-mon[82845]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/231 objects degraded (3.030%), 3 pgs degraded) 2024-03-04T16:07:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:39 smithi152 ceph-mon[82845]: Cluster is now healthy 2024-03-04T16:07:42.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:42 smithi152 ceph-mon[82845]: pgmap v62: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 35 B/s, 0 objects/s recovering 2024-03-04T16:07:42.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:42 smithi116 ceph-mon[94164]: pgmap v62: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 35 B/s, 0 objects/s recovering 2024-03-04T16:07:44.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:44 smithi152 ceph-mon[82845]: pgmap v63: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29 B/s, 0 objects/s recovering 2024-03-04T16:07:44.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:44 smithi116 ceph-mon[94164]: pgmap v63: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29 B/s, 0 objects/s recovering 2024-03-04T16:07:46.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:46 smithi116 ceph-mon[94164]: pgmap v64: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 26 B/s, 0 objects/s recovering 2024-03-04T16:07:46.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:46 smithi152 ceph-mon[82845]: pgmap v64: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 26 B/s, 0 objects/s recovering 2024-03-04T16:07:48.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:48 smithi152 ceph-mon[82845]: pgmap v65: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-04T16:07:48.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:48 smithi116 ceph-mon[94164]: pgmap v65: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-04T16:07:49.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:50 smithi116 ceph-mon[94164]: pgmap v66: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-04T16:07:50.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:50 smithi152 ceph-mon[82845]: pgmap v66: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-04T16:07:52.303 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:52 smithi116 ceph-mon[94164]: pgmap v67: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:52.303 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:07:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:52 smithi152 ceph-mon[82845]: pgmap v67: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:07:53.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:53 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:07:53.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:53 smithi116 ceph-mon[94164]: Upgrade: osd.1 is safe to restart 2024-03-04T16:07:53.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:53.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T16:07:53.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:53.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:53.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:53.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:53 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:07:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:53 smithi152 ceph-mon[82845]: Upgrade: osd.1 is safe to restart 2024-03-04T16:07:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T16:07:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:07:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:54.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:54 smithi152 ceph-mon[82845]: Upgrade: Updating osd.1 2024-03-04T16:07:54.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:54 smithi152 ceph-mon[82845]: Deploying daemon osd.1 on smithi116 2024-03-04T16:07:54.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:54 smithi152 ceph-mon[82845]: pgmap v68: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:54.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:54 smithi116 ceph-mon[94164]: Upgrade: Updating osd.1 2024-03-04T16:07:54.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:54 smithi116 ceph-mon[94164]: Deploying daemon osd.1 on smithi116 2024-03-04T16:07:54.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:54 smithi116 ceph-mon[94164]: pgmap v68: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:56 smithi116 ceph-mon[94164]: pgmap v69: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:56.476 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:56 smithi116 systemd[1]: Stopping Ceph osd.1 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:56.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:56 smithi152 ceph-mon[82845]: pgmap v69: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:56.975 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:56 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[56141]: 2024-03-04T16:07:56.549+0000 7fa8bca69700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:07:56.975 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:56 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[56141]: 2024-03-04T16:07:56.549+0000 7fa8bca69700 -1 osd.1 55 *** Got signal Terminated *** 2024-03-04T16:07:56.975 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:56 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[56141]: 2024-03-04T16:07:56.549+0000 7fa8bca69700 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:07:57.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:57 smithi116 ceph-mon[94164]: osd.1 marked itself down and dead 2024-03-04T16:07:57.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:57 smithi152 ceph-mon[82845]: osd.1 marked itself down and dead 2024-03-04T16:07:57.910 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:57 smithi116 podman[107769]: 2024-03-04 16:07:57.62289704 +0000 UTC m=+1.311903620 container died d8b733db1a468e32aef040ceb33031e864b89943df77b068b6c7515bf8a5c106 (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-03-04T16:07:58.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:58 smithi116 ceph-mon[94164]: pgmap v70: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:58.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:58 smithi116 ceph-mon[94164]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:07:58.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:58 smithi116 ceph-mon[94164]: osdmap e56: 6 total, 5 up, 6 in 2024-03-04T16:07:58.225 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:58 smithi116 podman[107769]: 2024-03-04 16:07:58.000393949 +0000 UTC m=+1.689400535 container cleanup d8b733db1a468e32aef040ceb33031e864b89943df77b068b6c7515bf8a5c106 (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-03-04T16:07:58.225 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:58 smithi116 bash[107769]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1 2024-03-04T16:07:58.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:58 smithi152 ceph-mon[82845]: pgmap v70: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:07:58.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:58 smithi152 ceph-mon[82845]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:07:58.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:58 smithi152 ceph-mon[82845]: osdmap e56: 6 total, 5 up, 6 in 2024-03-04T16:07:58.613 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:58 smithi116 podman[107788]: 2024-03-04 16:07:58.260011222 +0000 UTC m=+0.636339763 container remove d8b733db1a468e32aef040ceb33031e864b89943df77b068b6c7515bf8a5c106 (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-03-04T16:07:58.867 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:58 smithi116 podman[107884]: 2024-03-04 16:07:58.515838195 +0000 UTC m=+0.018055046 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:07:58.867 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:58 smithi116 podman[107884]: 2024-03-04 16:07:58.635742217 +0000 UTC m=+0.137959057 container create d35fc9502a55ebc4f88d270bcb8f7ee289ba497e3de23d7029aca5686ddb9d43 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-04T16:07:58.867 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:58 smithi116 podman[107884]: 2024-03-04 16:07:58.863148791 +0000 UTC m=+0.365365626 container init d35fc9502a55ebc4f88d270bcb8f7ee289ba497e3de23d7029aca5686ddb9d43 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:07:58.867 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:58 smithi116 podman[107884]: 2024-03-04 16:07:58.867385809 +0000 UTC m=+0.369602650 container start d35fc9502a55ebc4f88d270bcb8f7ee289ba497e3de23d7029aca5686ddb9d43 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222) 2024-03-04T16:07:59.130 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:58 smithi116 podman[107884]: 2024-03-04 16:07:58.97180656 +0000 UTC m=+0.474023409 container attach d35fc9502a55ebc4f88d270bcb8f7ee289ba497e3de23d7029aca5686ddb9d43 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-04T16:07:59.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:07:59 smithi152 ceph-mon[82845]: osdmap e57: 6 total, 5 up, 6 in 2024-03-04T16:07:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:07:59 smithi116 ceph-mon[94164]: osdmap e57: 6 total, 5 up, 6 in 2024-03-04T16:07:59.475 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:59 smithi116 podman[107884]: 2024-03-04 16:07:59.16107663 +0000 UTC m=+0.663293473 container died d35fc9502a55ebc4f88d270bcb8f7ee289ba497e3de23d7029aca5686ddb9d43 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-04T16:08:00.132 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:59 smithi116 podman[107910]: 2024-03-04 16:07:59.808720876 +0000 UTC m=+0.642708342 container remove d35fc9502a55ebc4f88d270bcb8f7ee289ba497e3de23d7029aca5686ddb9d43 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-04T16:08:00.132 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:59 smithi116 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-04T16:08:00.132 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:59 smithi116 systemd[1]: Stopped Ceph osd.1 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:08:00.133 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:07:59 smithi116 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.1.service: Consumed 1min 6.430s CPU time. 2024-03-04T16:08:00.133 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 systemd[1]: Starting Ceph osd.1 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:08:00.391 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:00 smithi116 ceph-mon[94164]: pgmap v73: 65 pgs: 11 active+undersized+degraded, 5 peering, 18 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 28/231 objects degraded (12.121%) 2024-03-04T16:08:00.391 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:00 smithi116 ceph-mon[94164]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-04T16:08:00.391 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:00 smithi116 ceph-mon[94164]: Health check failed: Degraded data redundancy: 28/231 objects degraded (12.121%), 11 pgs degraded (PG_DEGRADED) 2024-03-04T16:08:00.392 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 podman[108014]: 2024-03-04 16:08:00.156661897 +0000 UTC m=+0.018389438 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:08:00.392 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 podman[108014]: 2024-03-04 16:08:00.26805665 +0000 UTC m=+0.129784228 container create fd9ceb129174cc76a978a71848984c7ece19f97b6e2e253d329236b52032967a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:08:00.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:00 smithi152 ceph-mon[82845]: pgmap v73: 65 pgs: 11 active+undersized+degraded, 5 peering, 18 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 28/231 objects degraded (12.121%) 2024-03-04T16:08:00.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:00 smithi152 ceph-mon[82845]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-04T16:08:00.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:00 smithi152 ceph-mon[82845]: Health check failed: Degraded data redundancy: 28/231 objects degraded (12.121%), 11 pgs degraded (PG_DEGRADED) 2024-03-04T16:08:00.936 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 podman[108014]: 2024-03-04 16:08:00.495546165 +0000 UTC m=+0.357273707 container init fd9ceb129174cc76a978a71848984c7ece19f97b6e2e253d329236b52032967a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:08:00.936 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 podman[108014]: 2024-03-04 16:08:00.498674609 +0000 UTC m=+0.360402141 container start fd9ceb129174cc76a978a71848984c7ece19f97b6e2e253d329236b52032967a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222) 2024-03-04T16:08:00.936 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 podman[108014]: 2024-03-04 16:08:00.570302726 +0000 UTC m=+0.432030267 container attach fd9ceb129174cc76a978a71848984c7ece19f97b6e2e253d329236b52032967a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-04T16:08:01.225 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate[108028]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T16:08:01.225 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate[108028]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-04T16:08:01.225 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 bash[108014]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T16:08:01.225 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 bash[108014]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-04T16:08:01.225 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 bash[108014]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-04T16:08:01.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 bash[108014]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-04T16:08:01.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 bash[108014]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-04T16:08:01.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 bash[108014]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T16:08:01.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 bash[108014]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-04T16:08:01.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate[108028]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-04T16:08:01.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate[108028]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-04T16:08:01.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate[108028]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-04T16:08:01.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate[108028]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T16:08:01.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate[108028]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-04T16:08:01.226 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:00 smithi116 podman[108014]: 2024-03-04 16:08:00.960851405 +0000 UTC m=+0.822578931 container died fd9ceb129174cc76a978a71848984c7ece19f97b6e2e253d329236b52032967a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T16:08:01.844 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:01 smithi116 podman[108169]: 2024-03-04 16:08:01.524067556 +0000 UTC m=+0.558024166 container remove fd9ceb129174cc76a978a71848984c7ece19f97b6e2e253d329236b52032967a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-activate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-04T16:08:01.844 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:01 smithi116 podman[108232]: 2024-03-04 16:08:01.63877696 +0000 UTC m=+0.019993034 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:08:01.844 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:01 smithi116 podman[108232]: 2024-03-04 16:08:01.741484555 +0000 UTC m=+0.122700625 container create 3e403d77dd721861150a9374548e56db48d1c4d0a572cef21d8caa8db8227e47 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-04T16:08:02.133 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:02 smithi116 podman[108232]: 2024-03-04 16:08:02.019643647 +0000 UTC m=+0.400859717 container init 3e403d77dd721861150a9374548e56db48d1c4d0a572cef21d8caa8db8227e47 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD) 2024-03-04T16:08:02.134 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:02 smithi116 podman[108232]: 2024-03-04 16:08:02.022796132 +0000 UTC m=+0.404012202 container start 3e403d77dd721861150a9374548e56db48d1c4d0a572cef21d8caa8db8227e47 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-04T16:08:02.134 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:02 smithi116 ceph-osd[108252]: -- 172.21.15.116:0/4284702011 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555930625680 con 0x5559305ef000 2024-03-04T16:08:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:02 smithi116 ceph-mon[94164]: pgmap v74: 65 pgs: 11 active+undersized+degraded, 5 peering, 18 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 28/231 objects degraded (12.121%) 2024-03-04T16:08:02.475 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:02 smithi116 bash[108232]: 3e403d77dd721861150a9374548e56db48d1c4d0a572cef21d8caa8db8227e47 2024-03-04T16:08:02.475 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:02 smithi116 systemd[1]: Started Ceph osd.1 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:08:02.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:02 smithi152 ceph-mon[82845]: pgmap v74: 65 pgs: 11 active+undersized+degraded, 5 peering, 18 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 28/231 objects degraded (12.121%) 2024-03-04T16:08:02.976 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:02 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[108248]: 2024-03-04T16:08:02.565+0000 7f8f995be7c0 -1 Falling back to public interface 2024-03-04T16:08:03.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:03 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:03.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:03 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:03.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:03 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:03.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:03 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:03.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:03 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:03.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:03 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:04.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.275+0000 7f4d14d64700 1 -- 172.21.15.116:0/946382640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d101016a0 msgr2=0x7f4d10103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:04.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.275+0000 7f4d14d64700 1 --2- 172.21.15.116:0/946382640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d101016a0 0x7f4d10103a90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009910 tx=0x7f4d00009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:04.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.275+0000 7f4d14d64700 1 -- 172.21.15.116:0/946382640 shutdown_connections 2024-03-04T16:08:04.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.275+0000 7f4d14d64700 1 --2- 172.21.15.116:0/946382640 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d10103fd0 0x7f4d101063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.275+0000 7f4d14d64700 1 --2- 172.21.15.116:0/946382640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d101016a0 0x7f4d10103a90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.275+0000 7f4d14d64700 1 -- 172.21.15.116:0/946382640 >> 172.21.15.116:0/946382640 conn(0x7f4d100fb030 msgr2=0x7f4d100fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:04.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.275+0000 7f4d14d64700 1 -- 172.21.15.116:0/946382640 shutdown_connections 2024-03-04T16:08:04.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.275+0000 7f4d14d64700 1 -- 172.21.15.116:0/946382640 wait complete. 2024-03-04T16:08:04.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.276+0000 7f4d14d64700 1 Processor -- start 2024-03-04T16:08:04.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.276+0000 7f4d14d64700 1 -- start start 2024-03-04T16:08:04.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.276+0000 7f4d14d64700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d101016a0 0x7f4d1019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:04.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.276+0000 7f4d14d64700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d10103fd0 0x7f4d1019ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:04.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.276+0000 7f4d14d64700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1019d480 con 0x7f4d101016a0 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.276+0000 7f4d14d64700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1019d5c0 con 0x7f4d10103fd0 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.276+0000 7f4d0e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d101016a0 0x7f4d1019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.276+0000 7f4d0dd9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d10103fd0 0x7f4d1019ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.276+0000 7f4d0e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d101016a0 0x7f4d1019c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45984/0 (socket says 172.21.15.116:45984) 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.276+0000 7f4d0e59c700 1 -- 172.21.15.116:0/2803169678 learned_addr learned my addr 172.21.15.116:0/2803169678 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.277+0000 7f4d0e59c700 1 -- 172.21.15.116:0/2803169678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d10103fd0 msgr2=0x7f4d1019ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.277+0000 7f4d0e59c700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d10103fd0 0x7f4d1019ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.277+0000 7f4d0e59c700 1 -- 172.21.15.116:0/2803169678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d000095f0 con 0x7f4d101016a0 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.277+0000 7f4d0dd9b700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d10103fd0 0x7f4d1019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.277+0000 7f4d0e59c700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d101016a0 0x7f4d1019c920 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4d00006010 tx=0x7f4d0000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.277+0000 7f4cff7fe700 1 -- 172.21.15.116:0/2803169678 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d0001d070 con 0x7f4d101016a0 2024-03-04T16:08:04.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.277+0000 7f4cff7fe700 1 -- 172.21.15.116:0/2803169678 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4d00005dc0 con 0x7f4d101016a0 2024-03-04T16:08:04.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.277+0000 7f4d14d64700 1 -- 172.21.15.116:0/2803169678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d101a2010 con 0x7f4d101016a0 2024-03-04T16:08:04.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.277+0000 7f4cff7fe700 1 -- 172.21.15.116:0/2803169678 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d00021410 con 0x7f4d101016a0 2024-03-04T16:08:04.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.277+0000 7f4d14d64700 1 -- 172.21.15.116:0/2803169678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d101a2580 con 0x7f4d101016a0 2024-03-04T16:08:04.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.278+0000 7f4d14d64700 1 -- 172.21.15.116:0/2803169678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d100fcc20 con 0x7f4d101016a0 2024-03-04T16:08:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.279+0000 7f4cff7fe700 1 -- 172.21.15.116:0/2803169678 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f4d0000f460 con 0x7f4d101016a0 2024-03-04T16:08:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.279+0000 7f4cff7fe700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4cf806d8a0 0x7f4cf806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.280+0000 7f4cff7fe700 1 -- 172.21.15.116:0/2803169678 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4d0008e6c0 con 0x7f4d101016a0 2024-03-04T16:08:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.280+0000 7f4d0dd9b700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4cf806d8a0 0x7f4cf806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.280+0000 7f4d0dd9b700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4cf806d8a0 0x7f4cf806fd60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4d040104e0 tx=0x7f4d0400b3c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:04.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.287+0000 7f4cff7fe700 1 -- 172.21.15.116:0/2803169678 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4d0005ba70 con 0x7f4d101016a0 2024-03-04T16:08:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.509+0000 7f4d14d64700 1 -- 172.21.15.116:0/2803169678 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d100611d0 con 0x7f4cf806d8a0 2024-03-04T16:08:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.511+0000 7f4cff7fe700 1 -- 172.21.15.116:0/2803169678 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f4d100611d0 con 0x7f4cf806d8a0 2024-03-04T16:08:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 -- 172.21.15.116:0/2803169678 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4cf806d8a0 msgr2=0x7f4cf806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4cf806d8a0 0x7f4cf806fd60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4d040104e0 tx=0x7f4d0400b3c0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 -- 172.21.15.116:0/2803169678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d101016a0 msgr2=0x7f4d1019c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d101016a0 0x7f4d1019c920 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4d00006010 tx=0x7f4d0000bd70 comp rx=0 tx=0).stop 2024-03-04T16:08:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 -- 172.21.15.116:0/2803169678 shutdown_connections 2024-03-04T16:08:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4cf806d8a0 0x7f4cf806fd60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d101016a0 0x7f4d1019c920 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 --2- 172.21.15.116:0/2803169678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d10103fd0 0x7f4d1019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 -- 172.21.15.116:0/2803169678 >> 172.21.15.116:0/2803169678 conn(0x7f4d100fb030 msgr2=0x7f4d10104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:04.515 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:04 smithi116 ceph-mon[94164]: pgmap v75: 65 pgs: 12 active+undersized+degraded, 22 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:04.515 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:04 smithi116 ceph-mon[94164]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-04T16:08:04.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 -- 172.21.15.116:0/2803169678 shutdown_connections 2024-03-04T16:08:04.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.513+0000 7f4d14d64700 1 -- 172.21.15.116:0/2803169678 wait complete. 2024-03-04T16:08:04.528 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:08:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:04 smithi152 ceph-mon[82845]: pgmap v75: 65 pgs: 12 active+undersized+degraded, 22 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:04 smithi152 ceph-mon[82845]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-04T16:08:04.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.622+0000 7fe3229e6700 1 -- 172.21.15.116:0/1433591092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c100cb0 msgr2=0x7fe31c101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.622+0000 7fe3229e6700 1 --2- 172.21.15.116:0/1433591092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c100cb0 0x7fe31c101110 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe310009910 tx=0x7fe310009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.622+0000 7fe3229e6700 1 -- 172.21.15.116:0/1433591092 shutdown_connections 2024-03-04T16:08:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.622+0000 7fe3229e6700 1 --2- 172.21.15.116:0/1433591092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c100cb0 0x7fe31c101110 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.622+0000 7fe3229e6700 1 --2- 172.21.15.116:0/1433591092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe31c0ffab0 0x7fe31c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.622+0000 7fe3229e6700 1 -- 172.21.15.116:0/1433591092 >> 172.21.15.116:0/1433591092 conn(0x7fe31c0fb030 msgr2=0x7fe31c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.622+0000 7fe3229e6700 1 -- 172.21.15.116:0/1433591092 shutdown_connections 2024-03-04T16:08:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.622+0000 7fe3229e6700 1 -- 172.21.15.116:0/1433591092 wait complete. 2024-03-04T16:08:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.623+0000 7fe3229e6700 1 Processor -- start 2024-03-04T16:08:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.623+0000 7fe3229e6700 1 -- start start 2024-03-04T16:08:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.623+0000 7fe3229e6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c0ffab0 0x7fe31c103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.623+0000 7fe3229e6700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe31c100cb0 0x7fe31c102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.623+0000 7fe3229e6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c1025d0 con 0x7fe31c0ffab0 2024-03-04T16:08:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.623+0000 7fe3229e6700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c102710 con 0x7fe31c100cb0 2024-03-04T16:08:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.623+0000 7fe31bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c0ffab0 0x7fe31c103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.623+0000 7fe31bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c0ffab0 0x7fe31c103a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58304/0 (socket says 172.21.15.116:58304) 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.623+0000 7fe31bfff700 1 -- 172.21.15.116:0/3077252991 learned_addr learned my addr 172.21.15.116:0/3077252991 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.623+0000 7fe31b7fe700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe31c100cb0 0x7fe31c102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.624+0000 7fe31bfff700 1 -- 172.21.15.116:0/3077252991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe31c100cb0 msgr2=0x7fe31c102090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.624+0000 7fe31bfff700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe31c100cb0 0x7fe31c102090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.624+0000 7fe31bfff700 1 -- 172.21.15.116:0/3077252991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3100095f0 con 0x7fe31c0ffab0 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.624+0000 7fe31bfff700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c0ffab0 0x7fe31c103a40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00c8f0 tx=0x7fe30c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.624+0000 7fe3197fa700 1 -- 172.21.15.116:0/3077252991 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe30c0078e0 con 0x7fe31c0ffab0 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.624+0000 7fe3229e6700 1 -- 172.21.15.116:0/3077252991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31c1029f0 con 0x7fe31c0ffab0 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.624+0000 7fe3197fa700 1 -- 172.21.15.116:0/3077252991 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe30c00f450 con 0x7fe31c0ffab0 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.624+0000 7fe3197fa700 1 -- 172.21.15.116:0/3077252991 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe30c018610 con 0x7fe31c0ffab0 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.624+0000 7fe3229e6700 1 -- 172.21.15.116:0/3077252991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31c19e3a0 con 0x7fe31c0ffab0 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.625+0000 7fe3229e6700 1 -- 172.21.15.116:0/3077252991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c108610 con 0x7fe31c0ffab0 2024-03-04T16:08:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.625+0000 7fe3197fa700 1 -- 172.21.15.116:0/3077252991 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fe30c00f5c0 con 0x7fe31c0ffab0 2024-03-04T16:08:04.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.626+0000 7fe3197fa700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe30406d8a0 0x7fe30406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:04.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.626+0000 7fe3197fa700 1 -- 172.21.15.116:0/3077252991 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe30c08d3c0 con 0x7fe31c0ffab0 2024-03-04T16:08:04.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.626+0000 7fe31b7fe700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe30406d8a0 0x7fe30406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:04.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.626+0000 7fe31b7fe700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe30406d8a0 0x7fe30406fd60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe310006010 tx=0x7fe31000b560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.630+0000 7fe3197fa700 1 -- 172.21.15.116:0/3077252991 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe30c05a6f0 con 0x7fe31c0ffab0 2024-03-04T16:08:04.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.849+0000 7fe3229e6700 1 -- 172.21.15.116:0/3077252991 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe31c19e950 con 0x7fe30406d8a0 2024-03-04T16:08:04.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.850+0000 7fe3197fa700 1 -- 172.21.15.116:0/3077252991 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe31c19e950 con 0x7fe30406d8a0 2024-03-04T16:08:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 -- 172.21.15.116:0/3077252991 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe30406d8a0 msgr2=0x7fe30406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe30406d8a0 0x7fe30406fd60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe310006010 tx=0x7fe31000b560 comp rx=0 tx=0).stop 2024-03-04T16:08:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 -- 172.21.15.116:0/3077252991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c0ffab0 msgr2=0x7fe31c103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c0ffab0 0x7fe31c103a40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00c8f0 tx=0x7fe30c00cc00 comp rx=0 tx=0).stop 2024-03-04T16:08:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 -- 172.21.15.116:0/3077252991 shutdown_connections 2024-03-04T16:08:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe30406d8a0 0x7fe30406fd60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c0ffab0 0x7fe31c103a40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 --2- 172.21.15.116:0/3077252991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe31c100cb0 0x7fe31c102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 -- 172.21.15.116:0/3077252991 >> 172.21.15.116:0/3077252991 conn(0x7fe31c0fb030 msgr2=0x7fe31c068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 -- 172.21.15.116:0/3077252991 shutdown_connections 2024-03-04T16:08:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.852+0000 7fe3229e6700 1 -- 172.21.15.116:0/3077252991 wait complete. 2024-03-04T16:08:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.956+0000 7f26cf12c700 1 -- 172.21.15.116:0/1529147349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26c8100cb0 msgr2=0x7f26c8101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.956+0000 7f26cf12c700 1 --2- 172.21.15.116:0/1529147349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26c8100cb0 0x7f26c8101110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009910 tx=0x7f26b4009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.956+0000 7f26cf12c700 1 -- 172.21.15.116:0/1529147349 shutdown_connections 2024-03-04T16:08:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.956+0000 7f26cf12c700 1 --2- 172.21.15.116:0/1529147349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26c8100cb0 0x7f26c8101110 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.956+0000 7f26cf12c700 1 --2- 172.21.15.116:0/1529147349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26c80ffab0 0x7f26c80ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.956+0000 7f26cf12c700 1 -- 172.21.15.116:0/1529147349 >> 172.21.15.116:0/1529147349 conn(0x7f26c80fb030 msgr2=0x7f26c80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.956+0000 7f26cf12c700 1 -- 172.21.15.116:0/1529147349 shutdown_connections 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.956+0000 7f26cf12c700 1 -- 172.21.15.116:0/1529147349 wait complete. 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.956+0000 7f26cf12c700 1 Processor -- start 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.956+0000 7f26cf12c700 1 -- start start 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26cf12c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26c80ffab0 0x7f26c8195260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26cf12c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26c8100cb0 0x7f26c81957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26cf12c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c8195dc0 con 0x7f26c8100cb0 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26cf12c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c8195f00 con 0x7f26c80ffab0 2024-03-04T16:08:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26c7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26c8100cb0 0x7f26c81957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26c7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26c8100cb0 0x7f26c81957a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58320/0 (socket says 172.21.15.116:58320) 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26c7fff700 1 -- 172.21.15.116:0/2272059077 learned_addr learned my addr 172.21.15.116:0/2272059077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26ccec8700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26c80ffab0 0x7f26c8195260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26c7fff700 1 -- 172.21.15.116:0/2272059077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26c80ffab0 msgr2=0x7f26c8195260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26c7fff700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26c80ffab0 0x7f26c8195260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26c7fff700 1 -- 172.21.15.116:0/2272059077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b40095f0 con 0x7f26c8100cb0 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26c7fff700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26c8100cb0 0x7f26c81957a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f26b4006010 tx=0x7f26b400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.957+0000 7f26c5ffb700 1 -- 172.21.15.116:0/2272059077 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b401d070 con 0x7f26c8100cb0 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.958+0000 7f26c5ffb700 1 -- 172.21.15.116:0/2272059077 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f26b400f460 con 0x7f26c8100cb0 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.958+0000 7f26cf12c700 1 -- 172.21.15.116:0/2272059077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c819a950 con 0x7f26c8100cb0 2024-03-04T16:08:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.958+0000 7f26c5ffb700 1 -- 172.21.15.116:0/2272059077 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b4021640 con 0x7f26c8100cb0 2024-03-04T16:08:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.958+0000 7f26cf12c700 1 -- 172.21.15.116:0/2272059077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c819ae40 con 0x7f26c8100cb0 2024-03-04T16:08:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.959+0000 7f26cf12c700 1 -- 172.21.15.116:0/2272059077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c8108610 con 0x7f26c8100cb0 2024-03-04T16:08:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.959+0000 7f26c5ffb700 1 -- 172.21.15.116:0/2272059077 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f26b400fab0 con 0x7f26c8100cb0 2024-03-04T16:08:04.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.959+0000 7f26c5ffb700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f26b806d8a0 0x7f26b806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:04.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.960+0000 7f26c5ffb700 1 -- 172.21.15.116:0/2272059077 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f26b408e800 con 0x7f26c8100cb0 2024-03-04T16:08:04.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.960+0000 7f26ccec8700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f26b806d8a0 0x7f26b806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:04.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.960+0000 7f26ccec8700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f26b806d8a0 0x7f26b806fd60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f26c8100b10 tx=0x7f26bc00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:04.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:04.964+0000 7f26c5ffb700 1 -- 172.21.15.116:0/2272059077 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f26b405bb30 con 0x7f26c8100cb0 2024-03-04T16:08:05.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.186+0000 7f26cf12c700 1 -- 172.21.15.116:0/2272059077 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f26c8105600 con 0x7f26b806d8a0 2024-03-04T16:08:05.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.197+0000 7f26c5ffb700 1 -- 172.21.15.116:0/2272059077 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f26c8105600 con 0x7f26b806d8a0 2024-03-04T16:08:05.198 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:08:05.198 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (11m) 33s ago 12m 25.3M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:08:05.198 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (12m) 33s ago 12m 7876k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:08:05.198 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (11m) 56s ago 11m 7838k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:08:05.198 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (65s) 33s ago 12m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:08:05.198 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (60s) 56s ago 11m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:08:05.198 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (10m) 33s ago 12m 88.2M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (6m) 33s ago 6m 19.4M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (7m) 33s ago 7m 15.6M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (6m) 56s ago 6m 16.7M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (6m) 56s ago 6m 143M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (3m) 33s ago 14m 497M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (2m) 56s ago 11m 449M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (2m) 33s ago 14m 44.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (111s) 56s ago 11m 33.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (12m) 33s ago 12m 15.5M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (11m) 56s ago 11m 16.7M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (39s) 33s ago 10m 12.8M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 starting - - - 1216M 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (9m) 33s ago 9m 196M 1216M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (9m) 56s ago 9m 268M 2240M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:08:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (8m) 56s ago 8m 238M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:08:05.200 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (8m) 56s ago 8m 228M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:08:05.200 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (2m) 33s ago 12m 62.8M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:08:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.199+0000 7f26cf12c700 1 -- 172.21.15.116:0/2272059077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f26b806d8a0 msgr2=0x7f26b806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.199+0000 7f26cf12c700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f26b806d8a0 0x7f26b806fd60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f26c8100b10 tx=0x7f26bc00b410 comp rx=0 tx=0).stop 2024-03-04T16:08:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.199+0000 7f26cf12c700 1 -- 172.21.15.116:0/2272059077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26c8100cb0 msgr2=0x7f26c81957a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.199+0000 7f26cf12c700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26c8100cb0 0x7f26c81957a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f26b4006010 tx=0x7f26b400bd70 comp rx=0 tx=0).stop 2024-03-04T16:08:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.199+0000 7f26cf12c700 1 -- 172.21.15.116:0/2272059077 shutdown_connections 2024-03-04T16:08:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.199+0000 7f26cf12c700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f26b806d8a0 0x7f26b806fd60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.199+0000 7f26cf12c700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26c80ffab0 0x7f26c8195260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.199+0000 7f26cf12c700 1 --2- 172.21.15.116:0/2272059077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26c8100cb0 0x7f26c81957a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.199+0000 7f26cf12c700 1 -- 172.21.15.116:0/2272059077 >> 172.21.15.116:0/2272059077 conn(0x7f26c80fb030 msgr2=0x7f26c8103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.200+0000 7f26cf12c700 1 -- 172.21.15.116:0/2272059077 shutdown_connections 2024-03-04T16:08:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.200+0000 7f26cf12c700 1 -- 172.21.15.116:0/2272059077 wait complete. 2024-03-04T16:08:05.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.313+0000 7f2b9d255700 1 -- 172.21.15.116:0/2327690070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b98101dc0 msgr2=0x7f2b98102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.313+0000 7f2b9d255700 1 --2- 172.21.15.116:0/2327690070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b98101dc0 0x7f2b98102220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c009910 tx=0x7f2b8c009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.313+0000 7f2b9d255700 1 -- 172.21.15.116:0/2327690070 shutdown_connections 2024-03-04T16:08:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.313+0000 7f2b9d255700 1 --2- 172.21.15.116:0/2327690070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b98101dc0 0x7f2b98102220 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.313+0000 7f2b9d255700 1 --2- 172.21.15.116:0/2327690070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b98100bc0 0x7f2b98100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.313+0000 7f2b9d255700 1 -- 172.21.15.116:0/2327690070 >> 172.21.15.116:0/2327690070 conn(0x7f2b980fc140 msgr2=0x7f2b980fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.313+0000 7f2b9d255700 1 -- 172.21.15.116:0/2327690070 shutdown_connections 2024-03-04T16:08:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.313+0000 7f2b9d255700 1 -- 172.21.15.116:0/2327690070 wait complete. 2024-03-04T16:08:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9d255700 1 Processor -- start 2024-03-04T16:08:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9d255700 1 -- start start 2024-03-04T16:08:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9d255700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b98100bc0 0x7f2b98196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9d255700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b98101dc0 0x7f2b981968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9d255700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b98196e40 con 0x7f2b98101dc0 2024-03-04T16:08:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9d255700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b98196f80 con 0x7f2b98100bc0 2024-03-04T16:08:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b98101dc0 0x7f2b981968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b98101dc0 0x7f2b981968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58332/0 (socket says 172.21.15.116:58332) 2024-03-04T16:08:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9659c700 1 -- 172.21.15.116:0/2197503256 learned_addr learned my addr 172.21.15.116:0/2197503256 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b96d9d700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b98100bc0 0x7f2b98196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9659c700 1 -- 172.21.15.116:0/2197503256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b98100bc0 msgr2=0x7f2b98196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9659c700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b98100bc0 0x7f2b98196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b9659c700 1 -- 172.21.15.116:0/2197503256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c0095f0 con 0x7f2b98101dc0 2024-03-04T16:08:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.314+0000 7f2b96d9d700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b98100bc0 0x7f2b98196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:08:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.315+0000 7f2b9659c700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b98101dc0 0x7f2b981968b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00bd70 tx=0x7f2b8c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.315+0000 7f2b7ffff700 1 -- 172.21.15.116:0/2197503256 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b8c01d070 con 0x7f2b98101dc0 2024-03-04T16:08:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.315+0000 7f2b9d255700 1 -- 172.21.15.116:0/2197503256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9819b9e0 con 0x7f2b98101dc0 2024-03-04T16:08:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.315+0000 7f2b7ffff700 1 -- 172.21.15.116:0/2197503256 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2b8c022470 con 0x7f2b98101dc0 2024-03-04T16:08:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.315+0000 7f2b7ffff700 1 -- 172.21.15.116:0/2197503256 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b8c00f670 con 0x7f2b98101dc0 2024-03-04T16:08:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.315+0000 7f2b9d255700 1 -- 172.21.15.116:0/2197503256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9819bea0 con 0x7f2b98101dc0 2024-03-04T16:08:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.316+0000 7f2b9d255700 1 -- 172.21.15.116:0/2197503256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b98109720 con 0x7f2b98101dc0 2024-03-04T16:08:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.316+0000 7f2b7ffff700 1 -- 172.21.15.116:0/2197503256 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f2b8c022a50 con 0x7f2b98101dc0 2024-03-04T16:08:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.317+0000 7f2b7ffff700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2b8006d8a0 0x7f2b8006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.317+0000 7f2b7ffff700 1 -- 172.21.15.116:0/2197503256 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2b8c08ebb0 con 0x7f2b98101dc0 2024-03-04T16:08:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.317+0000 7f2b96d9d700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2b8006d8a0 0x7f2b8006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.317+0000 7f2b96d9d700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2b8006d8a0 0x7f2b8006fd60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2b88005fd0 tx=0x7f2b88005e40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.321+0000 7f2b7ffff700 1 -- 172.21.15.116:0/2197503256 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b8c095080 con 0x7f2b98101dc0 2024-03-04T16:08:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.613+0000 7f2b9d255700 1 -- 172.21.15.116:0/2197503256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2b98109910 con 0x7f2b98101dc0 2024-03-04T16:08:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.614+0000 7f2b7ffff700 1 -- 172.21.15.116:0/2197503256 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f2b8c027070 con 0x7f2b98101dc0 2024-03-04T16:08:05.615 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:08:05.615 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:08:05.615 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:08:05.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:08:05.615 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:08:05.615 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:08:05.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 8, 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 5 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:08:05.616 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:08:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 -- 172.21.15.116:0/2197503256 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2b8006d8a0 msgr2=0x7f2b8006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2b8006d8a0 0x7f2b8006fd60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2b88005fd0 tx=0x7f2b88005e40 comp rx=0 tx=0).stop 2024-03-04T16:08:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 -- 172.21.15.116:0/2197503256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b98101dc0 msgr2=0x7f2b981968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b98101dc0 0x7f2b981968b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00bd70 tx=0x7f2b8c00be50 comp rx=0 tx=0).stop 2024-03-04T16:08:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 -- 172.21.15.116:0/2197503256 shutdown_connections 2024-03-04T16:08:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2b8006d8a0 0x7f2b8006fd60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b98100bc0 0x7f2b98196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 --2- 172.21.15.116:0/2197503256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b98101dc0 0x7f2b981968b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 -- 172.21.15.116:0/2197503256 >> 172.21.15.116:0/2197503256 conn(0x7f2b980fc140 msgr2=0x7f2b98104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 -- 172.21.15.116:0/2197503256 shutdown_connections 2024-03-04T16:08:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.616+0000 7f2b9d255700 1 -- 172.21.15.116:0/2197503256 wait complete. 2024-03-04T16:08:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.723+0000 7fc373a73700 1 -- 172.21.15.116:0/2901865351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36c100bc0 msgr2=0x7fc36c100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.723+0000 7fc373a73700 1 --2- 172.21.15.116:0/2901865351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36c100bc0 0x7fc36c100fe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc368009910 tx=0x7fc368009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.723+0000 7fc373a73700 1 -- 172.21.15.116:0/2901865351 shutdown_connections 2024-03-04T16:08:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.724+0000 7fc373a73700 1 --2- 172.21.15.116:0/2901865351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc36c101dc0 0x7fc36c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.724+0000 7fc373a73700 1 --2- 172.21.15.116:0/2901865351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36c100bc0 0x7fc36c100fe0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.724+0000 7fc373a73700 1 -- 172.21.15.116:0/2901865351 >> 172.21.15.116:0/2901865351 conn(0x7fc36c0fc140 msgr2=0x7fc36c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.724+0000 7fc373a73700 1 -- 172.21.15.116:0/2901865351 shutdown_connections 2024-03-04T16:08:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.724+0000 7fc373a73700 1 -- 172.21.15.116:0/2901865351 wait complete. 2024-03-04T16:08:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.724+0000 7fc373a73700 1 Processor -- start 2024-03-04T16:08:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc373a73700 1 -- start start 2024-03-04T16:08:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc373a73700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc36c100bc0 0x7fc36c196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc373a73700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36c101dc0 0x7fc36c1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc373a73700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c196fe0 con 0x7fc36c101dc0 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc373a73700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c197120 con 0x7fc36c100bc0 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc37100e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36c101dc0 0x7fc36c1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc37100e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36c101dc0 0x7fc36c1969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58354/0 (socket says 172.21.15.116:58354) 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc37100e700 1 -- 172.21.15.116:0/2142646541 learned_addr learned my addr 172.21.15.116:0/2142646541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc37180f700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc36c100bc0 0x7fc36c196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc37100e700 1 -- 172.21.15.116:0/2142646541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc36c100bc0 msgr2=0x7fc36c196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc37100e700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc36c100bc0 0x7fc36c196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc37100e700 1 -- 172.21.15.116:0/2142646541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3680095f0 con 0x7fc36c101dc0 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.725+0000 7fc37100e700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36c101dc0 0x7fc36c1969c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00b4b0 tx=0x7fc35c00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.726+0000 7fc362ffd700 1 -- 172.21.15.116:0/2142646541 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc35c011800 con 0x7fc36c101dc0 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.726+0000 7fc373a73700 1 -- 172.21.15.116:0/2142646541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c19bbd0 con 0x7fc36c101dc0 2024-03-04T16:08:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.726+0000 7fc362ffd700 1 -- 172.21.15.116:0/2142646541 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc35c011e10 con 0x7fc36c101dc0 2024-03-04T16:08:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.726+0000 7fc362ffd700 1 -- 172.21.15.116:0/2142646541 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc35c00f2e0 con 0x7fc36c101dc0 2024-03-04T16:08:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.726+0000 7fc373a73700 1 -- 172.21.15.116:0/2142646541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c19c1a0 con 0x7fc36c101dc0 2024-03-04T16:08:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.727+0000 7fc373a73700 1 -- 172.21.15.116:0/2142646541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c1097a0 con 0x7fc36c101dc0 2024-03-04T16:08:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.728+0000 7fc362ffd700 1 -- 172.21.15.116:0/2142646541 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fc35c010380 con 0x7fc36c101dc0 2024-03-04T16:08:05.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.728+0000 7fc362ffd700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc35806d8f0 0x7fc35806fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:05.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.729+0000 7fc362ffd700 1 -- 172.21.15.116:0/2142646541 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc35c08d150 con 0x7fc36c101dc0 2024-03-04T16:08:05.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.729+0000 7fc37180f700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc35806d8f0 0x7fc35806fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:05.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.729+0000 7fc37180f700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc35806d8f0 0x7fc35806fdb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc368005090 tx=0x7fc3680058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.733+0000 7fc362ffd700 1 -- 172.21.15.116:0/2142646541 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc35c05a3e0 con 0x7fc36c101dc0 2024-03-04T16:08:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.997+0000 7fc373a73700 1 -- 172.21.15.116:0/2142646541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc36c19c860 con 0x7fc36c101dc0 2024-03-04T16:08:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:05.998+0000 7fc362ffd700 1 -- 172.21.15.116:0/2142646541 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1852 (secure 0 0 0) 0x7fc35c059f70 con 0x7fc36c101dc0 2024-03-04T16:08:05.999 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:08:05.999 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:08:05.999 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:08:05.999 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:08:05.999 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:08:05.999 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:08:05.999 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:08:05.999 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:08:05.999 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:08:06.000 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:08:06.001 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:08:06.002 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:08:06.002 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:08:06.002 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:08:06.002 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:08:06.002 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:08:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 -- 172.21.15.116:0/2142646541 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc35806d8f0 msgr2=0x7fc35806fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc35806d8f0 0x7fc35806fdb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc368005090 tx=0x7fc3680058e0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 -- 172.21.15.116:0/2142646541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36c101dc0 msgr2=0x7fc36c1969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36c101dc0 0x7fc36c1969c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00b4b0 tx=0x7fc35c00b7c0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 -- 172.21.15.116:0/2142646541 shutdown_connections 2024-03-04T16:08:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc35806d8f0 0x7fc35806fdb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc36c100bc0 0x7fc36c196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 --2- 172.21.15.116:0/2142646541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36c101dc0 0x7fc36c1969c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 -- 172.21.15.116:0/2142646541 >> 172.21.15.116:0/2142646541 conn(0x7fc36c0fc140 msgr2=0x7fc36c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 -- 172.21.15.116:0/2142646541 shutdown_connections 2024-03-04T16:08:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.000+0000 7fc373a73700 1 -- 172.21.15.116:0/2142646541 wait complete. 2024-03-04T16:08:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:08:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.111+0000 7f155981f700 1 -- 172.21.15.116:0/392751086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1554100bb0 msgr2=0x7f1554100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.111+0000 7f155981f700 1 --2- 172.21.15.116:0/392751086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1554100bb0 0x7f1554100fd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1548009910 tx=0x7f1548009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.111+0000 7f155981f700 1 -- 172.21.15.116:0/392751086 shutdown_connections 2024-03-04T16:08:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.111+0000 7f155981f700 1 --2- 172.21.15.116:0/392751086 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1554101db0 0x7f1554102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.111+0000 7f155981f700 1 --2- 172.21.15.116:0/392751086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1554100bb0 0x7f1554100fd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.111+0000 7f155981f700 1 -- 172.21.15.116:0/392751086 >> 172.21.15.116:0/392751086 conn(0x7f15540fc110 msgr2=0x7f15540fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.111+0000 7f155981f700 1 -- 172.21.15.116:0/392751086 shutdown_connections 2024-03-04T16:08:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.111+0000 7f155981f700 1 -- 172.21.15.116:0/392751086 wait complete. 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.112+0000 7f155981f700 1 Processor -- start 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f155981f700 1 -- start start 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f155981f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1554100bb0 0x7f1554074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f155981f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1554101db0 0x7f1554073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f155981f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15540736c0 con 0x7f1554101db0 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f155981f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1554073800 con 0x7f1554100bb0 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f15527fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1554101db0 0x7f1554073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f15527fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1554101db0 0x7f1554073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58370/0 (socket says 172.21.15.116:58370) 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f15527fc700 1 -- 172.21.15.116:0/1172702294 learned_addr learned my addr 172.21.15.116:0/1172702294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f1552ffd700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1554100bb0 0x7f1554074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f15527fc700 1 -- 172.21.15.116:0/1172702294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1554100bb0 msgr2=0x7f1554074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f15527fc700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1554100bb0 0x7f1554074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f15527fc700 1 -- 172.21.15.116:0/1172702294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480095f0 con 0x7f1554101db0 2024-03-04T16:08:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f1552ffd700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1554100bb0 0x7f1554074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:08:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.113+0000 7f15527fc700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1554101db0 0x7f1554073180 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f154400e920 tx=0x7f154400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.114+0000 7f155881d700 1 -- 172.21.15.116:0/1172702294 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1544009790 con 0x7f1554101db0 2024-03-04T16:08:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.114+0000 7f155881d700 1 -- 172.21.15.116:0/1172702294 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f154400cba0 con 0x7f1554101db0 2024-03-04T16:08:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.114+0000 7f155881d700 1 -- 172.21.15.116:0/1172702294 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1544018630 con 0x7f1554101db0 2024-03-04T16:08:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.114+0000 7f155981f700 1 -- 172.21.15.116:0/1172702294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1554073ae0 con 0x7f1554101db0 2024-03-04T16:08:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.114+0000 7f155981f700 1 -- 172.21.15.116:0/1172702294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155419e3e0 con 0x7f1554101db0 2024-03-04T16:08:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.115+0000 7f155981f700 1 -- 172.21.15.116:0/1172702294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1554109710 con 0x7f1554101db0 2024-03-04T16:08:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.115+0000 7f155881d700 1 -- 172.21.15.116:0/1172702294 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f154400cd10 con 0x7f1554101db0 2024-03-04T16:08:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.116+0000 7f155881d700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f153c06d8a0 0x7f153c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.116+0000 7f155881d700 1 -- 172.21.15.116:0/1172702294 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1544014070 con 0x7f1554101db0 2024-03-04T16:08:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.116+0000 7f1552ffd700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f153c06d8a0 0x7f153c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.117+0000 7f1552ffd700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f153c06d8a0 0x7f153c06fd60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f15480059d0 tx=0x7f1548005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.120+0000 7f155881d700 1 -- 172.21.15.116:0/1172702294 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1544052d20 con 0x7f1554101db0 2024-03-04T16:08:06.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.382+0000 7f155981f700 1 -- 172.21.15.116:0/1172702294 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1554106700 con 0x7f153c06d8a0 2024-03-04T16:08:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.383+0000 7f155881d700 1 -- 172.21.15.116:0/1172702294 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1554106700 con 0x7f153c06d8a0 2024-03-04T16:08:06.384 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:08:06.384 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:08:06.384 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:08:06.384 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:08:06.384 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:08:06.384 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:08:06.384 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:08:06.385 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:08:06.385 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:08:06.385 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "7/23 daemons upgraded", 2024-03-04T16:08:06.385 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:08:06.385 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:08:06.385 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:08:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 -- 172.21.15.116:0/1172702294 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f153c06d8a0 msgr2=0x7f153c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f153c06d8a0 0x7f153c06fd60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f15480059d0 tx=0x7f1548005940 comp rx=0 tx=0).stop 2024-03-04T16:08:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 -- 172.21.15.116:0/1172702294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1554101db0 msgr2=0x7f1554073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1554101db0 0x7f1554073180 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f154400e920 tx=0x7f154400ece0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 -- 172.21.15.116:0/1172702294 shutdown_connections 2024-03-04T16:08:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f153c06d8a0 0x7f153c06fd60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1554100bb0 0x7f1554074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 --2- 172.21.15.116:0/1172702294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1554101db0 0x7f1554073180 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 -- 172.21.15.116:0/1172702294 >> 172.21.15.116:0/1172702294 conn(0x7f15540fc110 msgr2=0x7f1554104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 -- 172.21.15.116:0/1172702294 shutdown_connections 2024-03-04T16:08:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.386+0000 7f155981f700 1 -- 172.21.15.116:0/1172702294 wait complete. 2024-03-04T16:08:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.503+0000 7f70432cf700 1 -- 172.21.15.116:0/1869446143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f703c101640 msgr2=0x7f703c103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:06.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.503+0000 7f70432cf700 1 --2- 172.21.15.116:0/1869446143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f703c101640 0x7f703c103a30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f702c009910 tx=0x7f702c009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.503+0000 7f70432cf700 1 -- 172.21.15.116:0/1869446143 shutdown_connections 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.503+0000 7f70432cf700 1 --2- 172.21.15.116:0/1869446143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f703c103f70 0x7f703c106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.503+0000 7f70432cf700 1 --2- 172.21.15.116:0/1869446143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f703c101640 0x7f703c103a30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.503+0000 7f70432cf700 1 -- 172.21.15.116:0/1869446143 >> 172.21.15.116:0/1869446143 conn(0x7f703c0fb030 msgr2=0x7f703c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.503+0000 7f70432cf700 1 -- 172.21.15.116:0/1869446143 shutdown_connections 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.503+0000 7f70432cf700 1 -- 172.21.15.116:0/1869446143 wait complete. 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f70432cf700 1 Processor -- start 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f70432cf700 1 -- start start 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f70432cf700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f703c101640 0x7f703c196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f70432cf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f703c103f70 0x7f703c196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f70432cf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c196e60 con 0x7f703c103f70 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f70432cf700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c196fa0 con 0x7f703c101640 2024-03-04T16:08:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f704086a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f703c103f70 0x7f703c196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f704086a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f703c103f70 0x7f703c196840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58400/0 (socket says 172.21.15.116:58400) 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f704086a700 1 -- 172.21.15.116:0/1351417152 learned_addr learned my addr 172.21.15.116:0/1351417152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f704106b700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f703c101640 0x7f703c196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f704086a700 1 -- 172.21.15.116:0/1351417152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f703c101640 msgr2=0x7f703c196300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f704086a700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f703c101640 0x7f703c196300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f704086a700 1 -- 172.21.15.116:0/1351417152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c0095f0 con 0x7f703c103f70 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.504+0000 7f704086a700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f703c103f70 0x7f703c196840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f703800d8d0 tx=0x7f703800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.505+0000 7f70327fc700 1 -- 172.21.15.116:0/1351417152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7038009940 con 0x7f703c103f70 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.505+0000 7f70327fc700 1 -- 172.21.15.116:0/1351417152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7038010460 con 0x7f703c103f70 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.505+0000 7f70432cf700 1 -- 172.21.15.116:0/1351417152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c19ba50 con 0x7f703c103f70 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.505+0000 7f70327fc700 1 -- 172.21.15.116:0/1351417152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f703800f5f0 con 0x7f703c103f70 2024-03-04T16:08:06.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.505+0000 7f70432cf700 1 -- 172.21.15.116:0/1351417152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c19bfa0 con 0x7f703c103f70 2024-03-04T16:08:06.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:06 smithi152 ceph-mon[82845]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:06.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:06 smithi152 ceph-mon[82845]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:06.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:06 smithi152 ceph-mon[82845]: pgmap v76: 65 pgs: 12 active+undersized+degraded, 22 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:06.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:06 smithi152 ceph-mon[82845]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:06.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:06 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2197503256' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:06.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:06 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2142646541' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:08:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.506+0000 7f70432cf700 1 -- 172.21.15.116:0/1351417152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703c190520 con 0x7f703c103f70 2024-03-04T16:08:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.506+0000 7f70327fc700 1 -- 172.21.15.116:0/1351417152 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f70380105d0 con 0x7f703c103f70 2024-03-04T16:08:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.507+0000 7f70327fc700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f702806d8a0 0x7f702806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.507+0000 7f70327fc700 1 -- 172.21.15.116:0/1351417152 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f703808d4c0 con 0x7f703c103f70 2024-03-04T16:08:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.507+0000 7f704106b700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f702806d8a0 0x7f702806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.507+0000 7f704106b700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f702806d8a0 0x7f702806fd60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f702c005b40 tx=0x7f702c005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.511+0000 7f70327fc700 1 -- 172.21.15.116:0/1351417152 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f703805a7f0 con 0x7f703c103f70 2024-03-04T16:08:06.631 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:06 smithi116 ceph-mon[94164]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:06.631 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:06 smithi116 ceph-mon[94164]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:06.631 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:06 smithi116 ceph-mon[94164]: pgmap v76: 65 pgs: 12 active+undersized+degraded, 22 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:06.631 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:06 smithi116 ceph-mon[94164]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:06.631 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:06 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2197503256' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:06.631 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:06 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2142646541' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:08:06.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.827+0000 7f70432cf700 1 -- 172.21.15.116:0/1351417152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f703c02cc70 con 0x7f703c103f70 2024-03-04T16:08:06.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.827+0000 7f70327fc700 1 -- 172.21.15.116:0/1351417152 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+936 (secure 0 0 0) 0x7f703805a380 con 0x7f703c103f70 2024-03-04T16:08:06.828 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 29/231 objects degraded (12.554%), 12 pgs degraded 2024-03-04T16:08:06.828 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: osd.1 (root=default,host=smithi116) is down 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 29/231 objects degraded (12.554%), 12 pgs degraded 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 1.0 is active+undersized+degraded, acting [3,0] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.0 is active+undersized+degraded, acting [3,0] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.2 is active+undersized+degraded, acting [5,0] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.4 is active+undersized+degraded, acting [0,4] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.a is active+undersized+degraded, acting [4,3] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.d is active+undersized+degraded, acting [3,2] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-03-04T16:08:06.829 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-03-04T16:08:06.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.830+0000 7f70432cf700 1 -- 172.21.15.116:0/1351417152 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f702806d8a0 msgr2=0x7f702806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:06.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.830+0000 7f70432cf700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f702806d8a0 0x7f702806fd60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f702c005b40 tx=0x7f702c005a90 comp rx=0 tx=0).stop 2024-03-04T16:08:06.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.830+0000 7f70432cf700 1 -- 172.21.15.116:0/1351417152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f703c103f70 msgr2=0x7f703c196840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:06.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.830+0000 7f70432cf700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f703c103f70 0x7f703c196840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f703800d8d0 tx=0x7f703800dc90 comp rx=0 tx=0).stop 2024-03-04T16:08:06.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.830+0000 7f70432cf700 1 -- 172.21.15.116:0/1351417152 shutdown_connections 2024-03-04T16:08:06.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.830+0000 7f70432cf700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f702806d8a0 0x7f702806fd60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.830+0000 7f70432cf700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f703c101640 0x7f703c196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.830+0000 7f70432cf700 1 --2- 172.21.15.116:0/1351417152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f703c103f70 0x7f703c196840 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:06.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.830+0000 7f70432cf700 1 -- 172.21.15.116:0/1351417152 >> 172.21.15.116:0/1351417152 conn(0x7f703c0fb030 msgr2=0x7f703c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:06.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.831+0000 7f70432cf700 1 -- 172.21.15.116:0/1351417152 shutdown_connections 2024-03-04T16:08:06.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:06.831+0000 7f70432cf700 1 -- 172.21.15.116:0/1351417152 wait complete. 2024-03-04T16:08:07.272 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:07 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[108248]: 2024-03-04T16:08:07.124+0000 7f8f995be7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-04T16:08:07.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:07 smithi152 ceph-mon[82845]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:07.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:07 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1351417152' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:08:07.627 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:07 smithi116 ceph-mon[94164]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:07.627 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:07 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1351417152' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:08:07.975 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:07 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[108248]: 2024-03-04T16:08:07.625+0000 7f8f995be7c0 -1 osd.1 55 log_to_monitors true 2024-03-04T16:08:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:08 smithi152 ceph-mon[82845]: pgmap v77: 65 pgs: 12 active+undersized+degraded, 22 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:08 smithi152 ceph-mon[82845]: from='osd.1 [v2:172.21.15.116:6810/22677649,v1:172.21.15.116:6811/22677649]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:08:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:08.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:08 smithi116 ceph-mon[94164]: pgmap v77: 65 pgs: 12 active+undersized+degraded, 22 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:08 smithi116 ceph-mon[94164]: from='osd.1 [v2:172.21.15.116:6810/22677649,v1:172.21.15.116:6811/22677649]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:08:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:08.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:09.435 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:08:09 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[108248]: 2024-03-04T16:08:09.139+0000 7f8f86a0e700 -1 osd.1 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:08:09.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:09 smithi116 ceph-mon[94164]: from='osd.1 [v2:172.21.15.116:6810/22677649,v1:172.21.15.116:6811/22677649]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T16:08:09.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:09 smithi116 ceph-mon[94164]: osdmap e58: 6 total, 5 up, 6 in 2024-03-04T16:08:09.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:09 smithi116 ceph-mon[94164]: from='osd.1 [v2:172.21.15.116:6810/22677649,v1:172.21.15.116:6811/22677649]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T16:08:09.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:09.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:09.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:09 smithi116 ceph-mon[94164]: Health check update: Degraded data redundancy: 29/231 objects degraded (12.554%), 12 pgs degraded (PG_DEGRADED) 2024-03-04T16:08:09.435 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:09 smithi116 ceph-mon[94164]: from='osd.1 [v2:172.21.15.116:6810/22677649,v1:172.21.15.116:6811/22677649]' entity='osd.1' 2024-03-04T16:08:09.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:09 smithi152 ceph-mon[82845]: from='osd.1 [v2:172.21.15.116:6810/22677649,v1:172.21.15.116:6811/22677649]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T16:08:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:09 smithi152 ceph-mon[82845]: osdmap e58: 6 total, 5 up, 6 in 2024-03-04T16:08:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:09 smithi152 ceph-mon[82845]: from='osd.1 [v2:172.21.15.116:6810/22677649,v1:172.21.15.116:6811/22677649]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T16:08:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:09 smithi152 ceph-mon[82845]: Health check update: Degraded data redundancy: 29/231 objects degraded (12.554%), 12 pgs degraded (PG_DEGRADED) 2024-03-04T16:08:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:09 smithi152 ceph-mon[82845]: from='osd.1 [v2:172.21.15.116:6810/22677649,v1:172.21.15.116:6811/22677649]' entity='osd.1' 2024-03-04T16:08:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:10 smithi152 ceph-mon[82845]: pgmap v79: 65 pgs: 12 active+undersized+degraded, 22 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:10 smithi152 ceph-mon[82845]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:08:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:10 smithi152 ceph-mon[82845]: osd.1 [v2:172.21.15.116:6810/22677649,v1:172.21.15.116:6811/22677649] boot 2024-03-04T16:08:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:10 smithi152 ceph-mon[82845]: osdmap e59: 6 total, 6 up, 6 in 2024-03-04T16:08:10.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:10 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:08:10.621 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:10 smithi116 ceph-mon[94164]: pgmap v79: 65 pgs: 12 active+undersized+degraded, 22 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:10.622 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:10 smithi116 ceph-mon[94164]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:08:10.622 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:10 smithi116 ceph-mon[94164]: osd.1 [v2:172.21.15.116:6810/22677649,v1:172.21.15.116:6811/22677649] boot 2024-03-04T16:08:10.622 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:10 smithi116 ceph-mon[94164]: osdmap e59: 6 total, 6 up, 6 in 2024-03-04T16:08:10.622 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:08:12.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:12 smithi116 ceph-mon[94164]: pgmap v81: 65 pgs: 12 active+undersized+degraded, 22 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:12.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:12 smithi116 ceph-mon[94164]: osdmap e60: 6 total, 6 up, 6 in 2024-03-04T16:08:12.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:12 smithi152 ceph-mon[82845]: pgmap v81: 65 pgs: 12 active+undersized+degraded, 22 active+undersized, 31 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:12.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:12 smithi152 ceph-mon[82845]: osdmap e60: 6 total, 6 up, 6 in 2024-03-04T16:08:13.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:13.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:13.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:13.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:13.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:13.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:13.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:13.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:13.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:13.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:08:13.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:08:13.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:13 smithi116 ceph-mon[94164]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-04T16:08:13.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:13.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:13 smithi152 ceph-mon[82845]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-04T16:08:14.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:14 smithi116 ceph-mon[94164]: pgmap v83: 65 pgs: 12 active+undersized+degraded, 20 active+undersized, 33 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:14.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:14 smithi152 ceph-mon[82845]: pgmap v83: 65 pgs: 12 active+undersized+degraded, 20 active+undersized, 33 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/231 objects degraded (12.554%) 2024-03-04T16:08:15.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:15 smithi152 ceph-mon[82845]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/231 objects degraded (12.554%), 12 pgs degraded) 2024-03-04T16:08:15.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:15 smithi152 ceph-mon[82845]: Cluster is now healthy 2024-03-04T16:08:15.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:15 smithi116 ceph-mon[94164]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/231 objects degraded (12.554%), 12 pgs degraded) 2024-03-04T16:08:15.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:15 smithi116 ceph-mon[94164]: Cluster is now healthy 2024-03-04T16:08:16.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:16 smithi152 ceph-mon[82845]: pgmap v84: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:16.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:16 smithi116 ceph-mon[94164]: pgmap v84: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:18.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:18 smithi152 ceph-mon[82845]: pgmap v85: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:18.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:18 smithi116 ceph-mon[94164]: pgmap v85: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:20.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:20 smithi152 ceph-mon[82845]: pgmap v86: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:20.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:20 smithi116 ceph-mon[94164]: pgmap v86: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:22.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:22 smithi152 ceph-mon[82845]: pgmap v87: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:22 smithi116 ceph-mon[94164]: pgmap v87: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:24 smithi152 ceph-mon[82845]: pgmap v88: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:24 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:24 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:24.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:24 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:24 smithi116 ceph-mon[94164]: pgmap v88: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:24 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:24 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:24 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:26.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:26 smithi152 ceph-mon[82845]: pgmap v89: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:26.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:26 smithi116 ceph-mon[94164]: pgmap v89: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:28.214 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:28 smithi116 ceph-mon[94164]: pgmap v90: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:28.214 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:08:28.214 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:28 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:08:28.214 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:28 smithi116 ceph-mon[94164]: Upgrade: osd.2 is safe to restart 2024-03-04T16:08:28.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:28 smithi152 ceph-mon[82845]: pgmap v90: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:28.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:08:28.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:28 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:08:28.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:28 smithi152 ceph-mon[82845]: Upgrade: osd.2 is safe to restart 2024-03-04T16:08:29.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:29 smithi116 ceph-mon[94164]: Upgrade: Updating osd.2 2024-03-04T16:08:29.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:29 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:29.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:29 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:08:29.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:29 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:29.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:29 smithi116 ceph-mon[94164]: Deploying daemon osd.2 on smithi116 2024-03-04T16:08:29.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:29 smithi116 ceph-mon[94164]: pgmap v91: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:30.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:29 smithi152 ceph-mon[82845]: Upgrade: Updating osd.2 2024-03-04T16:08:30.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:29 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:30.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:29 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:08:30.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:29 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:30.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:29 smithi152 ceph-mon[82845]: Deploying daemon osd.2 on smithi116 2024-03-04T16:08:30.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:29 smithi152 ceph-mon[82845]: pgmap v91: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:32.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:32 smithi116 ceph-mon[94164]: pgmap v92: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:32.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:31 smithi116 systemd[1]: Stopping Ceph osd.2 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:08:32.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:32 smithi152 ceph-mon[82845]: pgmap v92: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:32.725 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:32 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[63952]: 2024-03-04T16:08:32.257+0000 7ff6898fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:08:32.725 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:32 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[63952]: 2024-03-04T16:08:32.257+0000 7ff6898fa700 -1 osd.2 60 *** Got signal Terminated *** 2024-03-04T16:08:32.725 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:32 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[63952]: 2024-03-04T16:08:32.257+0000 7ff6898fa700 -1 osd.2 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:08:33.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:33 smithi152 ceph-mon[82845]: osd.2 marked itself down and dead 2024-03-04T16:08:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:33 smithi116 ceph-mon[94164]: osd.2 marked itself down and dead 2024-03-04T16:08:33.946 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:33 smithi116 podman[113644]: 2024-03-04 16:08:33.677275833 +0000 UTC m=+1.552485800 container died 9a07129d284ca1c61f61a7e9cf970b213095d95d33be8bb7fa309bef99a5c13f (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:08:34.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:34 smithi116 ceph-mon[94164]: pgmap v93: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:34.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:34 smithi116 ceph-mon[94164]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:08:34.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:34 smithi116 ceph-mon[94164]: osdmap e61: 6 total, 5 up, 6 in 2024-03-04T16:08:34.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:34 smithi116 podman[113644]: 2024-03-04 16:08:34.035560947 +0000 UTC m=+1.910770913 container cleanup 9a07129d284ca1c61f61a7e9cf970b213095d95d33be8bb7fa309bef99a5c13f (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:08:34.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:34 smithi116 bash[113644]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2 2024-03-04T16:08:34.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:34 smithi152 ceph-mon[82845]: pgmap v93: 65 pgs: 65 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:34.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:34 smithi152 ceph-mon[82845]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:08:34.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:34 smithi152 ceph-mon[82845]: osdmap e61: 6 total, 5 up, 6 in 2024-03-04T16:08:34.663 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:34 smithi116 podman[113662]: 2024-03-04 16:08:34.345629706 +0000 UTC m=+0.670661034 container remove 9a07129d284ca1c61f61a7e9cf970b213095d95d33be8bb7fa309bef99a5c13f (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-03-04T16:08:34.962 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:34 smithi116 podman[113754]: 2024-03-04 16:08:34.663296766 +0000 UTC m=+0.105356506 container create 25c363d102914022fe3b9e80b745613d6572642eaa9165a88eb7b0e88b6d3424 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:08:34.962 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:34 smithi116 podman[113754]: 2024-03-04 16:08:34.576143425 +0000 UTC m=+0.018203129 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:08:35.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:35 smithi116 ceph-mon[94164]: osdmap e62: 6 total, 5 up, 6 in 2024-03-04T16:08:35.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:35 smithi116 podman[113754]: 2024-03-04 16:08:35.057449699 +0000 UTC m=+0.499509412 container init 25c363d102914022fe3b9e80b745613d6572642eaa9165a88eb7b0e88b6d3424 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:08:35.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:35 smithi116 podman[113754]: 2024-03-04 16:08:35.060675403 +0000 UTC m=+0.502735105 container start 25c363d102914022fe3b9e80b745613d6572642eaa9165a88eb7b0e88b6d3424 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:08:35.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:35 smithi116 podman[113754]: 2024-03-04 16:08:35.132467939 +0000 UTC m=+0.574527648 container attach 25c363d102914022fe3b9e80b745613d6572642eaa9165a88eb7b0e88b6d3424 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:08:35.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:35 smithi152 ceph-mon[82845]: osdmap e62: 6 total, 5 up, 6 in 2024-03-04T16:08:35.669 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:35 smithi116 podman[113754]: 2024-03-04 16:08:35.356462522 +0000 UTC m=+0.798522221 container died 25c363d102914022fe3b9e80b745613d6572642eaa9165a88eb7b0e88b6d3424 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:08:36.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:36 smithi116 ceph-mon[94164]: pgmap v96: 65 pgs: 19 peering, 3 stale+active+clean, 43 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:36.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:36 smithi116 ceph-mon[94164]: Health check failed: Reduced data availability: 6 pgs peering (PG_AVAILABILITY) 2024-03-04T16:08:36.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:35 smithi116 podman[113780]: 2024-03-04 16:08:35.944655928 +0000 UTC m=+0.583109358 container remove 25c363d102914022fe3b9e80b745613d6572642eaa9165a88eb7b0e88b6d3424 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-04T16:08:36.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:35 smithi116 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-04T16:08:36.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:35 smithi116 systemd[1]: Stopped Ceph osd.2 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:08:36.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:35 smithi116 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.2.service: Consumed 55.275s CPU time. 2024-03-04T16:08:36.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:36 smithi116 systemd[1]: Starting Ceph osd.2 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:08:36.508 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:36 smithi152 ceph-mon[82845]: pgmap v96: 65 pgs: 19 peering, 3 stale+active+clean, 43 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:36.509 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:36 smithi152 ceph-mon[82845]: Health check failed: Reduced data availability: 6 pgs peering (PG_AVAILABILITY) 2024-03-04T16:08:36.707 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:36 smithi116 podman[113884]: 2024-03-04 16:08:36.29876422 +0000 UTC m=+0.017890398 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:08:36.707 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:36 smithi116 podman[113884]: 2024-03-04 16:08:36.421253273 +0000 UTC m=+0.140379492 container create 0f648fc13393ea5b7bb8e1800c4c32b42345d1f44f62d57a5050f5c6dc6b9351 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:08:36.707 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:36 smithi116 podman[113884]: 2024-03-04 16:08:36.623759864 +0000 UTC m=+0.342886052 container init 0f648fc13393ea5b7bb8e1800c4c32b42345d1f44f62d57a5050f5c6dc6b9351 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:08:36.707 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:36 smithi116 podman[113884]: 2024-03-04 16:08:36.626787389 +0000 UTC m=+0.345913570 container start 0f648fc13393ea5b7bb8e1800c4c32b42345d1f44f62d57a5050f5c6dc6b9351 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-04T16:08:36.707 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:36 smithi116 podman[113884]: 2024-03-04 16:08:36.706870959 +0000 UTC m=+0.425997135 container attach 0f648fc13393ea5b7bb8e1800c4c32b42345d1f44f62d57a5050f5c6dc6b9351 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=) 2024-03-04T16:08:36.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.946+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3299987161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0cc100c00 msgr2=0x7fe0cc101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.946+0000 7fe0d40ae700 1 --2- 172.21.15.116:0/3299987161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0cc100c00 0x7fe0cc101020 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009910 tx=0x7fe0c8009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.946+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3299987161 shutdown_connections 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.946+0000 7fe0d40ae700 1 --2- 172.21.15.116:0/3299987161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0cc101e00 0x7fe0cc102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.946+0000 7fe0d40ae700 1 --2- 172.21.15.116:0/3299987161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0cc100c00 0x7fe0cc101020 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.946+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3299987161 >> 172.21.15.116:0/3299987161 conn(0x7fe0cc0fc180 msgr2=0x7fe0cc0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.946+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3299987161 shutdown_connections 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.946+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3299987161 wait complete. 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.947+0000 7fe0d40ae700 1 Processor -- start 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.947+0000 7fe0d40ae700 1 -- start start 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.947+0000 7fe0d40ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0cc100c00 0x7fe0cc196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.947+0000 7fe0d40ae700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0cc101e00 0x7fe0cc1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:36.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.947+0000 7fe0d40ae700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc196ef0 con 0x7fe0cc100c00 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.947+0000 7fe0d40ae700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc197030 con 0x7fe0cc101e00 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.947+0000 7fe0d1e4a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0cc100c00 0x7fe0cc196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.947+0000 7fe0d1e4a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0cc100c00 0x7fe0cc196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50182/0 (socket says 172.21.15.116:50182) 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.947+0000 7fe0d1e4a700 1 -- 172.21.15.116:0/3683405720 learned_addr learned my addr 172.21.15.116:0/3683405720 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.948+0000 7fe0d1649700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0cc101e00 0x7fe0cc1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.948+0000 7fe0d1e4a700 1 -- 172.21.15.116:0/3683405720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0cc101e00 msgr2=0x7fe0cc1968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.948+0000 7fe0d1e4a700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0cc101e00 0x7fe0cc1968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.948+0000 7fe0d1e4a700 1 -- 172.21.15.116:0/3683405720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80095f0 con 0x7fe0cc100c00 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.948+0000 7fe0d1e4a700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0cc100c00 0x7fe0cc196390 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800bda0 tx=0x7fe0c800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.948+0000 7fe0c2ffd700 1 -- 172.21.15.116:0/3683405720 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0c801d070 con 0x7fe0cc100c00 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.948+0000 7fe0c2ffd700 1 -- 172.21.15.116:0/3683405720 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe0c800f460 con 0x7fe0cc100c00 2024-03-04T16:08:36.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.948+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3683405720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc19ba80 con 0x7fe0cc100c00 2024-03-04T16:08:36.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.948+0000 7fe0c2ffd700 1 -- 172.21.15.116:0/3683405720 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0c8021640 con 0x7fe0cc100c00 2024-03-04T16:08:36.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.948+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3683405720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0cc19bf70 con 0x7fe0cc100c00 2024-03-04T16:08:36.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.949+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3683405720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc109760 con 0x7fe0cc100c00 2024-03-04T16:08:36.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.950+0000 7fe0c2ffd700 1 -- 172.21.15.116:0/3683405720 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fe0c800f5d0 con 0x7fe0cc100c00 2024-03-04T16:08:36.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.950+0000 7fe0c2ffd700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe0b806d8a0 0x7fe0b806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:36.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.951+0000 7fe0c2ffd700 1 -- 172.21.15.116:0/3683405720 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe0c808e850 con 0x7fe0cc100c00 2024-03-04T16:08:36.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.951+0000 7fe0d1649700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe0b806d8a0 0x7fe0b806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:36.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.951+0000 7fe0d1649700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe0b806d8a0 0x7fe0b806fd60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00bbd0 tx=0x7fe0bc00b5b0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:36.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:36.957+0000 7fe0c2ffd700 1 -- 172.21.15.116:0/3683405720 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe0c805bb80 con 0x7fe0cc100c00 2024-03-04T16:08:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.197+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3683405720 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0cc106750 con 0x7fe0b806d8a0 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.198+0000 7fe0c2ffd700 1 -- 172.21.15.116:0/3683405720 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe0cc106750 con 0x7fe0b806d8a0 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.200+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3683405720 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe0b806d8a0 msgr2=0x7fe0b806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.200+0000 7fe0d40ae700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe0b806d8a0 0x7fe0b806fd60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00bbd0 tx=0x7fe0bc00b5b0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.200+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3683405720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0cc100c00 msgr2=0x7fe0cc196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.200+0000 7fe0d40ae700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0cc100c00 0x7fe0cc196390 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800bda0 tx=0x7fe0c800be80 comp rx=0 tx=0).stop 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.200+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3683405720 shutdown_connections 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.200+0000 7fe0d40ae700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe0b806d8a0 0x7fe0b806fd60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.200+0000 7fe0d40ae700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0cc100c00 0x7fe0cc196390 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.200+0000 7fe0d40ae700 1 --2- 172.21.15.116:0/3683405720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0cc101e00 0x7fe0cc1968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.200+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3683405720 >> 172.21.15.116:0/3683405720 conn(0x7fe0cc0fc180 msgr2=0x7fe0cc105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.200+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3683405720 shutdown_connections 2024-03-04T16:08:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.201+0000 7fe0d40ae700 1 -- 172.21.15.116:0/3683405720 wait complete. 2024-03-04T16:08:37.225 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:08:37.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate[113898]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T16:08:37.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate[113898]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-04T16:08:37.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 bash[113884]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T16:08:37.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 bash[113884]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-04T16:08:37.225 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 bash[113884]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-04T16:08:37.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 bash[113884]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-04T16:08:37.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 bash[113884]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-04T16:08:37.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 bash[113884]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T16:08:37.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 bash[113884]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-04T16:08:37.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate[113898]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-04T16:08:37.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate[113898]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-04T16:08:37.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate[113898]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-04T16:08:37.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate[113898]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T16:08:37.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate[113898]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-04T16:08:37.226 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 podman[113884]: 2024-03-04 16:08:37.098997797 +0000 UTC m=+0.818123964 container died 0f648fc13393ea5b7bb8e1800c4c32b42345d1f44f62d57a5050f5c6dc6b9351 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-04T16:08:37.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.304+0000 7efeeafb3700 1 -- 172.21.15.116:0/1428853670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4100be0 msgr2=0x7efee4101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.304+0000 7efeeafb3700 1 --2- 172.21.15.116:0/1428853670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4100be0 0x7efee4101000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efed8009910 tx=0x7efed8009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:37.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.304+0000 7efeeafb3700 1 -- 172.21.15.116:0/1428853670 shutdown_connections 2024-03-04T16:08:37.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.304+0000 7efeeafb3700 1 --2- 172.21.15.116:0/1428853670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee4101de0 0x7efee4102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.304+0000 7efeeafb3700 1 --2- 172.21.15.116:0/1428853670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4100be0 0x7efee4101000 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.304+0000 7efeeafb3700 1 -- 172.21.15.116:0/1428853670 >> 172.21.15.116:0/1428853670 conn(0x7efee40fc140 msgr2=0x7efee40fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:37.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.304+0000 7efeeafb3700 1 -- 172.21.15.116:0/1428853670 shutdown_connections 2024-03-04T16:08:37.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.304+0000 7efeeafb3700 1 -- 172.21.15.116:0/1428853670 wait complete. 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.305+0000 7efeeafb3700 1 Processor -- start 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.305+0000 7efeeafb3700 1 -- start start 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.305+0000 7efeeafb3700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee4100be0 0x7efee4196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.305+0000 7efeeafb3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101de0 0x7efee41968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.305+0000 7efeeafb3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee4196ef0 con 0x7efee4101de0 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.305+0000 7efeeafb3700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee4197030 con 0x7efee4100be0 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.305+0000 7efee3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101de0 0x7efee41968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.305+0000 7efee3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101de0 0x7efee41968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50202/0 (socket says 172.21.15.116:50202) 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.305+0000 7efee3fff700 1 -- 172.21.15.116:0/1861946707 learned_addr learned my addr 172.21.15.116:0/1861946707 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.305+0000 7efee8d4f700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee4100be0 0x7efee4196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.306+0000 7efee3fff700 1 -- 172.21.15.116:0/1861946707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee4100be0 msgr2=0x7efee4196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.306+0000 7efee3fff700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee4100be0 0x7efee4196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.306+0000 7efee3fff700 1 -- 172.21.15.116:0/1861946707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed80095f0 con 0x7efee4101de0 2024-03-04T16:08:37.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.306+0000 7efee3fff700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101de0 0x7efee41968d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efed000d8d0 tx=0x7efed000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:37.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.306+0000 7efee1ffb700 1 -- 172.21.15.116:0/1861946707 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efed00098a0 con 0x7efee4101de0 2024-03-04T16:08:37.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.306+0000 7efeeafb3700 1 -- 172.21.15.116:0/1861946707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee419bae0 con 0x7efee4101de0 2024-03-04T16:08:37.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.306+0000 7efee1ffb700 1 -- 172.21.15.116:0/1861946707 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efed0010460 con 0x7efee4101de0 2024-03-04T16:08:37.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.306+0000 7efee1ffb700 1 -- 172.21.15.116:0/1861946707 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efed000f5f0 con 0x7efee4101de0 2024-03-04T16:08:37.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.306+0000 7efeeafb3700 1 -- 172.21.15.116:0/1861946707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee419c030 con 0x7efee4101de0 2024-03-04T16:08:37.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.307+0000 7efeeafb3700 1 -- 172.21.15.116:0/1861946707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee4109740 con 0x7efee4101de0 2024-03-04T16:08:37.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.307+0000 7efee1ffb700 1 -- 172.21.15.116:0/1861946707 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7efed0009a00 con 0x7efee4101de0 2024-03-04T16:08:37.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.308+0000 7efee1ffb700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efed406d8a0 0x7efed406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:37.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.308+0000 7efee1ffb700 1 -- 172.21.15.116:0/1861946707 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efed008d400 con 0x7efee4101de0 2024-03-04T16:08:37.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.308+0000 7efee8d4f700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efed406d8a0 0x7efed406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:37.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.308+0000 7efee8d4f700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efed406d8a0 0x7efed406fd60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efed8006010 tx=0x7efed8005ec0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:37.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.312+0000 7efee1ffb700 1 -- 172.21.15.116:0/1861946707 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efed005a690 con 0x7efee4101de0 2024-03-04T16:08:37.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.529+0000 7efeeafb3700 1 -- 172.21.15.116:0/1861946707 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efee4106730 con 0x7efed406d8a0 2024-03-04T16:08:37.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.530+0000 7efee1ffb700 1 -- 172.21.15.116:0/1861946707 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7efee4106730 con 0x7efed406d8a0 2024-03-04T16:08:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 -- 172.21.15.116:0/1861946707 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efed406d8a0 msgr2=0x7efed406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efed406d8a0 0x7efed406fd60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efed8006010 tx=0x7efed8005ec0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 -- 172.21.15.116:0/1861946707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101de0 msgr2=0x7efee41968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101de0 0x7efee41968d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efed000d8d0 tx=0x7efed000dbe0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 -- 172.21.15.116:0/1861946707 shutdown_connections 2024-03-04T16:08:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efed406d8a0 0x7efed406fd60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee4100be0 0x7efee4196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 --2- 172.21.15.116:0/1861946707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101de0 0x7efee41968d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 -- 172.21.15.116:0/1861946707 >> 172.21.15.116:0/1861946707 conn(0x7efee40fc140 msgr2=0x7efee4105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 -- 172.21.15.116:0/1861946707 shutdown_connections 2024-03-04T16:08:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.532+0000 7efeeafb3700 1 -- 172.21.15.116:0/1861946707 wait complete. 2024-03-04T16:08:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.636+0000 7f232d05f700 1 -- 172.21.15.116:0/2195144277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23280ffab0 msgr2=0x7f23280ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.636+0000 7f232d05f700 1 --2- 172.21.15.116:0/2195144277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23280ffab0 0x7f23280ffed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2318009910 tx=0x7f2318009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.637+0000 7f232d05f700 1 -- 172.21.15.116:0/2195144277 shutdown_connections 2024-03-04T16:08:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.637+0000 7f232d05f700 1 --2- 172.21.15.116:0/2195144277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2328100cb0 0x7f2328101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.637+0000 7f232d05f700 1 --2- 172.21.15.116:0/2195144277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23280ffab0 0x7f23280ffed0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.637+0000 7f232d05f700 1 -- 172.21.15.116:0/2195144277 >> 172.21.15.116:0/2195144277 conn(0x7f23280fb030 msgr2=0x7f23280fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.637+0000 7f232d05f700 1 -- 172.21.15.116:0/2195144277 shutdown_connections 2024-03-04T16:08:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.637+0000 7f232d05f700 1 -- 172.21.15.116:0/2195144277 wait complete. 2024-03-04T16:08:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.637+0000 7f232d05f700 1 Processor -- start 2024-03-04T16:08:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232d05f700 1 -- start start 2024-03-04T16:08:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232d05f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f23280ffab0 0x7f2328196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232d05f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2328100cb0 0x7f2328196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232d05f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2328196e80 con 0x7f2328100cb0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232d05f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2328196fc0 con 0x7f23280ffab0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2328100cb0 0x7f2328196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2328100cb0 0x7f2328196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50222/0 (socket says 172.21.15.116:50222) 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232659c700 1 -- 172.21.15.116:0/2389911103 learned_addr learned my addr 172.21.15.116:0/2389911103 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f2326d9d700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f23280ffab0 0x7f2328196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232659c700 1 -- 172.21.15.116:0/2389911103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f23280ffab0 msgr2=0x7f2328196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232659c700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f23280ffab0 0x7f2328196320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232659c700 1 -- 172.21.15.116:0/2389911103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23180095f0 con 0x7f2328100cb0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232659c700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2328100cb0 0x7f2328196860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f231c00d900 tx=0x7f231c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f230ffff700 1 -- 172.21.15.116:0/2389911103 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f231c0041f0 con 0x7f2328100cb0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f232d05f700 1 -- 172.21.15.116:0/2389911103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232819ba70 con 0x7f2328100cb0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f230ffff700 1 -- 172.21.15.116:0/2389911103 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f231c004ce0 con 0x7f2328100cb0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.638+0000 7f230ffff700 1 -- 172.21.15.116:0/2389911103 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f231c00b750 con 0x7f2328100cb0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.639+0000 7f232d05f700 1 -- 172.21.15.116:0/2389911103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232819bfc0 con 0x7f2328100cb0 2024-03-04T16:08:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.639+0000 7f232d05f700 1 -- 172.21.15.116:0/2389911103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2328108610 con 0x7f2328100cb0 2024-03-04T16:08:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.640+0000 7f230ffff700 1 -- 172.21.15.116:0/2389911103 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f231c004e50 con 0x7f2328100cb0 2024-03-04T16:08:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.640+0000 7f230ffff700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f231006d8a0 0x7f231006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.641+0000 7f2326d9d700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f231006d8a0 0x7f231006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.641+0000 7f230ffff700 1 -- 172.21.15.116:0/2389911103 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f231c08d3f0 con 0x7f2328100cb0 2024-03-04T16:08:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.641+0000 7f2326d9d700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f231006d8a0 0x7f231006fd60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2318006010 tx=0x7f2318005ec0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.645+0000 7f230ffff700 1 -- 172.21.15.116:0/2389911103 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f231c05a720 con 0x7f2328100cb0 2024-03-04T16:08:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.860+0000 7f232d05f700 1 -- 172.21.15.116:0/2389911103 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2328068ae0 con 0x7f231006d8a0 2024-03-04T16:08:37.862 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 podman[114051]: 2024-03-04 16:08:37.694359772 +0000 UTC m=+0.590420121 container remove 0f648fc13393ea5b7bb8e1800c4c32b42345d1f44f62d57a5050f5c6dc6b9351 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:08:37.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.871+0000 7f230ffff700 1 -- 172.21.15.116:0/2389911103 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2328068ae0 con 0x7f231006d8a0 2024-03-04T16:08:37.872 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (11m) 30s ago 13m 25.4M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (13m) 30s ago 13m 7994k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (12m) 89s ago 12m 7838k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (98s) 30s ago 13m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (92s) 89s ago 12m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (11m) 30s ago 12m 87.2M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (7m) 30s ago 7m 19.7M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (7m) 30s ago 7m 15.9M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (7m) 89s ago 7m 16.7M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (7m) 89s ago 7m 143M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (4m) 30s ago 14m 499M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (3m) 89s ago 12m 449M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (2m) 30s ago 14m 45.4M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:08:37.873 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (2m) 89s ago 12m 33.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:08:37.874 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (13m) 30s ago 13m 15.5M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:08:37.874 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (12m) 89s ago 12m 16.7M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:08:37.874 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (72s) 30s ago 11m 87.0M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:08:37.874 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (36s) 30s ago 10m 15.1M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:08:37.874 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (10m) 30s ago 10m 197M 1216M 18.2.1 5be31c24972a 9a07129d284c 2024-03-04T16:08:37.874 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (9m) 89s ago 9m 268M 2240M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:08:37.874 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (9m) 89s ago 9m 238M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:08:37.874 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (9m) 89s ago 9m 228M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:08:37.874 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (3m) 30s ago 12m 62.9M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:08:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 -- 172.21.15.116:0/2389911103 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f231006d8a0 msgr2=0x7f231006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f231006d8a0 0x7f231006fd60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2318006010 tx=0x7f2318005ec0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 -- 172.21.15.116:0/2389911103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2328100cb0 msgr2=0x7f2328196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2328100cb0 0x7f2328196860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f231c00d900 tx=0x7f231c00dc10 comp rx=0 tx=0).stop 2024-03-04T16:08:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 -- 172.21.15.116:0/2389911103 shutdown_connections 2024-03-04T16:08:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f231006d8a0 0x7f231006fd60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f23280ffab0 0x7f2328196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 --2- 172.21.15.116:0/2389911103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2328100cb0 0x7f2328196860 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 -- 172.21.15.116:0/2389911103 >> 172.21.15.116:0/2389911103 conn(0x7f23280fb030 msgr2=0x7f23280683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 -- 172.21.15.116:0/2389911103 shutdown_connections 2024-03-04T16:08:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.874+0000 7f232d05f700 1 -- 172.21.15.116:0/2389911103 wait complete. 2024-03-04T16:08:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.978+0000 7fe029298700 1 -- 172.21.15.116:0/690647654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024100bb0 msgr2=0x7fe024100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.978+0000 7fe029298700 1 --2- 172.21.15.116:0/690647654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024100bb0 0x7fe024100fd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe014009910 tx=0x7fe014009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.978+0000 7fe029298700 1 -- 172.21.15.116:0/690647654 shutdown_connections 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.978+0000 7fe029298700 1 --2- 172.21.15.116:0/690647654 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe024101db0 0x7fe024102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.978+0000 7fe029298700 1 --2- 172.21.15.116:0/690647654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024100bb0 0x7fe024100fd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.978+0000 7fe029298700 1 -- 172.21.15.116:0/690647654 >> 172.21.15.116:0/690647654 conn(0x7fe0240fc110 msgr2=0x7fe0240fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.978+0000 7fe029298700 1 -- 172.21.15.116:0/690647654 shutdown_connections 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.978+0000 7fe029298700 1 -- 172.21.15.116:0/690647654 wait complete. 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe029298700 1 Processor -- start 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe029298700 1 -- start start 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe029298700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe024100bb0 0x7fe024074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe029298700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024101db0 0x7fe024073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe029298700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240736c0 con 0x7fe024101db0 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe029298700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe024073800 con 0x7fe024100bb0 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe0227fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024101db0 0x7fe024073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe022ffd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe024100bb0 0x7fe024074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe0227fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024101db0 0x7fe024073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50242/0 (socket says 172.21.15.116:50242) 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe0227fc700 1 -- 172.21.15.116:0/3415104377 learned_addr learned my addr 172.21.15.116:0/3415104377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe0227fc700 1 -- 172.21.15.116:0/3415104377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe024100bb0 msgr2=0x7fe024074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe0227fc700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe024100bb0 0x7fe024074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe0227fc700 1 -- 172.21.15.116:0/3415104377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0140095f0 con 0x7fe024101db0 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.979+0000 7fe022ffd700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe024100bb0 0x7fe024074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.980+0000 7fe0227fc700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024101db0 0x7fe024073180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c8f0 tx=0x7fe01800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.980+0000 7fe00bfff700 1 -- 172.21.15.116:0/3415104377 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe018004a00 con 0x7fe024101db0 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.980+0000 7fe00bfff700 1 -- 172.21.15.116:0/3415104377 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe0180054d0 con 0x7fe024101db0 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.980+0000 7fe00bfff700 1 -- 172.21.15.116:0/3415104377 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe018007ce0 con 0x7fe024101db0 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.980+0000 7fe029298700 1 -- 172.21.15.116:0/3415104377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe024073ae0 con 0x7fe024101db0 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.980+0000 7fe029298700 1 -- 172.21.15.116:0/3415104377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02419e400 con 0x7fe024101db0 2024-03-04T16:08:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.981+0000 7fe029298700 1 -- 172.21.15.116:0/3415104377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe024109730 con 0x7fe024101db0 2024-03-04T16:08:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.981+0000 7fe00bfff700 1 -- 172.21.15.116:0/3415104377 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fe018005020 con 0x7fe024101db0 2024-03-04T16:08:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.982+0000 7fe00bfff700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe00c06d8a0 0x7fe00c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.982+0000 7fe022ffd700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe00c06d8a0 0x7fe00c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.982+0000 7fe00bfff700 1 -- 172.21.15.116:0/3415104377 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe018013070 con 0x7fe024101db0 2024-03-04T16:08:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.982+0000 7fe022ffd700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe00c06d8a0 0x7fe00c06fd60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe014006010 tx=0x7fe014005670 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:37.986+0000 7fe00bfff700 1 -- 172.21.15.116:0/3415104377 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe01805a620 con 0x7fe024101db0 2024-03-04T16:08:38.116 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 podman[114161]: 2024-03-04 16:08:37.809981577 +0000 UTC m=+0.018438165 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:08:38.116 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:37 smithi116 podman[114161]: 2024-03-04 16:08:37.971127215 +0000 UTC m=+0.179583791 container create 43927e90a69fbe4f1217710261b7422787a96d4155963a6a99bd66f72f7c5579 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-04T16:08:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.270+0000 7fe029298700 1 -- 172.21.15.116:0/3415104377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe02419e9b0 con 0x7fe024101db0 2024-03-04T16:08:38.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.271+0000 7fe00bfff700 1 -- 172.21.15.116:0/3415104377 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fe01805a1b0 con 0x7fe024101db0 2024-03-04T16:08:38.272 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:08:38.272 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:08:38.272 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:08:38.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:08:38.272 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:08:38.272 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:08:38.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:08:38.272 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 7, 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:08:38.273 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:08:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 -- 172.21.15.116:0/3415104377 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe00c06d8a0 msgr2=0x7fe00c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe00c06d8a0 0x7fe00c06fd60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe014006010 tx=0x7fe014005670 comp rx=0 tx=0).stop 2024-03-04T16:08:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 -- 172.21.15.116:0/3415104377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024101db0 msgr2=0x7fe024073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024101db0 0x7fe024073180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c8f0 tx=0x7fe01800cc00 comp rx=0 tx=0).stop 2024-03-04T16:08:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 -- 172.21.15.116:0/3415104377 shutdown_connections 2024-03-04T16:08:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe00c06d8a0 0x7fe00c06fd60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe024100bb0 0x7fe024074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 --2- 172.21.15.116:0/3415104377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024101db0 0x7fe024073180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 -- 172.21.15.116:0/3415104377 >> 172.21.15.116:0/3415104377 conn(0x7fe0240fc110 msgr2=0x7fe024104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 -- 172.21.15.116:0/3415104377 shutdown_connections 2024-03-04T16:08:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.273+0000 7fe029298700 1 -- 172.21.15.116:0/3415104377 wait complete. 2024-03-04T16:08:38.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.379+0000 7f638af8c700 1 -- 172.21.15.116:0/1341714145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6384101e60 msgr2=0x7f63841022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:38.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.379+0000 7f638af8c700 1 --2- 172.21.15.116:0/1341714145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6384101e60 0x7f63841022c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6370009910 tx=0x7f6370009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:38.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.379+0000 7f638af8c700 1 -- 172.21.15.116:0/1341714145 shutdown_connections 2024-03-04T16:08:38.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 --2- 172.21.15.116:0/1341714145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6384101e60 0x7f63841022c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 --2- 172.21.15.116:0/1341714145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63840fee80 0x7f63840ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 -- 172.21.15.116:0/1341714145 >> 172.21.15.116:0/1341714145 conn(0x7f63840fa7c0 msgr2=0x7f63840fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:38.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 -- 172.21.15.116:0/1341714145 shutdown_connections 2024-03-04T16:08:38.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 -- 172.21.15.116:0/1341714145 wait complete. 2024-03-04T16:08:38.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:38 smithi116 ceph-mon[94164]: pgmap v97: 65 pgs: 19 peering, 3 stale+active+clean, 43 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:38.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:38 smithi116 ceph-mon[94164]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:38.381 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:38.382 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:38.382 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:38 smithi116 podman[114161]: 2024-03-04 16:08:38.224230769 +0000 UTC m=+0.432687362 container init 43927e90a69fbe4f1217710261b7422787a96d4155963a6a99bd66f72f7c5579 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-04T16:08:38.382 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:38 smithi116 podman[114161]: 2024-03-04 16:08:38.22809665 +0000 UTC m=+0.436553227 container start 43927e90a69fbe4f1217710261b7422787a96d4155963a6a99bd66f72f7c5579 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_CLEAN=True) 2024-03-04T16:08:38.382 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:38 smithi116 ceph-osd[114202]: -- 172.21.15.116:0/1296989242 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f0eb549680 con 0x55f0eb513000 2024-03-04T16:08:38.382 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:38 smithi116 bash[114161]: 43927e90a69fbe4f1217710261b7422787a96d4155963a6a99bd66f72f7c5579 2024-03-04T16:08:38.382 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:38 smithi116 systemd[1]: Started Ceph osd.2 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 Processor -- start 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 -- start start 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63840fee80 0x7f638419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6384101e60 0x7f638419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638419d4a0 con 0x7f63840fee80 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.380+0000 7f638af8c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638419d5e0 con 0x7f6384101e60 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f6388d28700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63840fee80 0x7f638419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f6388d28700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63840fee80 0x7f638419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50270/0 (socket says 172.21.15.116:50270) 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f6388d28700 1 -- 172.21.15.116:0/2727959889 learned_addr learned my addr 172.21.15.116:0/2727959889 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f6383fff700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6384101e60 0x7f638419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:38.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f6388d28700 1 -- 172.21.15.116:0/2727959889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6384101e60 msgr2=0x7f638419ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:38.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f6388d28700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6384101e60 0x7f638419ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f6388d28700 1 -- 172.21.15.116:0/2727959889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63700095f0 con 0x7f63840fee80 2024-03-04T16:08:38.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f6388d28700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63840fee80 0x7f638419c940 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f637800d900 tx=0x7f637800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:38.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f6381ffb700 1 -- 172.21.15.116:0/2727959889 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63780041f0 con 0x7f63840fee80 2024-03-04T16:08:38.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f638af8c700 1 -- 172.21.15.116:0/2727959889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63841a2090 con 0x7f63840fee80 2024-03-04T16:08:38.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.381+0000 7f6381ffb700 1 -- 172.21.15.116:0/2727959889 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6378004350 con 0x7f63840fee80 2024-03-04T16:08:38.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.382+0000 7f6381ffb700 1 -- 172.21.15.116:0/2727959889 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6378003cc0 con 0x7f63840fee80 2024-03-04T16:08:38.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.382+0000 7f638af8c700 1 -- 172.21.15.116:0/2727959889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63841a25e0 con 0x7f63840fee80 2024-03-04T16:08:38.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.383+0000 7f638af8c700 1 -- 172.21.15.116:0/2727959889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6384105f40 con 0x7f63840fee80 2024-03-04T16:08:38.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.383+0000 7f6381ffb700 1 -- 172.21.15.116:0/2727959889 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f6378009730 con 0x7f63840fee80 2024-03-04T16:08:38.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.384+0000 7f6381ffb700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f637406d8a0 0x7f637406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:38.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.384+0000 7f6381ffb700 1 -- 172.21.15.116:0/2727959889 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f637808d380 con 0x7f63840fee80 2024-03-04T16:08:38.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.384+0000 7f6383fff700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f637406d8a0 0x7f637406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:38.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.384+0000 7f6383fff700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f637406d8a0 0x7f637406fd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6370006010 tx=0x7f637000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:38.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.389+0000 7f6381ffb700 1 -- 172.21.15.116:0/2727959889 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f637805a6b0 con 0x7f63840fee80 2024-03-04T16:08:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:38 smithi152 ceph-mon[82845]: pgmap v97: 65 pgs: 19 peering, 3 stale+active+clean, 43 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:38 smithi152 ceph-mon[82845]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:38.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.657+0000 7f638af8c700 1 -- 172.21.15.116:0/2727959889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6384061ae0 con 0x7f63840fee80 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.658+0000 7f6381ffb700 1 -- 172.21.15.116:0/2727959889 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1852 (secure 0 0 0) 0x7f637805a240 con 0x7f63840fee80 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:08:38.659 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:08:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:08:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:08:38.662 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:08:38.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 -- 172.21.15.116:0/2727959889 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f637406d8a0 msgr2=0x7f637406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:38.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f637406d8a0 0x7f637406fd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6370006010 tx=0x7f637000b540 comp rx=0 tx=0).stop 2024-03-04T16:08:38.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 -- 172.21.15.116:0/2727959889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63840fee80 msgr2=0x7f638419c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:38.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63840fee80 0x7f638419c940 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f637800d900 tx=0x7f637800dc10 comp rx=0 tx=0).stop 2024-03-04T16:08:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 -- 172.21.15.116:0/2727959889 shutdown_connections 2024-03-04T16:08:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f637406d8a0 0x7f637406fd60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63840fee80 0x7f638419c940 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 --2- 172.21.15.116:0/2727959889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6384101e60 0x7f638419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 -- 172.21.15.116:0/2727959889 >> 172.21.15.116:0/2727959889 conn(0x7f63840fa7c0 msgr2=0x7f6384103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 -- 172.21.15.116:0/2727959889 shutdown_connections 2024-03-04T16:08:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.660+0000 7f638af8c700 1 -- 172.21.15.116:0/2727959889 wait complete. 2024-03-04T16:08:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:08:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.769+0000 7fe245192700 1 -- 172.21.15.116:0/516459229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe240102dd0 msgr2=0x7fe2401031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.769+0000 7fe245192700 1 --2- 172.21.15.116:0/516459229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe240102dd0 0x7fe2401031f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe234009910 tx=0x7fe234009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.769+0000 7fe245192700 1 -- 172.21.15.116:0/516459229 shutdown_connections 2024-03-04T16:08:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.769+0000 7fe245192700 1 --2- 172.21.15.116:0/516459229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe240103fd0 0x7fe240104430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.769+0000 7fe245192700 1 --2- 172.21.15.116:0/516459229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe240102dd0 0x7fe2401031f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.769+0000 7fe245192700 1 -- 172.21.15.116:0/516459229 >> 172.21.15.116:0/516459229 conn(0x7fe2400fe390 msgr2=0x7fe2401007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.769+0000 7fe245192700 1 -- 172.21.15.116:0/516459229 shutdown_connections 2024-03-04T16:08:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.769+0000 7fe245192700 1 -- 172.21.15.116:0/516459229 wait complete. 2024-03-04T16:08:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.770+0000 7fe245192700 1 Processor -- start 2024-03-04T16:08:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.770+0000 7fe245192700 1 -- start start 2024-03-04T16:08:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.770+0000 7fe245192700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe240102dd0 0x7fe240198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.770+0000 7fe245192700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe240103fd0 0x7fe240198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.770+0000 7fe245192700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2401990c0 con 0x7fe240102dd0 2024-03-04T16:08:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.770+0000 7fe245192700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe240199200 con 0x7fe240103fd0 2024-03-04T16:08:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.770+0000 7fe23ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe240102dd0 0x7fe240198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.770+0000 7fe23ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe240102dd0 0x7fe240198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50286/0 (socket says 172.21.15.116:50286) 2024-03-04T16:08:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.770+0000 7fe23ed9d700 1 -- 172.21.15.116:0/705868325 learned_addr learned my addr 172.21.15.116:0/705868325 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.771+0000 7fe23e59c700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe240103fd0 0x7fe240198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.771+0000 7fe23ed9d700 1 -- 172.21.15.116:0/705868325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe240103fd0 msgr2=0x7fe240198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.771+0000 7fe23ed9d700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe240103fd0 0x7fe240198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.771+0000 7fe23ed9d700 1 -- 172.21.15.116:0/705868325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2340095f0 con 0x7fe240102dd0 2024-03-04T16:08:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.771+0000 7fe23e59c700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe240103fd0 0x7fe240198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T16:08:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.771+0000 7fe23ed9d700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe240102dd0 0x7fe240198560 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe234005b40 tx=0x7fe23400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.771+0000 7fe227fff700 1 -- 172.21.15.116:0/705868325 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe23401d070 con 0x7fe240102dd0 2024-03-04T16:08:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.771+0000 7fe245192700 1 -- 172.21.15.116:0/705868325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24019dc50 con 0x7fe240102dd0 2024-03-04T16:08:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.771+0000 7fe245192700 1 -- 172.21.15.116:0/705868325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24019e140 con 0x7fe240102dd0 2024-03-04T16:08:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.771+0000 7fe227fff700 1 -- 172.21.15.116:0/705868325 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe234005dc0 con 0x7fe240102dd0 2024-03-04T16:08:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.772+0000 7fe227fff700 1 -- 172.21.15.116:0/705868325 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe234022470 con 0x7fe240102dd0 2024-03-04T16:08:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.772+0000 7fe245192700 1 -- 172.21.15.116:0/705868325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22c005320 con 0x7fe240102dd0 2024-03-04T16:08:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.773+0000 7fe227fff700 1 -- 172.21.15.116:0/705868325 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fe234003680 con 0x7fe240102dd0 2024-03-04T16:08:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.774+0000 7fe227fff700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe22806d8a0 0x7fe22806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.774+0000 7fe227fff700 1 -- 172.21.15.116:0/705868325 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe23408ec00 con 0x7fe240102dd0 2024-03-04T16:08:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.774+0000 7fe23e59c700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe22806d8a0 0x7fe22806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.774+0000 7fe23e59c700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe22806d8a0 0x7fe22806fd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe230011490 tx=0x7fe230009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:38.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:38.782+0000 7fe227fff700 1 -- 172.21.15.116:0/705868325 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe23405bfb0 con 0x7fe240102dd0 2024-03-04T16:08:38.974 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:38 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[114198]: 2024-03-04T16:08:38.770+0000 7ff23e81a7c0 -1 Falling back to public interface 2024-03-04T16:08:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.058+0000 7fe245192700 1 -- 172.21.15.116:0/705868325 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe22c000bf0 con 0x7fe22806d8a0 2024-03-04T16:08:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.059+0000 7fe227fff700 1 -- 172.21.15.116:0/705868325 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe22c000bf0 con 0x7fe22806d8a0 2024-03-04T16:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:08:39.060 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "8/23 daemons upgraded", 2024-03-04T16:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:08:39.061 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:08:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 -- 172.21.15.116:0/705868325 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe22806d8a0 msgr2=0x7fe22806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe22806d8a0 0x7fe22806fd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe230011490 tx=0x7fe230009260 comp rx=0 tx=0).stop 2024-03-04T16:08:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 -- 172.21.15.116:0/705868325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe240102dd0 msgr2=0x7fe240198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe240102dd0 0x7fe240198560 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe234005b40 tx=0x7fe23400fa00 comp rx=0 tx=0).stop 2024-03-04T16:08:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 -- 172.21.15.116:0/705868325 shutdown_connections 2024-03-04T16:08:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe22806d8a0 0x7fe22806fd60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe240102dd0 0x7fe240198560 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 --2- 172.21.15.116:0/705868325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe240103fd0 0x7fe240198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:39.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 -- 172.21.15.116:0/705868325 >> 172.21.15.116:0/705868325 conn(0x7fe2400fe390 msgr2=0x7fe240107200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:39.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 -- 172.21.15.116:0/705868325 shutdown_connections 2024-03-04T16:08:39.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.062+0000 7fe245192700 1 -- 172.21.15.116:0/705868325 wait complete. 2024-03-04T16:08:39.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.168+0000 7f6839daa700 1 -- 172.21.15.116:0/3547752058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68340fee90 msgr2=0x7f68340ff2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:39.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.168+0000 7f6839daa700 1 --2- 172.21.15.116:0/3547752058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68340fee90 0x7f68340ff2b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6824009910 tx=0x7f6824009c20 comp rx=0 tx=0).stop 2024-03-04T16:08:39.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.168+0000 7f6839daa700 1 -- 172.21.15.116:0/3547752058 shutdown_connections 2024-03-04T16:08:39.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.168+0000 7f6839daa700 1 --2- 172.21.15.116:0/3547752058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6834101e70 0x7f68341022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:39.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.168+0000 7f6839daa700 1 --2- 172.21.15.116:0/3547752058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68340fee90 0x7f68340ff2b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:39.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.168+0000 7f6839daa700 1 -- 172.21.15.116:0/3547752058 >> 172.21.15.116:0/3547752058 conn(0x7f68340fa7f0 msgr2=0x7f68340fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:39.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.168+0000 7f6839daa700 1 -- 172.21.15.116:0/3547752058 shutdown_connections 2024-03-04T16:08:39.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.168+0000 7f6839daa700 1 -- 172.21.15.116:0/3547752058 wait complete. 2024-03-04T16:08:39.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.169+0000 7f6839daa700 1 Processor -- start 2024-03-04T16:08:39.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.169+0000 7f6839daa700 1 -- start start 2024-03-04T16:08:39.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.169+0000 7f6839daa700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68340fee90 0x7f6834198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.169+0000 7f6839daa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6834101e70 0x7f6834198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.169+0000 7f6839daa700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68341990d0 con 0x7f6834101e70 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.169+0000 7f6839daa700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6834199210 con 0x7f68340fee90 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.169+0000 7f6832ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6834101e70 0x7f6834198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.169+0000 7f6832ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6834101e70 0x7f6834198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50298/0 (socket says 172.21.15.116:50298) 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.169+0000 7f6832ffd700 1 -- 172.21.15.116:0/3931529146 learned_addr learned my addr 172.21.15.116:0/3931529146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.169+0000 7f68337fe700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68340fee90 0x7f6834198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.170+0000 7f6832ffd700 1 -- 172.21.15.116:0/3931529146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68340fee90 msgr2=0x7f6834198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.170+0000 7f6832ffd700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68340fee90 0x7f6834198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.170+0000 7f6832ffd700 1 -- 172.21.15.116:0/3931529146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68240095f0 con 0x7f6834101e70 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.170+0000 7f6832ffd700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6834101e70 0x7f6834198ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f682800d8d0 tx=0x7f682800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.170+0000 7f6830ff9700 1 -- 172.21.15.116:0/3931529146 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68280098a0 con 0x7f6834101e70 2024-03-04T16:08:39.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.170+0000 7f6839daa700 1 -- 172.21.15.116:0/3931529146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683419dcc0 con 0x7f6834101e70 2024-03-04T16:08:39.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.170+0000 7f6830ff9700 1 -- 172.21.15.116:0/3931529146 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6828010460 con 0x7f6834101e70 2024-03-04T16:08:39.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.170+0000 7f6830ff9700 1 -- 172.21.15.116:0/3931529146 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f682800f5f0 con 0x7f6834101e70 2024-03-04T16:08:39.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.170+0000 7f6839daa700 1 -- 172.21.15.116:0/3931529146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683419e210 con 0x7f6834101e70 2024-03-04T16:08:39.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.171+0000 7f6839daa700 1 -- 172.21.15.116:0/3931529146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6834105f50 con 0x7f6834101e70 2024-03-04T16:08:39.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.172+0000 7f6830ff9700 1 -- 172.21.15.116:0/3931529146 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f6828009a00 con 0x7f6834101e70 2024-03-04T16:08:39.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.172+0000 7f6830ff9700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f681c06d8a0 0x7f681c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:08:39.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.172+0000 7f6830ff9700 1 -- 172.21.15.116:0/3931529146 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f682808d360 con 0x7f6834101e70 2024-03-04T16:08:39.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.172+0000 7f68337fe700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f681c06d8a0 0x7f681c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:08:39.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.172+0000 7f68337fe700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f681c06d8a0 0x7f681c06fd60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6824006010 tx=0x7f682400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:08:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.177+0000 7f6830ff9700 1 -- 172.21.15.116:0/3931529146 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f682805a690 con 0x7f6834101e70 2024-03-04T16:08:39.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:39 smithi116 ceph-mon[94164]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:39.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:39 smithi116 ceph-mon[94164]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:39.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:39 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3415104377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:39.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:39.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:39.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:39.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:39 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2727959889' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:08:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.457+0000 7f6839daa700 1 -- 172.21.15.116:0/3931529146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6834061a40 con 0x7f6834101e70 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.457+0000 7f6830ff9700 1 -- 172.21.15.116:0/3931529146 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+940 (secure 0 0 0) 0x7f682805a220 con 0x7f6834101e70 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 30/231 objects degraded (12.987%), 12 pgs degraded 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stdout: osd.2 (root=default,host=smithi116) is down 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 30/231 objects degraded (12.987%), 12 pgs degraded 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1 is active+undersized+degraded, acting [1,0] 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.3 is active+undersized+degraded, acting [5,1] 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.7 is active+undersized+degraded, acting [3,4] 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.c is active+undersized+degraded, acting [3,0] 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-03-04T16:08:39.458 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.e is active+undersized+degraded, acting [0,3] 2024-03-04T16:08:39.459 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-03-04T16:08:39.459 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-03-04T16:08:39.459 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.16 is active+undersized+degraded, acting [5,3] 2024-03-04T16:08:39.459 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.19 is active+undersized+degraded, acting [0,4] 2024-03-04T16:08:39.459 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1c is active+undersized+degraded, acting [4,5] 2024-03-04T16:08:39.459 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-03-04T16:08:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 -- 172.21.15.116:0/3931529146 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f681c06d8a0 msgr2=0x7f681c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f681c06d8a0 0x7f681c06fd60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6824006010 tx=0x7f682400b540 comp rx=0 tx=0).stop 2024-03-04T16:08:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 -- 172.21.15.116:0/3931529146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6834101e70 msgr2=0x7f6834198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:08:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6834101e70 0x7f6834198ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f682800d8d0 tx=0x7f682800dbe0 comp rx=0 tx=0).stop 2024-03-04T16:08:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 -- 172.21.15.116:0/3931529146 shutdown_connections 2024-03-04T16:08:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f681c06d8a0 0x7f681c06fd60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68340fee90 0x7f6834198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 --2- 172.21.15.116:0/3931529146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6834101e70 0x7f6834198ab0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:08:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 -- 172.21.15.116:0/3931529146 >> 172.21.15.116:0/3931529146 conn(0x7f68340fa7f0 msgr2=0x7f6834103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:08:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 -- 172.21.15.116:0/3931529146 shutdown_connections 2024-03-04T16:08:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:08:39.459+0000 7f6839daa700 1 -- 172.21.15.116:0/3931529146 wait complete. 2024-03-04T16:08:39.487 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:39 smithi152 ceph-mon[82845]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:39.487 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:39 smithi152 ceph-mon[82845]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:39.487 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:39 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3415104377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:39.487 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:39.487 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:39.487 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:39.487 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:39 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2727959889' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:08:40.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:40 smithi116 ceph-mon[94164]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30/231 objects degraded (12.987%) 2024-03-04T16:08:40.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:40 smithi116 ceph-mon[94164]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:40.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:40 smithi116 ceph-mon[94164]: Health check failed: Degraded data redundancy: 30/231 objects degraded (12.987%), 12 pgs degraded (PG_DEGRADED) 2024-03-04T16:08:40.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:40 smithi116 ceph-mon[94164]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs peering) 2024-03-04T16:08:40.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:40 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3931529146' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:08:40.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:40 smithi152 ceph-mon[82845]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30/231 objects degraded (12.987%) 2024-03-04T16:08:40.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:40 smithi152 ceph-mon[82845]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:40.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:40 smithi152 ceph-mon[82845]: Health check failed: Degraded data redundancy: 30/231 objects degraded (12.987%), 12 pgs degraded (PG_DEGRADED) 2024-03-04T16:08:40.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:40 smithi152 ceph-mon[82845]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs peering) 2024-03-04T16:08:40.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:40 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3931529146' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:08:42.283 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:42 smithi116 ceph-mon[94164]: pgmap v99: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30/231 objects degraded (12.987%) 2024-03-04T16:08:42.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:42 smithi152 ceph-mon[82845]: pgmap v99: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30/231 objects degraded (12.987%) 2024-03-04T16:08:43.475 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:43 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[114198]: 2024-03-04T16:08:43.255+0000 7ff23e81a7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-04T16:08:43.975 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:43 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[114198]: 2024-03-04T16:08:43.691+0000 7ff23e81a7c0 -1 osd.2 60 log_to_monitors true 2024-03-04T16:08:44.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:44 smithi116 ceph-mon[94164]: pgmap v100: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30/231 objects degraded (12.987%) 2024-03-04T16:08:44.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:44 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:44.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:44 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:44.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:44 smithi116 ceph-mon[94164]: from='osd.2 [v2:172.21.15.116:6818/1756826225,v1:172.21.15.116:6819/1756826225]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:08:44.535 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:44 smithi152 ceph-mon[82845]: pgmap v100: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30/231 objects degraded (12.987%) 2024-03-04T16:08:44.535 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:44 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:44.535 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:44 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:44.535 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:44 smithi152 ceph-mon[82845]: from='osd.2 [v2:172.21.15.116:6818/1756826225,v1:172.21.15.116:6819/1756826225]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:08:44.974 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:08:44 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[114198]: 2024-03-04T16:08:44.661+0000 7ff22fc72700 -1 osd.2 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:08:45.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:45 smithi116 ceph-mon[94164]: from='osd.2 [v2:172.21.15.116:6818/1756826225,v1:172.21.15.116:6819/1756826225]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T16:08:45.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:45 smithi116 ceph-mon[94164]: osdmap e63: 6 total, 5 up, 6 in 2024-03-04T16:08:45.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:45 smithi116 ceph-mon[94164]: from='osd.2 [v2:172.21.15.116:6818/1756826225,v1:172.21.15.116:6819/1756826225]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T16:08:45.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:45 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:45.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:45 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:45.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:45 smithi116 ceph-mon[94164]: pgmap v102: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 30/231 objects degraded (12.987%) 2024-03-04T16:08:46.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:45 smithi152 ceph-mon[82845]: from='osd.2 [v2:172.21.15.116:6818/1756826225,v1:172.21.15.116:6819/1756826225]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T16:08:46.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:45 smithi152 ceph-mon[82845]: osdmap e63: 6 total, 5 up, 6 in 2024-03-04T16:08:46.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:45 smithi152 ceph-mon[82845]: from='osd.2 [v2:172.21.15.116:6818/1756826225,v1:172.21.15.116:6819/1756826225]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T16:08:46.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:45 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:46.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:45 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:46.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:45 smithi152 ceph-mon[82845]: pgmap v102: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 30/231 objects degraded (12.987%) 2024-03-04T16:08:46.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:46 smithi116 ceph-mon[94164]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:08:46.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:46 smithi116 ceph-mon[94164]: osd.2 [v2:172.21.15.116:6818/1756826225,v1:172.21.15.116:6819/1756826225] boot 2024-03-04T16:08:46.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:46 smithi116 ceph-mon[94164]: osdmap e64: 6 total, 6 up, 6 in 2024-03-04T16:08:46.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:46 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:08:46.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:46 smithi116 ceph-mon[94164]: osdmap e65: 6 total, 6 up, 6 in 2024-03-04T16:08:47.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:46 smithi152 ceph-mon[82845]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:08:47.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:46 smithi152 ceph-mon[82845]: osd.2 [v2:172.21.15.116:6818/1756826225,v1:172.21.15.116:6819/1756826225] boot 2024-03-04T16:08:47.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:46 smithi152 ceph-mon[82845]: osdmap e64: 6 total, 6 up, 6 in 2024-03-04T16:08:47.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:46 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:08:47.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:46 smithi152 ceph-mon[82845]: osdmap e65: 6 total, 6 up, 6 in 2024-03-04T16:08:47.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:47 smithi116 ceph-mon[94164]: pgmap v105: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 30/231 objects degraded (12.987%) 2024-03-04T16:08:48.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:47 smithi152 ceph-mon[82845]: pgmap v105: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 30/231 objects degraded (12.987%) 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:08:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:49 smithi116 ceph-mon[94164]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-04T16:08:49.585 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:49.586 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:49.586 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:49.586 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:49.586 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:49.586 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:49.586 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:49.586 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:49.586 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:49.586 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:08:49.587 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:08:49.587 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:49 smithi152 ceph-mon[82845]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-04T16:08:50.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:50 smithi116 ceph-mon[94164]: pgmap v106: 65 pgs: 5 active+undersized+degraded, 4 active+undersized, 56 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 14/231 objects degraded (6.061%) 2024-03-04T16:08:50.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:50 smithi116 ceph-mon[94164]: Health check update: Degraded data redundancy: 14/231 objects degraded (6.061%), 5 pgs degraded (PG_DEGRADED) 2024-03-04T16:08:50.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:50 smithi152 ceph-mon[82845]: pgmap v106: 65 pgs: 5 active+undersized+degraded, 4 active+undersized, 56 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 14/231 objects degraded (6.061%) 2024-03-04T16:08:50.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:50 smithi152 ceph-mon[82845]: Health check update: Degraded data redundancy: 14/231 objects degraded (6.061%), 5 pgs degraded (PG_DEGRADED) 2024-03-04T16:08:51.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:51 smithi116 ceph-mon[94164]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/231 objects degraded (6.061%), 5 pgs degraded) 2024-03-04T16:08:51.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:51 smithi116 ceph-mon[94164]: Cluster is now healthy 2024-03-04T16:08:51.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:51 smithi152 ceph-mon[82845]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/231 objects degraded (6.061%), 5 pgs degraded) 2024-03-04T16:08:51.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:51 smithi152 ceph-mon[82845]: Cluster is now healthy 2024-03-04T16:08:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:52 smithi116 ceph-mon[94164]: pgmap v107: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:52 smithi152 ceph-mon[82845]: pgmap v107: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:54.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:54 smithi152 ceph-mon[82845]: pgmap v108: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:54.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:54 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:54.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:54 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:54.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:54 smithi116 ceph-mon[94164]: pgmap v108: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:54.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:54 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:54.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:54 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:56.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:56 smithi152 ceph-mon[82845]: pgmap v109: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:56 smithi116 ceph-mon[94164]: pgmap v109: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:58.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:58 smithi152 ceph-mon[82845]: pgmap v110: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:58.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:58 smithi116 ceph-mon[94164]: pgmap v110: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:08:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:08:59 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:08:59.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:08:59 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:00 smithi116 ceph-mon[94164]: pgmap v111: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:00.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:00 smithi152 ceph-mon[82845]: pgmap v111: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:02.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:02 smithi116 ceph-mon[94164]: pgmap v112: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:02.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:02 smithi152 ceph-mon[82845]: pgmap v112: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:04 smithi116 ceph-mon[94164]: pgmap v113: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:04 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:09:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:04 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:09:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:04 smithi116 ceph-mon[94164]: Upgrade: osd.3 is safe to restart 2024-03-04T16:09:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:04 smithi152 ceph-mon[82845]: pgmap v113: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:04 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:09:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:04 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:09:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:04 smithi152 ceph-mon[82845]: Upgrade: osd.3 is safe to restart 2024-03-04T16:09:06.321 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:06 smithi152 ceph-mon[82845]: Upgrade: Updating osd.3 2024-03-04T16:09:06.321 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:06 smithi152 ceph-mon[82845]: pgmap v114: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:06.321 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:06 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:06.321 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:06 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:09:06.321 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:06 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:06.321 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:06 smithi152 ceph-mon[82845]: Deploying daemon osd.3 on smithi152 2024-03-04T16:09:06.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:06 smithi116 ceph-mon[94164]: Upgrade: Updating osd.3 2024-03-04T16:09:06.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:06 smithi116 ceph-mon[94164]: pgmap v114: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:06.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:06 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:06.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:06 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:09:06.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:06 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:06.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:06 smithi116 ceph-mon[94164]: Deploying daemon osd.3 on smithi152 2024-03-04T16:09:08.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:08 smithi152 ceph-mon[82845]: pgmap v115: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:08 smithi116 ceph-mon[94164]: pgmap v115: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:08.635 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:08 smithi152 systemd[1]: Stopping Ceph osd.3 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:09:09.063 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:08 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[42907]: 2024-03-04T16:09:08.634+0000 7fbfd06ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:09:09.063 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:08 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[42907]: 2024-03-04T16:09:08.634+0000 7fbfd06ed700 -1 osd.3 65 *** Got signal Terminated *** 2024-03-04T16:09:09.063 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:08 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[42907]: 2024-03-04T16:09:08.634+0000 7fbfd06ed700 -1 osd.3 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:09:09.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:09.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:09 smithi152 ceph-mon[82845]: osd.3 marked itself down and dead 2024-03-04T16:09:09.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:09.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:09 smithi116 ceph-mon[94164]: osd.3 marked itself down and dead 2024-03-04T16:09:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.578+0000 7f515cc5b700 1 -- 172.21.15.116:0/3756523031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5158100cc0 msgr2=0x7f5158101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.578+0000 7f515cc5b700 1 --2- 172.21.15.116:0/3756523031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5158100cc0 0x7f5158101120 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5140009910 tx=0x7f5140009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.579+0000 7f515cc5b700 1 -- 172.21.15.116:0/3756523031 shutdown_connections 2024-03-04T16:09:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.579+0000 7f515cc5b700 1 --2- 172.21.15.116:0/3756523031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5158100cc0 0x7f5158101120 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.579+0000 7f515cc5b700 1 --2- 172.21.15.116:0/3756523031 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51580ffac0 0x7f51580ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.579+0000 7f515cc5b700 1 -- 172.21.15.116:0/3756523031 >> 172.21.15.116:0/3756523031 conn(0x7f51580fb060 msgr2=0x7f51580fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.579+0000 7f515cc5b700 1 -- 172.21.15.116:0/3756523031 shutdown_connections 2024-03-04T16:09:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.579+0000 7f515cc5b700 1 -- 172.21.15.116:0/3756523031 wait complete. 2024-03-04T16:09:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.579+0000 7f515cc5b700 1 Processor -- start 2024-03-04T16:09:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.579+0000 7f515cc5b700 1 -- start start 2024-03-04T16:09:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515cc5b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51580ffac0 0x7f5158195270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515cc5b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5158100cc0 0x7f51581957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515cc5b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5158195dd0 con 0x7f51580ffac0 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515cc5b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5158195f10 con 0x7f5158100cc0 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51580ffac0 0x7f5158195270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51580ffac0 0x7f5158195270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54178/0 (socket says 172.21.15.116:54178) 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515659c700 1 -- 172.21.15.116:0/794546558 learned_addr learned my addr 172.21.15.116:0/794546558 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f5155d9b700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5158100cc0 0x7f51581957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515659c700 1 -- 172.21.15.116:0/794546558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5158100cc0 msgr2=0x7f51581957b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515659c700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5158100cc0 0x7f51581957b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515659c700 1 -- 172.21.15.116:0/794546558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51400095f0 con 0x7f51580ffac0 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515659c700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51580ffac0 0x7f5158195270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f515000ada0 tx=0x7f515000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f514f7fe700 1 -- 172.21.15.116:0/794546558 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f515000f070 con 0x7f51580ffac0 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f514f7fe700 1 -- 172.21.15.116:0/794546558 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f515000ae90 con 0x7f51580ffac0 2024-03-04T16:09:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.580+0000 7f515cc5b700 1 -- 172.21.15.116:0/794546558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515819a9c0 con 0x7f51580ffac0 2024-03-04T16:09:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.581+0000 7f514f7fe700 1 -- 172.21.15.116:0/794546558 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51500083f0 con 0x7f51580ffac0 2024-03-04T16:09:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.581+0000 7f515cc5b700 1 -- 172.21.15.116:0/794546558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515819af90 con 0x7f51580ffac0 2024-03-04T16:09:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.581+0000 7f515cc5b700 1 -- 172.21.15.116:0/794546558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5158108620 con 0x7f51580ffac0 2024-03-04T16:09:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.582+0000 7f514f7fe700 1 -- 172.21.15.116:0/794546558 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f515000cad0 con 0x7f51580ffac0 2024-03-04T16:09:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.582+0000 7f514f7fe700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f514406d8a0 0x7f514406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.582+0000 7f514f7fe700 1 -- 172.21.15.116:0/794546558 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f515008cda0 con 0x7f51580ffac0 2024-03-04T16:09:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.583+0000 7f5155d9b700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f514406d8a0 0x7f514406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.583+0000 7f5155d9b700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f514406d8a0 0x7f514406fd60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5140006010 tx=0x7f51400058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.587+0000 7f514f7fe700 1 -- 172.21.15.116:0/794546558 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f515005a150 con 0x7f51580ffac0 2024-03-04T16:09:09.769 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:09 smithi152 podman[92493]: 2024-03-04 16:09:09.44167995 +0000 UTC m=+0.917783777 container died 57cf87525aba64a10ed84abbf9b22be0d809617c683be7ce8815cf10bd0c61ea (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-04T16:09:09.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.797+0000 7f515cc5b700 1 -- 172.21.15.116:0/794546558 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5158105610 con 0x7f514406d8a0 2024-03-04T16:09:09.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.798+0000 7f514f7fe700 1 -- 172.21.15.116:0/794546558 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f5158105610 con 0x7f514406d8a0 2024-03-04T16:09:09.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 -- 172.21.15.116:0/794546558 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f514406d8a0 msgr2=0x7f514406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:09.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f514406d8a0 0x7f514406fd60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5140006010 tx=0x7f51400058e0 comp rx=0 tx=0).stop 2024-03-04T16:09:09.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 -- 172.21.15.116:0/794546558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51580ffac0 msgr2=0x7f5158195270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:09.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51580ffac0 0x7f5158195270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f515000ada0 tx=0x7f515000c330 comp rx=0 tx=0).stop 2024-03-04T16:09:09.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 -- 172.21.15.116:0/794546558 shutdown_connections 2024-03-04T16:09:09.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f514406d8a0 0x7f514406fd60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:09.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51580ffac0 0x7f5158195270 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:09.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 --2- 172.21.15.116:0/794546558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5158100cc0 0x7f51581957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:09.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 -- 172.21.15.116:0/794546558 >> 172.21.15.116:0/794546558 conn(0x7f51580fb060 msgr2=0x7f5158103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:09.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 -- 172.21.15.116:0/794546558 shutdown_connections 2024-03-04T16:09:09.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.800+0000 7f515cc5b700 1 -- 172.21.15.116:0/794546558 wait complete. 2024-03-04T16:09:09.811 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:09:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.906+0000 7f06dc3f4700 1 -- 172.21.15.116:0/3690028715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4100bc0 msgr2=0x7f06d4100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.906+0000 7f06dc3f4700 1 --2- 172.21.15.116:0/3690028715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4100bc0 0x7f06d4100fe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009910 tx=0x7f06d0009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.906+0000 7f06dc3f4700 1 -- 172.21.15.116:0/3690028715 shutdown_connections 2024-03-04T16:09:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.906+0000 7f06dc3f4700 1 --2- 172.21.15.116:0/3690028715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4101dc0 0x7f06d4102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.906+0000 7f06dc3f4700 1 --2- 172.21.15.116:0/3690028715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4100bc0 0x7f06d4100fe0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.906+0000 7f06dc3f4700 1 -- 172.21.15.116:0/3690028715 >> 172.21.15.116:0/3690028715 conn(0x7f06d40fc140 msgr2=0x7f06d40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.906+0000 7f06dc3f4700 1 -- 172.21.15.116:0/3690028715 shutdown_connections 2024-03-04T16:09:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.906+0000 7f06dc3f4700 1 -- 172.21.15.116:0/3690028715 wait complete. 2024-03-04T16:09:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.906+0000 7f06dc3f4700 1 Processor -- start 2024-03-04T16:09:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.906+0000 7f06dc3f4700 1 -- start start 2024-03-04T16:09:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06dc3f4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4100bc0 0x7f06d4196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06dc3f4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4101dc0 0x7f06d41968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06dc3f4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d4196ef0 con 0x7f06d4101dc0 2024-03-04T16:09:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06dc3f4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d4197030 con 0x7f06d4100bc0 2024-03-04T16:09:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06d998f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4101dc0 0x7f06d41968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06d998f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4101dc0 0x7f06d41968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54184/0 (socket says 172.21.15.116:54184) 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06d998f700 1 -- 172.21.15.116:0/1706907826 learned_addr learned my addr 172.21.15.116:0/1706907826 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06da190700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4100bc0 0x7f06d4196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06d998f700 1 -- 172.21.15.116:0/1706907826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4100bc0 msgr2=0x7f06d4196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06d998f700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4100bc0 0x7f06d4196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06d998f700 1 -- 172.21.15.116:0/1706907826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d00095f0 con 0x7f06d4101dc0 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06d998f700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4101dc0 0x7f06d41968d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f06c400d8d0 tx=0x7f06c400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06cb7fe700 1 -- 172.21.15.116:0/1706907826 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06c40098a0 con 0x7f06d4101dc0 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06cb7fe700 1 -- 172.21.15.116:0/1706907826 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f06c4010460 con 0x7f06d4101dc0 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.907+0000 7f06dc3f4700 1 -- 172.21.15.116:0/1706907826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d419bae0 con 0x7f06d4101dc0 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.908+0000 7f06cb7fe700 1 -- 172.21.15.116:0/1706907826 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06c400f5f0 con 0x7f06d4101dc0 2024-03-04T16:09:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.908+0000 7f06dc3f4700 1 -- 172.21.15.116:0/1706907826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d419c030 con 0x7f06d4101dc0 2024-03-04T16:09:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.909+0000 7f06cb7fe700 1 -- 172.21.15.116:0/1706907826 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f06c4009a00 con 0x7f06d4101dc0 2024-03-04T16:09:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.909+0000 7f06dc3f4700 1 -- 172.21.15.116:0/1706907826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d4109720 con 0x7f06d4101dc0 2024-03-04T16:09:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.909+0000 7f06cb7fe700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f06c006d8a0 0x7f06c006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.910+0000 7f06cb7fe700 1 -- 172.21.15.116:0/1706907826 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f06c408d030 con 0x7f06d4101dc0 2024-03-04T16:09:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.910+0000 7f06da190700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f06c006d8a0 0x7f06c006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.910+0000 7f06da190700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f06c006d8a0 0x7f06c006fd60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f06d0006010 tx=0x7f06d000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:09.914+0000 7f06cb7fe700 1 -- 172.21.15.116:0/1706907826 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f06c405a3e0 con 0x7f06d4101dc0 2024-03-04T16:09:10.064 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:09 smithi152 podman[92493]: 2024-03-04 16:09:09.868632854 +0000 UTC m=+1.344736651 container cleanup 57cf87525aba64a10ed84abbf9b22be0d809617c683be7ce8815cf10bd0c61ea (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) 2024-03-04T16:09:10.064 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:09 smithi152 bash[92493]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3 2024-03-04T16:09:10.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.146+0000 7f06dc3f4700 1 -- 172.21.15.116:0/1706907826 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f06d4106710 con 0x7f06c006d8a0 2024-03-04T16:09:10.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.147+0000 7f06cb7fe700 1 -- 172.21.15.116:0/1706907826 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f06d4106710 con 0x7f06c006d8a0 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 -- 172.21.15.116:0/1706907826 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f06c006d8a0 msgr2=0x7f06c006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f06c006d8a0 0x7f06c006fd60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f06d0006010 tx=0x7f06d000b540 comp rx=0 tx=0).stop 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 -- 172.21.15.116:0/1706907826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4101dc0 msgr2=0x7f06d41968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4101dc0 0x7f06d41968d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f06c400d8d0 tx=0x7f06c400dbe0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 -- 172.21.15.116:0/1706907826 shutdown_connections 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f06c006d8a0 0x7f06c006fd60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4100bc0 0x7f06d4196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 --2- 172.21.15.116:0/1706907826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4101dc0 0x7f06d41968d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 -- 172.21.15.116:0/1706907826 >> 172.21.15.116:0/1706907826 conn(0x7f06d40fc140 msgr2=0x7f06d4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 -- 172.21.15.116:0/1706907826 shutdown_connections 2024-03-04T16:09:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.149+0000 7f06dc3f4700 1 -- 172.21.15.116:0/1706907826 wait complete. 2024-03-04T16:09:10.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:10 smithi116 ceph-mon[94164]: pgmap v116: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:10.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:10 smithi116 ceph-mon[94164]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:09:10.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:10 smithi116 ceph-mon[94164]: osdmap e66: 6 total, 5 up, 6 in 2024-03-04T16:09:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.253+0000 7faae9b53700 1 -- 172.21.15.116:0/195665797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae4101dd0 msgr2=0x7faae4102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.253+0000 7faae9b53700 1 --2- 172.21.15.116:0/195665797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae4101dd0 0x7faae4102230 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7faad8009910 tx=0x7faad8009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.253+0000 7faae9b53700 1 -- 172.21.15.116:0/195665797 shutdown_connections 2024-03-04T16:09:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.253+0000 7faae9b53700 1 --2- 172.21.15.116:0/195665797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae4101dd0 0x7faae4102230 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.253+0000 7faae9b53700 1 --2- 172.21.15.116:0/195665797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4100bd0 0x7faae4100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.253+0000 7faae9b53700 1 -- 172.21.15.116:0/195665797 >> 172.21.15.116:0/195665797 conn(0x7faae40fc170 msgr2=0x7faae40fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.253+0000 7faae9b53700 1 -- 172.21.15.116:0/195665797 shutdown_connections 2024-03-04T16:09:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.253+0000 7faae9b53700 1 -- 172.21.15.116:0/195665797 wait complete. 2024-03-04T16:09:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.253+0000 7faae9b53700 1 Processor -- start 2024-03-04T16:09:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.253+0000 7faae9b53700 1 -- start start 2024-03-04T16:09:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae9b53700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4100bd0 0x7faae4196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae9b53700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae4101dd0 0x7faae41968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae9b53700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae4196ee0 con 0x7faae4101dd0 2024-03-04T16:09:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae9b53700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae4197020 con 0x7faae4100bd0 2024-03-04T16:09:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae4101dd0 0x7faae41968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae4101dd0 0x7faae41968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54196/0 (socket says 172.21.15.116:54196) 2024-03-04T16:09:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae2ffd700 1 -- 172.21.15.116:0/2782055863 learned_addr learned my addr 172.21.15.116:0/2782055863 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae37fe700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4100bd0 0x7faae4196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae2ffd700 1 -- 172.21.15.116:0/2782055863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4100bd0 msgr2=0x7faae4196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae2ffd700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4100bd0 0x7faae4196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae2ffd700 1 -- 172.21.15.116:0/2782055863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad80095f0 con 0x7faae4101dd0 2024-03-04T16:09:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae2ffd700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae4101dd0 0x7faae41968c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faad800b5c0 tx=0x7faad800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae0ff9700 1 -- 172.21.15.116:0/2782055863 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faad801d070 con 0x7faae4101dd0 2024-03-04T16:09:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae0ff9700 1 -- 172.21.15.116:0/2782055863 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faad800f460 con 0x7faae4101dd0 2024-03-04T16:09:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae9b53700 1 -- 172.21.15.116:0/2782055863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae419ba70 con 0x7faae4101dd0 2024-03-04T16:09:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae0ff9700 1 -- 172.21.15.116:0/2782055863 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faad8021640 con 0x7faae4101dd0 2024-03-04T16:09:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.254+0000 7faae9b53700 1 -- 172.21.15.116:0/2782055863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae419bf60 con 0x7faae4101dd0 2024-03-04T16:09:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.255+0000 7faae9b53700 1 -- 172.21.15.116:0/2782055863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae4109730 con 0x7faae4101dd0 2024-03-04T16:09:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.257+0000 7faae0ff9700 1 -- 172.21.15.116:0/2782055863 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7faad800fa60 con 0x7faae4101dd0 2024-03-04T16:09:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.257+0000 7faae0ff9700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faacc06d8a0 0x7faacc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.257+0000 7faae0ff9700 1 -- 172.21.15.116:0/2782055863 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faad808eab0 con 0x7faae4101dd0 2024-03-04T16:09:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.258+0000 7faae37fe700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faacc06d8a0 0x7faacc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.258+0000 7faae37fe700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faacc06d8a0 0x7faacc06fd60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faae4101c30 tx=0x7faad400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:10.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.261+0000 7faae0ff9700 1 -- 172.21.15.116:0/2782055863 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faad805bf00 con 0x7faae4101dd0 2024-03-04T16:09:10.357 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:10 smithi152 podman[92511]: 2024-03-04 16:09:10.094915311 +0000 UTC m=+0.650766682 container remove 57cf87525aba64a10ed84abbf9b22be0d809617c683be7ce8815cf10bd0c61ea (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1) 2024-03-04T16:09:10.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:10 smithi152 ceph-mon[82845]: pgmap v116: 65 pgs: 65 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:10.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:10 smithi152 ceph-mon[82845]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:09:10.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:10 smithi152 ceph-mon[82845]: osdmap e66: 6 total, 5 up, 6 in 2024-03-04T16:09:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.463+0000 7faae9b53700 1 -- 172.21.15.116:0/2782055863 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faae4106720 con 0x7faacc06d8a0 2024-03-04T16:09:10.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.473+0000 7faae0ff9700 1 -- 172.21.15.116:0/2782055863 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7faae4106720 con 0x7faacc06d8a0 2024-03-04T16:09:10.474 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:09:10.474 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (12m) 26s ago 13m 25.4M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (14m) 26s ago 13m 8284k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (12m) 2m ago 12m 7838k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (2m) 26s ago 13m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (2m) 2m ago 12m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (12m) 26s ago 13m 87.8M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (7m) 26s ago 7m 19.9M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (8m) 26s ago 8m 16.1M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (7m) 2m ago 7m 16.7M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (8m) 2m ago 8m 143M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (4m) 26s ago 15m 503M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (3m) 2m ago 12m 449M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (3m) 26s ago 15m 46.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (2m) 2m ago 12m 33.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:09:10.475 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (13m) 26s ago 13m 15.4M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:09:10.476 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (12m) 2m ago 12m 16.7M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:09:10.476 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (104s) 26s ago 11m 87.1M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:09:10.476 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (68s) 26s ago 11m 101M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:09:10.476 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (32s) 26s ago 10m 12.7M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:09:10.476 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (10m) 2m ago 10m 268M 2240M 18.2.1 5be31c24972a 57cf87525aba 2024-03-04T16:09:10.476 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (10m) 2m ago 10m 238M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:09:10.476 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (9m) 2m ago 9m 228M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:09:10.476 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (4m) 26s ago 13m 63.0M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:09:10.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 -- 172.21.15.116:0/2782055863 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faacc06d8a0 msgr2=0x7faacc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faacc06d8a0 0x7faacc06fd60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faae4101c30 tx=0x7faad400b410 comp rx=0 tx=0).stop 2024-03-04T16:09:10.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 -- 172.21.15.116:0/2782055863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae4101dd0 msgr2=0x7faae41968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae4101dd0 0x7faae41968c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faad800b5c0 tx=0x7faad800bd70 comp rx=0 tx=0).stop 2024-03-04T16:09:10.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 -- 172.21.15.116:0/2782055863 shutdown_connections 2024-03-04T16:09:10.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faacc06d8a0 0x7faacc06fd60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4100bd0 0x7faae4196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 --2- 172.21.15.116:0/2782055863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae4101dd0 0x7faae41968c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 -- 172.21.15.116:0/2782055863 >> 172.21.15.116:0/2782055863 conn(0x7faae40fc170 msgr2=0x7faae4105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:10.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 -- 172.21.15.116:0/2782055863 shutdown_connections 2024-03-04T16:09:10.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.476+0000 7faae9b53700 1 -- 172.21.15.116:0/2782055863 wait complete. 2024-03-04T16:09:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.579+0000 7fb65058a700 1 -- 172.21.15.116:0/2081129552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb648101dc0 msgr2=0x7fb648102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.579+0000 7fb65058a700 1 --2- 172.21.15.116:0/2081129552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb648101dc0 0x7fb648102220 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb638009910 tx=0x7fb638009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.579+0000 7fb65058a700 1 -- 172.21.15.116:0/2081129552 shutdown_connections 2024-03-04T16:09:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.579+0000 7fb65058a700 1 --2- 172.21.15.116:0/2081129552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb648101dc0 0x7fb648102220 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.579+0000 7fb65058a700 1 --2- 172.21.15.116:0/2081129552 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb648100bc0 0x7fb648100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.579+0000 7fb65058a700 1 -- 172.21.15.116:0/2081129552 >> 172.21.15.116:0/2081129552 conn(0x7fb6480fc140 msgr2=0x7fb6480fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.579+0000 7fb65058a700 1 -- 172.21.15.116:0/2081129552 shutdown_connections 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.579+0000 7fb65058a700 1 -- 172.21.15.116:0/2081129552 wait complete. 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb65058a700 1 Processor -- start 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb65058a700 1 -- start start 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb65058a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb648100bc0 0x7fb6481963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb65058a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb648101dc0 0x7fb648196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb65058a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb648196f40 con 0x7fb648101dc0 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb65058a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb648197080 con 0x7fb648100bc0 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb64db25700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb648101dc0 0x7fb648196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb64db25700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb648101dc0 0x7fb648196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54220/0 (socket says 172.21.15.116:54220) 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb64db25700 1 -- 172.21.15.116:0/2485399895 learned_addr learned my addr 172.21.15.116:0/2485399895 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb64e326700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb648100bc0 0x7fb6481963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb64db25700 1 -- 172.21.15.116:0/2485399895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb648100bc0 msgr2=0x7fb6481963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb64db25700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb648100bc0 0x7fb6481963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb64db25700 1 -- 172.21.15.116:0/2485399895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6380095f0 con 0x7fb648101dc0 2024-03-04T16:09:10.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.580+0000 7fb64db25700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb648101dc0 0x7fb648196920 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb63800b5c0 tx=0x7fb63800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:10.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.581+0000 7fb63f7fe700 1 -- 172.21.15.116:0/2485399895 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63801d070 con 0x7fb648101dc0 2024-03-04T16:09:10.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.581+0000 7fb63f7fe700 1 -- 172.21.15.116:0/2485399895 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb638022470 con 0x7fb648101dc0 2024-03-04T16:09:10.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.581+0000 7fb65058a700 1 -- 172.21.15.116:0/2485399895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64819bad0 con 0x7fb648101dc0 2024-03-04T16:09:10.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.581+0000 7fb63f7fe700 1 -- 172.21.15.116:0/2485399895 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63800f690 con 0x7fb648101dc0 2024-03-04T16:09:10.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.581+0000 7fb65058a700 1 -- 172.21.15.116:0/2485399895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64819c040 con 0x7fb648101dc0 2024-03-04T16:09:10.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.582+0000 7fb65058a700 1 -- 172.21.15.116:0/2485399895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6481097c0 con 0x7fb648101dc0 2024-03-04T16:09:10.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.582+0000 7fb63f7fe700 1 -- 172.21.15.116:0/2485399895 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fb638004b70 con 0x7fb648101dc0 2024-03-04T16:09:10.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.583+0000 7fb63f7fe700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb63406d8a0 0x7fb63406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:10.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.583+0000 7fb63f7fe700 1 -- 172.21.15.116:0/2485399895 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb63808ec90 con 0x7fb648101dc0 2024-03-04T16:09:10.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.583+0000 7fb64e326700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb63406d8a0 0x7fb63406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:10.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.583+0000 7fb64e326700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb63406d8a0 0x7fb63406fd60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb644006fd0 tx=0x7fb644008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:10.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.590+0000 7fb63f7fe700 1 -- 172.21.15.116:0/2485399895 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb63805c210 con 0x7fb648101dc0 2024-03-04T16:09:10.840 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:10 smithi152 podman[92602]: 2024-03-04 16:09:10.435466158 +0000 UTC m=+0.018206096 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:09:10.840 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:10 smithi152 podman[92602]: 2024-03-04 16:09:10.571023103 +0000 UTC m=+0.153763029 container create 6ab4b28827439fb27ceeb7167e6d795d92ee9ccb58dec2638dc4f0f5904bb044 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-03-04T16:09:10.840 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:10 smithi152 podman[92602]: 2024-03-04 16:09:10.840568432 +0000 UTC m=+0.423308371 container init 6ab4b28827439fb27ceeb7167e6d795d92ee9ccb58dec2638dc4f0f5904bb044 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-04T16:09:10.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.865+0000 7fb65058a700 1 -- 172.21.15.116:0/2485399895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb64819c6d0 con 0x7fb648101dc0 2024-03-04T16:09:10.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.866+0000 7fb63f7fe700 1 -- 172.21.15.116:0/2485399895 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fb638027330 con 0x7fb648101dc0 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:09:10.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:09:10.868 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:09:10.868 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:09:10.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:09:10.868 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:09:10.868 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6, 2024-03-04T16:09:10.868 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 7 2024-03-04T16:09:10.868 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:09:10.868 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:09:10.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 -- 172.21.15.116:0/2485399895 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb63406d8a0 msgr2=0x7fb63406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb63406d8a0 0x7fb63406fd60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb644006fd0 tx=0x7fb644008040 comp rx=0 tx=0).stop 2024-03-04T16:09:10.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 -- 172.21.15.116:0/2485399895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb648101dc0 msgr2=0x7fb648196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb648101dc0 0x7fb648196920 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb63800b5c0 tx=0x7fb63800be20 comp rx=0 tx=0).stop 2024-03-04T16:09:10.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 -- 172.21.15.116:0/2485399895 shutdown_connections 2024-03-04T16:09:10.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb63406d8a0 0x7fb63406fd60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb648100bc0 0x7fb6481963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 --2- 172.21.15.116:0/2485399895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb648101dc0 0x7fb648196920 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 -- 172.21.15.116:0/2485399895 >> 172.21.15.116:0/2485399895 conn(0x7fb6480fc140 msgr2=0x7fb648104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:10.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 -- 172.21.15.116:0/2485399895 shutdown_connections 2024-03-04T16:09:10.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.868+0000 7fb65058a700 1 -- 172.21.15.116:0/2485399895 wait complete. 2024-03-04T16:09:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.971+0000 7f6223340700 1 -- 172.21.15.116:0/1665420098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f621c101dc0 msgr2=0x7f621c102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.971+0000 7f6223340700 1 --2- 172.21.15.116:0/1665420098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f621c101dc0 0x7f621c102220 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6218009910 tx=0x7f6218009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.971+0000 7f6223340700 1 -- 172.21.15.116:0/1665420098 shutdown_connections 2024-03-04T16:09:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.971+0000 7f6223340700 1 --2- 172.21.15.116:0/1665420098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f621c101dc0 0x7f621c102220 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.971+0000 7f6223340700 1 --2- 172.21.15.116:0/1665420098 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f621c100bc0 0x7f621c100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.971+0000 7f6223340700 1 -- 172.21.15.116:0/1665420098 >> 172.21.15.116:0/1665420098 conn(0x7f621c0fc140 msgr2=0x7f621c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.971+0000 7f6223340700 1 -- 172.21.15.116:0/1665420098 shutdown_connections 2024-03-04T16:09:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.971+0000 7f6223340700 1 -- 172.21.15.116:0/1665420098 wait complete. 2024-03-04T16:09:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.972+0000 7f6223340700 1 Processor -- start 2024-03-04T16:09:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.972+0000 7f6223340700 1 -- start start 2024-03-04T16:09:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.972+0000 7f6223340700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f621c100bc0 0x7f621c196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.972+0000 7f6223340700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f621c101dc0 0x7f621c1968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.972+0000 7f6223340700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c196ed0 con 0x7f621c101dc0 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.972+0000 7f6223340700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c197010 con 0x7f621c100bc0 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.972+0000 7f62208db700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f621c101dc0 0x7f621c1968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.972+0000 7f62208db700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f621c101dc0 0x7f621c1968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54246/0 (socket says 172.21.15.116:54246) 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.972+0000 7f62208db700 1 -- 172.21.15.116:0/2541161631 learned_addr learned my addr 172.21.15.116:0/2541161631 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.972+0000 7f62210dc700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f621c100bc0 0x7f621c196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.973+0000 7f62208db700 1 -- 172.21.15.116:0/2541161631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f621c100bc0 msgr2=0x7f621c196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.973+0000 7f62208db700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f621c100bc0 0x7f621c196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.973+0000 7f62208db700 1 -- 172.21.15.116:0/2541161631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180095f0 con 0x7f621c101dc0 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.973+0000 7f62208db700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f621c101dc0 0x7f621c1968b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6218000c00 tx=0x7f621800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.973+0000 7f62127fc700 1 -- 172.21.15.116:0/2541161631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f621801d070 con 0x7f621c101dc0 2024-03-04T16:09:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.973+0000 7f62127fc700 1 -- 172.21.15.116:0/2541161631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6218005dc0 con 0x7f621c101dc0 2024-03-04T16:09:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.973+0000 7f6223340700 1 -- 172.21.15.116:0/2541161631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f621c19ba60 con 0x7f621c101dc0 2024-03-04T16:09:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.973+0000 7f62127fc700 1 -- 172.21.15.116:0/2541161631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62180226a0 con 0x7f621c101dc0 2024-03-04T16:09:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.973+0000 7f6223340700 1 -- 172.21.15.116:0/2541161631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f621c19bf50 con 0x7f621c101dc0 2024-03-04T16:09:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.974+0000 7f6223340700 1 -- 172.21.15.116:0/2541161631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621c109720 con 0x7f621c101dc0 2024-03-04T16:09:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.974+0000 7f62127fc700 1 -- 172.21.15.116:0/2541161631 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f62180038f0 con 0x7f621c101dc0 2024-03-04T16:09:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.975+0000 7f62127fc700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f620806d8a0 0x7f620806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.975+0000 7f62127fc700 1 -- 172.21.15.116:0/2541161631 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f621808ea20 con 0x7f621c101dc0 2024-03-04T16:09:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.975+0000 7f62210dc700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f620806d8a0 0x7f620806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.975+0000 7f62210dc700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f620806d8a0 0x7f620806fd60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f621c101c20 tx=0x7f620c009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:10.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:10.979+0000 7f62127fc700 1 -- 172.21.15.116:0/2541161631 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f621805bd50 con 0x7f621c101dc0 2024-03-04T16:09:11.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:11 smithi152 ceph-mon[82845]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:11.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:11 smithi152 ceph-mon[82845]: osdmap e67: 6 total, 5 up, 6 in 2024-03-04T16:09:11.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:11 smithi152 ceph-mon[82845]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:11.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:11 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2485399895' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:11.108 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:10 smithi152 podman[92602]: 2024-03-04 16:09:10.843759167 +0000 UTC m=+0.426499097 container start 6ab4b28827439fb27ceeb7167e6d795d92ee9ccb58dec2638dc4f0f5904bb044 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=) 2024-03-04T16:09:11.108 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:10 smithi152 podman[92602]: 2024-03-04 16:09:10.948852391 +0000 UTC m=+0.531592319 container attach 6ab4b28827439fb27ceeb7167e6d795d92ee9ccb58dec2638dc4f0f5904bb044 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-04T16:09:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.217+0000 7f6223340700 1 -- 172.21.15.116:0/2541161631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f621c109910 con 0x7f621c101dc0 2024-03-04T16:09:11.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:11 smithi116 ceph-mon[94164]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:11.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:11 smithi116 ceph-mon[94164]: osdmap e67: 6 total, 5 up, 6 in 2024-03-04T16:09:11.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:11 smithi116 ceph-mon[94164]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:11.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:11 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2485399895' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.218+0000 7f62127fc700 1 -- 172.21.15.116:0/2541161631 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1852 (secure 0 0 0) 0x7f6218003680 con 0x7f621c101dc0 2024-03-04T16:09:11.219 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:09:11.220 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:09:11.221 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:09:11.222 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:09:11.223 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:09:11.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 -- 172.21.15.116:0/2541161631 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f620806d8a0 msgr2=0x7f620806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:11.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f620806d8a0 0x7f620806fd60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f621c101c20 tx=0x7f620c009260 comp rx=0 tx=0).stop 2024-03-04T16:09:11.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 -- 172.21.15.116:0/2541161631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f621c101dc0 msgr2=0x7f621c1968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f621c101dc0 0x7f621c1968b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6218000c00 tx=0x7f621800f690 comp rx=0 tx=0).stop 2024-03-04T16:09:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 -- 172.21.15.116:0/2541161631 shutdown_connections 2024-03-04T16:09:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f620806d8a0 0x7f620806fd60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f621c100bc0 0x7f621c196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 --2- 172.21.15.116:0/2541161631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f621c101dc0 0x7f621c1968b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 -- 172.21.15.116:0/2541161631 >> 172.21.15.116:0/2541161631 conn(0x7f621c0fc140 msgr2=0x7f621c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 -- 172.21.15.116:0/2541161631 shutdown_connections 2024-03-04T16:09:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.220+0000 7f6223340700 1 -- 172.21.15.116:0/2541161631 wait complete. 2024-03-04T16:09:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:09:11.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.325+0000 7fbb806cb700 1 -- 172.21.15.116:0/3329275368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb780ffaf0 msgr2=0x7fbb780fff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:11.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.325+0000 7fbb806cb700 1 --2- 172.21.15.116:0/3329275368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb780ffaf0 0x7fbb780fff10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009910 tx=0x7fbb68009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:11.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.325+0000 7fbb806cb700 1 -- 172.21.15.116:0/3329275368 shutdown_connections 2024-03-04T16:09:11.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.325+0000 7fbb806cb700 1 --2- 172.21.15.116:0/3329275368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb78100cf0 0x7fbb78101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.325+0000 7fbb806cb700 1 --2- 172.21.15.116:0/3329275368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb780ffaf0 0x7fbb780fff10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.325+0000 7fbb806cb700 1 -- 172.21.15.116:0/3329275368 >> 172.21.15.116:0/3329275368 conn(0x7fbb780fb070 msgr2=0x7fbb780fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:11.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.325+0000 7fbb806cb700 1 -- 172.21.15.116:0/3329275368 shutdown_connections 2024-03-04T16:09:11.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.325+0000 7fbb806cb700 1 -- 172.21.15.116:0/3329275368 wait complete. 2024-03-04T16:09:11.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb806cb700 1 Processor -- start 2024-03-04T16:09:11.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb806cb700 1 -- start start 2024-03-04T16:09:11.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb806cb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb780ffaf0 0x7fbb781952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:11.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb806cb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb78100cf0 0x7fbb78195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:11.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb806cb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78195d90 con 0x7fbb780ffaf0 2024-03-04T16:09:11.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb806cb700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78195ed0 con 0x7fbb78100cf0 2024-03-04T16:09:11.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb7e467700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb780ffaf0 0x7fbb781952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:11.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb7e467700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb780ffaf0 0x7fbb781952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54260/0 (socket says 172.21.15.116:54260) 2024-03-04T16:09:11.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb7e467700 1 -- 172.21.15.116:0/1426728844 learned_addr learned my addr 172.21.15.116:0/1426728844 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb7dc66700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb78100cf0 0x7fbb78195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb7e467700 1 -- 172.21.15.116:0/1426728844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb78100cf0 msgr2=0x7fbb78195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb7e467700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb78100cf0 0x7fbb78195800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb7e467700 1 -- 172.21.15.116:0/1426728844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb680095f0 con 0x7fbb780ffaf0 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.326+0000 7fbb7e467700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb780ffaf0 0x7fbb781952c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800bda0 tx=0x7fbb6800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.327+0000 7fbb6f7fe700 1 -- 172.21.15.116:0/1426728844 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb6801d070 con 0x7fbb780ffaf0 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.327+0000 7fbb6f7fe700 1 -- 172.21.15.116:0/1426728844 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbb68004c50 con 0x7fbb780ffaf0 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.327+0000 7fbb6f7fe700 1 -- 172.21.15.116:0/1426728844 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb6800f690 con 0x7fbb780ffaf0 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.327+0000 7fbb806cb700 1 -- 172.21.15.116:0/1426728844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb78068d90 con 0x7fbb780ffaf0 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.327+0000 7fbb806cb700 1 -- 172.21.15.116:0/1426728844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb78069250 con 0x7fbb780ffaf0 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.328+0000 7fbb806cb700 1 -- 172.21.15.116:0/1426728844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb78108650 con 0x7fbb780ffaf0 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.328+0000 7fbb6f7fe700 1 -- 172.21.15.116:0/1426728844 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fbb68022a20 con 0x7fbb780ffaf0 2024-03-04T16:09:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.329+0000 7fbb6f7fe700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbb6406d8a0 0x7fbb6406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.329+0000 7fbb6f7fe700 1 -- 172.21.15.116:0/1426728844 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbb6808ee70 con 0x7fbb780ffaf0 2024-03-04T16:09:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.329+0000 7fbb7dc66700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbb6406d8a0 0x7fbb6406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.329+0000 7fbb7dc66700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbb6406d8a0 0x7fbb6406fd60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbb74005ea0 tx=0x7fbb74005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:11.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.333+0000 7fbb6f7fe700 1 -- 172.21.15.116:0/1426728844 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbb6805c3f0 con 0x7fbb780ffaf0 2024-03-04T16:09:11.477 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:11 smithi152 podman[92602]: 2024-03-04 16:09:11.142402007 +0000 UTC m=+0.725141935 container died 6ab4b28827439fb27ceeb7167e6d795d92ee9ccb58dec2638dc4f0f5904bb044 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222) 2024-03-04T16:09:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.548+0000 7fbb806cb700 1 -- 172.21.15.116:0/1426728844 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb78105640 con 0x7fbb6406d8a0 2024-03-04T16:09:11.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.549+0000 7fbb6f7fe700 1 -- 172.21.15.116:0/1426728844 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fbb78105640 con 0x7fbb6406d8a0 2024-03-04T16:09:11.552 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:09:11.552 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:09:11.552 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:09:11.552 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:09:11.552 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:09:11.552 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:09:11.552 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:09:11.552 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:09:11.552 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:09:11.552 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "9/23 daemons upgraded", 2024-03-04T16:09:11.553 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:09:11.553 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:09:11.553 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:09:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 -- 172.21.15.116:0/1426728844 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbb6406d8a0 msgr2=0x7fbb6406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbb6406d8a0 0x7fbb6406fd60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbb74005ea0 tx=0x7fbb74005e30 comp rx=0 tx=0).stop 2024-03-04T16:09:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 -- 172.21.15.116:0/1426728844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb780ffaf0 msgr2=0x7fbb781952c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb780ffaf0 0x7fbb781952c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800bda0 tx=0x7fbb6800be80 comp rx=0 tx=0).stop 2024-03-04T16:09:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 -- 172.21.15.116:0/1426728844 shutdown_connections 2024-03-04T16:09:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbb6406d8a0 0x7fbb6406fd60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb780ffaf0 0x7fbb781952c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 --2- 172.21.15.116:0/1426728844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb78100cf0 0x7fbb78195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 -- 172.21.15.116:0/1426728844 >> 172.21.15.116:0/1426728844 conn(0x7fbb780fb070 msgr2=0x7fbb78103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 -- 172.21.15.116:0/1426728844 shutdown_connections 2024-03-04T16:09:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.551+0000 7fbb806cb700 1 -- 172.21.15.116:0/1426728844 wait complete. 2024-03-04T16:09:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.654+0000 7fb438f8b700 1 -- 172.21.15.116:0/3390180000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb434101de0 msgr2=0x7fb434102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.654+0000 7fb438f8b700 1 --2- 172.21.15.116:0/3390180000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb434101de0 0x7fb434102240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb424009910 tx=0x7fb424009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.654+0000 7fb438f8b700 1 -- 172.21.15.116:0/3390180000 shutdown_connections 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.654+0000 7fb438f8b700 1 --2- 172.21.15.116:0/3390180000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb434101de0 0x7fb434102240 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.654+0000 7fb438f8b700 1 --2- 172.21.15.116:0/3390180000 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb434100be0 0x7fb434101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.654+0000 7fb438f8b700 1 -- 172.21.15.116:0/3390180000 >> 172.21.15.116:0/3390180000 conn(0x7fb4340fc140 msgr2=0x7fb4340fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.655+0000 7fb438f8b700 1 -- 172.21.15.116:0/3390180000 shutdown_connections 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.655+0000 7fb438f8b700 1 -- 172.21.15.116:0/3390180000 wait complete. 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.655+0000 7fb438f8b700 1 Processor -- start 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.655+0000 7fb438f8b700 1 -- start start 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.655+0000 7fb438f8b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb434100be0 0x7fb434074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.655+0000 7fb438f8b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb434101de0 0x7fb434073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.655+0000 7fb438f8b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340736c0 con 0x7fb434101de0 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.655+0000 7fb438f8b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb434073800 con 0x7fb434100be0 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb431d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb434101de0 0x7fb434073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb431d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb434101de0 0x7fb434073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54270/0 (socket says 172.21.15.116:54270) 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb431d9b700 1 -- 172.21.15.116:0/3581680495 learned_addr learned my addr 172.21.15.116:0/3581680495 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb43259c700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb434100be0 0x7fb434074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb431d9b700 1 -- 172.21.15.116:0/3581680495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb434100be0 msgr2=0x7fb434074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb431d9b700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb434100be0 0x7fb434074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb431d9b700 1 -- 172.21.15.116:0/3581680495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4240095f0 con 0x7fb434101de0 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb431d9b700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb434101de0 0x7fb434073180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb424006010 tx=0x7fb42400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb4237fe700 1 -- 172.21.15.116:0/3581680495 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb42401d070 con 0x7fb434101de0 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb4237fe700 1 -- 172.21.15.116:0/3581680495 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb42400f460 con 0x7fb434101de0 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb438f8b700 1 -- 172.21.15.116:0/3581680495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb434073a80 con 0x7fb434101de0 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb4237fe700 1 -- 172.21.15.116:0/3581680495 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb424021640 con 0x7fb434101de0 2024-03-04T16:09:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.656+0000 7fb438f8b700 1 -- 172.21.15.116:0/3581680495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb434073f70 con 0x7fb434101de0 2024-03-04T16:09:11.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.657+0000 7fb438f8b700 1 -- 172.21.15.116:0/3581680495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb434109740 con 0x7fb434101de0 2024-03-04T16:09:11.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.658+0000 7fb4237fe700 1 -- 172.21.15.116:0/3581680495 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fb424003a60 con 0x7fb434101de0 2024-03-04T16:09:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.658+0000 7fb4237fe700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb41c06d8a0 0x7fb41c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.658+0000 7fb43259c700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb41c06d8a0 0x7fb41c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.658+0000 7fb4237fe700 1 -- 172.21.15.116:0/3581680495 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb42408e700 con 0x7fb434101de0 2024-03-04T16:09:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.658+0000 7fb43259c700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb41c06d8a0 0x7fb41c06fd60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb434101c40 tx=0x7fb42800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:11.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.663+0000 7fb4237fe700 1 -- 172.21.15.116:0/3581680495 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb42405bab0 con 0x7fb434101de0 2024-03-04T16:09:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.933+0000 7fb438f8b700 1 -- 172.21.15.116:0/3581680495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb434109930 con 0x7fb434101de0 2024-03-04T16:09:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.933+0000 7fb4237fe700 1 -- 172.21.15.116:0/3581680495 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+281 (secure 0 0 0) 0x7fb42405b640 con 0x7fb434101de0 2024-03-04T16:09:11.934 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 1 pg peering 2024-03-04T16:09:11.934 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-04T16:09:11.934 INFO:teuthology.orchestra.run.smithi116.stdout: osd.3 (root=default,host=smithi152) is down 2024-03-04T16:09:11.934 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering 2024-03-04T16:09:11.934 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.d is stuck peering for 2m, current state peering, last acting [2,4] 2024-03-04T16:09:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 -- 172.21.15.116:0/3581680495 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb41c06d8a0 msgr2=0x7fb41c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb41c06d8a0 0x7fb41c06fd60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb434101c40 tx=0x7fb42800b410 comp rx=0 tx=0).stop 2024-03-04T16:09:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 -- 172.21.15.116:0/3581680495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb434101de0 msgr2=0x7fb434073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb434101de0 0x7fb434073180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb424006010 tx=0x7fb42400bd70 comp rx=0 tx=0).stop 2024-03-04T16:09:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 -- 172.21.15.116:0/3581680495 shutdown_connections 2024-03-04T16:09:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb41c06d8a0 0x7fb41c06fd60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb434100be0 0x7fb434074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 --2- 172.21.15.116:0/3581680495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb434101de0 0x7fb434073180 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 -- 172.21.15.116:0/3581680495 >> 172.21.15.116:0/3581680495 conn(0x7fb4340fc140 msgr2=0x7fb434105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 -- 172.21.15.116:0/3581680495 shutdown_connections 2024-03-04T16:09:11.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:11.935+0000 7fb438f8b700 1 -- 172.21.15.116:0/3581680495 wait complete. 2024-03-04T16:09:12.078 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:11 smithi152 podman[92628]: 2024-03-04 16:09:11.761407848 +0000 UTC m=+0.613838927 container remove 6ab4b28827439fb27ceeb7167e6d795d92ee9ccb58dec2638dc4f0f5904bb044 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-04T16:09:12.078 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:11 smithi152 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-04T16:09:12.078 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:11 smithi152 systemd[1]: Stopped Ceph osd.3 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:09:12.078 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:11 smithi152 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.3.service: Consumed 1min 19.184s CPU time. 2024-03-04T16:09:12.078 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:11 smithi152 systemd[1]: Starting Ceph osd.3 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:09:12.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:12 smithi116 ceph-mon[94164]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:12.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:12 smithi116 ceph-mon[94164]: pgmap v119: 65 pgs: 4 peering, 15 stale+active+clean, 46 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:12.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:12 smithi116 ceph-mon[94164]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:09:12.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:12 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2541161631' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:09:12.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:12 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3581680495' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:09:12.329 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-mon[82845]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:12.329 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-mon[82845]: pgmap v119: 65 pgs: 4 peering, 15 stale+active+clean, 46 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:12.329 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-mon[82845]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:09:12.329 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2541161631' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:09:12.329 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3581680495' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:09:12.329 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 podman[92732]: 2024-03-04 16:09:12.114408582 +0000 UTC m=+0.017560896 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:09:12.329 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 podman[92732]: 2024-03-04 16:09:12.221290048 +0000 UTC m=+0.124442381 container create 24423c3552ca6cb83094b9e401484dc29ee38c7254889d33312d16eca11a1031 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-03-04T16:09:12.607 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 podman[92732]: 2024-03-04 16:09:12.44041884 +0000 UTC m=+0.343571146 container init 24423c3552ca6cb83094b9e401484dc29ee38c7254889d33312d16eca11a1031 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-04T16:09:12.607 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 podman[92732]: 2024-03-04 16:09:12.443579464 +0000 UTC m=+0.346731764 container start 24423c3552ca6cb83094b9e401484dc29ee38c7254889d33312d16eca11a1031 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:09:12.607 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 podman[92732]: 2024-03-04 16:09:12.540445734 +0000 UTC m=+0.443598039 container attach 24423c3552ca6cb83094b9e401484dc29ee38c7254889d33312d16eca11a1031 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:09:13.268 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:13 smithi152 ceph-mon[82845]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:13.269 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate[92747]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T16:09:13.269 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate[92747]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-04T16:09:13.269 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 bash[92732]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T16:09:13.269 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 bash[92732]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-04T16:09:13.269 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 bash[92732]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-04T16:09:13.269 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 bash[92732]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-04T16:09:13.270 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 bash[92732]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-03-04T16:09:13.270 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 bash[92732]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T16:09:13.270 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 bash[92732]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-04T16:09:13.270 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate[92747]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-04T16:09:13.270 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate[92747]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-04T16:09:13.270 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate[92747]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-03-04T16:09:13.270 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate[92747]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T16:09:13.270 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate[92747]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-04T16:09:13.271 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:12 smithi152 podman[92732]: 2024-03-04 16:09:12.917923956 +0000 UTC m=+0.821076240 container died 24423c3552ca6cb83094b9e401484dc29ee38c7254889d33312d16eca11a1031 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:09:13.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:13 smithi116 ceph-mon[94164]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:13.857 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:13 smithi152 podman[92883]: 2024-03-04 16:09:13.578473276 +0000 UTC m=+0.655643551 container remove 24423c3552ca6cb83094b9e401484dc29ee38c7254889d33312d16eca11a1031 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:09:13.858 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:13 smithi152 podman[92945]: 2024-03-04 16:09:13.695389997 +0000 UTC m=+0.018179917 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:09:13.858 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:13 smithi152 podman[92945]: 2024-03-04 16:09:13.812896979 +0000 UTC m=+0.135686932 container create 6b01e97c8f9b3597992f79c8e52df9cd368618e231cf196a74e53d5c11e464bf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True) 2024-03-04T16:09:14.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:14 smithi152 ceph-mon[82845]: pgmap v120: 65 pgs: 3 active+undersized+degraded, 4 peering, 12 stale+active+clean, 4 active+undersized, 42 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 7/231 objects degraded (3.030%) 2024-03-04T16:09:14.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:14 smithi152 ceph-mon[82845]: Health check failed: Degraded data redundancy: 7/231 objects degraded (3.030%), 3 pgs degraded (PG_DEGRADED) 2024-03-04T16:09:14.358 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:14 smithi152 podman[92945]: 2024-03-04 16:09:14.048559992 +0000 UTC m=+0.371349899 container init 6b01e97c8f9b3597992f79c8e52df9cd368618e231cf196a74e53d5c11e464bf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:09:14.358 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:14 smithi152 podman[92945]: 2024-03-04 16:09:14.0518483 +0000 UTC m=+0.374638217 container start 6b01e97c8f9b3597992f79c8e52df9cd368618e231cf196a74e53d5c11e464bf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:09:14.358 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:14 smithi152 ceph-osd[92964]: -- 172.21.15.152:0/3734664574 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56384f3f9680 con 0x56384f3c2c00 2024-03-04T16:09:14.358 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:14 smithi152 bash[92945]: 6b01e97c8f9b3597992f79c8e52df9cd368618e231cf196a74e53d5c11e464bf 2024-03-04T16:09:14.358 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:14 smithi152 systemd[1]: Started Ceph osd.3 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:09:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:14 smithi116 ceph-mon[94164]: pgmap v120: 65 pgs: 3 active+undersized+degraded, 4 peering, 12 stale+active+clean, 4 active+undersized, 42 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 7/231 objects degraded (3.030%) 2024-03-04T16:09:14.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:14 smithi116 ceph-mon[94164]: Health check failed: Degraded data redundancy: 7/231 objects degraded (3.030%), 3 pgs degraded (PG_DEGRADED) 2024-03-04T16:09:14.857 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:14 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[92960]: 2024-03-04T16:09:14.599+0000 7f68fb2747c0 -1 Falling back to public interface 2024-03-04T16:09:15.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:15.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:15.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:16 smithi116 ceph-mon[94164]: pgmap v121: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 55/231 objects degraded (23.810%) 2024-03-04T16:09:16.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:16 smithi116 ceph-mon[94164]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-04T16:09:16.529 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:16 smithi152 ceph-mon[82845]: pgmap v121: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 55/231 objects degraded (23.810%) 2024-03-04T16:09:16.530 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:16 smithi152 ceph-mon[82845]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-04T16:09:18.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:18 smithi116 ceph-mon[94164]: pgmap v122: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 55/231 objects degraded (23.810%) 2024-03-04T16:09:18.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:18 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:18.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:18 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:18.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:18 smithi152 ceph-mon[82845]: pgmap v122: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 55/231 objects degraded (23.810%) 2024-03-04T16:09:18.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:18 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:18.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:18 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:19.322 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:19 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[92960]: 2024-03-04T16:09:19.233+0000 7f68fb2747c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-04T16:09:19.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:19 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:19.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:19 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:19.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:19 smithi152 ceph-mon[82845]: Health check update: Degraded data redundancy: 55/231 objects degraded (23.810%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:09:19.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:19 smithi152 ceph-mon[82845]: pgmap v123: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 55/231 objects degraded (23.810%) 2024-03-04T16:09:19.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:19 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:19.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:19 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:19.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:19 smithi116 ceph-mon[94164]: Health check update: Degraded data redundancy: 55/231 objects degraded (23.810%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:09:19.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:19 smithi116 ceph-mon[94164]: pgmap v123: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 212 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 55/231 objects degraded (23.810%) 2024-03-04T16:09:20.089 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:19 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[92960]: 2024-03-04T16:09:19.789+0000 7f68fb2747c0 -1 osd.3 65 log_to_monitors true 2024-03-04T16:09:20.672 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:20 smithi152 ceph-mon[82845]: from='osd.3 [v2:172.21.15.152:6800/3040811013,v1:172.21.15.152:6801/3040811013]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:09:20.672 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:20 smithi152 ceph-mon[82845]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:09:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:20 smithi116 ceph-mon[94164]: from='osd.3 [v2:172.21.15.152:6800/3040811013,v1:172.21.15.152:6801/3040811013]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:09:20.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:20 smithi116 ceph-mon[94164]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:09:21.607 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:09:21 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[92960]: 2024-03-04T16:09:21.288+0000 7f68e86c4700 -1 osd.3 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:09:21.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:21 smithi152 ceph-mon[82845]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T16:09:21.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:21 smithi152 ceph-mon[82845]: from='osd.3 [v2:172.21.15.152:6800/3040811013,v1:172.21.15.152:6801/3040811013]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:09:21.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:21 smithi152 ceph-mon[82845]: osdmap e68: 6 total, 5 up, 6 in 2024-03-04T16:09:21.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:21 smithi152 ceph-mon[82845]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:09:21.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:21 smithi152 ceph-mon[82845]: pgmap v125: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 55/231 objects degraded (23.810%) 2024-03-04T16:09:21.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:21 smithi152 ceph-mon[82845]: from='osd.3 ' entity='osd.3' 2024-03-04T16:09:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:21 smithi116 ceph-mon[94164]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T16:09:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:21 smithi116 ceph-mon[94164]: from='osd.3 [v2:172.21.15.152:6800/3040811013,v1:172.21.15.152:6801/3040811013]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:09:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:21 smithi116 ceph-mon[94164]: osdmap e68: 6 total, 5 up, 6 in 2024-03-04T16:09:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:21 smithi116 ceph-mon[94164]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:09:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:21 smithi116 ceph-mon[94164]: pgmap v125: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 55/231 objects degraded (23.810%) 2024-03-04T16:09:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:21 smithi116 ceph-mon[94164]: from='osd.3 ' entity='osd.3' 2024-03-04T16:09:23.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:23.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:23.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: osd.3 [v2:172.21.15.152:6800/3040811013,v1:172.21.15.152:6801/3040811013] boot 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: osdmap e69: 6 total, 6 up, 6 in 2024-03-04T16:09:23.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:09:23.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-03-04T16:09:23.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:09:23.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: osd.3 [v2:172.21.15.152:6800/3040811013,v1:172.21.15.152:6801/3040811013] boot 2024-03-04T16:09:23.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: osdmap e69: 6 total, 6 up, 6 in 2024-03-04T16:09:23.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:09:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:24 smithi152 ceph-mon[82845]: pgmap v127: 65 pgs: 4 peering, 17 active+undersized+degraded, 17 active+undersized, 27 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 52/231 objects degraded (22.511%) 2024-03-04T16:09:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:24 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:24.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:24 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:24.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:24 smithi152 ceph-mon[82845]: osdmap e70: 6 total, 6 up, 6 in 2024-03-04T16:09:24.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:24 smithi152 ceph-mon[82845]: Health check update: Degraded data redundancy: 52/231 objects degraded (22.511%), 17 pgs degraded (PG_DEGRADED) 2024-03-04T16:09:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:24 smithi116 ceph-mon[94164]: pgmap v127: 65 pgs: 4 peering, 17 active+undersized+degraded, 17 active+undersized, 27 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 52/231 objects degraded (22.511%) 2024-03-04T16:09:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:24 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:24 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:24 smithi116 ceph-mon[94164]: osdmap e70: 6 total, 6 up, 6 in 2024-03-04T16:09:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:24 smithi116 ceph-mon[94164]: Health check update: Degraded data redundancy: 52/231 objects degraded (22.511%), 17 pgs degraded (PG_DEGRADED) 2024-03-04T16:09:26.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:26 smithi152 ceph-mon[82845]: pgmap v129: 65 pgs: 5 activating, 6 peering, 9 active+undersized+degraded, 7 active+undersized, 38 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26/231 objects degraded (11.255%) 2024-03-04T16:09:26.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:26 smithi116 ceph-mon[94164]: pgmap v129: 65 pgs: 5 activating, 6 peering, 9 active+undersized+degraded, 7 active+undersized, 38 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26/231 objects degraded (11.255%) 2024-03-04T16:09:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:27 smithi116 ceph-mon[94164]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/231 objects degraded (11.255%), 9 pgs degraded) 2024-03-04T16:09:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:27 smithi116 ceph-mon[94164]: Cluster is now healthy 2024-03-04T16:09:27.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:27 smithi152 ceph-mon[82845]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/231 objects degraded (11.255%), 9 pgs degraded) 2024-03-04T16:09:27.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:27 smithi152 ceph-mon[82845]: Cluster is now healthy 2024-03-04T16:09:28.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:28 smithi116 ceph-mon[94164]: pgmap v130: 65 pgs: 5 activating, 6 peering, 54 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:28.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:28 smithi152 ceph-mon[82845]: pgmap v130: 65 pgs: 5 activating, 6 peering, 54 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:30.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:30 smithi152 ceph-mon[82845]: pgmap v131: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:30.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:30 smithi116 ceph-mon[94164]: pgmap v131: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:32.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:32 smithi116 ceph-mon[94164]: pgmap v132: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:32 smithi152 ceph-mon[82845]: pgmap v132: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:34.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:34 smithi116 ceph-mon[94164]: pgmap v133: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:34.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:34 smithi152 ceph-mon[82845]: pgmap v133: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:36.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:36 smithi116 ceph-mon[94164]: pgmap v134: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:36.522 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:36 smithi152 ceph-mon[82845]: pgmap v134: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:38 smithi116 ceph-mon[94164]: pgmap v135: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:09:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:38 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:09:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:38 smithi116 ceph-mon[94164]: Upgrade: osd.4 is safe to restart 2024-03-04T16:09:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:38 smithi152 ceph-mon[82845]: pgmap v135: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:09:38.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:38 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:09:38.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:38 smithi152 ceph-mon[82845]: Upgrade: osd.4 is safe to restart 2024-03-04T16:09:38.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:38.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:38.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:40.083 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:39 smithi152 ceph-mon[82845]: Upgrade: Updating osd.4 2024-03-04T16:09:40.083 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:40.083 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:09:40.083 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:40.083 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:39 smithi152 ceph-mon[82845]: Deploying daemon osd.4 on smithi152 2024-03-04T16:09:40.083 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:39 smithi152 ceph-mon[82845]: pgmap v136: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:40.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:39 smithi116 ceph-mon[94164]: Upgrade: Updating osd.4 2024-03-04T16:09:40.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:40.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:09:40.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:40.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:39 smithi116 ceph-mon[94164]: Deploying daemon osd.4 on smithi152 2024-03-04T16:09:40.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:39 smithi116 ceph-mon[94164]: pgmap v136: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:42.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.051+0000 7f1137f9a700 1 -- 172.21.15.116:0/1693525676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11300ffdb0 msgr2=0x7f1130100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.051+0000 7f1137f9a700 1 --2- 172.21.15.116:0/1693525676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11300ffdb0 0x7f1130100230 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1124009910 tx=0x7f1124009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:42.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.052+0000 7f1137f9a700 1 -- 172.21.15.116:0/1693525676 shutdown_connections 2024-03-04T16:09:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.052+0000 7f1137f9a700 1 --2- 172.21.15.116:0/1693525676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11300ffdb0 0x7f1130100230 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.052+0000 7f1137f9a700 1 --2- 172.21.15.116:0/1693525676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11300ff450 0x7f11300ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.052+0000 7f1137f9a700 1 -- 172.21.15.116:0/1693525676 >> 172.21.15.116:0/1693525676 conn(0x7f11300fb050 msgr2=0x7f11300fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.052+0000 7f1137f9a700 1 -- 172.21.15.116:0/1693525676 shutdown_connections 2024-03-04T16:09:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.052+0000 7f1137f9a700 1 -- 172.21.15.116:0/1693525676 wait complete. 2024-03-04T16:09:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.052+0000 7f1137f9a700 1 Processor -- start 2024-03-04T16:09:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.052+0000 7f1137f9a700 1 -- start start 2024-03-04T16:09:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1137f9a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11300ff450 0x7f1130198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1137f9a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11300ffdb0 0x7f1130198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1137f9a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11301990d0 con 0x7f11300ffdb0 2024-03-04T16:09:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1137f9a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1130199210 con 0x7f11300ff450 2024-03-04T16:09:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1135535700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11300ffdb0 0x7f1130198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1135535700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11300ffdb0 0x7f1130198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42152/0 (socket says 172.21.15.116:42152) 2024-03-04T16:09:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1135535700 1 -- 172.21.15.116:0/1702365014 learned_addr learned my addr 172.21.15.116:0/1702365014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1135d36700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11300ff450 0x7f1130198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1135535700 1 -- 172.21.15.116:0/1702365014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11300ff450 msgr2=0x7f1130198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1135535700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11300ff450 0x7f1130198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1135535700 1 -- 172.21.15.116:0/1702365014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11240095f0 con 0x7f11300ffdb0 2024-03-04T16:09:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.053+0000 7f1135535700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11300ffdb0 0x7f1130198ab0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f112400bd70 tx=0x7f112400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.054+0000 7f1122ffd700 1 -- 172.21.15.116:0/1702365014 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f112401d070 con 0x7f11300ffdb0 2024-03-04T16:09:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.054+0000 7f1122ffd700 1 -- 172.21.15.116:0/1702365014 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1124005dc0 con 0x7f11300ffdb0 2024-03-04T16:09:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.054+0000 7f1122ffd700 1 -- 172.21.15.116:0/1702365014 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1124021410 con 0x7f11300ffdb0 2024-03-04T16:09:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.054+0000 7f1137f9a700 1 -- 172.21.15.116:0/1702365014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113019dc60 con 0x7f11300ffdb0 2024-03-04T16:09:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.054+0000 7f1137f9a700 1 -- 172.21.15.116:0/1702365014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113019e150 con 0x7f11300ffdb0 2024-03-04T16:09:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.055+0000 7f1137f9a700 1 -- 172.21.15.116:0/1702365014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113010b920 con 0x7f11300ffdb0 2024-03-04T16:09:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.055+0000 7f1122ffd700 1 -- 172.21.15.116:0/1702365014 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f112400f7d0 con 0x7f11300ffdb0 2024-03-04T16:09:42.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.056+0000 7f1122ffd700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f111c06d8a0 0x7f111c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:42.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.056+0000 7f1122ffd700 1 -- 172.21.15.116:0/1702365014 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f112408e730 con 0x7f11300ffdb0 2024-03-04T16:09:42.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.056+0000 7f1135d36700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f111c06d8a0 0x7f111c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:42.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.056+0000 7f1135d36700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f111c06d8a0 0x7f111c06fd60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f112c007c60 tx=0x7f112c0073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:42.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.060+0000 7f1122ffd700 1 -- 172.21.15.116:0/1702365014 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f112405ba00 con 0x7f11300ffdb0 2024-03-04T16:09:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.274+0000 7f1137f9a700 1 -- 172.21.15.116:0/1702365014 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f11301091a0 con 0x7f111c06d8a0 2024-03-04T16:09:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.275+0000 7f1122ffd700 1 -- 172.21.15.116:0/1702365014 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f11301091a0 con 0x7f111c06d8a0 2024-03-04T16:09:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.277+0000 7f1137f9a700 1 -- 172.21.15.116:0/1702365014 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f111c06d8a0 msgr2=0x7f111c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.277+0000 7f1137f9a700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f111c06d8a0 0x7f111c06fd60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f112c007c60 tx=0x7f112c0073d0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.277+0000 7f1137f9a700 1 -- 172.21.15.116:0/1702365014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11300ffdb0 msgr2=0x7f1130198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.277+0000 7f1137f9a700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11300ffdb0 0x7f1130198ab0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f112400bd70 tx=0x7f112400bda0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.278+0000 7f1137f9a700 1 -- 172.21.15.116:0/1702365014 shutdown_connections 2024-03-04T16:09:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.278+0000 7f1137f9a700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f111c06d8a0 0x7f111c06fd60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.278+0000 7f1137f9a700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11300ff450 0x7f1130198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.278+0000 7f1137f9a700 1 --2- 172.21.15.116:0/1702365014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11300ffdb0 0x7f1130198ab0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.278+0000 7f1137f9a700 1 -- 172.21.15.116:0/1702365014 >> 172.21.15.116:0/1702365014 conn(0x7f11300fb050 msgr2=0x7f1130107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.278+0000 7f1137f9a700 1 -- 172.21.15.116:0/1702365014 shutdown_connections 2024-03-04T16:09:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.278+0000 7f1137f9a700 1 -- 172.21.15.116:0/1702365014 wait complete. 2024-03-04T16:09:42.289 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:09:42.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:42 smithi152 ceph-mon[82845]: pgmap v137: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:42.358 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:42 smithi152 systemd[1]: Stopping Ceph osd.4 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:09:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.383+0000 7f9926c2c700 1 -- 172.21.15.116:0/2887700568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9920100cc0 msgr2=0x7f9920101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.383+0000 7f9926c2c700 1 --2- 172.21.15.116:0/2887700568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9920100cc0 0x7f9920101120 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f990c009910 tx=0x7f990c009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.383+0000 7f9926c2c700 1 -- 172.21.15.116:0/2887700568 shutdown_connections 2024-03-04T16:09:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.383+0000 7f9926c2c700 1 --2- 172.21.15.116:0/2887700568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9920100cc0 0x7f9920101120 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.383+0000 7f9926c2c700 1 --2- 172.21.15.116:0/2887700568 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99200ffac0 0x7f99200ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.383+0000 7f9926c2c700 1 -- 172.21.15.116:0/2887700568 >> 172.21.15.116:0/2887700568 conn(0x7f99200fb060 msgr2=0x7f99200fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:42.385 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:42 smithi116 ceph-mon[94164]: pgmap v137: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.383+0000 7f9926c2c700 1 -- 172.21.15.116:0/2887700568 shutdown_connections 2024-03-04T16:09:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.383+0000 7f9926c2c700 1 -- 172.21.15.116:0/2887700568 wait complete. 2024-03-04T16:09:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.384+0000 7f9926c2c700 1 Processor -- start 2024-03-04T16:09:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.384+0000 7f9926c2c700 1 -- start start 2024-03-04T16:09:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.384+0000 7f9926c2c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99200ffac0 0x7f9920195270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.384+0000 7f9926c2c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9920100cc0 0x7f99201957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.384+0000 7f9926c2c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9920195dd0 con 0x7f9920100cc0 2024-03-04T16:09:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.384+0000 7f9926c2c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9920195f10 con 0x7f99200ffac0 2024-03-04T16:09:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.384+0000 7f991ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9920100cc0 0x7f99201957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.384+0000 7f991ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9920100cc0 0x7f99201957b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42162/0 (socket says 172.21.15.116:42162) 2024-03-04T16:09:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.384+0000 7f991ffff700 1 -- 172.21.15.116:0/3763122842 learned_addr learned my addr 172.21.15.116:0/3763122842 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.385+0000 7f99249c8700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99200ffac0 0x7f9920195270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.385+0000 7f991ffff700 1 -- 172.21.15.116:0/3763122842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99200ffac0 msgr2=0x7f9920195270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.385+0000 7f991ffff700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99200ffac0 0x7f9920195270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.385+0000 7f991ffff700 1 -- 172.21.15.116:0/3763122842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c0095f0 con 0x7f9920100cc0 2024-03-04T16:09:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.385+0000 7f991ffff700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9920100cc0 0x7f99201957b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f990c006010 tx=0x7f990c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.385+0000 7f991dffb700 1 -- 172.21.15.116:0/3763122842 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f990c01d070 con 0x7f9920100cc0 2024-03-04T16:09:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.385+0000 7f991dffb700 1 -- 172.21.15.116:0/3763122842 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f990c00f460 con 0x7f9920100cc0 2024-03-04T16:09:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.385+0000 7f991dffb700 1 -- 172.21.15.116:0/3763122842 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f990c017630 con 0x7f9920100cc0 2024-03-04T16:09:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.385+0000 7f9926c2c700 1 -- 172.21.15.116:0/3763122842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992019a960 con 0x7f9920100cc0 2024-03-04T16:09:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.385+0000 7f9926c2c700 1 -- 172.21.15.116:0/3763122842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992019ad90 con 0x7f9920100cc0 2024-03-04T16:09:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.386+0000 7f9926c2c700 1 -- 172.21.15.116:0/3763122842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9920108620 con 0x7f9920100cc0 2024-03-04T16:09:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.386+0000 7f991dffb700 1 -- 172.21.15.116:0/3763122842 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f990c003a60 con 0x7f9920100cc0 2024-03-04T16:09:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.387+0000 7f991dffb700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f991006d8a0 0x7f991006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.387+0000 7f991dffb700 1 -- 172.21.15.116:0/3763122842 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f990c08ea10 con 0x7f9920100cc0 2024-03-04T16:09:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.387+0000 7f99249c8700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f991006d8a0 0x7f991006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.388+0000 7f99249c8700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f991006d8a0 0x7f991006fd60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9914005950 tx=0x7f99140058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.391+0000 7f991dffb700 1 -- 172.21.15.116:0/3763122842 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f990c05bd20 con 0x7f9920100cc0 2024-03-04T16:09:42.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.605+0000 7f9926c2c700 1 -- 172.21.15.116:0/3763122842 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9920105610 con 0x7f991006d8a0 2024-03-04T16:09:42.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.606+0000 7f991dffb700 1 -- 172.21.15.116:0/3763122842 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f9920105610 con 0x7f991006d8a0 2024-03-04T16:09:42.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 -- 172.21.15.116:0/3763122842 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f991006d8a0 msgr2=0x7f991006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f991006d8a0 0x7f991006fd60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9914005950 tx=0x7f99140058e0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 -- 172.21.15.116:0/3763122842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9920100cc0 msgr2=0x7f99201957b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9920100cc0 0x7f99201957b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f990c006010 tx=0x7f990c00be20 comp rx=0 tx=0).stop 2024-03-04T16:09:42.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 -- 172.21.15.116:0/3763122842 shutdown_connections 2024-03-04T16:09:42.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f991006d8a0 0x7f991006fd60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99200ffac0 0x7f9920195270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 --2- 172.21.15.116:0/3763122842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9920100cc0 0x7f99201957b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 -- 172.21.15.116:0/3763122842 >> 172.21.15.116:0/3763122842 conn(0x7f99200fb060 msgr2=0x7f9920103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:42.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 -- 172.21.15.116:0/3763122842 shutdown_connections 2024-03-04T16:09:42.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.608+0000 7f9926c2c700 1 -- 172.21.15.116:0/3763122842 wait complete. 2024-03-04T16:09:42.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.716+0000 7f6ea924c700 1 -- 172.21.15.116:0/2201305422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea4100bc0 msgr2=0x7f6ea4100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.716+0000 7f6ea924c700 1 --2- 172.21.15.116:0/2201305422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea4100bc0 0x7f6ea4100fe0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009910 tx=0x7f6e94009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:42.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.716+0000 7f6ea924c700 1 -- 172.21.15.116:0/2201305422 shutdown_connections 2024-03-04T16:09:42.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.716+0000 7f6ea924c700 1 --2- 172.21.15.116:0/2201305422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ea4101dc0 0x7f6ea4102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.716+0000 7f6ea924c700 1 --2- 172.21.15.116:0/2201305422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea4100bc0 0x7f6ea4100fe0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.716+0000 7f6ea924c700 1 -- 172.21.15.116:0/2201305422 >> 172.21.15.116:0/2201305422 conn(0x7f6ea40fc140 msgr2=0x7f6ea40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:42.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.716+0000 7f6ea924c700 1 -- 172.21.15.116:0/2201305422 shutdown_connections 2024-03-04T16:09:42.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.716+0000 7f6ea924c700 1 -- 172.21.15.116:0/2201305422 wait complete. 2024-03-04T16:09:42.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea924c700 1 Processor -- start 2024-03-04T16:09:42.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea924c700 1 -- start start 2024-03-04T16:09:42.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea924c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ea4100bc0 0x7f6ea4196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:42.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea924c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea4101dc0 0x7f6ea41968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:42.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea924c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4196ef0 con 0x7f6ea4101dc0 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea924c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4197030 con 0x7f6ea4100bc0 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea4101dc0 0x7f6ea41968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea4101dc0 0x7f6ea41968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42182/0 (socket says 172.21.15.116:42182) 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea259c700 1 -- 172.21.15.116:0/2804406954 learned_addr learned my addr 172.21.15.116:0/2804406954 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea2d9d700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ea4100bc0 0x7f6ea4196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea259c700 1 -- 172.21.15.116:0/2804406954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ea4100bc0 msgr2=0x7f6ea4196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea259c700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ea4100bc0 0x7f6ea4196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.717+0000 7f6ea259c700 1 -- 172.21.15.116:0/2804406954 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e940095f0 con 0x7f6ea4101dc0 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.718+0000 7f6ea259c700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea4101dc0 0x7f6ea41968d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800d8d0 tx=0x7f6e9800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.718+0000 7f6e8bfff700 1 -- 172.21.15.116:0/2804406954 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e980098a0 con 0x7f6ea4101dc0 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.718+0000 7f6e8bfff700 1 -- 172.21.15.116:0/2804406954 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6e98010460 con 0x7f6ea4101dc0 2024-03-04T16:09:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.718+0000 7f6ea924c700 1 -- 172.21.15.116:0/2804406954 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea419bae0 con 0x7f6ea4101dc0 2024-03-04T16:09:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.718+0000 7f6e8bfff700 1 -- 172.21.15.116:0/2804406954 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e9800f5f0 con 0x7f6ea4101dc0 2024-03-04T16:09:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.718+0000 7f6ea924c700 1 -- 172.21.15.116:0/2804406954 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea419c030 con 0x7f6ea4101dc0 2024-03-04T16:09:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.719+0000 7f6ea924c700 1 -- 172.21.15.116:0/2804406954 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea4109720 con 0x7f6ea4101dc0 2024-03-04T16:09:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.719+0000 7f6e8bfff700 1 -- 172.21.15.116:0/2804406954 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f6e98009a00 con 0x7f6ea4101dc0 2024-03-04T16:09:42.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.720+0000 7f6e8bfff700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e8c06d8a0 0x7f6e8c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:42.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.720+0000 7f6e8bfff700 1 -- 172.21.15.116:0/2804406954 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6e9808d360 con 0x7f6ea4101dc0 2024-03-04T16:09:42.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.720+0000 7f6ea2d9d700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e8c06d8a0 0x7f6e8c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:42.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.720+0000 7f6ea2d9d700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e8c06d8a0 0x7f6e8c06fd60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6e94006010 tx=0x7f6e94005ec0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.725+0000 7f6e8bfff700 1 -- 172.21.15.116:0/2804406954 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6e9805a690 con 0x7f6ea4101dc0 2024-03-04T16:09:42.857 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:42 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[49800]: 2024-03-04T16:09:42.362+0000 7f648de15700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:09:42.857 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:42 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[49800]: 2024-03-04T16:09:42.362+0000 7f648de15700 -1 osd.4 70 *** Got signal Terminated *** 2024-03-04T16:09:42.858 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:42 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[49800]: 2024-03-04T16:09:42.362+0000 7f648de15700 -1 osd.4 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:09:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.927+0000 7f6ea924c700 1 -- 172.21.15.116:0/2804406954 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ea4106710 con 0x7f6e8c06d8a0 2024-03-04T16:09:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.938+0000 7f6e8bfff700 1 -- 172.21.15.116:0/2804406954 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f6ea4106710 con 0x7f6e8c06d8a0 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (12m) 59s ago 14m 25.4M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (14m) 59s ago 14m 8284k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (13m) 25s ago 13m 8036k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (2m) 59s ago 14m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (2m) 25s ago 13m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (12m) 59s ago 13m 87.8M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (8m) 59s ago 8m 19.9M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (8m) 59s ago 8m 16.1M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (8m) 25s ago 8m 17.6M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:09:42.939 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (8m) 25s ago 8m 143M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (5m) 59s ago 15m 503M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (4m) 25s ago 13m 450M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (3m) 59s ago 15m 46.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (3m) 25s ago 13m 34.3M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (14m) 59s ago 14m 15.4M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (13m) 25s ago 13m 17.1M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (2m) 59s ago 12m 87.1M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (101s) 59s ago 11m 101M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (65s) 59s ago 11m 12.7M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (29s) 25s ago 11m 15.0M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6b01e97c8f9b 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (10m) 25s ago 10m 237M 2240M 18.2.1 5be31c24972a c40b387afb37 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (10m) 25s ago 10m 227M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:09:42.940 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (4m) 59s ago 13m 63.0M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:09:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 -- 172.21.15.116:0/2804406954 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e8c06d8a0 msgr2=0x7f6e8c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e8c06d8a0 0x7f6e8c06fd60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6e94006010 tx=0x7f6e94005ec0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 -- 172.21.15.116:0/2804406954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea4101dc0 msgr2=0x7f6ea41968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea4101dc0 0x7f6ea41968d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800d8d0 tx=0x7f6e9800dbe0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 -- 172.21.15.116:0/2804406954 shutdown_connections 2024-03-04T16:09:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e8c06d8a0 0x7f6e8c06fd60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ea4100bc0 0x7f6ea4196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 --2- 172.21.15.116:0/2804406954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea4101dc0 0x7f6ea41968d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 -- 172.21.15.116:0/2804406954 >> 172.21.15.116:0/2804406954 conn(0x7f6ea40fc140 msgr2=0x7f6ea4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 -- 172.21.15.116:0/2804406954 shutdown_connections 2024-03-04T16:09:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:42.940+0000 7f6ea924c700 1 -- 172.21.15.116:0/2804406954 wait complete. 2024-03-04T16:09:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.042+0000 7f218403b700 1 -- 172.21.15.116:0/2593863226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f217c101650 msgr2=0x7f217c103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.042+0000 7f218403b700 1 --2- 172.21.15.116:0/2593863226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f217c101650 0x7f217c103a40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2178009910 tx=0x7f2178009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.042+0000 7f218403b700 1 -- 172.21.15.116:0/2593863226 shutdown_connections 2024-03-04T16:09:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.042+0000 7f218403b700 1 --2- 172.21.15.116:0/2593863226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f217c103f80 0x7f217c106370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.042+0000 7f218403b700 1 --2- 172.21.15.116:0/2593863226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f217c101650 0x7f217c103a40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.042+0000 7f218403b700 1 -- 172.21.15.116:0/2593863226 >> 172.21.15.116:0/2593863226 conn(0x7f217c0fb060 msgr2=0x7f217c0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.042+0000 7f218403b700 1 -- 172.21.15.116:0/2593863226 shutdown_connections 2024-03-04T16:09:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.042+0000 7f218403b700 1 -- 172.21.15.116:0/2593863226 wait complete. 2024-03-04T16:09:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f218403b700 1 Processor -- start 2024-03-04T16:09:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f218403b700 1 -- start start 2024-03-04T16:09:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f218403b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f217c101650 0x7f217c100f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f218403b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f217c103f80 0x7f217c0ff600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f218403b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c0ffb40 con 0x7f217c103f80 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f218403b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c0ffc80 con 0x7f217c101650 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f21815d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f217c103f80 0x7f217c0ff600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f21815d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f217c103f80 0x7f217c0ff600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42198/0 (socket says 172.21.15.116:42198) 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f21815d6700 1 -- 172.21.15.116:0/2266722820 learned_addr learned my addr 172.21.15.116:0/2266722820 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f21815d6700 1 -- 172.21.15.116:0/2266722820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f217c101650 msgr2=0x7f217c100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f21815d6700 1 --2- 172.21.15.116:0/2266722820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f217c101650 0x7f217c100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.043+0000 7f21815d6700 1 -- 172.21.15.116:0/2266722820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21780095f0 con 0x7f217c103f80 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.044+0000 7f21815d6700 1 --2- 172.21.15.116:0/2266722820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f217c103f80 0x7f217c0ff600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c8a0 tx=0x7f216c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.044+0000 7f2172ffd700 1 -- 172.21.15.116:0/2266722820 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f216c00cea0 con 0x7f217c103f80 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.044+0000 7f2172ffd700 1 -- 172.21.15.116:0/2266722820 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f216c004830 con 0x7f217c103f80 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.044+0000 7f218403b700 1 -- 172.21.15.116:0/2266722820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f217c0fff60 con 0x7f217c103f80 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.044+0000 7f2172ffd700 1 -- 172.21.15.116:0/2266722820 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f216c0055b0 con 0x7f217c103f80 2024-03-04T16:09:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.044+0000 7f218403b700 1 -- 172.21.15.116:0/2266722820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217c19e3e0 con 0x7f217c103f80 2024-03-04T16:09:43.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.045+0000 7f218403b700 1 -- 172.21.15.116:0/2266722820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217c190530 con 0x7f217c103f80 2024-03-04T16:09:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.045+0000 7f2172ffd700 1 -- 172.21.15.116:0/2266722820 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f216c007ad0 con 0x7f217c103f80 2024-03-04T16:09:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.046+0000 7f2172ffd700 1 --2- 172.21.15.116:0/2266722820 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f216806d8a0 0x7f216806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.046+0000 7f2172ffd700 1 -- 172.21.15.116:0/2266722820 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f216c08d1f0 con 0x7f217c103f80 2024-03-04T16:09:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.046+0000 7f2181dd7700 1 --2- 172.21.15.116:0/2266722820 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f216806d8a0 0x7f216806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.046+0000 7f2181dd7700 1 --2- 172.21.15.116:0/2266722820 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f216806d8a0 0x7f216806fd60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f217800b5c0 tx=0x7f2178005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:43.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.050+0000 7f2172ffd700 1 -- 172.21.15.116:0/2266722820 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f216c05a520 con 0x7f217c103f80 2024-03-04T16:09:43.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.330+0000 7f218403b700 1 -- 172.21.15.116:0/2266722820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f217c0619a0 con 0x7f217c103f80 2024-03-04T16:09:43.331 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:43 smithi116 ceph-mon[94164]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:43.331 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:43 smithi116 ceph-mon[94164]: osd.4 marked itself down and dead 2024-03-04T16:09:43.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.331+0000 7f2172ffd700 1 -- 172.21.15.116:0/2266722820 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f216c05a0b0 con 0x7f217c103f80 2024-03-04T16:09:43.332 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:09:43.332 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:09:43.332 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:09:43.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:09:43.332 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:09:43.332 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:09:43.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:09:43.332 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:09:43.332 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-03-04T16:09:43.333 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:09:43.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:09:43.333 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:09:43.333 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:09:43.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:09:43.333 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:09:43.333 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-03-04T16:09:43.333 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 8 2024-03-04T16:09:43.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:09:43.333 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:09:43.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 -- 172.21.15.116:0/2266722820 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f216806d8a0 msgr2=0x7f216806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:43.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 --2- 172.21.15.116:0/2266722820 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f216806d8a0 0x7f216806fd60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f217800b5c0 tx=0x7f2178005960 comp rx=0 tx=0).stop 2024-03-04T16:09:43.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 -- 172.21.15.116:0/2266722820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f217c103f80 msgr2=0x7f217c0ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:43.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 --2- 172.21.15.116:0/2266722820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f217c103f80 0x7f217c0ff600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c8a0 tx=0x7f216c00cc60 comp rx=0 tx=0).stop 2024-03-04T16:09:43.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 -- 172.21.15.116:0/2266722820 shutdown_connections 2024-03-04T16:09:43.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 --2- 172.21.15.116:0/2266722820 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f216806d8a0 0x7f216806fd60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 --2- 172.21.15.116:0/2266722820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f217c101650 0x7f217c100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 --2- 172.21.15.116:0/2266722820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f217c103f80 0x7f217c0ff600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 -- 172.21.15.116:0/2266722820 >> 172.21.15.116:0/2266722820 conn(0x7f217c0fb060 msgr2=0x7f217c0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 -- 172.21.15.116:0/2266722820 shutdown_connections 2024-03-04T16:09:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.333+0000 7f218403b700 1 -- 172.21.15.116:0/2266722820 wait complete. 2024-03-04T16:09:43.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.437+0000 7ff610ebe700 1 -- 172.21.15.116:0/2911451874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff60c0ffab0 msgr2=0x7ff60c0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.437+0000 7ff610ebe700 1 --2- 172.21.15.116:0/2911451874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff60c0ffab0 0x7ff60c0ffed0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009910 tx=0x7ff5fc009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.437+0000 7ff610ebe700 1 -- 172.21.15.116:0/2911451874 shutdown_connections 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.437+0000 7ff610ebe700 1 --2- 172.21.15.116:0/2911451874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff60c100cb0 0x7ff60c101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.437+0000 7ff610ebe700 1 --2- 172.21.15.116:0/2911451874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff60c0ffab0 0x7ff60c0ffed0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.437+0000 7ff610ebe700 1 -- 172.21.15.116:0/2911451874 >> 172.21.15.116:0/2911451874 conn(0x7ff60c0fb030 msgr2=0x7ff60c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.437+0000 7ff610ebe700 1 -- 172.21.15.116:0/2911451874 shutdown_connections 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.437+0000 7ff610ebe700 1 -- 172.21.15.116:0/2911451874 wait complete. 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.437+0000 7ff610ebe700 1 Processor -- start 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff610ebe700 1 -- start start 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff610ebe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff60c0ffab0 0x7ff60c1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff610ebe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff60c100cb0 0x7ff60c196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff610ebe700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c196f50 con 0x7ff60c0ffab0 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff610ebe700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c197090 con 0x7ff60c100cb0 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff60a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff60c0ffab0 0x7ff60c1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff60a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff60c0ffab0 0x7ff60c1963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42212/0 (socket says 172.21.15.116:42212) 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff60a59c700 1 -- 172.21.15.116:0/1665312056 learned_addr learned my addr 172.21.15.116:0/1665312056 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff609d9b700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff60c100cb0 0x7ff60c196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff60a59c700 1 -- 172.21.15.116:0/1665312056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff60c100cb0 msgr2=0x7ff60c196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff60a59c700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff60c100cb0 0x7ff60c196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff60a59c700 1 -- 172.21.15.116:0/1665312056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0095f0 con 0x7ff60c0ffab0 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff609d9b700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff60c100cb0 0x7ff60c196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff60a59c700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff60c0ffab0 0x7ff60c1963f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00bda0 tx=0x7ff5fc00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.438+0000 7ff5fb7fe700 1 -- 172.21.15.116:0/1665312056 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5fc01d070 con 0x7ff60c0ffab0 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.439+0000 7ff610ebe700 1 -- 172.21.15.116:0/1665312056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c19bae0 con 0x7ff60c0ffab0 2024-03-04T16:09:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.439+0000 7ff5fb7fe700 1 -- 172.21.15.116:0/1665312056 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff5fc004c50 con 0x7ff60c0ffab0 2024-03-04T16:09:43.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.439+0000 7ff5fb7fe700 1 -- 172.21.15.116:0/1665312056 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5fc00f690 con 0x7ff60c0ffab0 2024-03-04T16:09:43.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.439+0000 7ff610ebe700 1 -- 172.21.15.116:0/1665312056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c19c050 con 0x7ff60c0ffab0 2024-03-04T16:09:43.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.439+0000 7ff610ebe700 1 -- 172.21.15.116:0/1665312056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c108690 con 0x7ff60c0ffab0 2024-03-04T16:09:43.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.440+0000 7ff5fb7fe700 1 -- 172.21.15.116:0/1665312056 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7ff5fc0229b0 con 0x7ff60c0ffab0 2024-03-04T16:09:43.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.441+0000 7ff5fb7fe700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff5f406d8a0 0x7ff5f406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:43.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.441+0000 7ff609d9b700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff5f406d8a0 0x7ff5f406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:43.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.441+0000 7ff5fb7fe700 1 -- 172.21.15.116:0/1665312056 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff5fc08ec00 con 0x7ff60c0ffab0 2024-03-04T16:09:43.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.441+0000 7ff609d9b700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff5f406d8a0 0x7ff5f406fd60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff6000099d0 tx=0x7ff600008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:43.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.445+0000 7ff5fb7fe700 1 -- 172.21.15.116:0/1665312056 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff5fc094020 con 0x7ff60c0ffab0 2024-03-04T16:09:43.548 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:43 smithi152 ceph-mon[82845]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:43.549 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:43 smithi152 ceph-mon[82845]: osd.4 marked itself down and dead 2024-03-04T16:09:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.681+0000 7ff610ebe700 1 -- 172.21.15.116:0/1665312056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff60c108880 con 0x7ff60c0ffab0 2024-03-04T16:09:43.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.681+0000 7ff5fb7fe700 1 -- 172.21.15.116:0/1665312056 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1852 (secure 0 0 0) 0x7ff5fc027750 con 0x7ff60c0ffab0 2024-03-04T16:09:43.682 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:09:43.682 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:09:43.682 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:09:43.682 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:09:43.682 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:09:43.682 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:09:43.683 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:09:43.684 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:09:43.685 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:09:43.685 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:09:43.685 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:09:43.685 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:09:43.685 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:09:43.685 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:09:43.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 -- 172.21.15.116:0/1665312056 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff5f406d8a0 msgr2=0x7ff5f406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff5f406d8a0 0x7ff5f406fd60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff6000099d0 tx=0x7ff600008040 comp rx=0 tx=0).stop 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 -- 172.21.15.116:0/1665312056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff60c0ffab0 msgr2=0x7ff60c1963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff60c0ffab0 0x7ff60c1963f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00bda0 tx=0x7ff5fc00be80 comp rx=0 tx=0).stop 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 -- 172.21.15.116:0/1665312056 shutdown_connections 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff5f406d8a0 0x7ff5f406fd60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff60c0ffab0 0x7ff60c1963f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 --2- 172.21.15.116:0/1665312056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff60c100cb0 0x7ff60c196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 -- 172.21.15.116:0/1665312056 >> 172.21.15.116:0/1665312056 conn(0x7ff60c0fb030 msgr2=0x7ff60c068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 -- 172.21.15.116:0/1665312056 shutdown_connections 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.684+0000 7ff610ebe700 1 -- 172.21.15.116:0/1665312056 wait complete. 2024-03-04T16:09:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:09:43.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.787+0000 7ff66e244700 1 -- 172.21.15.116:0/1803624818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff668100bc0 msgr2=0x7ff668100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.787+0000 7ff66e244700 1 --2- 172.21.15.116:0/1803624818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff668100bc0 0x7ff668100fe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff65800b0a0 tx=0x7ff65800b3b0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.787+0000 7ff66e244700 1 -- 172.21.15.116:0/1803624818 shutdown_connections 2024-03-04T16:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.787+0000 7ff66e244700 1 --2- 172.21.15.116:0/1803624818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff668101dc0 0x7ff668102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.787+0000 7ff66e244700 1 --2- 172.21.15.116:0/1803624818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff668100bc0 0x7ff668100fe0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.787+0000 7ff66e244700 1 -- 172.21.15.116:0/1803624818 >> 172.21.15.116:0/1803624818 conn(0x7ff6680fc140 msgr2=0x7ff6680fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.787+0000 7ff66e244700 1 -- 172.21.15.116:0/1803624818 shutdown_connections 2024-03-04T16:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.787+0000 7ff66e244700 1 -- 172.21.15.116:0/1803624818 wait complete. 2024-03-04T16:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.788+0000 7ff66e244700 1 Processor -- start 2024-03-04T16:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.788+0000 7ff66e244700 1 -- start start 2024-03-04T16:09:43.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.788+0000 7ff66e244700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff668100bc0 0x7ff668198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:43.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.788+0000 7ff66e244700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff668101dc0 0x7ff668198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:43.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.788+0000 7ff66e244700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff668199060 con 0x7ff668100bc0 2024-03-04T16:09:43.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.788+0000 7ff66e244700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6681991a0 con 0x7ff668101dc0 2024-03-04T16:09:43.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.788+0000 7ff6677fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff668100bc0 0x7ff668198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:43.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.788+0000 7ff6677fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff668100bc0 0x7ff668198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42218/0 (socket says 172.21.15.116:42218) 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.788+0000 7ff6677fe700 1 -- 172.21.15.116:0/2459671195 learned_addr learned my addr 172.21.15.116:0/2459671195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.789+0000 7ff666ffd700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff668101dc0 0x7ff668198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.789+0000 7ff6677fe700 1 -- 172.21.15.116:0/2459671195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff668101dc0 msgr2=0x7ff668198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.789+0000 7ff6677fe700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff668101dc0 0x7ff668198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.789+0000 7ff6677fe700 1 -- 172.21.15.116:0/2459671195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff658009d00 con 0x7ff668100bc0 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.789+0000 7ff6677fe700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff668100bc0 0x7ff668198590 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff6580077b0 tx=0x7ff658003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.789+0000 7ff664ff9700 1 -- 172.21.15.116:0/2459671195 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff65800e040 con 0x7ff668100bc0 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.789+0000 7ff664ff9700 1 -- 172.21.15.116:0/2459671195 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff65800ce60 con 0x7ff668100bc0 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.789+0000 7ff66e244700 1 -- 172.21.15.116:0/2459671195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66819dc00 con 0x7ff668100bc0 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.789+0000 7ff664ff9700 1 -- 172.21.15.116:0/2459671195 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff65801db10 con 0x7ff668100bc0 2024-03-04T16:09:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.789+0000 7ff66e244700 1 -- 172.21.15.116:0/2459671195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66819e0c0 con 0x7ff668100bc0 2024-03-04T16:09:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.790+0000 7ff66e244700 1 -- 172.21.15.116:0/2459671195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff668073810 con 0x7ff668100bc0 2024-03-04T16:09:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.790+0000 7ff664ff9700 1 -- 172.21.15.116:0/2459671195 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7ff65801dc90 con 0x7ff668100bc0 2024-03-04T16:09:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.791+0000 7ff664ff9700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff65006d8a0 0x7ff65006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.791+0000 7ff664ff9700 1 -- 172.21.15.116:0/2459671195 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff65808f030 con 0x7ff668100bc0 2024-03-04T16:09:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.791+0000 7ff666ffd700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff65006d8a0 0x7ff65006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.791+0000 7ff666ffd700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff65006d8a0 0x7ff65006fd60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff65c005ea0 tx=0x7ff65c005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:43.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:43.796+0000 7ff664ff9700 1 -- 172.21.15.116:0/2459671195 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff65805c360 con 0x7ff668100bc0 2024-03-04T16:09:43.813 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:43 smithi152 podman[97404]: 2024-03-04 16:09:43.549078121 +0000 UTC m=+1.303887550 container died c40b387afb374f5d68cdfcff5fc0a4b97d1bfbc774a1f178b5962d1a033428bf (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) 2024-03-04T16:09:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.007+0000 7ff66e244700 1 -- 172.21.15.116:0/2459671195 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6680fd460 con 0x7ff65006d8a0 2024-03-04T16:09:44.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.008+0000 7ff664ff9700 1 -- 172.21.15.116:0/2459671195 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7ff6680fd460 con 0x7ff65006d8a0 2024-03-04T16:09:44.009 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:09:44.009 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:09:44.009 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:09:44.009 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:09:44.009 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:09:44.009 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:09:44.010 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:09:44.010 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:09:44.010 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:09:44.010 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "10/23 daemons upgraded", 2024-03-04T16:09:44.010 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:09:44.010 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:09:44.010 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:09:44.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.010+0000 7ff66e244700 1 -- 172.21.15.116:0/2459671195 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff65006d8a0 msgr2=0x7ff65006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:44.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.010+0000 7ff66e244700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff65006d8a0 0x7ff65006fd60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff65c005ea0 tx=0x7ff65c005e30 comp rx=0 tx=0).stop 2024-03-04T16:09:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.010+0000 7ff66e244700 1 -- 172.21.15.116:0/2459671195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff668100bc0 msgr2=0x7ff668198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.010+0000 7ff66e244700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff668100bc0 0x7ff668198590 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff6580077b0 tx=0x7ff658003ce0 comp rx=0 tx=0).stop 2024-03-04T16:09:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.011+0000 7ff66e244700 1 -- 172.21.15.116:0/2459671195 shutdown_connections 2024-03-04T16:09:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.011+0000 7ff66e244700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ff65006d8a0 0x7ff65006fd60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.011+0000 7ff66e244700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff668100bc0 0x7ff668198590 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.011+0000 7ff66e244700 1 --2- 172.21.15.116:0/2459671195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff668101dc0 0x7ff668198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.011+0000 7ff66e244700 1 -- 172.21.15.116:0/2459671195 >> 172.21.15.116:0/2459671195 conn(0x7ff6680fc140 msgr2=0x7ff6680fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.011+0000 7ff66e244700 1 -- 172.21.15.116:0/2459671195 shutdown_connections 2024-03-04T16:09:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.011+0000 7ff66e244700 1 -- 172.21.15.116:0/2459671195 wait complete. 2024-03-04T16:09:44.107 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:43 smithi152 podman[97404]: 2024-03-04 16:09:43.90449285 +0000 UTC m=+1.659302286 container cleanup c40b387afb374f5d68cdfcff5fc0a4b97d1bfbc774a1f178b5962d1a033428bf (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-03-04T16:09:44.107 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:43 smithi152 bash[97404]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4 2024-03-04T16:09:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.114+0000 7f3695a91700 1 -- 172.21.15.116:0/1926957765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36900ffac0 msgr2=0x7f36900ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.114+0000 7f3695a91700 1 --2- 172.21.15.116:0/1926957765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36900ffac0 0x7f36900ffee0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3680009910 tx=0x7f3680009c20 comp rx=0 tx=0).stop 2024-03-04T16:09:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.114+0000 7f3695a91700 1 -- 172.21.15.116:0/1926957765 shutdown_connections 2024-03-04T16:09:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.114+0000 7f3695a91700 1 --2- 172.21.15.116:0/1926957765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3690100cc0 0x7f3690101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.114+0000 7f3695a91700 1 --2- 172.21.15.116:0/1926957765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36900ffac0 0x7f36900ffee0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.114+0000 7f3695a91700 1 -- 172.21.15.116:0/1926957765 >> 172.21.15.116:0/1926957765 conn(0x7f36900fb060 msgr2=0x7f36900fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.114+0000 7f3695a91700 1 -- 172.21.15.116:0/1926957765 shutdown_connections 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.114+0000 7f3695a91700 1 -- 172.21.15.116:0/1926957765 wait complete. 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.114+0000 7f3695a91700 1 Processor -- start 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f3695a91700 1 -- start start 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f3695a91700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36900ffac0 0x7f3690069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f3695a91700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3690100cc0 0x7f369019b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f3695a91700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369019b6b0 con 0x7f3690100cc0 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f3695a91700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369019b820 con 0x7f36900ffac0 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f368effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3690100cc0 0x7f369019b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f368effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3690100cc0 0x7f369019b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42240/0 (socket says 172.21.15.116:42240) 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f368effd700 1 -- 172.21.15.116:0/3462237908 learned_addr learned my addr 172.21.15.116:0/3462237908 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f368f7fe700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36900ffac0 0x7f3690069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f368effd700 1 -- 172.21.15.116:0/3462237908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36900ffac0 msgr2=0x7f3690069930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f368effd700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36900ffac0 0x7f3690069930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f368effd700 1 -- 172.21.15.116:0/3462237908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36800095f0 con 0x7f3690100cc0 2024-03-04T16:09:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f368effd700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3690100cc0 0x7f369019b0e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f368400e920 tx=0x7f368400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f368cff9700 1 -- 172.21.15.116:0/3462237908 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f368400c9d0 con 0x7f3690100cc0 2024-03-04T16:09:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f368cff9700 1 -- 172.21.15.116:0/3462237908 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3684010430 con 0x7f3690100cc0 2024-03-04T16:09:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.115+0000 7f3695a91700 1 -- 172.21.15.116:0/3462237908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369019bb00 con 0x7f3690100cc0 2024-03-04T16:09:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.116+0000 7f368cff9700 1 -- 172.21.15.116:0/3462237908 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3684018650 con 0x7f3690100cc0 2024-03-04T16:09:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.116+0000 7f3695a91700 1 -- 172.21.15.116:0/3462237908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369019c050 con 0x7f3690100cc0 2024-03-04T16:09:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.117+0000 7f3695a91700 1 -- 172.21.15.116:0/3462237908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3690108620 con 0x7f3690100cc0 2024-03-04T16:09:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.117+0000 7f368cff9700 1 -- 172.21.15.116:0/3462237908 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f36840105a0 con 0x7f3690100cc0 2024-03-04T16:09:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.117+0000 7f368cff9700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f367806d8a0 0x7f367806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:09:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.118+0000 7f368cff9700 1 -- 172.21.15.116:0/3462237908 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3684014070 con 0x7f3690100cc0 2024-03-04T16:09:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.118+0000 7f368f7fe700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f367806d8a0 0x7f367806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:09:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.118+0000 7f368f7fe700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f367806d8a0 0x7f367806fd60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3680009fd0 tx=0x7f3680005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:09:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.124+0000 7f368cff9700 1 -- 172.21.15.116:0/3462237908 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3684052d20 con 0x7f3690100cc0 2024-03-04T16:09:44.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.416+0000 7f3695a91700 1 -- 172.21.15.116:0/3462237908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f369019c660 con 0x7f3690100cc0 2024-03-04T16:09:44.418 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:44 smithi116 ceph-mon[94164]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:44.418 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:44 smithi116 ceph-mon[94164]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:44.418 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:44 smithi116 ceph-mon[94164]: pgmap v138: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:44.418 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:44 smithi116 ceph-mon[94164]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:09:44.418 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:44 smithi116 ceph-mon[94164]: osdmap e71: 6 total, 5 up, 6 in 2024-03-04T16:09:44.418 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:44 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2266722820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:44.418 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:44 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1665312056' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:09:44.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.417+0000 7f368cff9700 1 -- 172.21.15.116:0/3462237908 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+100 (secure 0 0 0) 0x7f368405ab00 con 0x7f3690100cc0 2024-03-04T16:09:44.419 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 osds down 2024-03-04T16:09:44.419 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-04T16:09:44.419 INFO:teuthology.orchestra.run.smithi116.stdout: osd.4 (root=default,host=smithi152) is down 2024-03-04T16:09:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 -- 172.21.15.116:0/3462237908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f367806d8a0 msgr2=0x7f367806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f367806d8a0 0x7f367806fd60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3680009fd0 tx=0x7f3680005a90 comp rx=0 tx=0).stop 2024-03-04T16:09:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 -- 172.21.15.116:0/3462237908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3690100cc0 msgr2=0x7f369019b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:09:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3690100cc0 0x7f369019b0e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f368400e920 tx=0x7f368400ec30 comp rx=0 tx=0).stop 2024-03-04T16:09:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 -- 172.21.15.116:0/3462237908 shutdown_connections 2024-03-04T16:09:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f367806d8a0 0x7f367806fd60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36900ffac0 0x7f3690069930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 --2- 172.21.15.116:0/3462237908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3690100cc0 0x7f369019b0e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:09:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 -- 172.21.15.116:0/3462237908 >> 172.21.15.116:0/3462237908 conn(0x7f36900fb060 msgr2=0x7f3690103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:09:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 -- 172.21.15.116:0/3462237908 shutdown_connections 2024-03-04T16:09:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:09:44.419+0000 7f3695a91700 1 -- 172.21.15.116:0/3462237908 wait complete. 2024-03-04T16:09:44.547 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:44 smithi152 podman[97423]: 2024-03-04 16:09:44.223149491 +0000 UTC m=+0.677425774 container remove c40b387afb374f5d68cdfcff5fc0a4b97d1bfbc774a1f178b5962d1a033428bf (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-04T16:09:44.548 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:44 smithi152 ceph-mon[82845]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:44.548 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:44 smithi152 ceph-mon[82845]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:44.548 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:44 smithi152 ceph-mon[82845]: pgmap v138: 65 pgs: 65 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:44.548 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:44 smithi152 ceph-mon[82845]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:09:44.548 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:44 smithi152 ceph-mon[82845]: osdmap e71: 6 total, 5 up, 6 in 2024-03-04T16:09:44.548 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:44 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2266722820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:44.549 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:44 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1665312056' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:09:44.809 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:44 smithi152 podman[97517]: 2024-03-04 16:09:44.450311927 +0000 UTC m=+0.017790732 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:09:44.810 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:44 smithi152 podman[97517]: 2024-03-04 16:09:44.579541659 +0000 UTC m=+0.147020455 container create 7d4b3fd6bd47097ec02bee836d6e45637e639b7925632a994c524efe4c7c8ddf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:09:44.810 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:44 smithi152 podman[97517]: 2024-03-04 16:09:44.809957279 +0000 UTC m=+0.377436087 container init 7d4b3fd6bd47097ec02bee836d6e45637e639b7925632a994c524efe4c7c8ddf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:09:45.104 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:44 smithi152 podman[97517]: 2024-03-04 16:09:44.813193603 +0000 UTC m=+0.380672400 container start 7d4b3fd6bd47097ec02bee836d6e45637e639b7925632a994c524efe4c7c8ddf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:09:45.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:45 smithi152 ceph-mon[82845]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:45.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:45 smithi152 ceph-mon[82845]: osdmap e72: 6 total, 5 up, 6 in 2024-03-04T16:09:45.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:45 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3462237908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:09:45.358 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:45 smithi152 podman[97517]: 2024-03-04 16:09:45.160261802 +0000 UTC m=+0.727740612 container attach 7d4b3fd6bd47097ec02bee836d6e45637e639b7925632a994c524efe4c7c8ddf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) 2024-03-04T16:09:45.358 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:45 smithi152 podman[97517]: 2024-03-04 16:09:45.160493182 +0000 UTC m=+0.727971973 container died 7d4b3fd6bd47097ec02bee836d6e45637e639b7925632a994c524efe4c7c8ddf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:09:45.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:45 smithi116 ceph-mon[94164]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:45.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:45 smithi116 ceph-mon[94164]: osdmap e72: 6 total, 5 up, 6 in 2024-03-04T16:09:45.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:45 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3462237908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:09:46.107 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:45 smithi152 podman[97543]: 2024-03-04 16:09:45.764712888 +0000 UTC m=+0.654766712 container remove 7d4b3fd6bd47097ec02bee836d6e45637e639b7925632a994c524efe4c7c8ddf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-04T16:09:46.108 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:45 smithi152 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-04T16:09:46.108 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:45 smithi152 systemd[1]: Stopped Ceph osd.4 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:09:46.108 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:45 smithi152 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.4.service: Consumed 1min 5.453s CPU time. 2024-03-04T16:09:46.108 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:45 smithi152 systemd[1]: Starting Ceph osd.4 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:09:46.399 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:46 smithi152 ceph-mon[82845]: pgmap v141: 65 pgs: 13 peering, 3 stale+active+clean, 2 active+undersized, 47 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:46.400 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:46 smithi152 ceph-mon[82845]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-04T16:09:46.400 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 podman[97647]: 2024-03-04 16:09:46.118255897 +0000 UTC m=+0.017966392 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:09:46.400 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 podman[97647]: 2024-03-04 16:09:46.257899688 +0000 UTC m=+0.157610174 container create cc731aa80258a2931712afdf357d831cb846bb302b302ad624093578f6e976b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:09:46.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:46 smithi116 ceph-mon[94164]: pgmap v141: 65 pgs: 13 peering, 3 stale+active+clean, 2 active+undersized, 47 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-04T16:09:46.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:46 smithi116 ceph-mon[94164]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-04T16:09:46.857 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 podman[97647]: 2024-03-04 16:09:46.52753781 +0000 UTC m=+0.427248296 container init cc731aa80258a2931712afdf357d831cb846bb302b302ad624093578f6e976b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:09:46.857 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 podman[97647]: 2024-03-04 16:09:46.530610045 +0000 UTC m=+0.430320526 container start cc731aa80258a2931712afdf357d831cb846bb302b302ad624093578f6e976b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-04T16:09:46.858 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 podman[97647]: 2024-03-04 16:09:46.602336601 +0000 UTC m=+0.502047102 container attach cc731aa80258a2931712afdf357d831cb846bb302b302ad624093578f6e976b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-04T16:09:47.289 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:47 smithi152 ceph-mon[82845]: Health check failed: Degraded data redundancy: 18/231 objects degraded (7.792%), 5 pgs degraded (PG_DEGRADED) 2024-03-04T16:09:47.289 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate[97662]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-04T16:09:47.289 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate[97662]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-04T16:09:47.289 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 bash[97647]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-04T16:09:47.289 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 bash[97647]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-04T16:09:47.289 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 bash[97647]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-04T16:09:47.289 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 bash[97647]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-04T16:09:47.290 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 bash[97647]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-03-04T16:09:47.290 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 bash[97647]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-04T16:09:47.290 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 bash[97647]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-04T16:09:47.290 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate[97662]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-04T16:09:47.290 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate[97662]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-04T16:09:47.290 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate[97662]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-03-04T16:09:47.290 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate[97662]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-04T16:09:47.290 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate[97662]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-04T16:09:47.290 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:46 smithi152 podman[97647]: 2024-03-04 16:09:46.994933636 +0000 UTC m=+0.894644109 container died cc731aa80258a2931712afdf357d831cb846bb302b302ad624093578f6e976b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-04T16:09:47.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:47 smithi116 ceph-mon[94164]: Health check failed: Degraded data redundancy: 18/231 objects degraded (7.792%), 5 pgs degraded (PG_DEGRADED) 2024-03-04T16:09:47.908 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:47 smithi152 podman[97782]: 2024-03-04 16:09:47.62376192 +0000 UTC m=+0.620218641 container remove cc731aa80258a2931712afdf357d831cb846bb302b302ad624093578f6e976b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-04T16:09:47.908 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:47 smithi152 podman[97842]: 2024-03-04 16:09:47.739642922 +0000 UTC m=+0.018116790 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:09:47.908 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:47 smithi152 podman[97842]: 2024-03-04 16:09:47.908098496 +0000 UTC m=+0.186572389 container create 5ecc30e5a86ccd93283723000d5b0979b90653f857da0dade47bc6a81149e57d (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-03-04T16:09:48.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:48 smithi116 ceph-mon[94164]: pgmap v142: 65 pgs: 5 active+undersized+degraded, 13 peering, 2 stale+active+clean, 8 active+undersized, 37 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 18/231 objects degraded (7.792%) 2024-03-04T16:09:48.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:48 smithi152 ceph-mon[82845]: pgmap v142: 65 pgs: 5 active+undersized+degraded, 13 peering, 2 stale+active+clean, 8 active+undersized, 37 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 18/231 objects degraded (7.792%) 2024-03-04T16:09:48.608 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:48 smithi152 podman[97842]: 2024-03-04 16:09:48.160649038 +0000 UTC m=+0.439122888 container init 5ecc30e5a86ccd93283723000d5b0979b90653f857da0dade47bc6a81149e57d (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-04T16:09:48.608 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:48 smithi152 podman[97842]: 2024-03-04 16:09:48.163739036 +0000 UTC m=+0.442212889 container start 5ecc30e5a86ccd93283723000d5b0979b90653f857da0dade47bc6a81149e57d (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-04T16:09:48.608 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:48 smithi152 bash[97842]: 5ecc30e5a86ccd93283723000d5b0979b90653f857da0dade47bc6a81149e57d 2024-03-04T16:09:48.608 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:48 smithi152 systemd[1]: Started Ceph osd.4 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:09:49.107 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:48 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:09:48.711+0000 7efcb8cf27c0 -1 Falling back to public interface 2024-03-04T16:09:49.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:49.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:49.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:49.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:49.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:49.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:50.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:50 smithi152 ceph-mon[82845]: pgmap v143: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 40/231 objects degraded (17.316%) 2024-03-04T16:09:50.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:50 smithi152 ceph-mon[82845]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-04T16:09:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:50 smithi116 ceph-mon[94164]: pgmap v143: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 40/231 objects degraded (17.316%) 2024-03-04T16:09:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:50 smithi116 ceph-mon[94164]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-04T16:09:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:52 smithi152 ceph-mon[82845]: pgmap v144: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 40/231 objects degraded (17.316%) 2024-03-04T16:09:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:52 smithi116 ceph-mon[94164]: pgmap v144: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 40/231 objects degraded (17.316%) 2024-03-04T16:09:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:52.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:53.518 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:53 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:09:53.236+0000 7efcb8cf27c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-04T16:09:53.769 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:53.769 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:53.769 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:53 smithi152 ceph-mon[82845]: pgmap v145: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 40/231 objects degraded (17.316%) 2024-03-04T16:09:53.769 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:53.769 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:54.107 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:53 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:09:53.767+0000 7efcb8cf27c0 -1 osd.4 70 log_to_monitors true 2024-03-04T16:09:54.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:54.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:54.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:53 smithi116 ceph-mon[94164]: pgmap v145: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 212 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 40/231 objects degraded (17.316%) 2024-03-04T16:09:54.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:54.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:55.029 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:54 smithi152 ceph-mon[82845]: from='osd.4 [v2:172.21.15.152:6808/878265314,v1:172.21.15.152:6809/878265314]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:09:55.029 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:54 smithi152 ceph-mon[82845]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:09:55.029 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:54 smithi152 ceph-mon[82845]: Health check update: Degraded data redundancy: 40/231 objects degraded (17.316%), 15 pgs degraded (PG_DEGRADED) 2024-03-04T16:09:55.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:54 smithi116 ceph-mon[94164]: from='osd.4 [v2:172.21.15.152:6808/878265314,v1:172.21.15.152:6809/878265314]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:09:55.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:54 smithi116 ceph-mon[94164]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:09:55.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:54 smithi116 ceph-mon[94164]: Health check update: Degraded data redundancy: 40/231 objects degraded (17.316%), 15 pgs degraded (PG_DEGRADED) 2024-03-04T16:09:56.108 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:09:55 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:09:55.646+0000 7efca6142700 -1 osd.4 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:09:56.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:55 smithi152 ceph-mon[82845]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T16:09:56.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:55 smithi152 ceph-mon[82845]: from='osd.4 [v2:172.21.15.152:6808/878265314,v1:172.21.15.152:6809/878265314]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:09:56.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:55 smithi152 ceph-mon[82845]: osdmap e73: 6 total, 5 up, 6 in 2024-03-04T16:09:56.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:55 smithi152 ceph-mon[82845]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:09:56.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:55 smithi152 ceph-mon[82845]: pgmap v147: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 212 MiB data, 971 MiB used, 535 GiB / 536 GiB avail; 40/231 objects degraded (17.316%) 2024-03-04T16:09:56.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:55 smithi152 ceph-mon[82845]: from='osd.4 ' entity='osd.4' 2024-03-04T16:09:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:55 smithi116 ceph-mon[94164]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T16:09:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:55 smithi116 ceph-mon[94164]: from='osd.4 [v2:172.21.15.152:6808/878265314,v1:172.21.15.152:6809/878265314]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:09:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:55 smithi116 ceph-mon[94164]: osdmap e73: 6 total, 5 up, 6 in 2024-03-04T16:09:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:55 smithi116 ceph-mon[94164]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:09:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:55 smithi116 ceph-mon[94164]: pgmap v147: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 212 MiB data, 971 MiB used, 535 GiB / 536 GiB avail; 40/231 objects degraded (17.316%) 2024-03-04T16:09:56.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:55 smithi116 ceph-mon[94164]: from='osd.4 ' entity='osd.4' 2024-03-04T16:09:57.788 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:57.788 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:57.788 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:57.788 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:57.788 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:57.788 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:57.788 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:09:57.789 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:57.789 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:57.789 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: osd.4 [v2:172.21.15.152:6808/878265314,v1:172.21.15.152:6809/878265314] boot 2024-03-04T16:09:57.789 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: osdmap e74: 6 total, 6 up, 6 in 2024-03-04T16:09:57.789 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:57.789 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:09:57.789 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:09:57.789 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:09:57.789 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-04T16:09:57.789 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:57 smithi116 ceph-mon[94164]: pgmap v149: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 212 MiB data, 971 MiB used, 535 GiB / 536 GiB avail; 40/231 objects degraded (17.316%) 2024-03-04T16:09:57.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: osd.4 [v2:172.21.15.152:6808/878265314,v1:172.21.15.152:6809/878265314] boot 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: osdmap e74: 6 total, 6 up, 6 in 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:09:57.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:09:57.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:09:57.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-04T16:09:57.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:57 smithi152 ceph-mon[82845]: pgmap v149: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 212 MiB data, 971 MiB used, 535 GiB / 536 GiB avail; 40/231 objects degraded (17.316%) 2024-03-04T16:09:58.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:58 smithi116 ceph-mon[94164]: osdmap e75: 6 total, 6 up, 6 in 2024-03-04T16:09:59.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:58 smithi152 ceph-mon[82845]: osdmap e75: 6 total, 6 up, 6 in 2024-03-04T16:09:59.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:09:59 smithi116 ceph-mon[94164]: pgmap v151: 65 pgs: 6 peering, 7 active+undersized+degraded, 10 active+undersized, 42 active+clean; 212 MiB data, 971 MiB used, 535 GiB / 536 GiB avail; 22/231 objects degraded (9.524%) 2024-03-04T16:10:00.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:09:59 smithi152 ceph-mon[82845]: pgmap v151: 65 pgs: 6 peering, 7 active+undersized+degraded, 10 active+undersized, 42 active+clean; 212 MiB data, 971 MiB used, 535 GiB / 536 GiB avail; 22/231 objects degraded (9.524%) 2024-03-04T16:10:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:00 smithi116 ceph-mon[94164]: Health check update: Degraded data redundancy: 22/231 objects degraded (9.524%), 7 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:00 smithi116 ceph-mon[94164]: Health detail: HEALTH_WARN Degraded data redundancy: 22/231 objects degraded (9.524%), 7 pgs degraded 2024-03-04T16:10:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:00 smithi116 ceph-mon[94164]: [WRN] PG_DEGRADED: Degraded data redundancy: 22/231 objects degraded (9.524%), 7 pgs degraded 2024-03-04T16:10:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:00 smithi116 ceph-mon[94164]: pg 2.5 is active+undersized+degraded, acting [3,0] 2024-03-04T16:10:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:00 smithi116 ceph-mon[94164]: pg 2.7 is active+undersized+degraded, acting [3,2] 2024-03-04T16:10:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:00 smithi116 ceph-mon[94164]: pg 2.b is active+undersized+degraded, acting [3,5] 2024-03-04T16:10:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:00 smithi116 ceph-mon[94164]: pg 2.f is active+undersized+degraded, acting [0,5] 2024-03-04T16:10:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:00 smithi116 ceph-mon[94164]: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-03-04T16:10:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:00 smithi116 ceph-mon[94164]: pg 2.1a is active+undersized+degraded, acting [3,5] 2024-03-04T16:10:00.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:00 smithi116 ceph-mon[94164]: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-03-04T16:10:01.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:00 smithi152 ceph-mon[82845]: Health check update: Degraded data redundancy: 22/231 objects degraded (9.524%), 7 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:01.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:00 smithi152 ceph-mon[82845]: Health detail: HEALTH_WARN Degraded data redundancy: 22/231 objects degraded (9.524%), 7 pgs degraded 2024-03-04T16:10:01.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:00 smithi152 ceph-mon[82845]: [WRN] PG_DEGRADED: Degraded data redundancy: 22/231 objects degraded (9.524%), 7 pgs degraded 2024-03-04T16:10:01.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:00 smithi152 ceph-mon[82845]: pg 2.5 is active+undersized+degraded, acting [3,0] 2024-03-04T16:10:01.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:00 smithi152 ceph-mon[82845]: pg 2.7 is active+undersized+degraded, acting [3,2] 2024-03-04T16:10:01.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:00 smithi152 ceph-mon[82845]: pg 2.b is active+undersized+degraded, acting [3,5] 2024-03-04T16:10:01.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:00 smithi152 ceph-mon[82845]: pg 2.f is active+undersized+degraded, acting [0,5] 2024-03-04T16:10:01.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:00 smithi152 ceph-mon[82845]: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-03-04T16:10:01.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:00 smithi152 ceph-mon[82845]: pg 2.1a is active+undersized+degraded, acting [3,5] 2024-03-04T16:10:01.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:00 smithi152 ceph-mon[82845]: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-03-04T16:10:01.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:01 smithi116 ceph-mon[94164]: pgmap v152: 65 pgs: 6 peering, 59 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:02.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:01 smithi152 ceph-mon[82845]: pgmap v152: 65 pgs: 6 peering, 59 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:02.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:02 smithi116 ceph-mon[94164]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/231 objects degraded (9.524%), 7 pgs degraded) 2024-03-04T16:10:02.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:02 smithi116 ceph-mon[94164]: Cluster is now healthy 2024-03-04T16:10:03.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:02 smithi152 ceph-mon[82845]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/231 objects degraded (9.524%), 7 pgs degraded) 2024-03-04T16:10:03.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:02 smithi152 ceph-mon[82845]: Cluster is now healthy 2024-03-04T16:10:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:03 smithi116 ceph-mon[94164]: pgmap v153: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:04.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:03 smithi152 ceph-mon[82845]: pgmap v153: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:06.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:06 smithi116 ceph-mon[94164]: pgmap v154: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:06.509 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:06 smithi152 ceph-mon[82845]: pgmap v154: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:08.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:08 smithi116 ceph-mon[94164]: pgmap v155: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:08.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:08.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:08 smithi152 ceph-mon[82845]: pgmap v155: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:09.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:09.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:10.405 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:10 smithi152 ceph-mon[82845]: pgmap v156: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:10.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:10 smithi116 ceph-mon[94164]: pgmap v156: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:12.470 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:12 smithi152 ceph-mon[82845]: pgmap v157: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:12.470 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:12 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:10:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:12 smithi116 ceph-mon[94164]: pgmap v157: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:12 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:10:13.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:13 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:10:13.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:13 smithi116 ceph-mon[94164]: Upgrade: osd.5 is safe to restart 2024-03-04T16:10:13.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:13 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:10:13.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:13 smithi152 ceph-mon[82845]: Upgrade: osd.5 is safe to restart 2024-03-04T16:10:14.344 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:14 smithi152 ceph-mon[82845]: pgmap v158: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:14.344 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:14 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:14.344 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:14 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:10:14.344 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:14 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:14.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:14 smithi116 ceph-mon[94164]: pgmap v158: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:14.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:14 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:14.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:14 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:10:14.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:14 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:14.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.538+0000 7f28a7f84700 1 -- 172.21.15.116:0/617418010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a0103f70 msgr2=0x7f28a0106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.538+0000 7f28a7f84700 1 --2- 172.21.15.116:0/617418010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a0103f70 0x7f28a0106360 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2890009910 tx=0x7f2890009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.538+0000 7f28a7f84700 1 -- 172.21.15.116:0/617418010 shutdown_connections 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.538+0000 7f28a7f84700 1 --2- 172.21.15.116:0/617418010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a0103f70 0x7f28a0106360 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.538+0000 7f28a7f84700 1 --2- 172.21.15.116:0/617418010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28a0101640 0x7f28a0103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.538+0000 7f28a7f84700 1 -- 172.21.15.116:0/617418010 >> 172.21.15.116:0/617418010 conn(0x7f28a00fb030 msgr2=0x7f28a00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.538+0000 7f28a7f84700 1 -- 172.21.15.116:0/617418010 shutdown_connections 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.538+0000 7f28a7f84700 1 -- 172.21.15.116:0/617418010 wait complete. 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.538+0000 7f28a7f84700 1 Processor -- start 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.538+0000 7f28a7f84700 1 -- start start 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a7f84700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a0101640 0x7f28a01963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a7f84700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28a0103f70 0x7f28a01968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a7f84700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a0196f00 con 0x7f28a0101640 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a7f84700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a0197040 con 0x7f28a0103f70 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a5d20700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a0101640 0x7f28a01963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a5d20700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a0101640 0x7f28a01963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58736/0 (socket says 172.21.15.116:58736) 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a551f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28a0103f70 0x7f28a01968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a5d20700 1 -- 172.21.15.116:0/2331800009 learned_addr learned my addr 172.21.15.116:0/2331800009 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a5d20700 1 -- 172.21.15.116:0/2331800009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28a0103f70 msgr2=0x7f28a01968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a5d20700 1 --2- 172.21.15.116:0/2331800009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28a0103f70 0x7f28a01968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a5d20700 1 -- 172.21.15.116:0/2331800009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28900095f0 con 0x7f28a0101640 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a5d20700 1 --2- 172.21.15.116:0/2331800009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a0101640 0x7f28a01963a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f289c00b4b0 tx=0x7f289c00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f2896ffd700 1 -- 172.21.15.116:0/2331800009 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f289c011800 con 0x7f28a0101640 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a7f84700 1 -- 172.21.15.116:0/2331800009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a019baf0 con 0x7f28a0101640 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f2896ffd700 1 -- 172.21.15.116:0/2331800009 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f289c011e10 con 0x7f28a0101640 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f2896ffd700 1 -- 172.21.15.116:0/2331800009 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f289c00f2e0 con 0x7f28a0101640 2024-03-04T16:10:14.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.539+0000 7f28a7f84700 1 -- 172.21.15.116:0/2331800009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a019c040 con 0x7f28a0101640 2024-03-04T16:10:14.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.540+0000 7f28a7f84700 1 -- 172.21.15.116:0/2331800009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a01905a0 con 0x7f28a0101640 2024-03-04T16:10:14.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.541+0000 7f2896ffd700 1 -- 172.21.15.116:0/2331800009 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f289c010380 con 0x7f28a0101640 2024-03-04T16:10:14.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.541+0000 7f2896ffd700 1 --2- 172.21.15.116:0/2331800009 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f288c06d8a0 0x7f288c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:14.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.541+0000 7f2896ffd700 1 -- 172.21.15.116:0/2331800009 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f289c08d080 con 0x7f28a0101640 2024-03-04T16:10:14.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.541+0000 7f28a551f700 1 --2- 172.21.15.116:0/2331800009 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f288c06d8a0 0x7f288c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:14.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.542+0000 7f28a551f700 1 --2- 172.21.15.116:0/2331800009 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f288c06d8a0 0x7f288c06fd60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2890006010 tx=0x7f289000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:14.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.546+0000 7f2896ffd700 1 -- 172.21.15.116:0/2331800009 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f289c05a3b0 con 0x7f28a0101640 2024-03-04T16:10:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.758+0000 7f28a7f84700 1 -- 172.21.15.116:0/2331800009 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f28a00611d0 con 0x7f288c06d8a0 2024-03-04T16:10:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.759+0000 7f2896ffd700 1 -- 172.21.15.116:0/2331800009 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f28a00611d0 con 0x7f288c06d8a0 2024-03-04T16:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 -- 172.21.15.116:0/2331800009 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f288c06d8a0 msgr2=0x7f288c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 --2- 172.21.15.116:0/2331800009 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f288c06d8a0 0x7f288c06fd60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2890006010 tx=0x7f289000b540 comp rx=0 tx=0).stop 2024-03-04T16:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 -- 172.21.15.116:0/2331800009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a0101640 msgr2=0x7f28a01963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 --2- 172.21.15.116:0/2331800009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a0101640 0x7f28a01963a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f289c00b4b0 tx=0x7f289c00b7c0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 -- 172.21.15.116:0/2331800009 shutdown_connections 2024-03-04T16:10:14.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 --2- 172.21.15.116:0/2331800009 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f288c06d8a0 0x7f288c06fd60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 --2- 172.21.15.116:0/2331800009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a0101640 0x7f28a01963a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 --2- 172.21.15.116:0/2331800009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28a0103f70 0x7f28a01968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 -- 172.21.15.116:0/2331800009 >> 172.21.15.116:0/2331800009 conn(0x7f28a00fb030 msgr2=0x7f28a00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:14.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 -- 172.21.15.116:0/2331800009 shutdown_connections 2024-03-04T16:10:14.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.761+0000 7f28a7f84700 1 -- 172.21.15.116:0/2331800009 wait complete. 2024-03-04T16:10:14.772 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.865+0000 7fdca45de700 1 -- 172.21.15.116:0/3830654360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc9c0ffac0 msgr2=0x7fdc9c0ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.865+0000 7fdca45de700 1 --2- 172.21.15.116:0/3830654360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc9c0ffac0 0x7fdc9c0ffee0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400b0a0 tx=0x7fdc9400b3b0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.865+0000 7fdca45de700 1 -- 172.21.15.116:0/3830654360 shutdown_connections 2024-03-04T16:10:14.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.865+0000 7fdca45de700 1 --2- 172.21.15.116:0/3830654360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc9c100cc0 0x7fdc9c101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.865+0000 7fdca45de700 1 --2- 172.21.15.116:0/3830654360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc9c0ffac0 0x7fdc9c0ffee0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.865+0000 7fdca45de700 1 -- 172.21.15.116:0/3830654360 >> 172.21.15.116:0/3830654360 conn(0x7fdc9c0fb060 msgr2=0x7fdc9c0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:14.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.865+0000 7fdca45de700 1 -- 172.21.15.116:0/3830654360 shutdown_connections 2024-03-04T16:10:14.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.865+0000 7fdca45de700 1 -- 172.21.15.116:0/3830654360 wait complete. 2024-03-04T16:10:14.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.866+0000 7fdca45de700 1 Processor -- start 2024-03-04T16:10:14.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.866+0000 7fdca45de700 1 -- start start 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.866+0000 7fdca45de700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc9c0ffac0 0x7fdc9c069910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.866+0000 7fdca45de700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc9c100cc0 0x7fdc9c19b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.866+0000 7fdca45de700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c19b730 con 0x7fdc9c100cc0 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.866+0000 7fdca45de700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c19b870 con 0x7fdc9c0ffac0 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.866+0000 7fdca1b79700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc9c100cc0 0x7fdc9c19b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.866+0000 7fdca1b79700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc9c100cc0 0x7fdc9c19b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52860/0 (socket says 172.21.15.116:52860) 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.866+0000 7fdca1b79700 1 -- 172.21.15.116:0/1752182692 learned_addr learned my addr 172.21.15.116:0/1752182692 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.866+0000 7fdca237a700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc9c0ffac0 0x7fdc9c069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.867+0000 7fdca1b79700 1 -- 172.21.15.116:0/1752182692 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc9c0ffac0 msgr2=0x7fdc9c069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.867+0000 7fdca1b79700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc9c0ffac0 0x7fdc9c069910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.867+0000 7fdca1b79700 1 -- 172.21.15.116:0/1752182692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc94009d00 con 0x7fdc9c100cc0 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.867+0000 7fdca1b79700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc9c100cc0 0x7fdc9c19b160 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00d8d0 tx=0x7fdc8c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.867+0000 7fdc937fe700 1 -- 172.21.15.116:0/1752182692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc8c0098a0 con 0x7fdc9c100cc0 2024-03-04T16:10:14.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.867+0000 7fdc937fe700 1 -- 172.21.15.116:0/1752182692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdc8c010460 con 0x7fdc9c100cc0 2024-03-04T16:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.867+0000 7fdca45de700 1 -- 172.21.15.116:0/1752182692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9c19bb50 con 0x7fdc9c100cc0 2024-03-04T16:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.867+0000 7fdc937fe700 1 -- 172.21.15.116:0/1752182692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc8c00f5f0 con 0x7fdc9c100cc0 2024-03-04T16:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.867+0000 7fdca45de700 1 -- 172.21.15.116:0/1752182692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9c19c120 con 0x7fdc9c100cc0 2024-03-04T16:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.868+0000 7fdca45de700 1 -- 172.21.15.116:0/1752182692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9c108620 con 0x7fdc9c100cc0 2024-03-04T16:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.869+0000 7fdc937fe700 1 -- 172.21.15.116:0/1752182692 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fdc8c009a00 con 0x7fdc9c100cc0 2024-03-04T16:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.869+0000 7fdc937fe700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdc8806d8a0 0x7fdc8806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.869+0000 7fdc937fe700 1 -- 172.21.15.116:0/1752182692 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdc8c08d380 con 0x7fdc9c100cc0 2024-03-04T16:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.869+0000 7fdca237a700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdc8806d8a0 0x7fdc8806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.870+0000 7fdca237a700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdc8806d8a0 0x7fdc8806fd60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009fd0 tx=0x7fdc94000f40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:14.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:14.874+0000 7fdc937fe700 1 -- 172.21.15.116:0/1752182692 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdc8c05a6b0 con 0x7fdc9c100cc0 2024-03-04T16:10:15.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.086+0000 7fdca45de700 1 -- 172.21.15.116:0/1752182692 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc9c105610 con 0x7fdc8806d8a0 2024-03-04T16:10:15.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.087+0000 7fdc937fe700 1 -- 172.21.15.116:0/1752182692 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fdc9c105610 con 0x7fdc8806d8a0 2024-03-04T16:10:15.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.089+0000 7fdca45de700 1 -- 172.21.15.116:0/1752182692 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdc8806d8a0 msgr2=0x7fdc8806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.089+0000 7fdca45de700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdc8806d8a0 0x7fdc8806fd60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009fd0 tx=0x7fdc94000f40 comp rx=0 tx=0).stop 2024-03-04T16:10:15.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.089+0000 7fdca45de700 1 -- 172.21.15.116:0/1752182692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc9c100cc0 msgr2=0x7fdc9c19b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.089+0000 7fdca45de700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc9c100cc0 0x7fdc9c19b160 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00d8d0 tx=0x7fdc8c00dbe0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.089+0000 7fdca45de700 1 -- 172.21.15.116:0/1752182692 shutdown_connections 2024-03-04T16:10:15.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.089+0000 7fdca45de700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdc8806d8a0 0x7fdc8806fd60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.089+0000 7fdca45de700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc9c0ffac0 0x7fdc9c069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.089+0000 7fdca45de700 1 --2- 172.21.15.116:0/1752182692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc9c100cc0 0x7fdc9c19b160 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.089+0000 7fdca45de700 1 -- 172.21.15.116:0/1752182692 >> 172.21.15.116:0/1752182692 conn(0x7fdc9c0fb060 msgr2=0x7fdc9c103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:15.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.089+0000 7fdca45de700 1 -- 172.21.15.116:0/1752182692 shutdown_connections 2024-03-04T16:10:15.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.090+0000 7fdca45de700 1 -- 172.21.15.116:0/1752182692 wait complete. 2024-03-04T16:10:15.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.194+0000 7f3c318bb700 1 -- 172.21.15.116:0/3711557122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c2c100bc0 msgr2=0x7f3c2c100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.194+0000 7f3c318bb700 1 --2- 172.21.15.116:0/3711557122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c2c100bc0 0x7f3c2c100fe0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400b0a0 tx=0x7f3c2400b3b0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.194+0000 7f3c318bb700 1 -- 172.21.15.116:0/3711557122 shutdown_connections 2024-03-04T16:10:15.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.194+0000 7f3c318bb700 1 --2- 172.21.15.116:0/3711557122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c2c101dc0 0x7f3c2c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.194+0000 7f3c318bb700 1 --2- 172.21.15.116:0/3711557122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c2c100bc0 0x7f3c2c100fe0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.194+0000 7f3c318bb700 1 -- 172.21.15.116:0/3711557122 >> 172.21.15.116:0/3711557122 conn(0x7f3c2c0fc140 msgr2=0x7f3c2c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.194+0000 7f3c318bb700 1 -- 172.21.15.116:0/3711557122 shutdown_connections 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.194+0000 7f3c318bb700 1 -- 172.21.15.116:0/3711557122 wait complete. 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.194+0000 7f3c318bb700 1 Processor -- start 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c318bb700 1 -- start start 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c318bb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c2c100bc0 0x7f3c2c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c318bb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c2c101dc0 0x7f3c2c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c318bb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0736c0 con 0x7f3c2c101dc0 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c318bb700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c073800 con 0x7f3c2c100bc0 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c2a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c2c101dc0 0x7f3c2c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c2affd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c2c100bc0 0x7f3c2c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c2a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c2c101dc0 0x7f3c2c073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52876/0 (socket says 172.21.15.116:52876) 2024-03-04T16:10:15.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c2a7fc700 1 -- 172.21.15.116:0/4277055052 learned_addr learned my addr 172.21.15.116:0/4277055052 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:15.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c2a7fc700 1 -- 172.21.15.116:0/4277055052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c2c100bc0 msgr2=0x7f3c2c074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c2a7fc700 1 --2- 172.21.15.116:0/4277055052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c2c100bc0 0x7f3c2c074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c2a7fc700 1 -- 172.21.15.116:0/4277055052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c24009d00 con 0x7f3c2c101dc0 2024-03-04T16:10:15.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c2a7fc700 1 --2- 172.21.15.116:0/4277055052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c2c101dc0 0x7f3c2c073180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400e920 tx=0x7f3c1400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:15.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.195+0000 7f3c308b9700 1 -- 172.21.15.116:0/4277055052 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c1400c9d0 con 0x7f3c2c101dc0 2024-03-04T16:10:15.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.196+0000 7f3c308b9700 1 -- 172.21.15.116:0/4277055052 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3c14010430 con 0x7f3c2c101dc0 2024-03-04T16:10:15.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.196+0000 7f3c318bb700 1 -- 172.21.15.116:0/4277055052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2c073ae0 con 0x7f3c2c101dc0 2024-03-04T16:10:15.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.196+0000 7f3c308b9700 1 -- 172.21.15.116:0/4277055052 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c14018650 con 0x7f3c2c101dc0 2024-03-04T16:10:15.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.196+0000 7f3c318bb700 1 -- 172.21.15.116:0/4277055052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2c19e420 con 0x7f3c2c101dc0 2024-03-04T16:10:15.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.197+0000 7f3c318bb700 1 -- 172.21.15.116:0/4277055052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2c109720 con 0x7f3c2c101dc0 2024-03-04T16:10:15.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.197+0000 7f3c308b9700 1 -- 172.21.15.116:0/4277055052 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f3c140105a0 con 0x7f3c2c101dc0 2024-03-04T16:10:15.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.197+0000 7f3c308b9700 1 --2- 172.21.15.116:0/4277055052 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c1806d8a0 0x7f3c1806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:15.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.198+0000 7f3c308b9700 1 -- 172.21.15.116:0/4277055052 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3c14014070 con 0x7f3c2c101dc0 2024-03-04T16:10:15.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.198+0000 7f3c2affd700 1 --2- 172.21.15.116:0/4277055052 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c1806d8a0 0x7f3c1806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:15.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.198+0000 7f3c2affd700 1 --2- 172.21.15.116:0/4277055052 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c1806d8a0 0x7f3c1806fd60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009fd0 tx=0x7f3c24006000 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:15.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.202+0000 7f3c308b9700 1 -- 172.21.15.116:0/4277055052 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3c14052d20 con 0x7f3c2c101dc0 2024-03-04T16:10:15.400 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:15 smithi152 ceph-mon[82845]: Upgrade: Updating osd.5 2024-03-04T16:10:15.400 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:15 smithi152 ceph-mon[82845]: Deploying daemon osd.5 on smithi152 2024-03-04T16:10:15.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.402+0000 7f3c318bb700 1 -- 172.21.15.116:0/4277055052 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c2c106710 con 0x7f3c1806d8a0 2024-03-04T16:10:15.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.412+0000 7f3c308b9700 1 -- 172.21.15.116:0/4277055052 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3c2c106710 con 0x7f3c1806d8a0 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (13m) 91s ago 14m 25.4M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (15m) 91s ago 15m 8284k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (13m) 23s ago 13m 8292k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (3m) 91s ago 15m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (3m) 23s ago 13m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (13m) 91s ago 14m 87.8M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (9m) 91s ago 9m 19.9M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (9m) 91s ago 9m 16.1M - 18.2.1 5be31c24972a 0488284b6637 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (9m) 23s ago 9m 17.9M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (9m) 23s ago 9m 143M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:10:15.413 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (5m) 91s ago 16m 503M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (5m) 23s ago 13m 450M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (4m) 91s ago 16m 46.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (4m) 23s ago 13m 35.5M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (14m) 91s ago 14m 15.4M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (13m) 23s ago 13m 16.5M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (2m) 91s ago 12m 87.1M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (2m) 91s ago 12m 101M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (97s) 91s ago 11m 12.7M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (61s) 23s ago 11m 108M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6b01e97c8f9b 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (27s) 23s ago 11m 15.0M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 5ecc30e5a86c 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (10m) 23s ago 10m 228M 2240M 18.2.1 5be31c24972a 99a749ab705b 2024-03-04T16:10:15.414 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (5m) 91s ago 14m 63.0M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:10:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.414+0000 7f3c318bb700 1 -- 172.21.15.116:0/4277055052 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c1806d8a0 msgr2=0x7f3c1806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.414+0000 7f3c318bb700 1 --2- 172.21.15.116:0/4277055052 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c1806d8a0 0x7f3c1806fd60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009fd0 tx=0x7f3c24006000 comp rx=0 tx=0).stop 2024-03-04T16:10:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.414+0000 7f3c318bb700 1 -- 172.21.15.116:0/4277055052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c2c101dc0 msgr2=0x7f3c2c073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.414+0000 7f3c318bb700 1 --2- 172.21.15.116:0/4277055052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c2c101dc0 0x7f3c2c073180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400e920 tx=0x7f3c1400ec30 comp rx=0 tx=0).stop 2024-03-04T16:10:15.415 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:15 smithi116 ceph-mon[94164]: Upgrade: Updating osd.5 2024-03-04T16:10:15.415 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:15 smithi116 ceph-mon[94164]: Deploying daemon osd.5 on smithi152 2024-03-04T16:10:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.415+0000 7f3c318bb700 1 -- 172.21.15.116:0/4277055052 shutdown_connections 2024-03-04T16:10:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.415+0000 7f3c318bb700 1 --2- 172.21.15.116:0/4277055052 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c1806d8a0 0x7f3c1806fd60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.415+0000 7f3c318bb700 1 --2- 172.21.15.116:0/4277055052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c2c100bc0 0x7f3c2c074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.415+0000 7f3c318bb700 1 --2- 172.21.15.116:0/4277055052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c2c101dc0 0x7f3c2c073180 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.415+0000 7f3c318bb700 1 -- 172.21.15.116:0/4277055052 >> 172.21.15.116:0/4277055052 conn(0x7f3c2c0fc140 msgr2=0x7f3c2c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.415+0000 7f3c318bb700 1 -- 172.21.15.116:0/4277055052 shutdown_connections 2024-03-04T16:10:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.415+0000 7f3c318bb700 1 -- 172.21.15.116:0/4277055052 wait complete. 2024-03-04T16:10:15.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.518+0000 7f278b9df700 1 -- 172.21.15.116:0/4210149621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27840fec20 msgr2=0x7f27840ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.518+0000 7f278b9df700 1 --2- 172.21.15.116:0/4210149621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27840fec20 0x7f27840ff080 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2780009910 tx=0x7f2780009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:15.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.518+0000 7f278b9df700 1 -- 172.21.15.116:0/4210149621 shutdown_connections 2024-03-04T16:10:15.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.518+0000 7f278b9df700 1 --2- 172.21.15.116:0/4210149621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27840fec20 0x7f27840ff080 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.518+0000 7f278b9df700 1 --2- 172.21.15.116:0/4210149621 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27841021a0 0x7f27841025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.518+0000 7f278b9df700 1 -- 172.21.15.116:0/4210149621 >> 172.21.15.116:0/4210149621 conn(0x7f27840fa7c0 msgr2=0x7f27840fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:15.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.518+0000 7f278b9df700 1 -- 172.21.15.116:0/4210149621 shutdown_connections 2024-03-04T16:10:15.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.518+0000 7f278b9df700 1 -- 172.21.15.116:0/4210149621 wait complete. 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278b9df700 1 Processor -- start 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278b9df700 1 -- start start 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278b9df700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27840fec20 0x7f2784196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278b9df700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27841021a0 0x7f2784196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278b9df700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2784196f60 con 0x7f27840fec20 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278b9df700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27841970a0 con 0x7f27841021a0 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278977b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27840fec20 0x7f2784196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278977b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27840fec20 0x7f2784196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52894/0 (socket says 172.21.15.116:52894) 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278977b700 1 -- 172.21.15.116:0/3956912012 learned_addr learned my addr 172.21.15.116:0/3956912012 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f2788f7a700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27841021a0 0x7f2784196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:15.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278977b700 1 -- 172.21.15.116:0/3956912012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27841021a0 msgr2=0x7f2784196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278977b700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27841021a0 0x7f2784196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278977b700 1 -- 172.21.15.116:0/3956912012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27800095f0 con 0x7f27840fec20 2024-03-04T16:10:15.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278977b700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27840fec20 0x7f2784196400 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f277400d900 tx=0x7f277400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:15.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f277a7fc700 1 -- 172.21.15.116:0/3956912012 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277400b920 con 0x7f27840fec20 2024-03-04T16:10:15.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f277a7fc700 1 -- 172.21.15.116:0/3956912012 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2774010460 con 0x7f27840fec20 2024-03-04T16:10:15.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278b9df700 1 -- 172.21.15.116:0/3956912012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278419bb50 con 0x7f27840fec20 2024-03-04T16:10:15.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f277a7fc700 1 -- 172.21.15.116:0/3956912012 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277400f590 con 0x7f27840fec20 2024-03-04T16:10:15.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.519+0000 7f278b9df700 1 -- 172.21.15.116:0/3956912012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278419c0a0 con 0x7f27840fec20 2024-03-04T16:10:15.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.520+0000 7f278b9df700 1 -- 172.21.15.116:0/3956912012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2784109730 con 0x7f27840fec20 2024-03-04T16:10:15.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.521+0000 7f277a7fc700 1 -- 172.21.15.116:0/3956912012 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f27740105d0 con 0x7f27840fec20 2024-03-04T16:10:15.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.521+0000 7f277a7fc700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f277006d8a0 0x7f277006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:15.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.521+0000 7f277a7fc700 1 -- 172.21.15.116:0/3956912012 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f277408d4d0 con 0x7f27840fec20 2024-03-04T16:10:15.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.521+0000 7f2788f7a700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f277006d8a0 0x7f277006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:15.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.522+0000 7f2788f7a700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f277006d8a0 0x7f277006fd60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2780006010 tx=0x7f278000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:15.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.526+0000 7f277a7fc700 1 -- 172.21.15.116:0/3956912012 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f277405a800 con 0x7f27840fec20 2024-03-04T16:10:15.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.798+0000 7f278b9df700 1 -- 172.21.15.116:0/3956912012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f278419bce0 con 0x7f27840fec20 2024-03-04T16:10:15.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.799+0000 7f277a7fc700 1 -- 172.21.15.116:0/3956912012 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f278419bce0 con 0x7f27840fec20 2024-03-04T16:10:15.800 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:10:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:10:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:10:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:10:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:10:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:10:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:10:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:10:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-03-04T16:10:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 5 2024-03-04T16:10:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:10:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:10:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-03-04T16:10:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:10:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:10:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-03-04T16:10:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 9 2024-03-04T16:10:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:10:15.801 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:10:15.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 -- 172.21.15.116:0/3956912012 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f277006d8a0 msgr2=0x7f277006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f277006d8a0 0x7f277006fd60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2780006010 tx=0x7f278000b540 comp rx=0 tx=0).stop 2024-03-04T16:10:15.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 -- 172.21.15.116:0/3956912012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27840fec20 msgr2=0x7f2784196400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27840fec20 0x7f2784196400 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f277400d900 tx=0x7f277400dc10 comp rx=0 tx=0).stop 2024-03-04T16:10:15.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 -- 172.21.15.116:0/3956912012 shutdown_connections 2024-03-04T16:10:15.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f277006d8a0 0x7f277006fd60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27840fec20 0x7f2784196400 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 --2- 172.21.15.116:0/3956912012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27841021a0 0x7f2784196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 -- 172.21.15.116:0/3956912012 >> 172.21.15.116:0/3956912012 conn(0x7f27840fa7c0 msgr2=0x7f2784105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:15.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 -- 172.21.15.116:0/3956912012 shutdown_connections 2024-03-04T16:10:15.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.801+0000 7f278b9df700 1 -- 172.21.15.116:0/3956912012 wait complete. 2024-03-04T16:10:15.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.904+0000 7fb1d94e2700 1 -- 172.21.15.116:0/1921616749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d4100be0 msgr2=0x7fb1d4101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.904+0000 7fb1d94e2700 1 --2- 172.21.15.116:0/1921616749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d4100be0 0x7fb1d4101000 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009910 tx=0x7fb1c4009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:15.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.904+0000 7fb1d94e2700 1 -- 172.21.15.116:0/1921616749 shutdown_connections 2024-03-04T16:10:15.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.904+0000 7fb1d94e2700 1 --2- 172.21.15.116:0/1921616749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1d4101de0 0x7fb1d4102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.904+0000 7fb1d94e2700 1 --2- 172.21.15.116:0/1921616749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d4100be0 0x7fb1d4101000 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.904+0000 7fb1d94e2700 1 -- 172.21.15.116:0/1921616749 >> 172.21.15.116:0/1921616749 conn(0x7fb1d40fc140 msgr2=0x7fb1d40fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:15.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.904+0000 7fb1d94e2700 1 -- 172.21.15.116:0/1921616749 shutdown_connections 2024-03-04T16:10:15.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.904+0000 7fb1d94e2700 1 -- 172.21.15.116:0/1921616749 wait complete. 2024-03-04T16:10:15.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.904+0000 7fb1d94e2700 1 Processor -- start 2024-03-04T16:10:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d94e2700 1 -- start start 2024-03-04T16:10:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d94e2700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1d4100be0 0x7fb1d4196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d94e2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d4101de0 0x7fb1d41968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d94e2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d4196ef0 con 0x7fb1d4101de0 2024-03-04T16:10:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d94e2700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d4197030 con 0x7fb1d4100be0 2024-03-04T16:10:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d27fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d4101de0 0x7fb1d41968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d27fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d4101de0 0x7fb1d41968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52910/0 (socket says 172.21.15.116:52910) 2024-03-04T16:10:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d27fc700 1 -- 172.21.15.116:0/2641158711 learned_addr learned my addr 172.21.15.116:0/2641158711 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d2ffd700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1d4100be0 0x7fb1d4196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d27fc700 1 -- 172.21.15.116:0/2641158711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1d4100be0 msgr2=0x7fb1d4196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d27fc700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1d4100be0 0x7fb1d4196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d27fc700 1 -- 172.21.15.116:0/2641158711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c40095f0 con 0x7fb1d4101de0 2024-03-04T16:10:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1d27fc700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d4101de0 0x7fb1d41968d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800e920 tx=0x7fb1c800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.905+0000 7fb1bbfff700 1 -- 172.21.15.116:0/2641158711 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1c800c9d0 con 0x7fb1d4101de0 2024-03-04T16:10:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.906+0000 7fb1bbfff700 1 -- 172.21.15.116:0/2641158711 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb1c8010430 con 0x7fb1d4101de0 2024-03-04T16:10:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.906+0000 7fb1d94e2700 1 -- 172.21.15.116:0/2641158711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d419bae0 con 0x7fb1d4101de0 2024-03-04T16:10:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.906+0000 7fb1bbfff700 1 -- 172.21.15.116:0/2641158711 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1c8018650 con 0x7fb1d4101de0 2024-03-04T16:10:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.906+0000 7fb1d94e2700 1 -- 172.21.15.116:0/2641158711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d419c030 con 0x7fb1d4101de0 2024-03-04T16:10:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.906+0000 7fb1d94e2700 1 -- 172.21.15.116:0/2641158711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d4109740 con 0x7fb1d4101de0 2024-03-04T16:10:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.907+0000 7fb1bbfff700 1 -- 172.21.15.116:0/2641158711 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fb1c80105a0 con 0x7fb1d4101de0 2024-03-04T16:10:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.907+0000 7fb1bbfff700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb1bc06d8a0 0x7fb1bc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.908+0000 7fb1d2ffd700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb1bc06d8a0 0x7fb1bc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.908+0000 7fb1bbfff700 1 -- 172.21.15.116:0/2641158711 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb1c8014070 con 0x7fb1d4101de0 2024-03-04T16:10:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.908+0000 7fb1d2ffd700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb1bc06d8a0 0x7fb1bc06fd60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400b6f0 tx=0x7fb1c4005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:15.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:15.912+0000 7fb1bbfff700 1 -- 172.21.15.116:0/2641158711 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb1c8052d20 con 0x7fb1d4101de0 2024-03-04T16:10:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.144+0000 7fb1d94e2700 1 -- 172.21.15.116:0/2641158711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb1d4109930 con 0x7fb1d4101de0 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.144+0000 7fb1bbfff700 1 -- 172.21.15.116:0/2641158711 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1852 (secure 0 0 0) 0x7fb1c805ab00 con 0x7fb1d4101de0 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stdout:e14 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:10:16.145 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:10:16.146 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:10:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:10:16.148 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:10:16.148 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:10:16.148 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/4270616896,v1:172.21.15.116:6827/4270616896] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:10:16.148 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:10:16.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.146+0000 7fb1d94e2700 1 -- 172.21.15.116:0/2641158711 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb1bc06d8a0 msgr2=0x7fb1bc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.146+0000 7fb1d94e2700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb1bc06d8a0 0x7fb1bc06fd60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400b6f0 tx=0x7fb1c4005940 comp rx=0 tx=0).stop 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.146+0000 7fb1d94e2700 1 -- 172.21.15.116:0/2641158711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d4101de0 msgr2=0x7fb1d41968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.146+0000 7fb1d94e2700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d4101de0 0x7fb1d41968d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800e920 tx=0x7fb1c800ec30 comp rx=0 tx=0).stop 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.146+0000 7fb1d94e2700 1 -- 172.21.15.116:0/2641158711 shutdown_connections 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.146+0000 7fb1d94e2700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb1bc06d8a0 0x7fb1bc06fd60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.146+0000 7fb1d94e2700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1d4100be0 0x7fb1d4196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.146+0000 7fb1d94e2700 1 --2- 172.21.15.116:0/2641158711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d4101de0 0x7fb1d41968d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.146+0000 7fb1d94e2700 1 -- 172.21.15.116:0/2641158711 >> 172.21.15.116:0/2641158711 conn(0x7fb1d40fc140 msgr2=0x7fb1d4105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.147+0000 7fb1d94e2700 1 -- 172.21.15.116:0/2641158711 shutdown_connections 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.147+0000 7fb1d94e2700 1 -- 172.21.15.116:0/2641158711 wait complete. 2024-03-04T16:10:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-03-04T16:10:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.252+0000 7f7bc9d22700 1 -- 172.21.15.116:0/2730301585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc4103ff0 msgr2=0x7f7bc41063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.252+0000 7f7bc9d22700 1 --2- 172.21.15.116:0/2730301585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc4103ff0 0x7f7bc41063e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009910 tx=0x7f7bac009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.252+0000 7f7bc9d22700 1 -- 172.21.15.116:0/2730301585 shutdown_connections 2024-03-04T16:10:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.252+0000 7f7bc9d22700 1 --2- 172.21.15.116:0/2730301585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc4103ff0 0x7f7bc41063e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.252+0000 7f7bc9d22700 1 --2- 172.21.15.116:0/2730301585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bc41016c0 0x7f7bc4103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.252+0000 7f7bc9d22700 1 -- 172.21.15.116:0/2730301585 >> 172.21.15.116:0/2730301585 conn(0x7f7bc40fb030 msgr2=0x7f7bc40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.252+0000 7f7bc9d22700 1 -- 172.21.15.116:0/2730301585 shutdown_connections 2024-03-04T16:10:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.252+0000 7f7bc9d22700 1 -- 172.21.15.116:0/2730301585 wait complete. 2024-03-04T16:10:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.252+0000 7f7bc9d22700 1 Processor -- start 2024-03-04T16:10:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.252+0000 7f7bc9d22700 1 -- start start 2024-03-04T16:10:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc9d22700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc41016c0 0x7f7bc41962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc9d22700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bc4103ff0 0x7f7bc4196800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc9d22700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc4196d90 con 0x7f7bc41016c0 2024-03-04T16:10:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc9d22700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc4196ed0 con 0x7f7bc4103ff0 2024-03-04T16:10:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc41016c0 0x7f7bc41962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc41016c0 0x7f7bc41962c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52928/0 (socket says 172.21.15.116:52928) 2024-03-04T16:10:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc37fe700 1 -- 172.21.15.116:0/3343028461 learned_addr learned my addr 172.21.15.116:0/3343028461 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bbaffd700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bc4103ff0 0x7f7bc4196800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc37fe700 1 -- 172.21.15.116:0/3343028461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bc4103ff0 msgr2=0x7f7bc4196800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc37fe700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bc4103ff0 0x7f7bc4196800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc37fe700 1 -- 172.21.15.116:0/3343028461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac0095f0 con 0x7f7bc41016c0 2024-03-04T16:10:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc37fe700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc41016c0 0x7f7bc41962c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400c930 tx=0x7f7bb400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc17fa700 1 -- 172.21.15.116:0/3343028461 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bb4007ad0 con 0x7f7bc41016c0 2024-03-04T16:10:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc17fa700 1 -- 172.21.15.116:0/3343028461 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7bb400ce80 con 0x7f7bc41016c0 2024-03-04T16:10:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc9d22700 1 -- 172.21.15.116:0/3343028461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc419b990 con 0x7f7bc41016c0 2024-03-04T16:10:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc17fa700 1 -- 172.21.15.116:0/3343028461 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bb4018630 con 0x7f7bc41016c0 2024-03-04T16:10:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.253+0000 7f7bc9d22700 1 -- 172.21.15.116:0/3343028461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc419beb0 con 0x7f7bc41016c0 2024-03-04T16:10:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.254+0000 7f7bc9d22700 1 -- 172.21.15.116:0/3343028461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc4190520 con 0x7f7bc41016c0 2024-03-04T16:10:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.255+0000 7f7bc17fa700 1 -- 172.21.15.116:0/3343028461 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f7bb4007c30 con 0x7f7bc41016c0 2024-03-04T16:10:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.255+0000 7f7bc17fa700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f7bb006d8a0 0x7f7bb006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.255+0000 7f7bc17fa700 1 -- 172.21.15.116:0/3343028461 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7bb408d470 con 0x7f7bc41016c0 2024-03-04T16:10:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.255+0000 7f7bbaffd700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f7bb006d8a0 0x7f7bb006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.256+0000 7f7bbaffd700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f7bb006d8a0 0x7f7bb006fd60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00b5c0 tx=0x7f7bac0052a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.260+0000 7f7bc17fa700 1 -- 172.21.15.116:0/3343028461 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7bb405a7a0 con 0x7f7bc41016c0 2024-03-04T16:10:16.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.470+0000 7f7bc9d22700 1 -- 172.21.15.116:0/3343028461 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bc40611d0 con 0x7f7bb006d8a0 2024-03-04T16:10:16.471 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:16 smithi116 ceph-mon[94164]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:16.471 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:16 smithi116 ceph-mon[94164]: pgmap v159: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:16.471 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:16 smithi116 ceph-mon[94164]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:16.471 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:16 smithi116 ceph-mon[94164]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:16.471 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:16 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3956912012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:16.472 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:16 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2641158711' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:10:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.471+0000 7f7bc17fa700 1 -- 172.21.15.116:0/3343028461 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f7bc40611d0 con 0x7f7bb006d8a0 2024-03-04T16:10:16.472 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:10:16.472 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:10:16.472 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:10:16.472 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:10:16.472 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:10:16.473 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:10:16.473 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:10:16.473 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:10:16.473 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:10:16.473 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "11/23 daemons upgraded", 2024-03-04T16:10:16.473 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:10:16.473 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:10:16.473 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:10:16.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 -- 172.21.15.116:0/3343028461 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f7bb006d8a0 msgr2=0x7f7bb006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f7bb006d8a0 0x7f7bb006fd60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00b5c0 tx=0x7f7bac0052a0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 -- 172.21.15.116:0/3343028461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc41016c0 msgr2=0x7f7bc41962c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc41016c0 0x7f7bc41962c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400c930 tx=0x7f7bb400cc40 comp rx=0 tx=0).stop 2024-03-04T16:10:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 -- 172.21.15.116:0/3343028461 shutdown_connections 2024-03-04T16:10:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f7bb006d8a0 0x7f7bb006fd60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc41016c0 0x7f7bc41962c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 --2- 172.21.15.116:0/3343028461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bc4103ff0 0x7f7bc4196800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 -- 172.21.15.116:0/3343028461 >> 172.21.15.116:0/3343028461 conn(0x7f7bc40fb030 msgr2=0x7f7bc40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 -- 172.21.15.116:0/3343028461 shutdown_connections 2024-03-04T16:10:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.474+0000 7f7bc9d22700 1 -- 172.21.15.116:0/3343028461 wait complete. 2024-03-04T16:10:16.533 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:16 smithi152 ceph-mon[82845]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:16.534 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:16 smithi152 ceph-mon[82845]: pgmap v159: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:16.534 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:16 smithi152 ceph-mon[82845]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:16.534 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:16 smithi152 ceph-mon[82845]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:16.534 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:16 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3956912012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:16.534 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:16 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2641158711' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:10:16.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.578+0000 7f18a873c700 1 -- 172.21.15.116:0/180231516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a00ffdf0 msgr2=0x7f18a0100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:16.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.578+0000 7f18a873c700 1 --2- 172.21.15.116:0/180231516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a00ffdf0 0x7f18a0100270 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1890009910 tx=0x7f1890009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:16.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.578+0000 7f18a873c700 1 -- 172.21.15.116:0/180231516 shutdown_connections 2024-03-04T16:10:16.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.578+0000 7f18a873c700 1 --2- 172.21.15.116:0/180231516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a00ffdf0 0x7f18a0100270 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.578+0000 7f18a873c700 1 --2- 172.21.15.116:0/180231516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18a00ff490 0x7f18a00ff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.578+0000 7f18a873c700 1 -- 172.21.15.116:0/180231516 >> 172.21.15.116:0/180231516 conn(0x7f18a00fb050 msgr2=0x7f18a00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.578+0000 7f18a873c700 1 -- 172.21.15.116:0/180231516 shutdown_connections 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.578+0000 7f18a873c700 1 -- 172.21.15.116:0/180231516 wait complete. 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.578+0000 7f18a873c700 1 Processor -- start 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.578+0000 7f18a873c700 1 -- start start 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a873c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18a00ff490 0x7f18a0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a873c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a00ffdf0 0x7f18a0198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a873c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a0199090 con 0x7f18a00ffdf0 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a873c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a01991d0 con 0x7f18a00ff490 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a5cd7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a00ffdf0 0x7f18a0198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a5cd7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a00ffdf0 0x7f18a0198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52934/0 (socket says 172.21.15.116:52934) 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a5cd7700 1 -- 172.21.15.116:0/1736010115 learned_addr learned my addr 172.21.15.116:0/1736010115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a64d8700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18a00ff490 0x7f18a0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:16.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a5cd7700 1 -- 172.21.15.116:0/1736010115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18a00ff490 msgr2=0x7f18a0198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a5cd7700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18a00ff490 0x7f18a0198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a5cd7700 1 -- 172.21.15.116:0/1736010115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18900095f0 con 0x7f18a00ffdf0 2024-03-04T16:10:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a5cd7700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a00ffdf0 0x7f18a0198a70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1890000c00 tx=0x7f189000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18977fe700 1 -- 172.21.15.116:0/1736010115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f189001d070 con 0x7f18a00ffdf0 2024-03-04T16:10:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18977fe700 1 -- 172.21.15.116:0/1736010115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1890005dc0 con 0x7f18a00ffdf0 2024-03-04T16:10:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a873c700 1 -- 172.21.15.116:0/1736010115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a019dc20 con 0x7f18a00ffdf0 2024-03-04T16:10:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18977fe700 1 -- 172.21.15.116:0/1736010115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18900226a0 con 0x7f18a00ffdf0 2024-03-04T16:10:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.579+0000 7f18a873c700 1 -- 172.21.15.116:0/1736010115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a019e110 con 0x7f18a00ffdf0 2024-03-04T16:10:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.580+0000 7f18a873c700 1 -- 172.21.15.116:0/1736010115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a010b910 con 0x7f18a00ffdf0 2024-03-04T16:10:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.581+0000 7f18977fe700 1 -- 172.21.15.116:0/1736010115 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f18900038f0 con 0x7f18a00ffdf0 2024-03-04T16:10:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.581+0000 7f18977fe700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f188c06d8a0 0x7f188c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.581+0000 7f18977fe700 1 -- 172.21.15.116:0/1736010115 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f189008e990 con 0x7f18a00ffdf0 2024-03-04T16:10:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.581+0000 7f18a64d8700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f188c06d8a0 0x7f188c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.582+0000 7f18a64d8700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f188c06d8a0 0x7f188c06fd60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f189800f540 tx=0x7f189800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.586+0000 7f18977fe700 1 -- 172.21.15.116:0/1736010115 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f189005bcc0 con 0x7f18a00ffdf0 2024-03-04T16:10:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.859+0000 7f18a873c700 1 -- 172.21.15.116:0/1736010115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f18a010bb00 con 0x7f18a00ffdf0 2024-03-04T16:10:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.860+0000 7f18977fe700 1 -- 172.21.15.116:0/1736010115 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f1890003680 con 0x7f18a00ffdf0 2024-03-04T16:10:16.861 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:10:16.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 -- 172.21.15.116:0/1736010115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f188c06d8a0 msgr2=0x7f188c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:16.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f188c06d8a0 0x7f188c06fd60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f189800f540 tx=0x7f189800b040 comp rx=0 tx=0).stop 2024-03-04T16:10:16.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 -- 172.21.15.116:0/1736010115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a00ffdf0 msgr2=0x7f18a0198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:16.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a00ffdf0 0x7f18a0198a70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1890000c00 tx=0x7f189000f690 comp rx=0 tx=0).stop 2024-03-04T16:10:16.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 -- 172.21.15.116:0/1736010115 shutdown_connections 2024-03-04T16:10:16.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f188c06d8a0 0x7f188c06fd60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18a00ff490 0x7f18a0198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 --2- 172.21.15.116:0/1736010115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a00ffdf0 0x7f18a0198a70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:16.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 -- 172.21.15.116:0/1736010115 >> 172.21.15.116:0/1736010115 conn(0x7f18a00fb050 msgr2=0x7f18a0107ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:16.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 -- 172.21.15.116:0/1736010115 shutdown_connections 2024-03-04T16:10:16.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:16.862+0000 7f18a873c700 1 -- 172.21.15.116:0/1736010115 wait complete. 2024-03-04T16:10:17.170 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:16 smithi152 systemd[1]: Stopping Ceph osd.5 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:10:17.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:17 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1736010115' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:10:17.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:17 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1736010115' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:10:17.607 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:17 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[56761]: 2024-03-04T16:10:17.195+0000 7fbb6f819700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:10:17.607 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:17 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[56761]: 2024-03-04T16:10:17.195+0000 7fbb6f819700 -1 osd.5 75 *** Got signal Terminated *** 2024-03-04T16:10:17.608 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:17 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[56761]: 2024-03-04T16:10:17.195+0000 7fbb6f819700 -1 osd.5 75 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:10:18.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:18 smithi116 ceph-mon[94164]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:18.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:18 smithi116 ceph-mon[94164]: pgmap v160: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:18.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:18 smithi116 ceph-mon[94164]: osd.5 marked itself down and dead 2024-03-04T16:10:18.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:18 smithi152 ceph-mon[82845]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:18.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:18 smithi152 ceph-mon[82845]: pgmap v160: 65 pgs: 65 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:18.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:18 smithi152 ceph-mon[82845]: osd.5 marked itself down and dead 2024-03-04T16:10:19.015 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:18 smithi152 podman[102304]: 2024-03-04 16:10:18.732254891 +0000 UTC m=+1.662175309 container died 99a749ab705b87abd0bbd50ea1a548655806934b8c1eb3e4ab67e2e43ad55463 (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1) 2024-03-04T16:10:19.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:19 smithi152 ceph-mon[82845]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:10:19.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:19 smithi152 ceph-mon[82845]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-04T16:10:19.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:19 smithi152 ceph-mon[82845]: osdmap e76: 6 total, 5 up, 6 in 2024-03-04T16:10:19.357 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:19 smithi152 podman[102304]: 2024-03-04 16:10:19.130185603 +0000 UTC m=+2.060106018 container cleanup 99a749ab705b87abd0bbd50ea1a548655806934b8c1eb3e4ab67e2e43ad55463 (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-03-04T16:10:19.358 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:19 smithi152 bash[102304]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5 2024-03-04T16:10:19.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:19 smithi116 ceph-mon[94164]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:10:19.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:19 smithi116 ceph-mon[94164]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-04T16:10:19.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:19 smithi116 ceph-mon[94164]: osdmap e76: 6 total, 5 up, 6 in 2024-03-04T16:10:19.672 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:19 smithi152 podman[102323]: 2024-03-04 16:10:19.381755451 +0000 UTC m=+0.648632870 container remove 99a749ab705b87abd0bbd50ea1a548655806934b8c1eb3e4ab67e2e43ad55463 (image=quay.io/ceph/ceph@sha256:ad0cdfebcf37ce5b53cd2ac0dff0139660c294299187cb382582f4189b434eaf, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222) 2024-03-04T16:10:19.935 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:19 smithi152 podman[102412]: 2024-03-04 16:10:19.575815402 +0000 UTC m=+0.018867210 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:10:19.936 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:19 smithi152 podman[102412]: 2024-03-04 16:10:19.674377194 +0000 UTC m=+0.117429000 container create c80ea29c78ceb2591eabb203f67821e527440908044786dd7f797fad688afef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:10:19.936 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:19 smithi152 podman[102412]: 2024-03-04 16:10:19.935828975 +0000 UTC m=+0.378880782 container init c80ea29c78ceb2591eabb203f67821e527440908044786dd7f797fad688afef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-04T16:10:20.229 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:20 smithi152 ceph-mon[82845]: pgmap v162: 65 pgs: 3 peering, 12 stale+active+clean, 50 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:20.229 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:20 smithi152 ceph-mon[82845]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:10:20.229 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:20 smithi152 ceph-mon[82845]: osdmap e77: 6 total, 5 up, 6 in 2024-03-04T16:10:20.229 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:19 smithi152 podman[102412]: 2024-03-04 16:10:19.93918158 +0000 UTC m=+0.382233388 container start c80ea29c78ceb2591eabb203f67821e527440908044786dd7f797fad688afef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-03-04T16:10:20.229 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:20 smithi152 podman[102412]: 2024-03-04 16:10:20.02746463 +0000 UTC m=+0.470516450 container attach c80ea29c78ceb2591eabb203f67821e527440908044786dd7f797fad688afef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-04T16:10:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:20 smithi116 ceph-mon[94164]: pgmap v162: 65 pgs: 3 peering, 12 stale+active+clean, 50 active+clean; 212 MiB data, 975 MiB used, 535 GiB / 536 GiB avail 2024-03-04T16:10:20.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:20 smithi116 ceph-mon[94164]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:10:20.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:20 smithi116 ceph-mon[94164]: osdmap e77: 6 total, 5 up, 6 in 2024-03-04T16:10:20.514 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:20 smithi152 podman[102412]: 2024-03-04 16:10:20.229669374 +0000 UTC m=+0.672721186 container died c80ea29c78ceb2591eabb203f67821e527440908044786dd7f797fad688afef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-04T16:10:21.227 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:20 smithi152 podman[102439]: 2024-03-04 16:10:20.890312316 +0000 UTC m=+0.653422576 container remove c80ea29c78ceb2591eabb203f67821e527440908044786dd7f797fad688afef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-04T16:10:21.227 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:20 smithi152 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-04T16:10:21.227 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:20 smithi152 systemd[1]: Stopped Ceph osd.5 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:10:21.227 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:20 smithi152 systemd[1]: ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.5.service: Consumed 1min 7.352s CPU time. 2024-03-04T16:10:21.228 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:21 smithi152 systemd[1]: Starting Ceph osd.5 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:10:21.486 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:21 smithi152 ceph-mon[82845]: Health check failed: Degraded data redundancy: 34/231 objects degraded (14.719%), 11 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:21.487 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:21 smithi152 podman[102539]: 2024-03-04 16:10:21.230539431 +0000 UTC m=+0.019201345 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:10:21.487 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:21 smithi152 podman[102539]: 2024-03-04 16:10:21.358510571 +0000 UTC m=+0.147172484 container create 75060c66f241fa6bdee7af29e3b738983c5b5ef005adabf56d1a1f448290b185 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-04T16:10:21.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:21 smithi116 ceph-mon[94164]: Health check failed: Degraded data redundancy: 34/231 objects degraded (14.719%), 11 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:21.857 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:21 smithi152 podman[102539]: 2024-03-04 16:10:21.586110703 +0000 UTC m=+0.374772617 container init 75060c66f241fa6bdee7af29e3b738983c5b5ef005adabf56d1a1f448290b185 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=) 2024-03-04T16:10:21.858 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:21 smithi152 podman[102539]: 2024-03-04 16:10:21.589399196 +0000 UTC m=+0.378061118 container start 75060c66f241fa6bdee7af29e3b738983c5b5ef005adabf56d1a1f448290b185 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:10:21.858 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:21 smithi152 podman[102539]: 2024-03-04 16:10:21.669411253 +0000 UTC m=+0.458073185 container attach 75060c66f241fa6bdee7af29e3b738983c5b5ef005adabf56d1a1f448290b185 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:10:22.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:22 smithi152 ceph-mon[82845]: pgmap v164: 65 pgs: 11 active+undersized+degraded, 3 peering, 2 stale+active+clean, 11 active+undersized, 38 active+clean; 212 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 34/231 objects degraded (14.719%) 2024-03-04T16:10:22.358 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate[102555]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-04T16:10:22.358 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate[102555]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-04T16:10:22.358 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 bash[102539]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-04T16:10:22.358 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 bash[102539]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-04T16:10:22.358 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 bash[102539]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-04T16:10:22.358 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 bash[102539]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-04T16:10:22.358 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 bash[102539]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-03-04T16:10:22.358 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 bash[102539]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-04T16:10:22.358 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 bash[102539]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-04T16:10:22.359 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate[102555]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-04T16:10:22.359 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate[102555]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-04T16:10:22.359 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate[102555]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-03-04T16:10:22.359 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate[102555]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-04T16:10:22.359 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate[102555]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-04T16:10:22.359 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 podman[102539]: 2024-03-04 16:10:22.056593616 +0000 UTC m=+0.845255529 container died 75060c66f241fa6bdee7af29e3b738983c5b5ef005adabf56d1a1f448290b185 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-04T16:10:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:22 smithi116 ceph-mon[94164]: pgmap v164: 65 pgs: 11 active+undersized+degraded, 3 peering, 2 stale+active+clean, 11 active+undersized, 38 active+clean; 212 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 34/231 objects degraded (14.719%) 2024-03-04T16:10:23.042 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 podman[102681]: 2024-03-04 16:10:22.699026763 +0000 UTC m=+0.637202614 container remove 75060c66f241fa6bdee7af29e3b738983c5b5ef005adabf56d1a1f448290b185 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-03-04T16:10:23.043 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 podman[102746]: 2024-03-04 16:10:22.814411418 +0000 UTC m=+0.018590282 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:10:23.043 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:22 smithi152 podman[102746]: 2024-03-04 16:10:22.925094259 +0000 UTC m=+0.129273154 container create 1083700b476d11f516829fbff21ed94dc01632106053c9336e91960db89bc22a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-03-04T16:10:23.607 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:23 smithi152 podman[102746]: 2024-03-04 16:10:23.135963078 +0000 UTC m=+0.340141927 container init 1083700b476d11f516829fbff21ed94dc01632106053c9336e91960db89bc22a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-04T16:10:23.608 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:23 smithi152 podman[102746]: 2024-03-04 16:10:23.139125131 +0000 UTC m=+0.343303990 container start 1083700b476d11f516829fbff21ed94dc01632106053c9336e91960db89bc22a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0) 2024-03-04T16:10:23.608 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:23 smithi152 ceph-osd[102764]: -- 172.21.15.152:0/219758876 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562f47d4f680 con 0x562f47d18c00 2024-03-04T16:10:23.608 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:23 smithi152 bash[102746]: 1083700b476d11f516829fbff21ed94dc01632106053c9336e91960db89bc22a 2024-03-04T16:10:23.608 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:23 smithi152 systemd[1]: Started Ceph osd.5 for 4960653c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:10:24.088 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:23 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:10:23.686+0000 7ff9c9ccf7c0 -1 Falling back to public interface 2024-03-04T16:10:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:24 smithi152 ceph-mon[82845]: pgmap v165: 65 pgs: 11 active+undersized+degraded, 3 peering, 14 active+undersized, 37 active+clean; 212 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 34/231 objects degraded (14.719%) 2024-03-04T16:10:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:24 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:24 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:24 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:24.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:24 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:24.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:24 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:24 smithi116 ceph-mon[94164]: pgmap v165: 65 pgs: 11 active+undersized+degraded, 3 peering, 14 active+undersized, 37 active+clean; 212 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 34/231 objects degraded (14.719%) 2024-03-04T16:10:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:24 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:24.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:24 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:24 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:24 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:24.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:24 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:25 smithi116 ceph-mon[94164]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-04T16:10:25.536 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:25 smithi152 ceph-mon[82845]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-04T16:10:26.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:26 smithi116 ceph-mon[94164]: pgmap v166: 65 pgs: 12 active+undersized+degraded, 16 active+undersized, 37 active+clean; 212 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 35/231 objects degraded (15.152%) 2024-03-04T16:10:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:26 smithi152 ceph-mon[82845]: pgmap v166: 65 pgs: 12 active+undersized+degraded, 16 active+undersized, 37 active+clean; 212 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 35/231 objects degraded (15.152%) 2024-03-04T16:10:27.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:27 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:27.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:27 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:27.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:27 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:27.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:27 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:28.380 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:28 smithi152 ceph-mon[82845]: pgmap v167: 65 pgs: 12 active+undersized+degraded, 16 active+undersized, 37 active+clean; 212 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 35/231 objects degraded (15.152%) 2024-03-04T16:10:28.380 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:28.380 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:28.380 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:28 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:10:28.180+0000 7ff9c9ccf7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-04T16:10:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:28 smithi116 ceph-mon[94164]: pgmap v167: 65 pgs: 12 active+undersized+degraded, 16 active+undersized, 37 active+clean; 212 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 35/231 objects degraded (15.152%) 2024-03-04T16:10:28.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:28.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:29.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:28 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:10:28.651+0000 7ff9c9ccf7c0 -1 osd.5 75 log_to_monitors true 2024-03-04T16:10:29.398 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:29 smithi152 ceph-mon[82845]: from='osd.5 [v2:172.21.15.152:6816/1180242057,v1:172.21.15.152:6817/1180242057]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:10:29.398 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:29 smithi152 ceph-mon[82845]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:10:29.398 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:29 smithi152 ceph-mon[82845]: Health check update: Degraded data redundancy: 35/231 objects degraded (15.152%), 12 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:29.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:29 smithi116 ceph-mon[94164]: from='osd.5 [v2:172.21.15.152:6816/1180242057,v1:172.21.15.152:6817/1180242057]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:10:29.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:29 smithi116 ceph-mon[94164]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:10:29.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:29 smithi116 ceph-mon[94164]: Health check update: Degraded data redundancy: 35/231 objects degraded (15.152%), 12 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:30.357 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:10:30 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:10:30.190+0000 7ff9b711f700 -1 osd.5 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:10:30.638 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:30 smithi152 ceph-mon[82845]: pgmap v168: 65 pgs: 12 active+undersized+degraded, 16 active+undersized, 37 active+clean; 212 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 35/231 objects degraded (15.152%) 2024-03-04T16:10:30.638 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:30 smithi152 ceph-mon[82845]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:10:30.639 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:30 smithi152 ceph-mon[82845]: osdmap e78: 6 total, 5 up, 6 in 2024-03-04T16:10:30.639 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:30 smithi152 ceph-mon[82845]: from='osd.5 [v2:172.21.15.152:6816/1180242057,v1:172.21.15.152:6817/1180242057]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:10:30.639 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:30 smithi152 ceph-mon[82845]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:10:30.639 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:30 smithi152 ceph-mon[82845]: from='osd.5 ' entity='osd.5' 2024-03-04T16:10:30.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:30 smithi116 ceph-mon[94164]: pgmap v168: 65 pgs: 12 active+undersized+degraded, 16 active+undersized, 37 active+clean; 212 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 35/231 objects degraded (15.152%) 2024-03-04T16:10:30.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:30 smithi116 ceph-mon[94164]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:10:30.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:30 smithi116 ceph-mon[94164]: osdmap e78: 6 total, 5 up, 6 in 2024-03-04T16:10:30.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:30 smithi116 ceph-mon[94164]: from='osd.5 [v2:172.21.15.152:6816/1180242057,v1:172.21.15.152:6817/1180242057]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:10:30.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:30 smithi116 ceph-mon[94164]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T16:10:30.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:30 smithi116 ceph-mon[94164]: from='osd.5 ' entity='osd.5' 2024-03-04T16:10:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:31 smithi152 ceph-mon[82845]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:10:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:31 smithi152 ceph-mon[82845]: osd.5 [v2:172.21.15.152:6816/1180242057,v1:172.21.15.152:6817/1180242057] boot 2024-03-04T16:10:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:31 smithi152 ceph-mon[82845]: osdmap e79: 6 total, 6 up, 6 in 2024-03-04T16:10:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:31.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:31 smithi116 ceph-mon[94164]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:10:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:31 smithi116 ceph-mon[94164]: osd.5 [v2:172.21.15.152:6816/1180242057,v1:172.21.15.152:6817/1180242057] boot 2024-03-04T16:10:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:31 smithi116 ceph-mon[94164]: osdmap e79: 6 total, 6 up, 6 in 2024-03-04T16:10:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:10:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:10:31.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: pgmap v170: 65 pgs: 12 active+undersized+degraded, 16 active+undersized, 37 active+clean; 212 MiB data, 895 MiB used, 536 GiB / 536 GiB avail; 35/231 objects degraded (15.152%) 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-04T16:10:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: osdmap e80: 6 total, 6 up, 6 in 2024-03-04T16:10:32.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi116.pjrmzv"]}]: dispatch 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: pgmap v170: 65 pgs: 12 active+undersized+degraded, 16 active+undersized, 37 active+clean; 212 MiB data, 895 MiB used, 536 GiB / 536 GiB avail; 35/231 objects degraded (15.152%) 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-04T16:10:32.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: osdmap e80: 6 total, 6 up, 6 in 2024-03-04T16:10:32.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi116.pjrmzv"]}]: dispatch 2024-03-04T16:10:33.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:33 smithi116 ceph-mon[94164]: Upgrade: Setting container_image for all osd 2024-03-04T16:10:33.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:33 smithi116 ceph-mon[94164]: Upgrade: Setting require_osd_release to 19 squid 2024-03-04T16:10:33.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:33 smithi116 ceph-mon[94164]: Upgrade: It appears safe to stop mds.cephfs.smithi116.pjrmzv 2024-03-04T16:10:33.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:33 smithi152 ceph-mon[82845]: Upgrade: Setting container_image for all osd 2024-03-04T16:10:33.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:33 smithi152 ceph-mon[82845]: Upgrade: Setting require_osd_release to 19 squid 2024-03-04T16:10:33.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:33 smithi152 ceph-mon[82845]: Upgrade: It appears safe to stop mds.cephfs.smithi116.pjrmzv 2024-03-04T16:10:34.695 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:34 smithi116 ceph-mon[94164]: pgmap v173: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 895 MiB used, 536 GiB / 536 GiB avail; 35/231 objects degraded (15.152%) 2024-03-04T16:10:34.695 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:34 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:34.695 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:34 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.pjrmzv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:10:34.695 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:34 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:34.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:34 smithi152 ceph-mon[82845]: pgmap v173: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 212 MiB data, 895 MiB used, 536 GiB / 536 GiB avail; 35/231 objects degraded (15.152%) 2024-03-04T16:10:34.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:34 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:34.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:34 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.pjrmzv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:10:34.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:34 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:35.619 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:35 smithi116 ceph-mon[94164]: Upgrade: Updating mds.cephfs.smithi116.pjrmzv 2024-03-04T16:10:35.619 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:35 smithi116 ceph-mon[94164]: Deploying daemon mds.cephfs.smithi116.pjrmzv on smithi116 2024-03-04T16:10:35.672 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:35 smithi152 ceph-mon[82845]: Upgrade: Updating mds.cephfs.smithi116.pjrmzv 2024-03-04T16:10:35.672 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:35 smithi152 ceph-mon[82845]: Deploying daemon mds.cephfs.smithi116.pjrmzv on smithi116 2024-03-04T16:10:36.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:36 smithi116 ceph-mon[94164]: pgmap v174: 65 pgs: 5 active+undersized+degraded, 2 active+undersized, 58 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail; 18/231 objects degraded (7.792%) 2024-03-04T16:10:36.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:36 smithi116 ceph-mon[94164]: Health check update: Degraded data redundancy: 18/231 objects degraded (7.792%), 5 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:36.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:36 smithi152 ceph-mon[82845]: pgmap v174: 65 pgs: 5 active+undersized+degraded, 2 active+undersized, 58 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail; 18/231 objects degraded (7.792%) 2024-03-04T16:10:36.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:36 smithi152 ceph-mon[82845]: Health check update: Degraded data redundancy: 18/231 objects degraded (7.792%), 5 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:37 smithi152 ceph-mon[82845]: osdmap e81: 6 total, 6 up, 6 in 2024-03-04T16:10:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:37 smithi152 ceph-mon[82845]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 2 up:standby 2024-03-04T16:10:37.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:37 smithi116 ceph-mon[94164]: osdmap e81: 6 total, 6 up, 6 in 2024-03-04T16:10:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:37 smithi116 ceph-mon[94164]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 2 up:standby 2024-03-04T16:10:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:38 smithi116 ceph-mon[94164]: pgmap v176: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:38 smithi116 ceph-mon[94164]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/231 objects degraded (7.792%), 5 pgs degraded) 2024-03-04T16:10:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:38 smithi116 ceph-mon[94164]: Cluster is now healthy 2024-03-04T16:10:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:38.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:38 smithi152 ceph-mon[82845]: pgmap v176: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:38 smithi152 ceph-mon[82845]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/231 objects degraded (7.792%), 5 pgs degraded) 2024-03-04T16:10:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:38 smithi152 ceph-mon[82845]: Cluster is now healthy 2024-03-04T16:10:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:38.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:39.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:10:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:10:40.616 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:40 smithi116 ceph-mon[94164]: pgmap v177: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:40.728 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:40 smithi152 ceph-mon[82845]: pgmap v177: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:42.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:42 smithi116 ceph-mon[94164]: pgmap v178: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:42.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:42.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:42.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:42.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:42 smithi116 ceph-mon[94164]: mds.? [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] up:boot 2024-03-04T16:10:42.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:42 smithi116 ceph-mon[94164]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 3 up:standby 2024-03-04T16:10:42.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:10:42.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:42 smithi152 ceph-mon[82845]: pgmap v178: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:42.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:42.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:42.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:42.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:42 smithi152 ceph-mon[82845]: mds.? [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] up:boot 2024-03-04T16:10:42.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:42 smithi152 ceph-mon[82845]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 3 up:standby 2024-03-04T16:10:42.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:10:44.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:44 smithi116 ceph-mon[94164]: pgmap v179: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:44.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:44 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:44.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:44 smithi152 ceph-mon[82845]: pgmap v179: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:44.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:44 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:46.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:46 smithi116 ceph-mon[94164]: pgmap v180: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:46.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:46 smithi152 ceph-mon[82845]: pgmap v180: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.973+0000 7f76d917c700 1 -- 172.21.15.116:0/4165030074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4103fd0 msgr2=0x7f76d41063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.973+0000 7f76d917c700 1 --2- 172.21.15.116:0/4165030074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4103fd0 0x7f76d41063c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009910 tx=0x7f76c4009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.973+0000 7f76d917c700 1 -- 172.21.15.116:0/4165030074 shutdown_connections 2024-03-04T16:10:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.973+0000 7f76d917c700 1 --2- 172.21.15.116:0/4165030074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4103fd0 0x7f76d41063c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.973+0000 7f76d917c700 1 --2- 172.21.15.116:0/4165030074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f76d41016a0 0x7f76d4103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.973+0000 7f76d917c700 1 -- 172.21.15.116:0/4165030074 >> 172.21.15.116:0/4165030074 conn(0x7f76d40fb030 msgr2=0x7f76d40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.973+0000 7f76d917c700 1 -- 172.21.15.116:0/4165030074 shutdown_connections 2024-03-04T16:10:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.973+0000 7f76d917c700 1 -- 172.21.15.116:0/4165030074 wait complete. 2024-03-04T16:10:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d917c700 1 Processor -- start 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d917c700 1 -- start start 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d917c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f76d41016a0 0x7f76d4196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d917c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4103fd0 0x7f76d4196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d917c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d4196e00 con 0x7f76d4103fd0 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d917c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d4196f40 con 0x7f76d41016a0 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4103fd0 0x7f76d4196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4103fd0 0x7f76d4196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52800/0 (socket says 172.21.15.116:52800) 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d259c700 1 -- 172.21.15.116:0/3661705595 learned_addr learned my addr 172.21.15.116:0/3661705595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d259c700 1 -- 172.21.15.116:0/3661705595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f76d41016a0 msgr2=0x7f76d4196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d259c700 1 --2- 172.21.15.116:0/3661705595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f76d41016a0 0x7f76d4196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.974+0000 7f76d259c700 1 -- 172.21.15.116:0/3661705595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c40095f0 con 0x7f76d4103fd0 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.975+0000 7f76d259c700 1 --2- 172.21.15.116:0/3661705595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4103fd0 0x7f76d4196870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f76c400bd70 tx=0x7f76c400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.975+0000 7f76bbfff700 1 -- 172.21.15.116:0/3661705595 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76c401d070 con 0x7f76d4103fd0 2024-03-04T16:10:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.975+0000 7f76bbfff700 1 -- 172.21.15.116:0/3661705595 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f76c4022470 con 0x7f76d4103fd0 2024-03-04T16:10:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.975+0000 7f76bbfff700 1 -- 172.21.15.116:0/3661705595 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76c400f670 con 0x7f76d4103fd0 2024-03-04T16:10:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.975+0000 7f76d917c700 1 -- 172.21.15.116:0/3661705595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d419b9a0 con 0x7f76d4103fd0 2024-03-04T16:10:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.975+0000 7f76d917c700 1 -- 172.21.15.116:0/3661705595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d419be10 con 0x7f76d4103fd0 2024-03-04T16:10:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.976+0000 7f76d917c700 1 -- 172.21.15.116:0/3661705595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d4190520 con 0x7f76d4103fd0 2024-03-04T16:10:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.976+0000 7f76bbfff700 1 -- 172.21.15.116:0/3661705595 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f76c4022a50 con 0x7f76d4103fd0 2024-03-04T16:10:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.977+0000 7f76bbfff700 1 --2- 172.21.15.116:0/3661705595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f76bc06d8a0 0x7f76bc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:46.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.977+0000 7f76bbfff700 1 -- 172.21.15.116:0/3661705595 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f76c408ebb0 con 0x7f76d4103fd0 2024-03-04T16:10:46.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.977+0000 7f76d2d9d700 1 --2- 172.21.15.116:0/3661705595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f76bc06d8a0 0x7f76bc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:46.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.977+0000 7f76d2d9d700 1 --2- 172.21.15.116:0/3661705595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f76bc06d8a0 0x7f76bc06fd60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f76c8005fd0 tx=0x7f76c8005d00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:46.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:46.981+0000 7f76bbfff700 1 -- 172.21.15.116:0/3661705595 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f76c4095080 con 0x7f76d4103fd0 2024-03-04T16:10:47.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.196+0000 7f76d917c700 1 -- 172.21.15.116:0/3661705595 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f76d40611d0 con 0x7f76bc06d8a0 2024-03-04T16:10:47.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.197+0000 7f76bbfff700 1 -- 172.21.15.116:0/3661705595 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f76d40611d0 con 0x7f76bc06d8a0 2024-03-04T16:10:47.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.199+0000 7f76d917c700 1 -- 172.21.15.116:0/3661705595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f76bc06d8a0 msgr2=0x7f76bc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:47.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.199+0000 7f76d917c700 1 --2- 172.21.15.116:0/3661705595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f76bc06d8a0 0x7f76bc06fd60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f76c8005fd0 tx=0x7f76c8005d00 comp rx=0 tx=0).stop 2024-03-04T16:10:47.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.199+0000 7f76d917c700 1 -- 172.21.15.116:0/3661705595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4103fd0 msgr2=0x7f76d4196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:47.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.199+0000 7f76d917c700 1 --2- 172.21.15.116:0/3661705595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4103fd0 0x7f76d4196870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f76c400bd70 tx=0x7f76c400be50 comp rx=0 tx=0).stop 2024-03-04T16:10:47.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.199+0000 7f76d917c700 1 -- 172.21.15.116:0/3661705595 shutdown_connections 2024-03-04T16:10:47.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.200+0000 7f76d917c700 1 --2- 172.21.15.116:0/3661705595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f76bc06d8a0 0x7f76bc06fd60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.200+0000 7f76d917c700 1 --2- 172.21.15.116:0/3661705595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f76d41016a0 0x7f76d4196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.200+0000 7f76d917c700 1 --2- 172.21.15.116:0/3661705595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4103fd0 0x7f76d4196870 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.200+0000 7f76d917c700 1 -- 172.21.15.116:0/3661705595 >> 172.21.15.116:0/3661705595 conn(0x7f76d40fb030 msgr2=0x7f76d40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:47.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.200+0000 7f76d917c700 1 -- 172.21.15.116:0/3661705595 shutdown_connections 2024-03-04T16:10:47.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.200+0000 7f76d917c700 1 -- 172.21.15.116:0/3661705595 wait complete. 2024-03-04T16:10:47.211 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.307+0000 7fed42810700 1 -- 172.21.15.116:0/3365284944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3c101db0 msgr2=0x7fed3c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.307+0000 7fed42810700 1 --2- 172.21.15.116:0/3365284944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3c101db0 0x7fed3c102210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fed30009910 tx=0x7fed30009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.307+0000 7fed42810700 1 -- 172.21.15.116:0/3365284944 shutdown_connections 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.307+0000 7fed42810700 1 --2- 172.21.15.116:0/3365284944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3c101db0 0x7fed3c102210 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.307+0000 7fed42810700 1 --2- 172.21.15.116:0/3365284944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed3c100bb0 0x7fed3c100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.307+0000 7fed42810700 1 -- 172.21.15.116:0/3365284944 >> 172.21.15.116:0/3365284944 conn(0x7fed3c0fc110 msgr2=0x7fed3c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.307+0000 7fed42810700 1 -- 172.21.15.116:0/3365284944 shutdown_connections 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.307+0000 7fed42810700 1 -- 172.21.15.116:0/3365284944 wait complete. 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.308+0000 7fed42810700 1 Processor -- start 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.308+0000 7fed42810700 1 -- start start 2024-03-04T16:10:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.308+0000 7fed42810700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed3c100bb0 0x7fed3c1963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.308+0000 7fed42810700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3c101db0 0x7fed3c196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.308+0000 7fed42810700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c196f30 con 0x7fed3c101db0 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.308+0000 7fed42810700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c197070 con 0x7fed3c100bb0 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.308+0000 7fed3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3c101db0 0x7fed3c196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.308+0000 7fed3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3c101db0 0x7fed3c196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52814/0 (socket says 172.21.15.116:52814) 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.308+0000 7fed3b7fe700 1 -- 172.21.15.116:0/214699191 learned_addr learned my addr 172.21.15.116:0/214699191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.309+0000 7fed3b7fe700 1 -- 172.21.15.116:0/214699191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed3c100bb0 msgr2=0x7fed3c1963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.309+0000 7fed3bfff700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed3c100bb0 0x7fed3c1963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.309+0000 7fed3b7fe700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed3c100bb0 0x7fed3c1963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.309+0000 7fed3b7fe700 1 -- 172.21.15.116:0/214699191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed300095f0 con 0x7fed3c101db0 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.309+0000 7fed3b7fe700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3c101db0 0x7fed3c196910 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fed3000b5c0 tx=0x7fed3000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.309+0000 7fed397fa700 1 -- 172.21.15.116:0/214699191 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed3001d070 con 0x7fed3c101db0 2024-03-04T16:10:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.309+0000 7fed397fa700 1 -- 172.21.15.116:0/214699191 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fed30022470 con 0x7fed3c101db0 2024-03-04T16:10:47.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.309+0000 7fed42810700 1 -- 172.21.15.116:0/214699191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3c19bac0 con 0x7fed3c101db0 2024-03-04T16:10:47.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.309+0000 7fed397fa700 1 -- 172.21.15.116:0/214699191 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed3000f690 con 0x7fed3c101db0 2024-03-04T16:10:47.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.309+0000 7fed42810700 1 -- 172.21.15.116:0/214699191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3c19c030 con 0x7fed3c101db0 2024-03-04T16:10:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.310+0000 7fed42810700 1 -- 172.21.15.116:0/214699191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3c1097b0 con 0x7fed3c101db0 2024-03-04T16:10:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.311+0000 7fed397fa700 1 -- 172.21.15.116:0/214699191 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fed30004b70 con 0x7fed3c101db0 2024-03-04T16:10:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.311+0000 7fed397fa700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed2406d8a0 0x7fed2406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.311+0000 7fed397fa700 1 -- 172.21.15.116:0/214699191 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fed3008ec90 con 0x7fed3c101db0 2024-03-04T16:10:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.311+0000 7fed3bfff700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed2406d8a0 0x7fed2406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.311+0000 7fed3bfff700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed2406d8a0 0x7fed2406fd60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fed2c006fd0 tx=0x7fed2c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.316+0000 7fed397fa700 1 -- 172.21.15.116:0/214699191 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fed3005c210 con 0x7fed3c101db0 2024-03-04T16:10:47.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.535+0000 7fed42810700 1 -- 172.21.15.116:0/214699191 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed3c106700 con 0x7fed2406d8a0 2024-03-04T16:10:47.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.536+0000 7fed397fa700 1 -- 172.21.15.116:0/214699191 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fed3c106700 con 0x7fed2406d8a0 2024-03-04T16:10:47.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.538+0000 7fed42810700 1 -- 172.21.15.116:0/214699191 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed2406d8a0 msgr2=0x7fed2406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:47.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.538+0000 7fed42810700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed2406d8a0 0x7fed2406fd60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fed2c006fd0 tx=0x7fed2c008040 comp rx=0 tx=0).stop 2024-03-04T16:10:47.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.538+0000 7fed42810700 1 -- 172.21.15.116:0/214699191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3c101db0 msgr2=0x7fed3c196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:47.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.538+0000 7fed42810700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3c101db0 0x7fed3c196910 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fed3000b5c0 tx=0x7fed3000be20 comp rx=0 tx=0).stop 2024-03-04T16:10:47.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.538+0000 7fed42810700 1 -- 172.21.15.116:0/214699191 shutdown_connections 2024-03-04T16:10:47.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.538+0000 7fed42810700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed2406d8a0 0x7fed2406fd60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.538+0000 7fed42810700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed3c100bb0 0x7fed3c1963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.538+0000 7fed42810700 1 --2- 172.21.15.116:0/214699191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3c101db0 0x7fed3c196910 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.539+0000 7fed42810700 1 -- 172.21.15.116:0/214699191 >> 172.21.15.116:0/214699191 conn(0x7fed3c0fc110 msgr2=0x7fed3c104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:47.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.539+0000 7fed42810700 1 -- 172.21.15.116:0/214699191 shutdown_connections 2024-03-04T16:10:47.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.539+0000 7fed42810700 1 -- 172.21.15.116:0/214699191 wait complete. 2024-03-04T16:10:47.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:47 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:47.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:47 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:47.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.646+0000 7f830350c700 1 -- 172.21.15.116:0/300595329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82fc100cb0 msgr2=0x7f82fc101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:47.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.646+0000 7f830350c700 1 --2- 172.21.15.116:0/300595329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82fc100cb0 0x7f82fc101110 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009910 tx=0x7f82ec009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:47.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.647+0000 7f830350c700 1 -- 172.21.15.116:0/300595329 shutdown_connections 2024-03-04T16:10:47.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.647+0000 7f830350c700 1 --2- 172.21.15.116:0/300595329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82fc100cb0 0x7f82fc101110 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.647+0000 7f830350c700 1 --2- 172.21.15.116:0/300595329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82fc0ffab0 0x7f82fc0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.647+0000 7f830350c700 1 -- 172.21.15.116:0/300595329 >> 172.21.15.116:0/300595329 conn(0x7f82fc0fb050 msgr2=0x7f82fc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:47.648 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:47 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:47.648 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:47 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.647+0000 7f830350c700 1 -- 172.21.15.116:0/300595329 shutdown_connections 2024-03-04T16:10:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.647+0000 7f830350c700 1 -- 172.21.15.116:0/300595329 wait complete. 2024-03-04T16:10:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.647+0000 7f830350c700 1 Processor -- start 2024-03-04T16:10:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f830350c700 1 -- start start 2024-03-04T16:10:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f830350c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82fc0ffab0 0x7f82fc196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f830350c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82fc100cb0 0x7f82fc1966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f830350c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc196c50 con 0x7f82fc100cb0 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f830350c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc196d90 con 0x7f82fc0ffab0 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f8300aa7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82fc100cb0 0x7f82fc1966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f8300aa7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82fc100cb0 0x7f82fc1966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52828/0 (socket says 172.21.15.116:52828) 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f8300aa7700 1 -- 172.21.15.116:0/3572901042 learned_addr learned my addr 172.21.15.116:0/3572901042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f83012a8700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82fc0ffab0 0x7f82fc196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f8300aa7700 1 -- 172.21.15.116:0/3572901042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82fc0ffab0 msgr2=0x7f82fc196180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f8300aa7700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82fc0ffab0 0x7f82fc196180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f8300aa7700 1 -- 172.21.15.116:0/3572901042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec0095f0 con 0x7f82fc100cb0 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.648+0000 7f8300aa7700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82fc100cb0 0x7f82fc1966c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0059d0 tx=0x7f82ec004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.649+0000 7f82f27fc700 1 -- 172.21.15.116:0/3572901042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82ec01d070 con 0x7f82fc100cb0 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.649+0000 7f830350c700 1 -- 172.21.15.116:0/3572901042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82fc19b7f0 con 0x7f82fc100cb0 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.649+0000 7f82f27fc700 1 -- 172.21.15.116:0/3572901042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f82ec005dc0 con 0x7f82fc100cb0 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.649+0000 7f82f27fc700 1 -- 172.21.15.116:0/3572901042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82ec00f970 con 0x7f82fc100cb0 2024-03-04T16:10:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.649+0000 7f830350c700 1 -- 172.21.15.116:0/3572901042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82fc19bcb0 con 0x7f82fc100cb0 2024-03-04T16:10:47.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.650+0000 7f830350c700 1 -- 172.21.15.116:0/3572901042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82fc108610 con 0x7f82fc100cb0 2024-03-04T16:10:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.651+0000 7f82f27fc700 1 -- 172.21.15.116:0/3572901042 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f82ec022cd0 con 0x7f82fc100cb0 2024-03-04T16:10:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.651+0000 7f82f27fc700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f82e806d8a0 0x7f82e806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.651+0000 7f82f27fc700 1 -- 172.21.15.116:0/3572901042 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f82ec08ee60 con 0x7f82fc100cb0 2024-03-04T16:10:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.651+0000 7f83012a8700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f82e806d8a0 0x7f82e806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.652+0000 7f83012a8700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f82e806d8a0 0x7f82e806fd60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f82f8005fd0 tx=0x7f82f8005e50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:47.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.657+0000 7f82f27fc700 1 -- 172.21.15.116:0/3572901042 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82ec05c360 con 0x7f82fc100cb0 2024-03-04T16:10:47.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.871+0000 7f830350c700 1 -- 172.21.15.116:0/3572901042 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f82fc069470 con 0x7f82e806d8a0 2024-03-04T16:10:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.883+0000 7f82f27fc700 1 -- 172.21.15.116:0/3572901042 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f82fc069470 con 0x7f82e806d8a0 2024-03-04T16:10:47.884 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:10:47.884 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (13m) 1s ago 15m 25.4M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:10:47.884 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (15m) 1s ago 15m 8644k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:10:47.884 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (14m) 21s ago 14m 8548k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:10:47.884 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (3m) 1s ago 15m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:10:47.884 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (3m) 21s ago 14m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:10:47.884 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (13m) 1s ago 15m 86.6M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:10:47.884 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (9m) 1s ago 9m 20.7M - 18.2.1 5be31c24972a 47b310ba20f4 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (7s) 1s ago 9m 13.8M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4bf8fc434069 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (9m) 21s ago 9m 18.1M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (9m) 21s ago 9m 143M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (6m) 1s ago 16m 506M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (5m) 21s ago 14m 450M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (5m) 1s ago 16m 50.2M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (4m) 21s ago 14m 36.8M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (15m) 1s ago 15m 15.4M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (14m) 21s ago 14m 16.7M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (3m) 1s ago 13m 91.5M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (2m) 1s ago 13m 104M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (2m) 1s ago 12m 85.0M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (94s) 21s ago 12m 109M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6b01e97c8f9b 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (60s) 21s ago 11m 105M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 5ecc30e5a86c 2024-03-04T16:10:47.885 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (25s) 21s ago 11m 12.6M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 1083700b476d 2024-03-04T16:10:47.886 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (5m) 1s ago 14m 73.6M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:10:47.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.885+0000 7f830350c700 1 -- 172.21.15.116:0/3572901042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f82e806d8a0 msgr2=0x7f82e806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:47.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.886+0000 7f830350c700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f82e806d8a0 0x7f82e806fd60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f82f8005fd0 tx=0x7f82f8005e50 comp rx=0 tx=0).stop 2024-03-04T16:10:47.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.886+0000 7f830350c700 1 -- 172.21.15.116:0/3572901042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82fc100cb0 msgr2=0x7f82fc1966c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:47.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.886+0000 7f830350c700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82fc100cb0 0x7f82fc1966c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0059d0 tx=0x7f82ec004970 comp rx=0 tx=0).stop 2024-03-04T16:10:47.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.886+0000 7f830350c700 1 -- 172.21.15.116:0/3572901042 shutdown_connections 2024-03-04T16:10:47.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.886+0000 7f830350c700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f82e806d8a0 0x7f82e806fd60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.886+0000 7f830350c700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82fc0ffab0 0x7f82fc196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.886+0000 7f830350c700 1 --2- 172.21.15.116:0/3572901042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82fc100cb0 0x7f82fc1966c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:47.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.886+0000 7f830350c700 1 -- 172.21.15.116:0/3572901042 >> 172.21.15.116:0/3572901042 conn(0x7f82fc0fb050 msgr2=0x7f82fc068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:47.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.886+0000 7f830350c700 1 -- 172.21.15.116:0/3572901042 shutdown_connections 2024-03-04T16:10:47.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:47.886+0000 7f830350c700 1 -- 172.21.15.116:0/3572901042 wait complete. 2024-03-04T16:10:48.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.005+0000 7efcb8aa0700 1 -- 172.21.15.116:0/2543977069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4101db0 msgr2=0x7efcb4102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:48.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.005+0000 7efcb8aa0700 1 --2- 172.21.15.116:0/2543977069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4101db0 0x7efcb4102210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009910 tx=0x7efc9c009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:48.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.005+0000 7efcb8aa0700 1 -- 172.21.15.116:0/2543977069 shutdown_connections 2024-03-04T16:10:48.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.005+0000 7efcb8aa0700 1 --2- 172.21.15.116:0/2543977069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4101db0 0x7efcb4102210 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.005+0000 7efcb8aa0700 1 --2- 172.21.15.116:0/2543977069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb4100bb0 0x7efcb4100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.005+0000 7efcb8aa0700 1 -- 172.21.15.116:0/2543977069 >> 172.21.15.116:0/2543977069 conn(0x7efcb40fc110 msgr2=0x7efcb40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:48.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.005+0000 7efcb8aa0700 1 -- 172.21.15.116:0/2543977069 shutdown_connections 2024-03-04T16:10:48.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.005+0000 7efcb8aa0700 1 -- 172.21.15.116:0/2543977069 wait complete. 2024-03-04T16:10:48.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb8aa0700 1 Processor -- start 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb8aa0700 1 -- start start 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb8aa0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb4100bb0 0x7efcb4196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb8aa0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4101db0 0x7efcb41968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb8aa0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb4196ec0 con 0x7efcb4101db0 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb8aa0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb4197000 con 0x7efcb4100bb0 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb1d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4101db0 0x7efcb41968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb1d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4101db0 0x7efcb41968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52846/0 (socket says 172.21.15.116:52846) 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb1d9b700 1 -- 172.21.15.116:0/1313627163 learned_addr learned my addr 172.21.15.116:0/1313627163 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb259c700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb4100bb0 0x7efcb4196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb1d9b700 1 -- 172.21.15.116:0/1313627163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb4100bb0 msgr2=0x7efcb4196360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb1d9b700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb4100bb0 0x7efcb4196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb1d9b700 1 -- 172.21.15.116:0/1313627163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c0095f0 con 0x7efcb4101db0 2024-03-04T16:10:48.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.006+0000 7efcb1d9b700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4101db0 0x7efcb41968a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b5c0 tx=0x7efc9c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:48.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.007+0000 7efcab7fe700 1 -- 172.21.15.116:0/1313627163 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc9c01d070 con 0x7efcb4101db0 2024-03-04T16:10:48.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.007+0000 7efcab7fe700 1 -- 172.21.15.116:0/1313627163 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efc9c022470 con 0x7efcb4101db0 2024-03-04T16:10:48.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.007+0000 7efcb8aa0700 1 -- 172.21.15.116:0/1313627163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb419ba50 con 0x7efcb4101db0 2024-03-04T16:10:48.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.007+0000 7efcab7fe700 1 -- 172.21.15.116:0/1313627163 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc9c00f690 con 0x7efcb4101db0 2024-03-04T16:10:48.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.007+0000 7efcb8aa0700 1 -- 172.21.15.116:0/1313627163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb419bf40 con 0x7efcb4101db0 2024-03-04T16:10:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.008+0000 7efcb8aa0700 1 -- 172.21.15.116:0/1313627163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb4109710 con 0x7efcb4101db0 2024-03-04T16:10:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.008+0000 7efcab7fe700 1 -- 172.21.15.116:0/1313627163 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7efc9c004b70 con 0x7efcb4101db0 2024-03-04T16:10:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.008+0000 7efcab7fe700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efca006d8a0 0x7efca006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.009+0000 7efcab7fe700 1 -- 172.21.15.116:0/1313627163 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efc9c08ec50 con 0x7efcb4101db0 2024-03-04T16:10:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.009+0000 7efcb259c700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efca006d8a0 0x7efca006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.009+0000 7efcb259c700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efca006d8a0 0x7efca006fd60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efcb4101c10 tx=0x7efca4005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:48.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.013+0000 7efcab7fe700 1 -- 172.21.15.116:0/1313627163 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc9c05c150 con 0x7efcb4101db0 2024-03-04T16:10:48.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.300+0000 7efcb8aa0700 1 -- 172.21.15.116:0/1313627163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efcb41098d0 con 0x7efcb4101db0 2024-03-04T16:10:48.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.301+0000 7efcab7fe700 1 -- 172.21.15.116:0/1313627163 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7efc9c027340 con 0x7efcb4101db0 2024-03-04T16:10:48.302 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:10:48.302 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:10:48.302 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:10:48.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:10:48.302 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:10:48.302 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:10:48.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 11 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:10:48.303 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:10:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.303+0000 7efcb8aa0700 1 -- 172.21.15.116:0/1313627163 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efca006d8a0 msgr2=0x7efca006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.303+0000 7efcb8aa0700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efca006d8a0 0x7efca006fd60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efcb4101c10 tx=0x7efca4005cf0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.303+0000 7efcb8aa0700 1 -- 172.21.15.116:0/1313627163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4101db0 msgr2=0x7efcb41968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.303+0000 7efcb8aa0700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4101db0 0x7efcb41968a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b5c0 tx=0x7efc9c00be20 comp rx=0 tx=0).stop 2024-03-04T16:10:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.303+0000 7efcb8aa0700 1 -- 172.21.15.116:0/1313627163 shutdown_connections 2024-03-04T16:10:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.303+0000 7efcb8aa0700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7efca006d8a0 0x7efca006fd60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.303+0000 7efcb8aa0700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb4100bb0 0x7efcb4196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.303+0000 7efcb8aa0700 1 --2- 172.21.15.116:0/1313627163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4101db0 0x7efcb41968a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.303+0000 7efcb8aa0700 1 -- 172.21.15.116:0/1313627163 >> 172.21.15.116:0/1313627163 conn(0x7efcb40fc110 msgr2=0x7efcb4104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:48.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.304+0000 7efcb8aa0700 1 -- 172.21.15.116:0/1313627163 shutdown_connections 2024-03-04T16:10:48.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.304+0000 7efcb8aa0700 1 -- 172.21.15.116:0/1313627163 wait complete. 2024-03-04T16:10:48.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.420+0000 7f51f8c33700 1 -- 172.21.15.116:0/2157488015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f4100cd0 msgr2=0x7f51f4101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:48.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.420+0000 7f51f8c33700 1 --2- 172.21.15.116:0/2157488015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f4100cd0 0x7f51f4101130 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009910 tx=0x7f51e8009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:48.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.420+0000 7f51f8c33700 1 -- 172.21.15.116:0/2157488015 shutdown_connections 2024-03-04T16:10:48.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.420+0000 7f51f8c33700 1 --2- 172.21.15.116:0/2157488015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f4100cd0 0x7f51f4101130 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.420+0000 7f51f8c33700 1 --2- 172.21.15.116:0/2157488015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f40ffad0 0x7f51f40ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.420+0000 7f51f8c33700 1 -- 172.21.15.116:0/2157488015 >> 172.21.15.116:0/2157488015 conn(0x7f51f40fb050 msgr2=0x7f51f40fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:48.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.420+0000 7f51f8c33700 1 -- 172.21.15.116:0/2157488015 shutdown_connections 2024-03-04T16:10:48.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.420+0000 7f51f8c33700 1 -- 172.21.15.116:0/2157488015 wait complete. 2024-03-04T16:10:48.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.421+0000 7f51f8c33700 1 Processor -- start 2024-03-04T16:10:48.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.421+0000 7f51f8c33700 1 -- start start 2024-03-04T16:10:48.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.421+0000 7f51f8c33700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f40ffad0 0x7f51f40698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:48.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.421+0000 7f51f8c33700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f4100cd0 0x7f51f419b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:48.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.421+0000 7f51f8c33700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f419b750 con 0x7f51f40ffad0 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.421+0000 7f51f8c33700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f419b890 con 0x7f51f4100cd0 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.421+0000 7f51f259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f40ffad0 0x7f51f40698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51f259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f40ffad0 0x7f51f40698f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52862/0 (socket says 172.21.15.116:52862) 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51f259c700 1 -- 172.21.15.116:0/273853923 learned_addr learned my addr 172.21.15.116:0/273853923 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51f1d9b700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f4100cd0 0x7f51f419b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51f259c700 1 -- 172.21.15.116:0/273853923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f4100cd0 msgr2=0x7f51f419b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51f259c700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f4100cd0 0x7f51f419b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51f259c700 1 -- 172.21.15.116:0/273853923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e80095f0 con 0x7f51f40ffad0 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51f259c700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f40ffad0 0x7f51f40698f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f51e400b480 tx=0x7f51e400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51e37fe700 1 -- 172.21.15.116:0/273853923 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51e40118c0 con 0x7f51f40ffad0 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51e37fe700 1 -- 172.21.15.116:0/273853923 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f51e4010380 con 0x7f51f40ffad0 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51e37fe700 1 -- 172.21.15.116:0/273853923 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51e400f4f0 con 0x7f51f40ffad0 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51f8c33700 1 -- 172.21.15.116:0/273853923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f419bb70 con 0x7f51f40ffad0 2024-03-04T16:10:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.422+0000 7f51f8c33700 1 -- 172.21.15.116:0/273853923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f419c140 con 0x7f51f40ffad0 2024-03-04T16:10:48.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.423+0000 7f51f8c33700 1 -- 172.21.15.116:0/273853923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f41086d0 con 0x7f51f40ffad0 2024-03-04T16:10:48.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.424+0000 7f51e37fe700 1 -- 172.21.15.116:0/273853923 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f51e40104f0 con 0x7f51f40ffad0 2024-03-04T16:10:48.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.424+0000 7f51e37fe700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f51dc06d8a0 0x7f51dc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:48.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.424+0000 7f51e37fe700 1 -- 172.21.15.116:0/273853923 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f51e408d080 con 0x7f51f40ffad0 2024-03-04T16:10:48.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.424+0000 7f51f1d9b700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f51dc06d8a0 0x7f51dc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:48.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.425+0000 7f51f1d9b700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f51dc06d8a0 0x7f51dc06fd60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f51f4069e70 tx=0x7f51e8005920 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:48.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.429+0000 7f51e37fe700 1 -- 172.21.15.116:0/273853923 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f51e405a430 con 0x7f51f40ffad0 2024-03-04T16:10:48.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:48 smithi152 ceph-mon[82845]: pgmap v181: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:48.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:48 smithi152 ceph-mon[82845]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:48.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:48 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:48.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:48 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:48 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1313627163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:48.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.668+0000 7f51f8c33700 1 -- 172.21.15.116:0/273853923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f51f41088f0 con 0x7f51f40ffad0 2024-03-04T16:10:48.669 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:48 smithi116 ceph-mon[94164]: pgmap v181: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:48.669 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:48 smithi116 ceph-mon[94164]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:48.669 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:48 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:48.669 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:48 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:48.669 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:48 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1313627163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.668+0000 7f51e37fe700 1 -- 172.21.15.116:0/273853923 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1850 (secure 0 0 0) 0x7f51e4059fc0 con 0x7f51f40ffad0 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:e16 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 14 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:10:48.670 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:01:18.395823+0000 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 47 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24247} 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:10:48.671 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{0:24247} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.152:6824/287206196,v1:172.21.15.152:6825/287206196] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:14486} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.116:6828/3470200572,v1:172.21.15.116:6829/3470200572] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:24257} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:10:48.672 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:34400} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:10:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.671+0000 7f51f8c33700 1 -- 172.21.15.116:0/273853923 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f51dc06d8a0 msgr2=0x7f51dc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.671+0000 7f51f8c33700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f51dc06d8a0 0x7f51dc06fd60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f51f4069e70 tx=0x7f51e8005920 comp rx=0 tx=0).stop 2024-03-04T16:10:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.671+0000 7f51f8c33700 1 -- 172.21.15.116:0/273853923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f40ffad0 msgr2=0x7f51f40698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.671+0000 7f51f8c33700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f40ffad0 0x7f51f40698f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f51e400b480 tx=0x7f51e400b840 comp rx=0 tx=0).stop 2024-03-04T16:10:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.671+0000 7f51f8c33700 1 -- 172.21.15.116:0/273853923 shutdown_connections 2024-03-04T16:10:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.671+0000 7f51f8c33700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f51dc06d8a0 0x7f51dc06fd60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.671+0000 7f51f8c33700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f40ffad0 0x7f51f40698f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.671+0000 7f51f8c33700 1 --2- 172.21.15.116:0/273853923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f4100cd0 0x7f51f419b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.671+0000 7f51f8c33700 1 -- 172.21.15.116:0/273853923 >> 172.21.15.116:0/273853923 conn(0x7f51f40fb050 msgr2=0x7f51f4103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.672+0000 7f51f8c33700 1 -- 172.21.15.116:0/273853923 shutdown_connections 2024-03-04T16:10:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.672+0000 7f51f8c33700 1 -- 172.21.15.116:0/273853923 wait complete. 2024-03-04T16:10:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2024-03-04T16:10:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.778+0000 7f53a0f45700 1 -- 172.21.15.116:0/225590389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f539c074dc0 msgr2=0x7f539c073220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.778+0000 7f53a0f45700 1 --2- 172.21.15.116:0/225590389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f539c074dc0 0x7f539c073220 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f538c009910 tx=0x7f538c009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.779+0000 7f53a0f45700 1 -- 172.21.15.116:0/225590389 shutdown_connections 2024-03-04T16:10:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.779+0000 7f53a0f45700 1 --2- 172.21.15.116:0/225590389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f539c0737f0 0x7f539c073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.779+0000 7f53a0f45700 1 --2- 172.21.15.116:0/225590389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f539c074dc0 0x7f539c073220 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.779+0000 7f53a0f45700 1 -- 172.21.15.116:0/225590389 >> 172.21.15.116:0/225590389 conn(0x7f539c0fc140 msgr2=0x7f539c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.779+0000 7f53a0f45700 1 -- 172.21.15.116:0/225590389 shutdown_connections 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.779+0000 7f53a0f45700 1 -- 172.21.15.116:0/225590389 wait complete. 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.779+0000 7f53a0f45700 1 Processor -- start 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.779+0000 7f53a0f45700 1 -- start start 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.779+0000 7f53a0f45700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f539c0737f0 0x7f539c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f53a0f45700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f539c074dc0 0x7f539c19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f53a0f45700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539c19d580 con 0x7f539c0737f0 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f53a0f45700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539c19d6c0 con 0x7f539c074dc0 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f539a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f539c0737f0 0x7f539c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f539a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f539c0737f0 0x7f539c19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52882/0 (socket says 172.21.15.116:52882) 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f539a59c700 1 -- 172.21.15.116:0/1592528123 learned_addr learned my addr 172.21.15.116:0/1592528123 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f5399d9b700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f539c074dc0 0x7f539c19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f539a59c700 1 -- 172.21.15.116:0/1592528123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f539c074dc0 msgr2=0x7f539c19cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f539a59c700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f539c074dc0 0x7f539c19cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f539a59c700 1 -- 172.21.15.116:0/1592528123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c0095f0 con 0x7f539c0737f0 2024-03-04T16:10:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f539a59c700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f539c0737f0 0x7f539c19ca20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f538c00bda0 tx=0x7f538c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f538b7fe700 1 -- 172.21.15.116:0/1592528123 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f538c01d070 con 0x7f539c0737f0 2024-03-04T16:10:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f538b7fe700 1 -- 172.21.15.116:0/1592528123 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f538c004c50 con 0x7f539c0737f0 2024-03-04T16:10:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f53a0f45700 1 -- 172.21.15.116:0/1592528123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539c1a2110 con 0x7f539c0737f0 2024-03-04T16:10:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f538b7fe700 1 -- 172.21.15.116:0/1592528123 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f538c00f690 con 0x7f539c0737f0 2024-03-04T16:10:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.780+0000 7f53a0f45700 1 -- 172.21.15.116:0/1592528123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539c1a2680 con 0x7f539c0737f0 2024-03-04T16:10:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.781+0000 7f53a0f45700 1 -- 172.21.15.116:0/1592528123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539c10fdd0 con 0x7f539c0737f0 2024-03-04T16:10:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.782+0000 7f538b7fe700 1 -- 172.21.15.116:0/1592528123 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f538c00f900 con 0x7f539c0737f0 2024-03-04T16:10:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.782+0000 7f538b7fe700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f538406d8a0 0x7f538406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.782+0000 7f538b7fe700 1 -- 172.21.15.116:0/1592528123 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f538c08ecf0 con 0x7f539c0737f0 2024-03-04T16:10:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.782+0000 7f5399d9b700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f538406d8a0 0x7f538406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:48.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.783+0000 7f5399d9b700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f538406d8a0 0x7f538406fd60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5390006fd0 tx=0x7f5390008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:48.787+0000 7f538b7fe700 1 -- 172.21.15.116:0/1592528123 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f538c05c020 con 0x7f539c0737f0 2024-03-04T16:10:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.001+0000 7f53a0f45700 1 -- 172.21.15.116:0/1592528123 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f539c103c70 con 0x7f538406d8a0 2024-03-04T16:10:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.002+0000 7f538b7fe700 1 -- 172.21.15.116:0/1592528123 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f539c103c70 con 0x7f538406d8a0 2024-03-04T16:10:49.002 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr", 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "osd" 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "13/23 daemons upgraded", 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mds daemons", 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:10:49.003 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 -- 172.21.15.116:0/1592528123 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f538406d8a0 msgr2=0x7f538406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f538406d8a0 0x7f538406fd60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5390006fd0 tx=0x7f5390008040 comp rx=0 tx=0).stop 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 -- 172.21.15.116:0/1592528123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f539c0737f0 msgr2=0x7f539c19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f539c0737f0 0x7f539c19ca20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f538c00bda0 tx=0x7f538c00be80 comp rx=0 tx=0).stop 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 -- 172.21.15.116:0/1592528123 shutdown_connections 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f538406d8a0 0x7f538406fd60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f539c0737f0 0x7f539c19ca20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 --2- 172.21.15.116:0/1592528123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f539c074dc0 0x7f539c19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 -- 172.21.15.116:0/1592528123 >> 172.21.15.116:0/1592528123 conn(0x7f539c0fc140 msgr2=0x7f539c102550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 -- 172.21.15.116:0/1592528123 shutdown_connections 2024-03-04T16:10:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.004+0000 7f53a0f45700 1 -- 172.21.15.116:0/1592528123 wait complete. 2024-03-04T16:10:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.108+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/1996788308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a40ff460 msgr2=0x7fe1a40ff880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.108+0000 7fe1a8f0b700 1 --2- 172.21.15.116:0/1996788308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a40ff460 0x7fe1a40ff880 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe194009910 tx=0x7fe194009c20 comp rx=0 tx=0).stop 2024-03-04T16:10:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.108+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/1996788308 shutdown_connections 2024-03-04T16:10:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.108+0000 7fe1a8f0b700 1 --2- 172.21.15.116:0/1996788308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1a40ffdc0 0x7fe1a4100240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.108+0000 7fe1a8f0b700 1 --2- 172.21.15.116:0/1996788308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a40ff460 0x7fe1a40ff880 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.108+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/1996788308 >> 172.21.15.116:0/1996788308 conn(0x7fe1a40fb080 msgr2=0x7fe1a40fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.108+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/1996788308 shutdown_connections 2024-03-04T16:10:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.108+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/1996788308 wait complete. 2024-03-04T16:10:49.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.109+0000 7fe1a8f0b700 1 Processor -- start 2024-03-04T16:10:49.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.109+0000 7fe1a8f0b700 1 -- start start 2024-03-04T16:10:49.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.109+0000 7fe1a8f0b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1a40ff460 0x7fe1a4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.109+0000 7fe1a8f0b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a40ffdc0 0x7fe1a4198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.109+0000 7fe1a8f0b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a4199090 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.109+0000 7fe1a8f0b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a41991d0 con 0x7fe1a40ff460 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.109+0000 7fe1a1d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a40ffdc0 0x7fe1a4198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.109+0000 7fe1a1d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a40ffdc0 0x7fe1a4198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52892/0 (socket says 172.21.15.116:52892) 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.109+0000 7fe1a1d9b700 1 -- 172.21.15.116:0/3628474411 learned_addr learned my addr 172.21.15.116:0/3628474411 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.109+0000 7fe1a259c700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1a40ff460 0x7fe1a4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.110+0000 7fe1a1d9b700 1 -- 172.21.15.116:0/3628474411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1a40ff460 msgr2=0x7fe1a4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.110+0000 7fe1a1d9b700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1a40ff460 0x7fe1a4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.110+0000 7fe1a1d9b700 1 -- 172.21.15.116:0/3628474411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1940095f0 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.110+0000 7fe1a1d9b700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a40ffdc0 0x7fe1a4198a70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe19800c8a0 tx=0x7fe19800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.110+0000 7fe1937fe700 1 -- 172.21.15.116:0/3628474411 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1980078e0 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.110+0000 7fe1937fe700 1 -- 172.21.15.116:0/3628474411 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe19800f450 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.110+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/3628474411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a419dc80 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.110+0000 7fe1937fe700 1 -- 172.21.15.116:0/3628474411 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe19800e5e0 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.110+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/3628474411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a419e1d0 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.111+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/3628474411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a410b930 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.112+0000 7fe1937fe700 1 -- 172.21.15.116:0/3628474411 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fe198007a40 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.112+0000 7fe1937fe700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe18c06d8a0 0x7fe18c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:10:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.112+0000 7fe1937fe700 1 -- 172.21.15.116:0/3628474411 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe19808d4c0 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.112+0000 7fe1a259c700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe18c06d8a0 0x7fe18c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:10:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.113+0000 7fe1a259c700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe18c06d8a0 0x7fe18c06fd60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe194006010 tx=0x7fe19400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:10:49.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.117+0000 7fe1937fe700 1 -- 172.21.15.116:0/3628474411 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe19805a7f0 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.399+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/3628474411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe1a410bb20 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.399+0000 7fe1937fe700 1 -- 172.21.15.116:0/3628474411 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fe19805a380 con 0x7fe1a40ffdc0 2024-03-04T16:10:49.400 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:10:49.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/3628474411 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe18c06d8a0 msgr2=0x7fe18c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:49.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe18c06d8a0 0x7fe18c06fd60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe194006010 tx=0x7fe19400b540 comp rx=0 tx=0).stop 2024-03-04T16:10:49.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/3628474411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a40ffdc0 msgr2=0x7fe1a4198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:10:49.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a40ffdc0 0x7fe1a4198a70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe19800c8a0 tx=0x7fe19800cbb0 comp rx=0 tx=0).stop 2024-03-04T16:10:49.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/3628474411 shutdown_connections 2024-03-04T16:10:49.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe18c06d8a0 0x7fe18c06fd60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:49.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1a40ff460 0x7fe1a4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:49.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 --2- 172.21.15.116:0/3628474411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a40ffdc0 0x7fe1a4198a70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:10:49.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/3628474411 >> 172.21.15.116:0/3628474411 conn(0x7fe1a40fb080 msgr2=0x7fe1a4107a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:10:49.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/3628474411 shutdown_connections 2024-03-04T16:10:49.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:10:49.401+0000 7fe1a8f0b700 1 -- 172.21.15.116:0/3628474411 wait complete. 2024-03-04T16:10:49.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:49 smithi152 ceph-mon[82845]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:49.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:49 smithi152 ceph-mon[82845]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:49.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:49 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/273853923' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:10:49.660 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:49 smithi116 ceph-mon[94164]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:49.660 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:49 smithi116 ceph-mon[94164]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:49.660 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:49 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/273853923' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:10:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:50 smithi116 ceph-mon[94164]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:50 smithi116 ceph-mon[94164]: pgmap v182: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:50.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:50 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3628474411' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:10:50.842 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:50 smithi152 ceph-mon[82845]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:50.843 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:50 smithi152 ceph-mon[82845]: pgmap v182: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:50.843 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:50 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3628474411' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:10:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: pgmap v183: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: Detected new or changed devices on smithi116 2024-03-04T16:10:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:10:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi116.gkenwy"]}]: dispatch 2024-03-04T16:10:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:52 smithi116 ceph-mon[94164]: Upgrade: It appears safe to stop mds.cephfs.smithi116.gkenwy 2024-03-04T16:10:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: pgmap v183: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: Detected new or changed devices on smithi116 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi116.gkenwy"]}]: dispatch 2024-03-04T16:10:52.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:52 smithi152 ceph-mon[82845]: Upgrade: It appears safe to stop mds.cephfs.smithi116.gkenwy 2024-03-04T16:10:53.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:53 smithi116 ceph-mon[94164]: Upgrade: Updating mds.cephfs.smithi116.gkenwy 2024-03-04T16:10:53.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:53.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.gkenwy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:10:53.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:53.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:53 smithi116 ceph-mon[94164]: Deploying daemon mds.cephfs.smithi116.gkenwy on smithi116 2024-03-04T16:10:53.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:53 smithi116 ceph-mon[94164]: pgmap v184: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:53.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:54.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:53 smithi152 ceph-mon[82845]: Upgrade: Updating mds.cephfs.smithi116.gkenwy 2024-03-04T16:10:54.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:10:54.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi116.gkenwy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:10:54.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:54.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:53 smithi152 ceph-mon[82845]: Deploying daemon mds.cephfs.smithi116.gkenwy on smithi116 2024-03-04T16:10:54.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:53 smithi152 ceph-mon[82845]: pgmap v184: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:54.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:56 smithi152 ceph-mon[82845]: osdmap e82: 6 total, 6 up, 6 in 2024-03-04T16:10:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:56 smithi152 ceph-mon[82845]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 2 up:standby 2024-03-04T16:10:56.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:56 smithi152 ceph-mon[82845]: pgmap v186: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:56 smithi116 ceph-mon[94164]: osdmap e82: 6 total, 6 up, 6 in 2024-03-04T16:10:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:56 smithi116 ceph-mon[94164]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 2 up:standby 2024-03-04T16:10:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:56 smithi116 ceph-mon[94164]: pgmap v186: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:58.423 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:10:58 smithi116 ceph-mon[94164]: pgmap v187: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:10:58.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:10:58 smithi152 ceph-mon[82845]: pgmap v187: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:00.388 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:00 smithi116 ceph-mon[94164]: pgmap v188: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:00.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:00 smithi152 ceph-mon[82845]: pgmap v188: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:01.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:01 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:01.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:01 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:01.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:01 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:01.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:01 smithi116 ceph-mon[94164]: pgmap v189: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:01.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:01 smithi116 ceph-mon[94164]: mds.? [v2:172.21.15.116:6828/786823228,v1:172.21.15.116:6829/786823228] up:boot 2024-03-04T16:11:01.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:01 smithi116 ceph-mon[94164]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 3 up:standby 2024-03-04T16:11:01.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:01 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:11:02.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:01 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:02.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:01 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:02.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:01 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:02.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:01 smithi152 ceph-mon[82845]: pgmap v189: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:02.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:01 smithi152 ceph-mon[82845]: mds.? [v2:172.21.15.116:6828/786823228,v1:172.21.15.116:6829/786823228] up:boot 2024-03-04T16:11:02.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:01 smithi152 ceph-mon[82845]: fsmap cephfs:1 {0=cephfs.smithi152.tlpshy=up:active} 3 up:standby 2024-03-04T16:11:02.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:01 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:11:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:04 smithi116 ceph-mon[94164]: pgmap v190: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:04 smithi152 ceph-mon[82845]: pgmap v190: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:06.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:06 smithi152 ceph-mon[82845]: pgmap v191: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:06.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:06 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:06.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:06 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:06.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:06 smithi116 ceph-mon[94164]: pgmap v191: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:06.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:06 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:06.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:06 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:08.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:07 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:08.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:07 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:08.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:07 smithi116 ceph-mon[94164]: pgmap v192: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:08.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:07 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:08.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:07 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:08.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:07 smithi152 ceph-mon[82845]: pgmap v192: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:09.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:09.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:10.307 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:10 smithi116 ceph-mon[94164]: pgmap v193: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:10.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:10 smithi152 ceph-mon[82845]: pgmap v193: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:11.626 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:11.626 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:11.627 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:11.627 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:11.627 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:11.627 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:11.627 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:11.627 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:11.628 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:11.628 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:11.628 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi152.tlpshy"]}]: dispatch 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:11.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi152.tlpshy"]}]: dispatch 2024-03-04T16:11:12.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:12 smithi152 ceph-mon[82845]: Upgrade: It appears safe to stop mds.cephfs.smithi152.tlpshy 2024-03-04T16:11:12.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:12 smithi152 ceph-mon[82845]: pgmap v194: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:12.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:12 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:12.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:12 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.tlpshy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:11:12.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:12 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:12.654 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:12 smithi116 ceph-mon[94164]: Upgrade: It appears safe to stop mds.cephfs.smithi152.tlpshy 2024-03-04T16:11:12.654 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:12 smithi116 ceph-mon[94164]: pgmap v194: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:12.654 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:12 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:12.655 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:12 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.tlpshy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:11:12.655 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:12 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:13.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:13 smithi152 ceph-mon[82845]: Upgrade: Updating mds.cephfs.smithi152.tlpshy 2024-03-04T16:11:13.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:13 smithi152 ceph-mon[82845]: Deploying daemon mds.cephfs.smithi152.tlpshy on smithi152 2024-03-04T16:11:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:13 smithi116 ceph-mon[94164]: Upgrade: Updating mds.cephfs.smithi152.tlpshy 2024-03-04T16:11:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:13 smithi116 ceph-mon[94164]: Deploying daemon mds.cephfs.smithi152.tlpshy on smithi152 2024-03-04T16:11:14.444 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:14 smithi152 ceph-mon[82845]: pgmap v195: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:14.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:14 smithi116 ceph-mon[94164]: pgmap v195: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:11:14.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:14 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116[94159]: 2024-03-04T16:11:14.443+0000 7fa4b3157700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T16:11:15.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:15 smithi116 ceph-mon[94164]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T16:11:15.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:15 smithi116 ceph-mon[94164]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T16:11:15.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:15 smithi116 ceph-mon[94164]: osdmap e83: 6 total, 6 up, 6 in 2024-03-04T16:11:15.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:15 smithi116 ceph-mon[94164]: Standby daemon mds.cephfs.smithi152.kqoybf assigned to filesystem cephfs as rank 0 2024-03-04T16:11:15.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:15 smithi116 ceph-mon[94164]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T16:11:15.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:15 smithi116 ceph-mon[94164]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-03-04T16:11:15.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:15 smithi116 ceph-mon[94164]: fsmap cephfs:1/1 {0=cephfs.smithi152.kqoybf=up:replay} 2 up:standby 2024-03-04T16:11:15.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:15 smithi152 ceph-mon[82845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T16:11:15.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:15 smithi152 ceph-mon[82845]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T16:11:15.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:15 smithi152 ceph-mon[82845]: osdmap e83: 6 total, 6 up, 6 in 2024-03-04T16:11:15.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:15 smithi152 ceph-mon[82845]: Standby daemon mds.cephfs.smithi152.kqoybf assigned to filesystem cephfs as rank 0 2024-03-04T16:11:15.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:15 smithi152 ceph-mon[82845]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T16:11:15.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:15 smithi152 ceph-mon[82845]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-03-04T16:11:15.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:15 smithi152 ceph-mon[82845]: fsmap cephfs:1/1 {0=cephfs.smithi152.kqoybf=up:replay} 2 up:standby 2024-03-04T16:11:16.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:16 smithi116 ceph-mon[94164]: pgmap v197: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 0 op/s 2024-03-04T16:11:16.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:16 smithi152 ceph-mon[82845]: pgmap v197: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 0 op/s 2024-03-04T16:11:18.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:18 smithi116 ceph-mon[94164]: pgmap v198: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s rd, 1 op/s 2024-03-04T16:11:18.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:18 smithi152 ceph-mon[82845]: pgmap v198: 65 pgs: 65 active+clean; 212 MiB data, 899 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s rd, 1 op/s 2024-03-04T16:11:19.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.519+0000 7fde9c90c700 1 -- 172.21.15.116:0/3294349729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde98101dc0 msgr2=0x7fde98102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:19.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.519+0000 7fde9c90c700 1 --2- 172.21.15.116:0/3294349729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde98101dc0 0x7fde98102220 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fde8c009910 tx=0x7fde8c009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.519+0000 7fde9c90c700 1 -- 172.21.15.116:0/3294349729 shutdown_connections 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.519+0000 7fde9c90c700 1 --2- 172.21.15.116:0/3294349729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde98101dc0 0x7fde98102220 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.519+0000 7fde9c90c700 1 --2- 172.21.15.116:0/3294349729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100bc0 0x7fde98100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.519+0000 7fde9c90c700 1 -- 172.21.15.116:0/3294349729 >> 172.21.15.116:0/3294349729 conn(0x7fde980fc140 msgr2=0x7fde980fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.519+0000 7fde9c90c700 1 -- 172.21.15.116:0/3294349729 shutdown_connections 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.519+0000 7fde9c90c700 1 -- 172.21.15.116:0/3294349729 wait complete. 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde9c90c700 1 Processor -- start 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde9c90c700 1 -- start start 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde9c90c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100bc0 0x7fde98196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde9c90c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde98101dc0 0x7fde981968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde9c90c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde98196ed0 con 0x7fde98101dc0 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde9c90c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde98197010 con 0x7fde98100bc0 2024-03-04T16:11:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde95d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde98101dc0 0x7fde981968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde9659c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100bc0 0x7fde98196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde95d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde98101dc0 0x7fde981968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59004/0 (socket says 172.21.15.116:59004) 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde9659c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100bc0 0x7fde98196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:60890/0 (socket says 172.21.15.116:60890) 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde95d9b700 1 -- 172.21.15.116:0/986742275 learned_addr learned my addr 172.21.15.116:0/986742275 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde95d9b700 1 -- 172.21.15.116:0/986742275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100bc0 msgr2=0x7fde98196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde95d9b700 1 --2- 172.21.15.116:0/986742275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100bc0 0x7fde98196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.520+0000 7fde95d9b700 1 -- 172.21.15.116:0/986742275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0095f0 con 0x7fde98101dc0 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.521+0000 7fde95d9b700 1 --2- 172.21.15.116:0/986742275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde98101dc0 0x7fde981968b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fde8c005b40 tx=0x7fde8c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.521+0000 7fde877fe700 1 -- 172.21.15.116:0/986742275 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde8c01d070 con 0x7fde98101dc0 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.521+0000 7fde9c90c700 1 -- 172.21.15.116:0/986742275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9819ba60 con 0x7fde98101dc0 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.521+0000 7fde877fe700 1 -- 172.21.15.116:0/986742275 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fde8c00f460 con 0x7fde98101dc0 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.521+0000 7fde877fe700 1 -- 172.21.15.116:0/986742275 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde8c021620 con 0x7fde98101dc0 2024-03-04T16:11:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.521+0000 7fde9c90c700 1 -- 172.21.15.116:0/986742275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde9819bf50 con 0x7fde98101dc0 2024-03-04T16:11:19.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.522+0000 7fde9c90c700 1 -- 172.21.15.116:0/986742275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde98109720 con 0x7fde98101dc0 2024-03-04T16:11:19.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.522+0000 7fde877fe700 1 -- 172.21.15.116:0/986742275 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fde8c00fa40 con 0x7fde98101dc0 2024-03-04T16:11:19.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.523+0000 7fde877fe700 1 --2- 172.21.15.116:0/986742275 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fde8006d8a0 0x7fde8006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:19.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.523+0000 7fde9659c700 1 --2- 172.21.15.116:0/986742275 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fde8006d8a0 0x7fde8006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:19.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.523+0000 7fde877fe700 1 -- 172.21.15.116:0/986742275 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fde8c08e7b0 con 0x7fde98101dc0 2024-03-04T16:11:19.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.523+0000 7fde9659c700 1 --2- 172.21.15.116:0/986742275 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fde8006d8a0 0x7fde8006fd60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde98101c20 tx=0x7fde88007400 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:19.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.528+0000 7fde877fe700 1 -- 172.21.15.116:0/986742275 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fde8c05ba90 con 0x7fde98101dc0 2024-03-04T16:11:19.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.742+0000 7fde9c90c700 1 -- 172.21.15.116:0/986742275 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde9819c6a0 con 0x7fde8006d8a0 2024-03-04T16:11:19.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.743+0000 7fde877fe700 1 -- 172.21.15.116:0/986742275 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fde9819c6a0 con 0x7fde8006d8a0 2024-03-04T16:11:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 -- 172.21.15.116:0/986742275 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fde8006d8a0 msgr2=0x7fde8006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 --2- 172.21.15.116:0/986742275 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fde8006d8a0 0x7fde8006fd60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde98101c20 tx=0x7fde88007400 comp rx=0 tx=0).stop 2024-03-04T16:11:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 -- 172.21.15.116:0/986742275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde98101dc0 msgr2=0x7fde981968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 --2- 172.21.15.116:0/986742275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde98101dc0 0x7fde981968b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fde8c005b40 tx=0x7fde8c00bd70 comp rx=0 tx=0).stop 2024-03-04T16:11:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 -- 172.21.15.116:0/986742275 shutdown_connections 2024-03-04T16:11:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 --2- 172.21.15.116:0/986742275 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fde8006d8a0 0x7fde8006fd60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:19.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 --2- 172.21.15.116:0/986742275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100bc0 0x7fde98196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:19.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 --2- 172.21.15.116:0/986742275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde98101dc0 0x7fde981968b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:19.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 -- 172.21.15.116:0/986742275 >> 172.21.15.116:0/986742275 conn(0x7fde980fc140 msgr2=0x7fde98104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:19.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 -- 172.21.15.116:0/986742275 shutdown_connections 2024-03-04T16:11:19.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.745+0000 7fde9c90c700 1 -- 172.21.15.116:0/986742275 wait complete. 2024-03-04T16:11:19.756 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:11:19.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.849+0000 7f4c9439d700 1 -- 172.21.15.116:0/1743580063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c8c101dc0 msgr2=0x7f4c8c102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:19.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.849+0000 7f4c9439d700 1 --2- 172.21.15.116:0/1743580063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c8c101dc0 0x7f4c8c102220 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009910 tx=0x7f4c7c009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:19.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.849+0000 7f4c9439d700 1 -- 172.21.15.116:0/1743580063 shutdown_connections 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.849+0000 7f4c9439d700 1 --2- 172.21.15.116:0/1743580063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c8c101dc0 0x7f4c8c102220 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.849+0000 7f4c9439d700 1 --2- 172.21.15.116:0/1743580063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c8c100bc0 0x7f4c8c100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.849+0000 7f4c9439d700 1 -- 172.21.15.116:0/1743580063 >> 172.21.15.116:0/1743580063 conn(0x7f4c8c0fc140 msgr2=0x7f4c8c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.849+0000 7f4c9439d700 1 -- 172.21.15.116:0/1743580063 shutdown_connections 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.849+0000 7f4c9439d700 1 -- 172.21.15.116:0/1743580063 wait complete. 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.850+0000 7f4c9439d700 1 Processor -- start 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.850+0000 7f4c9439d700 1 -- start start 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.850+0000 7f4c9439d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c8c100bc0 0x7f4c8c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.850+0000 7f4c9439d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c8c101dc0 0x7f4c8c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.850+0000 7f4c9439d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c0736c0 con 0x7f4c8c101dc0 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.850+0000 7f4c9439d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c073800 con 0x7f4c8c100bc0 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.850+0000 7f4c91938700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c8c101dc0 0x7f4c8c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.850+0000 7f4c91938700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c8c101dc0 0x7f4c8c073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59022/0 (socket says 172.21.15.116:59022) 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.850+0000 7f4c91938700 1 -- 172.21.15.116:0/3645303731 learned_addr learned my addr 172.21.15.116:0/3645303731 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.850+0000 7f4c92139700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c8c100bc0 0x7f4c8c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.851+0000 7f4c91938700 1 -- 172.21.15.116:0/3645303731 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c8c100bc0 msgr2=0x7f4c8c074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.851+0000 7f4c91938700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c8c100bc0 0x7f4c8c074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.851+0000 7f4c91938700 1 -- 172.21.15.116:0/3645303731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c0095f0 con 0x7f4c8c101dc0 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.851+0000 7f4c91938700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c8c101dc0 0x7f4c8c073180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c006010 tx=0x7f4c7c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.851+0000 7f4c837fe700 1 -- 172.21.15.116:0/3645303731 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c7c01d070 con 0x7f4c8c101dc0 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.851+0000 7f4c837fe700 1 -- 172.21.15.116:0/3645303731 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4c7c00f460 con 0x7f4c8c101dc0 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.851+0000 7f4c837fe700 1 -- 172.21.15.116:0/3645303731 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c7c021640 con 0x7f4c8c101dc0 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.851+0000 7f4c9439d700 1 -- 172.21.15.116:0/3645303731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8c073a80 con 0x7f4c8c101dc0 2024-03-04T16:11:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.851+0000 7f4c9439d700 1 -- 172.21.15.116:0/3645303731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8c073f70 con 0x7f4c8c101dc0 2024-03-04T16:11:19.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.852+0000 7f4c9439d700 1 -- 172.21.15.116:0/3645303731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8c109720 con 0x7f4c8c101dc0 2024-03-04T16:11:19.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.853+0000 7f4c837fe700 1 -- 172.21.15.116:0/3645303731 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f4c7c00fab0 con 0x7f4c8c101dc0 2024-03-04T16:11:19.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.853+0000 7f4c837fe700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c7806d8a0 0x7f4c7806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:19.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.853+0000 7f4c837fe700 1 -- 172.21.15.116:0/3645303731 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4c7c08e780 con 0x7f4c8c101dc0 2024-03-04T16:11:19.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.853+0000 7f4c92139700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c7806d8a0 0x7f4c7806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:19.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.853+0000 7f4c92139700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c7806d8a0 0x7f4c7806fd60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c101c20 tx=0x7f4c8800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:19.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:19.858+0000 7f4c837fe700 1 -- 172.21.15.116:0/3645303731 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c7c05bae0 con 0x7f4c8c101dc0 2024-03-04T16:11:20.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.068+0000 7f4c9439d700 1 -- 172.21.15.116:0/3645303731 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c8c106710 con 0x7f4c7806d8a0 2024-03-04T16:11:20.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.069+0000 7f4c837fe700 1 -- 172.21.15.116:0/3645303731 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4c8c106710 con 0x7f4c7806d8a0 2024-03-04T16:11:20.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.071+0000 7f4c9439d700 1 -- 172.21.15.116:0/3645303731 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c7806d8a0 msgr2=0x7f4c7806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.071+0000 7f4c9439d700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c7806d8a0 0x7f4c7806fd60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c101c20 tx=0x7f4c8800b410 comp rx=0 tx=0).stop 2024-03-04T16:11:20.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.071+0000 7f4c9439d700 1 -- 172.21.15.116:0/3645303731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c8c101dc0 msgr2=0x7f4c8c073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.071+0000 7f4c9439d700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c8c101dc0 0x7f4c8c073180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c006010 tx=0x7f4c7c00bd70 comp rx=0 tx=0).stop 2024-03-04T16:11:20.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.072+0000 7f4c9439d700 1 -- 172.21.15.116:0/3645303731 shutdown_connections 2024-03-04T16:11:20.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.072+0000 7f4c9439d700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c7806d8a0 0x7f4c7806fd60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.072+0000 7f4c9439d700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c8c100bc0 0x7f4c8c074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.072+0000 7f4c9439d700 1 --2- 172.21.15.116:0/3645303731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c8c101dc0 0x7f4c8c073180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.072+0000 7f4c9439d700 1 -- 172.21.15.116:0/3645303731 >> 172.21.15.116:0/3645303731 conn(0x7f4c8c0fc140 msgr2=0x7f4c8c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:20.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.072+0000 7f4c9439d700 1 -- 172.21.15.116:0/3645303731 shutdown_connections 2024-03-04T16:11:20.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.072+0000 7f4c9439d700 1 -- 172.21.15.116:0/3645303731 wait complete. 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.174+0000 7f8be0038700 1 -- 172.21.15.116:0/914799369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd8103fa0 msgr2=0x7f8bd8106390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.174+0000 7f8be0038700 1 --2- 172.21.15.116:0/914799369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd8103fa0 0x7f8bd8106390 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009910 tx=0x7f8bc8009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.174+0000 7f8be0038700 1 -- 172.21.15.116:0/914799369 shutdown_connections 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.174+0000 7f8be0038700 1 --2- 172.21.15.116:0/914799369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd8103fa0 0x7f8bd8106390 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.174+0000 7f8be0038700 1 --2- 172.21.15.116:0/914799369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bd8101670 0x7f8bd8103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.174+0000 7f8be0038700 1 -- 172.21.15.116:0/914799369 >> 172.21.15.116:0/914799369 conn(0x7f8bd80fb080 msgr2=0x7f8bd80fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.174+0000 7f8be0038700 1 -- 172.21.15.116:0/914799369 shutdown_connections 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.174+0000 7f8be0038700 1 -- 172.21.15.116:0/914799369 wait complete. 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.175+0000 7f8be0038700 1 Processor -- start 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.175+0000 7f8be0038700 1 -- start start 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.175+0000 7f8be0038700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd8101670 0x7f8bd8100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.175+0000 7f8be0038700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bd8103fa0 0x7f8bd80ff620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.175+0000 7f8be0038700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd80ffb60 con 0x7f8bd8101670 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.175+0000 7f8be0038700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd80ffca0 con 0x7f8bd8103fa0 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.175+0000 7f8bdddd4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd8101670 0x7f8bd8100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.175+0000 7f8bdddd4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd8101670 0x7f8bd8100fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59028/0 (socket says 172.21.15.116:59028) 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.175+0000 7f8bdddd4700 1 -- 172.21.15.116:0/3533468628 learned_addr learned my addr 172.21.15.116:0/3533468628 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.176+0000 7f8bdd5d3700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bd8103fa0 0x7f8bd80ff620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.176+0000 7f8bdddd4700 1 -- 172.21.15.116:0/3533468628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bd8103fa0 msgr2=0x7f8bd80ff620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.176+0000 7f8bdddd4700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bd8103fa0 0x7f8bd80ff620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.176+0000 7f8bdddd4700 1 -- 172.21.15.116:0/3533468628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80095f0 con 0x7f8bd8101670 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.176+0000 7f8bdddd4700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd8101670 0x7f8bd8100fd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8bd400d900 tx=0x7f8bd400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.176+0000 7f8bceffd700 1 -- 172.21.15.116:0/3533468628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bd4004a00 con 0x7f8bd8101670 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.176+0000 7f8bceffd700 1 -- 172.21.15.116:0/3533468628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8bd40054d0 con 0x7f8bd8101670 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.176+0000 7f8be0038700 1 -- 172.21.15.116:0/3533468628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd80fff80 con 0x7f8bd8101670 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.176+0000 7f8bceffd700 1 -- 172.21.15.116:0/3533468628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bd4009d00 con 0x7f8bd8101670 2024-03-04T16:11:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.176+0000 7f8be0038700 1 -- 172.21.15.116:0/3533468628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd819e3b0 con 0x7f8bd8101670 2024-03-04T16:11:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.177+0000 7f8be0038700 1 -- 172.21.15.116:0/3533468628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd8190570 con 0x7f8bd8101670 2024-03-04T16:11:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.178+0000 7f8bceffd700 1 -- 172.21.15.116:0/3533468628 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f8bd4005020 con 0x7f8bd8101670 2024-03-04T16:11:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.178+0000 7f8bceffd700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8bc406d8a0 0x7f8bc406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.178+0000 7f8bceffd700 1 -- 172.21.15.116:0/3533468628 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8bd408d390 con 0x7f8bd8101670 2024-03-04T16:11:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.178+0000 7f8bdd5d3700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8bc406d8a0 0x7f8bc406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.179+0000 7f8bdd5d3700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8bc406d8a0 0x7f8bc406fd60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8006010 tx=0x7f8bc800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.183+0000 7f8bceffd700 1 -- 172.21.15.116:0/3533468628 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8bd405a6f0 con 0x7f8bd8101670 2024-03-04T16:11:20.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.388+0000 7f8be0038700 1 -- 172.21.15.116:0/3533468628 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bd80611d0 con 0x7f8bc406d8a0 2024-03-04T16:11:20.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.400+0000 7f8bceffd700 1 -- 172.21.15.116:0/3533468628 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f8bd80611d0 con 0x7f8bc406d8a0 2024-03-04T16:11:20.400 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:11:20.400 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (14m) 14s ago 15m 25.5M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (16m) 14s ago 16m 8674k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (15m) 54s ago 15m 8548k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (4m) 14s ago 16m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (4m) 54s ago 14m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (14m) 14s ago 15m 87.8M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (20s) 14s ago 10m 12.2M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9d5061a093c6 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (39s) 14s ago 10m 14.8M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4bf8fc434069 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (10m) 54s ago 10m 18.1M - 18.2.1 5be31c24972a 0c2230a5114e 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (10m) 54s ago 10m 143M - 18.2.1 5be31c24972a ea030a249d64 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (6m) 14s ago 17m 507M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (6m) 54s ago 14m 450M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (5m) 14s ago 17m 51.0M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (5m) 54s ago 14m 36.8M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (16m) 14s ago 16m 15.6M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:11:20.401 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (14m) 54s ago 14m 16.7M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:11:20.402 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (3m) 14s ago 14m 92.0M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:11:20.402 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (3m) 14s ago 13m 104M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:11:20.402 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (2m) 14s ago 13m 85.0M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:11:20.402 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (2m) 54s ago 12m 109M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6b01e97c8f9b 2024-03-04T16:11:20.402 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (92s) 54s ago 12m 105M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 5ecc30e5a86c 2024-03-04T16:11:20.402 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (57s) 54s ago 11m 12.6M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 1083700b476d 2024-03-04T16:11:20.402 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (6m) 14s ago 15m 73.6M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 -- 172.21.15.116:0/3533468628 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8bc406d8a0 msgr2=0x7f8bc406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8bc406d8a0 0x7f8bc406fd60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8006010 tx=0x7f8bc800b540 comp rx=0 tx=0).stop 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 -- 172.21.15.116:0/3533468628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd8101670 msgr2=0x7f8bd8100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd8101670 0x7f8bd8100fd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8bd400d900 tx=0x7f8bd400dc10 comp rx=0 tx=0).stop 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 -- 172.21.15.116:0/3533468628 shutdown_connections 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8bc406d8a0 0x7f8bc406fd60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd8101670 0x7f8bd8100fd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 --2- 172.21.15.116:0/3533468628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bd8103fa0 0x7f8bd80ff620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 -- 172.21.15.116:0/3533468628 >> 172.21.15.116:0/3533468628 conn(0x7f8bd80fb080 msgr2=0x7f8bd80fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 -- 172.21.15.116:0/3533468628 shutdown_connections 2024-03-04T16:11:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.402+0000 7f8be0038700 1 -- 172.21.15.116:0/3533468628 wait complete. 2024-03-04T16:11:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:20 smithi116 ceph-mon[94164]: pgmap v199: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 7.0 MiB/s rd, 3 op/s 2024-03-04T16:11:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.507+0000 7f629cfec700 1 -- 172.21.15.116:0/378084018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298100cb0 msgr2=0x7f6298101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.507+0000 7f629cfec700 1 --2- 172.21.15.116:0/378084018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298100cb0 0x7f6298101110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f628c009910 tx=0x7f628c009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.507+0000 7f629cfec700 1 -- 172.21.15.116:0/378084018 shutdown_connections 2024-03-04T16:11:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.507+0000 7f629cfec700 1 --2- 172.21.15.116:0/378084018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298100cb0 0x7f6298101110 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.507+0000 7f629cfec700 1 --2- 172.21.15.116:0/378084018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62980ffab0 0x7f62980ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.507+0000 7f629cfec700 1 -- 172.21.15.116:0/378084018 >> 172.21.15.116:0/378084018 conn(0x7f62980fb030 msgr2=0x7f62980fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.507+0000 7f629cfec700 1 -- 172.21.15.116:0/378084018 shutdown_connections 2024-03-04T16:11:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.507+0000 7f629cfec700 1 -- 172.21.15.116:0/378084018 wait complete. 2024-03-04T16:11:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.507+0000 7f629cfec700 1 Processor -- start 2024-03-04T16:11:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.507+0000 7f629cfec700 1 -- start start 2024-03-04T16:11:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629cfec700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62980ffab0 0x7f6298196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629cfec700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6298100cb0 0x7f62981968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629cfec700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6298196ec0 con 0x7f62980ffab0 2024-03-04T16:11:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629cfec700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6298197000 con 0x7f6298100cb0 2024-03-04T16:11:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62980ffab0 0x7f6298196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62980ffab0 0x7f6298196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59056/0 (socket says 172.21.15.116:59056) 2024-03-04T16:11:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629659c700 1 -- 172.21.15.116:0/1415638907 learned_addr learned my addr 172.21.15.116:0/1415638907 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f6295d9b700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6298100cb0 0x7f62981968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629659c700 1 -- 172.21.15.116:0/1415638907 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6298100cb0 msgr2=0x7f62981968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629659c700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6298100cb0 0x7f62981968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629659c700 1 -- 172.21.15.116:0/1415638907 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c0095f0 con 0x7f62980ffab0 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629659c700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62980ffab0 0x7f6298196360 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f628800d900 tx=0x7f628800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f62877fe700 1 -- 172.21.15.116:0/1415638907 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62880041f0 con 0x7f62980ffab0 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629cfec700 1 -- 172.21.15.116:0/1415638907 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629819bab0 con 0x7f62980ffab0 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f62877fe700 1 -- 172.21.15.116:0/1415638907 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6288004350 con 0x7f62980ffab0 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f62877fe700 1 -- 172.21.15.116:0/1415638907 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6288003d20 con 0x7f62980ffab0 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.508+0000 7f629cfec700 1 -- 172.21.15.116:0/1415638907 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629819c000 con 0x7f62980ffab0 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.509+0000 7f629cfec700 1 -- 172.21.15.116:0/1415638907 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6298109710 con 0x7f62980ffab0 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.510+0000 7f62877fe700 1 -- 172.21.15.116:0/1415638907 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f6288009730 con 0x7f62980ffab0 2024-03-04T16:11:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.510+0000 7f62877fe700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f628006d8a0 0x7f628006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.510+0000 7f6295d9b700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f628006d8a0 0x7f628006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.510+0000 7f62877fe700 1 -- 172.21.15.116:0/1415638907 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f628808d3f0 con 0x7f62980ffab0 2024-03-04T16:11:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.511+0000 7f6295d9b700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f628006d8a0 0x7f628006fd60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f628c006010 tx=0x7f628c00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.515+0000 7f62877fe700 1 -- 172.21.15.116:0/1415638907 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f628805a6d0 con 0x7f62980ffab0 2024-03-04T16:11:20.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.793+0000 7f629cfec700 1 -- 172.21.15.116:0/1415638907 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f629819c610 con 0x7f62980ffab0 2024-03-04T16:11:20.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.793+0000 7f62877fe700 1 -- 172.21.15.116:0/1415638907 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f6288003e80 con 0x7f62980ffab0 2024-03-04T16:11:20.794 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:11:20.794 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:11:20.794 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:11:20.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:11:20.795 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:11:20.796 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-03-04T16:11:20.796 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 12 2024-03-04T16:11:20.796 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:11:20.796 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 -- 172.21.15.116:0/1415638907 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f628006d8a0 msgr2=0x7f628006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f628006d8a0 0x7f628006fd60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f628c006010 tx=0x7f628c00b540 comp rx=0 tx=0).stop 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 -- 172.21.15.116:0/1415638907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62980ffab0 msgr2=0x7f6298196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62980ffab0 0x7f6298196360 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f628800d900 tx=0x7f628800dc10 comp rx=0 tx=0).stop 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 -- 172.21.15.116:0/1415638907 shutdown_connections 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f628006d8a0 0x7f628006fd60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62980ffab0 0x7f6298196360 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 --2- 172.21.15.116:0/1415638907 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6298100cb0 0x7f62981968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 -- 172.21.15.116:0/1415638907 >> 172.21.15.116:0/1415638907 conn(0x7f62980fb030 msgr2=0x7f6298103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 -- 172.21.15.116:0/1415638907 shutdown_connections 2024-03-04T16:11:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.796+0000 7f629cfec700 1 -- 172.21.15.116:0/1415638907 wait complete. 2024-03-04T16:11:20.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:20 smithi152 ceph-mon[82845]: pgmap v199: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 7.0 MiB/s rd, 3 op/s 2024-03-04T16:11:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.897+0000 7f3c564ec700 1 -- 172.21.15.116:0/1309802916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c50100bb0 msgr2=0x7f3c50100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.897+0000 7f3c564ec700 1 --2- 172.21.15.116:0/1309802916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c50100bb0 0x7f3c50100fd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009910 tx=0x7f3c40009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.898+0000 7f3c564ec700 1 -- 172.21.15.116:0/1309802916 shutdown_connections 2024-03-04T16:11:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.898+0000 7f3c564ec700 1 --2- 172.21.15.116:0/1309802916 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c50101db0 0x7f3c50102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.898+0000 7f3c564ec700 1 --2- 172.21.15.116:0/1309802916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c50100bb0 0x7f3c50100fd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.898+0000 7f3c564ec700 1 -- 172.21.15.116:0/1309802916 >> 172.21.15.116:0/1309802916 conn(0x7f3c500fc110 msgr2=0x7f3c500fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.898+0000 7f3c564ec700 1 -- 172.21.15.116:0/1309802916 shutdown_connections 2024-03-04T16:11:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.898+0000 7f3c564ec700 1 -- 172.21.15.116:0/1309802916 wait complete. 2024-03-04T16:11:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.898+0000 7f3c564ec700 1 Processor -- start 2024-03-04T16:11:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.898+0000 7f3c564ec700 1 -- start start 2024-03-04T16:11:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c564ec700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c50100bb0 0x7f3c50074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c564ec700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c50101db0 0x7f3c50073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c564ec700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c500736c0 con 0x7f3c50100bb0 2024-03-04T16:11:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c564ec700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c50073800 con 0x7f3c50101db0 2024-03-04T16:11:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c50100bb0 0x7f3c50074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c50100bb0 0x7f3c50074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59074/0 (socket says 172.21.15.116:59074) 2024-03-04T16:11:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4ffff700 1 -- 172.21.15.116:0/191895006 learned_addr learned my addr 172.21.15.116:0/191895006 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4f7fe700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c50101db0 0x7f3c50073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4ffff700 1 -- 172.21.15.116:0/191895006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c50101db0 msgr2=0x7f3c50073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4ffff700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c50101db0 0x7f3c50073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4ffff700 1 -- 172.21.15.116:0/191895006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c400095f0 con 0x7f3c50100bb0 2024-03-04T16:11:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4ffff700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c50100bb0 0x7f3c50074b30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000bda0 tx=0x7f3c4000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4d7fa700 1 -- 172.21.15.116:0/191895006 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c4001d070 con 0x7f3c50100bb0 2024-03-04T16:11:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4d7fa700 1 -- 172.21.15.116:0/191895006 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3c40005dc0 con 0x7f3c50100bb0 2024-03-04T16:11:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c564ec700 1 -- 172.21.15.116:0/191895006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c50073a80 con 0x7f3c50100bb0 2024-03-04T16:11:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.899+0000 7f3c4d7fa700 1 -- 172.21.15.116:0/191895006 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c40021410 con 0x7f3c50100bb0 2024-03-04T16:11:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.900+0000 7f3c564ec700 1 -- 172.21.15.116:0/191895006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c50073f70 con 0x7f3c50100bb0 2024-03-04T16:11:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.901+0000 7f3c564ec700 1 -- 172.21.15.116:0/191895006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c50109710 con 0x7f3c50100bb0 2024-03-04T16:11:20.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.901+0000 7f3c4d7fa700 1 -- 172.21.15.116:0/191895006 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f3c4000f7f0 con 0x7f3c50100bb0 2024-03-04T16:11:20.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.901+0000 7f3c4d7fa700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c3806d8a0 0x7f3c3806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:20.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.902+0000 7f3c4d7fa700 1 -- 172.21.15.116:0/191895006 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3c4008e770 con 0x7f3c50100bb0 2024-03-04T16:11:20.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.902+0000 7f3c4f7fe700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c3806d8a0 0x7f3c3806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:20.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.902+0000 7f3c4f7fe700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c3806d8a0 0x7f3c3806fd60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3c44005950 tx=0x7f3c4400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:20.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:20.906+0000 7f3c4d7fa700 1 -- 172.21.15.116:0/191895006 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3c4005bad0 con 0x7f3c50100bb0 2024-03-04T16:11:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.147+0000 7f3c564ec700 1 -- 172.21.15.116:0/191895006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3c5019e870 con 0x7f3c50100bb0 2024-03-04T16:11:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.147+0000 7f3c4d7fa700 1 -- 172.21.15.116:0/191895006 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1672 (secure 0 0 0) 0x7f3c4005b660 con 0x7f3c50100bb0 2024-03-04T16:11:21.148 INFO:teuthology.orchestra.run.smithi116.stdout:e20 2024-03-04T16:11:21.148 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:11:21.148 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:11:21.148 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:11:21.148 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:11:21.148 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 20 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:11:14.449915+0000 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:11:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 83 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24257} 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{0:24257} state up:replay seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] compat {c=[1],r=[1],i=[7ff]}] 2024-03-04T16:11:21.150 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:11:21.151 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:11:21.151 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:11:21.151 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:11:21.151 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{-1:34400} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:11:21.151 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6828/786823228,v1:172.21.15.116:6829/786823228] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.149+0000 7f3c564ec700 1 -- 172.21.15.116:0/191895006 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c3806d8a0 msgr2=0x7f3c3806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.149+0000 7f3c564ec700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c3806d8a0 0x7f3c3806fd60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3c44005950 tx=0x7f3c4400b410 comp rx=0 tx=0).stop 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.149+0000 7f3c564ec700 1 -- 172.21.15.116:0/191895006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c50100bb0 msgr2=0x7f3c50074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.149+0000 7f3c564ec700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c50100bb0 0x7f3c50074b30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000bda0 tx=0x7f3c4000bdd0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.149+0000 7f3c564ec700 1 -- 172.21.15.116:0/191895006 shutdown_connections 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.149+0000 7f3c564ec700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3c3806d8a0 0x7f3c3806fd60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.149+0000 7f3c564ec700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c50100bb0 0x7f3c50074b30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.149+0000 7f3c564ec700 1 --2- 172.21.15.116:0/191895006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3c50101db0 0x7f3c50073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.149+0000 7f3c564ec700 1 -- 172.21.15.116:0/191895006 >> 172.21.15.116:0/191895006 conn(0x7f3c500fc110 msgr2=0x7f3c50104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.150+0000 7f3c564ec700 1 -- 172.21.15.116:0/191895006 shutdown_connections 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.150+0000 7f3c564ec700 1 -- 172.21.15.116:0/191895006 wait complete. 2024-03-04T16:11:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 20 2024-03-04T16:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.254+0000 7f5fea659700 1 -- 172.21.15.116:0/746462216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe40737f0 msgr2=0x7f5fe4073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.254+0000 7f5fea659700 1 --2- 172.21.15.116:0/746462216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe40737f0 0x7f5fe4073c70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009910 tx=0x7f5fd8009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.254+0000 7f5fea659700 1 -- 172.21.15.116:0/746462216 shutdown_connections 2024-03-04T16:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.254+0000 7f5fea659700 1 --2- 172.21.15.116:0/746462216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe40737f0 0x7f5fe4073c70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.254+0000 7f5fea659700 1 --2- 172.21.15.116:0/746462216 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5fe4074dc0 0x7f5fe4073220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.254+0000 7f5fea659700 1 -- 172.21.15.116:0/746462216 >> 172.21.15.116:0/746462216 conn(0x7f5fe40fc140 msgr2=0x7f5fe40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.254+0000 7f5fea659700 1 -- 172.21.15.116:0/746462216 shutdown_connections 2024-03-04T16:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.254+0000 7f5fea659700 1 -- 172.21.15.116:0/746462216 wait complete. 2024-03-04T16:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.254+0000 7f5fea659700 1 Processor -- start 2024-03-04T16:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.254+0000 7f5fea659700 1 -- start start 2024-03-04T16:11:21.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fea659700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe40737f0 0x7f5fe41985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:21.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fea659700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5fe4074dc0 0x7f5fe4198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:21.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fea659700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4199120 con 0x7f5fe40737f0 2024-03-04T16:11:21.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fea659700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4199260 con 0x7f5fe4074dc0 2024-03-04T16:11:21.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe40737f0 0x7f5fe41985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:21.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe40737f0 0x7f5fe41985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59090/0 (socket says 172.21.15.116:59090) 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe3fff700 1 -- 172.21.15.116:0/1012523864 learned_addr learned my addr 172.21.15.116:0/1012523864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe37fe700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5fe4074dc0 0x7f5fe4198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe3fff700 1 -- 172.21.15.116:0/1012523864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5fe4074dc0 msgr2=0x7f5fe4198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe3fff700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5fe4074dc0 0x7f5fe4198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe3fff700 1 -- 172.21.15.116:0/1012523864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd80095f0 con 0x7f5fe40737f0 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe3fff700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe40737f0 0x7f5fe41985c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400e8f0 tx=0x7f5fd400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe17fa700 1 -- 172.21.15.116:0/1012523864 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd4009790 con 0x7f5fe40737f0 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe17fa700 1 -- 172.21.15.116:0/1012523864 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5fd400c810 con 0x7f5fe40737f0 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fea659700 1 -- 172.21.15.116:0/1012523864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe419dd10 con 0x7f5fe40737f0 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fe17fa700 1 -- 172.21.15.116:0/1012523864 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd40055b0 con 0x7f5fe40737f0 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.255+0000 7f5fea659700 1 -- 172.21.15.116:0/1012523864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe419e200 con 0x7f5fe40737f0 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.256+0000 7f5fea659700 1 -- 172.21.15.116:0/1012523864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe410b950 con 0x7f5fe40737f0 2024-03-04T16:11:21.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.257+0000 7f5fe17fa700 1 -- 172.21.15.116:0/1012523864 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f5fd400c980 con 0x7f5fe40737f0 2024-03-04T16:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.257+0000 7f5fe17fa700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5fcc06d8a0 0x7f5fcc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.258+0000 7f5fe37fe700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5fcc06d8a0 0x7f5fcc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.258+0000 7f5fe17fa700 1 -- 172.21.15.116:0/1012523864 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f5fd4014070 con 0x7f5fe40737f0 2024-03-04T16:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.258+0000 7f5fe37fe700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5fcc06d8a0 0x7f5fcc06fd60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800b6f0 tx=0x7f5fd8005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:21.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.262+0000 7f5fe17fa700 1 -- 172.21.15.116:0/1012523864 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5fd405aa50 con 0x7f5fe40737f0 2024-03-04T16:11:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.473+0000 7f5fea659700 1 -- 172.21.15.116:0/1012523864 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fe41080c0 con 0x7f5fcc06d8a0 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.474+0000 7f5fe17fa700 1 -- 172.21.15.116:0/1012523864 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5fe41080c0 con 0x7f5fcc06d8a0 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr", 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stdout: "osd" 2024-03-04T16:11:21.475 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:11:21.476 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "14/23 daemons upgraded", 2024-03-04T16:11:21.476 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mds daemons", 2024-03-04T16:11:21.476 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:11:21.476 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:11:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.476+0000 7f5fea659700 1 -- 172.21.15.116:0/1012523864 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5fcc06d8a0 msgr2=0x7f5fcc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.476+0000 7f5fea659700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5fcc06d8a0 0x7f5fcc06fd60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800b6f0 tx=0x7f5fd8005940 comp rx=0 tx=0).stop 2024-03-04T16:11:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.476+0000 7f5fea659700 1 -- 172.21.15.116:0/1012523864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe40737f0 msgr2=0x7f5fe41985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.476+0000 7f5fea659700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe40737f0 0x7f5fe41985c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400e8f0 tx=0x7f5fd400ec00 comp rx=0 tx=0).stop 2024-03-04T16:11:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.477+0000 7f5fea659700 1 -- 172.21.15.116:0/1012523864 shutdown_connections 2024-03-04T16:11:21.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.477+0000 7f5fea659700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5fcc06d8a0 0x7f5fcc06fd60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.477+0000 7f5fea659700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe40737f0 0x7f5fe41985c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.477+0000 7f5fea659700 1 --2- 172.21.15.116:0/1012523864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5fe4074dc0 0x7f5fe4198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.477+0000 7f5fea659700 1 -- 172.21.15.116:0/1012523864 >> 172.21.15.116:0/1012523864 conn(0x7f5fe40fc140 msgr2=0x7f5fe41069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:21.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.477+0000 7f5fea659700 1 -- 172.21.15.116:0/1012523864 shutdown_connections 2024-03-04T16:11:21.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.477+0000 7f5fea659700 1 -- 172.21.15.116:0/1012523864 wait complete. 2024-03-04T16:11:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.580+0000 7f0510976700 1 -- 172.21.15.116:0/1793673316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f050c100bc0 msgr2=0x7f050c100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.580+0000 7f0510976700 1 --2- 172.21.15.116:0/1793673316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f050c100bc0 0x7f050c100fe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009910 tx=0x7f04fc009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.581+0000 7f0510976700 1 -- 172.21.15.116:0/1793673316 shutdown_connections 2024-03-04T16:11:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.581+0000 7f0510976700 1 --2- 172.21.15.116:0/1793673316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f050c101dc0 0x7f050c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.581+0000 7f0510976700 1 --2- 172.21.15.116:0/1793673316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f050c100bc0 0x7f050c100fe0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.581+0000 7f0510976700 1 -- 172.21.15.116:0/1793673316 >> 172.21.15.116:0/1793673316 conn(0x7f050c0fc140 msgr2=0x7f050c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.581+0000 7f0510976700 1 -- 172.21.15.116:0/1793673316 shutdown_connections 2024-03-04T16:11:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.581+0000 7f0510976700 1 -- 172.21.15.116:0/1793673316 wait complete. 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.581+0000 7f0510976700 1 Processor -- start 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.581+0000 7f0510976700 1 -- start start 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f0510976700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f050c100bc0 0x7f050c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f0510976700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f050c101dc0 0x7f050c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f0510976700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0736c0 con 0x7f050c100bc0 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f0510976700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c073800 con 0x7f050c101dc0 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f050a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f050c100bc0 0x7f050c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f0509d9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f050c101dc0 0x7f050c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f050a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f050c100bc0 0x7f050c074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59106/0 (socket says 172.21.15.116:59106) 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f050a59c700 1 -- 172.21.15.116:0/3301015458 learned_addr learned my addr 172.21.15.116:0/3301015458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f050a59c700 1 -- 172.21.15.116:0/3301015458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f050c101dc0 msgr2=0x7f050c073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f050a59c700 1 --2- 172.21.15.116:0/3301015458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f050c101dc0 0x7f050c073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f050a59c700 1 -- 172.21.15.116:0/3301015458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc0095f0 con 0x7f050c100bc0 2024-03-04T16:11:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f050a59c700 1 --2- 172.21.15.116:0/3301015458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f050c100bc0 0x7f050c074b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00bda0 tx=0x7f04fc00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f04fb7fe700 1 -- 172.21.15.116:0/3301015458 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04fc01d070 con 0x7f050c100bc0 2024-03-04T16:11:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f04fb7fe700 1 -- 172.21.15.116:0/3301015458 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f04fc00f460 con 0x7f050c100bc0 2024-03-04T16:11:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f0510976700 1 -- 172.21.15.116:0/3301015458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c073a80 con 0x7f050c100bc0 2024-03-04T16:11:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f04fb7fe700 1 -- 172.21.15.116:0/3301015458 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04fc021620 con 0x7f050c100bc0 2024-03-04T16:11:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.582+0000 7f0510976700 1 -- 172.21.15.116:0/3301015458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050c073f70 con 0x7f050c100bc0 2024-03-04T16:11:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.583+0000 7f0510976700 1 -- 172.21.15.116:0/3301015458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050c109720 con 0x7f050c100bc0 2024-03-04T16:11:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.584+0000 7f04fb7fe700 1 -- 172.21.15.116:0/3301015458 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f04fc00f5d0 con 0x7f050c100bc0 2024-03-04T16:11:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.584+0000 7f04fb7fe700 1 --2- 172.21.15.116:0/3301015458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04f406d8a0 0x7f04f406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.584+0000 7f04fb7fe700 1 -- 172.21.15.116:0/3301015458 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f04fc08e7d0 con 0x7f050c100bc0 2024-03-04T16:11:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.584+0000 7f0509d9b700 1 --2- 172.21.15.116:0/3301015458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04f406d8a0 0x7f04f406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.585+0000 7f0509d9b700 1 --2- 172.21.15.116:0/3301015458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04f406d8a0 0x7f04f406fd60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0500005950 tx=0x7f050000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.589+0000 7f04fb7fe700 1 -- 172.21.15.116:0/3301015458 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f04fc05bb30 con 0x7f050c100bc0 2024-03-04T16:11:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:21 smithi152 ceph-mon[82845]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:21.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:21 smithi152 ceph-mon[82845]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:21.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:21 smithi152 ceph-mon[82845]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:21.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:21 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1415638907' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:21.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:21 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/191895006' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:11:21.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:21 smithi116 ceph-mon[94164]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:21.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:21 smithi116 ceph-mon[94164]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:21.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:21 smithi116 ceph-mon[94164]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:21 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1415638907' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:21.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:21 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/191895006' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:11:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.865+0000 7f0510976700 1 -- 172.21.15.116:0/3301015458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f050c19e810 con 0x7f050c100bc0 2024-03-04T16:11:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.865+0000 7f04fb7fe700 1 -- 172.21.15.116:0/3301015458 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+107 (secure 0 0 0) 0x7f04fc05b6c0 con 0x7f050c100bc0 2024-03-04T16:11:21.866 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 filesystem is degraded 2024-03-04T16:11:21.866 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-04T16:11:21.866 INFO:teuthology.orchestra.run.smithi116.stdout: fs cephfs is degraded 2024-03-04T16:11:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 -- 172.21.15.116:0/3301015458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04f406d8a0 msgr2=0x7f04f406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 --2- 172.21.15.116:0/3301015458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04f406d8a0 0x7f04f406fd60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0500005950 tx=0x7f050000b410 comp rx=0 tx=0).stop 2024-03-04T16:11:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 -- 172.21.15.116:0/3301015458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f050c100bc0 msgr2=0x7f050c074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 --2- 172.21.15.116:0/3301015458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f050c100bc0 0x7f050c074b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00bda0 tx=0x7f04fc00bdd0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 -- 172.21.15.116:0/3301015458 shutdown_connections 2024-03-04T16:11:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 --2- 172.21.15.116:0/3301015458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04f406d8a0 0x7f04f406fd60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 --2- 172.21.15.116:0/3301015458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f050c100bc0 0x7f050c074b30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 --2- 172.21.15.116:0/3301015458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f050c101dc0 0x7f050c073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 -- 172.21.15.116:0/3301015458 >> 172.21.15.116:0/3301015458 conn(0x7f050c0fc140 msgr2=0x7f050c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 -- 172.21.15.116:0/3301015458 shutdown_connections 2024-03-04T16:11:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:21.867+0000 7f0510976700 1 -- 172.21.15.116:0/3301015458 wait complete. 2024-03-04T16:11:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:22 smithi116 ceph-mon[94164]: pgmap v200: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 5 op/s 2024-03-04T16:11:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:22 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3301015458' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:11:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:22 smithi152 ceph-mon[82845]: pgmap v200: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 5 op/s 2024-03-04T16:11:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:22 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3301015458' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:11:23.687 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:23 smithi116 ceph-mon[94164]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:23.687 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:23.687 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:23.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:23 smithi152 ceph-mon[82845]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:23.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:23.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:24.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:24 smithi116 ceph-mon[94164]: pgmap v201: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 5 op/s 2024-03-04T16:11:24.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:24 smithi152 ceph-mon[82845]: pgmap v201: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 5 op/s 2024-03-04T16:11:25.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:25 smithi116 ceph-mon[94164]: reconnect by client.24297 192.168.0.1:0/2088783451 after 0.00099999 2024-03-04T16:11:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:25 smithi116 ceph-mon[94164]: reconnect by client.14538 192.168.0.1:0/1893604702 after 0.00099999 2024-03-04T16:11:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:25 smithi116 ceph-mon[94164]: mds.? [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] up:reconnect 2024-03-04T16:11:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:25 smithi116 ceph-mon[94164]: fsmap cephfs:1/1 {0=cephfs.smithi152.kqoybf=up:reconnect} 2 up:standby 2024-03-04T16:11:25.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:25 smithi116 ceph-mon[94164]: pgmap v202: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 6 op/s 2024-03-04T16:11:25.769 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:25 smithi152 ceph-mon[82845]: reconnect by client.24297 192.168.0.1:0/2088783451 after 0.00099999 2024-03-04T16:11:25.770 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:25 smithi152 ceph-mon[82845]: reconnect by client.14538 192.168.0.1:0/1893604702 after 0.00099999 2024-03-04T16:11:25.770 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:25 smithi152 ceph-mon[82845]: mds.? [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] up:reconnect 2024-03-04T16:11:25.770 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:25 smithi152 ceph-mon[82845]: fsmap cephfs:1/1 {0=cephfs.smithi152.kqoybf=up:reconnect} 2 up:standby 2024-03-04T16:11:25.770 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:25 smithi152 ceph-mon[82845]: pgmap v202: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 6 op/s 2024-03-04T16:11:26.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:26 smithi116 ceph-mon[94164]: mds.? [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] up:rejoin 2024-03-04T16:11:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:26 smithi116 ceph-mon[94164]: fsmap cephfs:1/1 {0=cephfs.smithi152.kqoybf=up:rejoin} 2 up:standby 2024-03-04T16:11:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:26 smithi116 ceph-mon[94164]: daemon mds.cephfs.smithi152.kqoybf is now active in filesystem cephfs as rank 0 2024-03-04T16:11:26.762 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:26 smithi152 ceph-mon[82845]: mds.? [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] up:rejoin 2024-03-04T16:11:26.762 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:26 smithi152 ceph-mon[82845]: fsmap cephfs:1/1 {0=cephfs.smithi152.kqoybf=up:rejoin} 2 up:standby 2024-03-04T16:11:26.762 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:26 smithi152 ceph-mon[82845]: daemon mds.cephfs.smithi152.kqoybf is now active in filesystem cephfs as rank 0 2024-03-04T16:11:27.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:27 smithi116 ceph-mon[94164]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T16:11:27.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:27 smithi116 ceph-mon[94164]: Cluster is now healthy 2024-03-04T16:11:27.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:27 smithi116 ceph-mon[94164]: mds.? [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] up:active 2024-03-04T16:11:27.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:27 smithi116 ceph-mon[94164]: fsmap cephfs:1 {0=cephfs.smithi152.kqoybf=up:active} 2 up:standby 2024-03-04T16:11:27.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:27 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:27.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:27 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:27.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:27 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:27.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:27 smithi116 ceph-mon[94164]: pgmap v203: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 85 B/s wr, 5 op/s 2024-03-04T16:11:27.822 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:27 smithi152 ceph-mon[82845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T16:11:27.822 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:27 smithi152 ceph-mon[82845]: Cluster is now healthy 2024-03-04T16:11:27.822 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:27 smithi152 ceph-mon[82845]: mds.? [v2:172.21.15.152:6826/1672192052,v1:172.21.15.152:6827/1672192052] up:active 2024-03-04T16:11:27.822 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:27 smithi152 ceph-mon[82845]: fsmap cephfs:1 {0=cephfs.smithi152.kqoybf=up:active} 2 up:standby 2024-03-04T16:11:27.823 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:27 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:27.823 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:27 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:27.823 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:27 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:27.823 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:27 smithi152 ceph-mon[82845]: pgmap v203: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 85 B/s wr, 5 op/s 2024-03-04T16:11:28.710 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:28 smithi152 ceph-mon[82845]: mds.? [v2:172.21.15.152:6824/3736721533,v1:172.21.15.152:6825/3736721533] up:boot 2024-03-04T16:11:28.710 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:28 smithi152 ceph-mon[82845]: fsmap cephfs:1 {0=cephfs.smithi152.kqoybf=up:active} 3 up:standby 2024-03-04T16:11:28.710 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:11:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:28 smithi116 ceph-mon[94164]: mds.? [v2:172.21.15.152:6824/3736721533,v1:172.21.15.152:6825/3736721533] up:boot 2024-03-04T16:11:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:28 smithi116 ceph-mon[94164]: fsmap cephfs:1 {0=cephfs.smithi152.kqoybf=up:active} 3 up:standby 2024-03-04T16:11:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:11:29.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:29 smithi152 ceph-mon[82845]: pgmap v204: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 170 B/s wr, 5 op/s 2024-03-04T16:11:29.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:29 smithi116 ceph-mon[94164]: pgmap v204: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 170 B/s wr, 5 op/s 2024-03-04T16:11:31.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:31.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:31.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:31.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:32.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:32 smithi152 ceph-mon[82845]: pgmap v205: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 170 B/s wr, 6 op/s 2024-03-04T16:11:32.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:32.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:32.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:32 smithi116 ceph-mon[94164]: pgmap v205: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 170 B/s wr, 6 op/s 2024-03-04T16:11:32.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:32.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:34 smithi116 ceph-mon[94164]: pgmap v206: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 170 B/s wr, 3 op/s 2024-03-04T16:11:34.521 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:34 smithi152 ceph-mon[82845]: pgmap v206: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 170 B/s wr, 3 op/s 2024-03-04T16:11:35.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: Detected new or changed devices on smithi152 2024-03-04T16:11:35.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:35.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:35.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:35.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:35.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:35.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:35.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:35.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:35.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:35.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:35.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi152.kqoybf"]}]: dispatch 2024-03-04T16:11:35.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: Upgrade: It appears safe to stop mds.cephfs.smithi152.kqoybf 2024-03-04T16:11:35.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:35 smithi152 ceph-mon[82845]: pgmap v207: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 4.3 KiB/s wr, 5 op/s 2024-03-04T16:11:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: Detected new or changed devices on smithi152 2024-03-04T16:11:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi152.kqoybf"]}]: dispatch 2024-03-04T16:11:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: Upgrade: It appears safe to stop mds.cephfs.smithi152.kqoybf 2024-03-04T16:11:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:35 smithi116 ceph-mon[94164]: pgmap v207: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 4.3 KiB/s wr, 5 op/s 2024-03-04T16:11:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:37 smithi152 ceph-mon[82845]: Upgrade: Updating mds.cephfs.smithi152.kqoybf 2024-03-04T16:11:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.kqoybf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:11:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:37 smithi152 ceph-mon[82845]: Deploying daemon mds.cephfs.smithi152.kqoybf on smithi152 2024-03-04T16:11:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:37 smithi116 ceph-mon[94164]: Upgrade: Updating mds.cephfs.smithi152.kqoybf 2024-03-04T16:11:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.kqoybf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T16:11:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:37 smithi116 ceph-mon[94164]: Deploying daemon mds.cephfs.smithi152.kqoybf on smithi152 2024-03-04T16:11:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:38 smithi116 ceph-mon[94164]: pgmap v208: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 345 KiB/s rd, 4.3 KiB/s wr, 5 op/s 2024-03-04T16:11:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:38.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:38 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116[94159]: 2024-03-04T16:11:38.458+0000 7fa4b3157700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T16:11:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:38 smithi152 ceph-mon[82845]: pgmap v208: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 345 KiB/s rd, 4.3 KiB/s wr, 5 op/s 2024-03-04T16:11:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:39.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:39 smithi116 ceph-mon[94164]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T16:11:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:39 smithi116 ceph-mon[94164]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T16:11:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:39 smithi116 ceph-mon[94164]: osdmap e84: 6 total, 6 up, 6 in 2024-03-04T16:11:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:39 smithi116 ceph-mon[94164]: Standby daemon mds.cephfs.smithi116.pjrmzv assigned to filesystem cephfs as rank 0 2024-03-04T16:11:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:39 smithi116 ceph-mon[94164]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T16:11:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:39 smithi116 ceph-mon[94164]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-03-04T16:11:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:39 smithi116 ceph-mon[94164]: fsmap cephfs:1/1 {0=cephfs.smithi116.pjrmzv=up:replay} 2 up:standby 2024-03-04T16:11:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:11:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:39 smithi152 ceph-mon[82845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T16:11:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:39 smithi152 ceph-mon[82845]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T16:11:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:39 smithi152 ceph-mon[82845]: osdmap e84: 6 total, 6 up, 6 in 2024-03-04T16:11:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:39 smithi152 ceph-mon[82845]: Standby daemon mds.cephfs.smithi116.pjrmzv assigned to filesystem cephfs as rank 0 2024-03-04T16:11:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:39 smithi152 ceph-mon[82845]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T16:11:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:39 smithi152 ceph-mon[82845]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-03-04T16:11:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:39 smithi152 ceph-mon[82845]: fsmap cephfs:1/1 {0=cephfs.smithi116.pjrmzv=up:replay} 2 up:standby 2024-03-04T16:11:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:11:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:40 smithi116 ceph-mon[94164]: pgmap v210: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 5.0 KiB/s wr, 5 op/s 2024-03-04T16:11:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:40 smithi152 ceph-mon[82845]: pgmap v210: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 5.0 KiB/s wr, 5 op/s 2024-03-04T16:11:42.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:42 smithi116 ceph-mon[94164]: pgmap v211: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 6.2 MiB/s rd, 5.0 KiB/s wr, 4 op/s 2024-03-04T16:11:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:42 smithi152 ceph-mon[82845]: pgmap v211: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 6.2 MiB/s rd, 5.0 KiB/s wr, 4 op/s 2024-03-04T16:11:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:44 smithi116 ceph-mon[94164]: pgmap v212: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 7.4 MiB/s rd, 5.0 KiB/s wr, 5 op/s 2024-03-04T16:11:44.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:44 smithi152 ceph-mon[82845]: pgmap v212: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 7.4 MiB/s rd, 5.0 KiB/s wr, 5 op/s 2024-03-04T16:11:45.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:45 smithi116 ceph-mon[94164]: pgmap v213: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 0 B/s wr, 5 op/s 2024-03-04T16:11:45.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:45 smithi152 ceph-mon[82845]: pgmap v213: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 0 B/s wr, 5 op/s 2024-03-04T16:11:48.230 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:48 smithi152 ceph-mon[82845]: pgmap v214: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 0 B/s wr, 6 op/s 2024-03-04T16:11:48.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:48 smithi116 ceph-mon[94164]: pgmap v214: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 0 B/s wr, 6 op/s 2024-03-04T16:11:49.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:49 smithi152 ceph-mon[82845]: reconnect by client.14538 192.168.0.1:0/1893604702 after 0.000999985 2024-03-04T16:11:49.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:49 smithi152 ceph-mon[82845]: reconnect by client.24297 192.168.0.1:0/2088783451 after 0.000999985 2024-03-04T16:11:49.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:49 smithi152 ceph-mon[82845]: mds.? [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] up:reconnect 2024-03-04T16:11:49.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:49 smithi152 ceph-mon[82845]: fsmap cephfs:1/1 {0=cephfs.smithi116.pjrmzv=up:reconnect} 2 up:standby 2024-03-04T16:11:49.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:49.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:49.358 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:49 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:49 smithi116 ceph-mon[94164]: reconnect by client.14538 192.168.0.1:0/1893604702 after 0.000999985 2024-03-04T16:11:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:49 smithi116 ceph-mon[94164]: reconnect by client.24297 192.168.0.1:0/2088783451 after 0.000999985 2024-03-04T16:11:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:49 smithi116 ceph-mon[94164]: mds.? [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] up:reconnect 2024-03-04T16:11:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:49 smithi116 ceph-mon[94164]: fsmap cephfs:1/1 {0=cephfs.smithi116.pjrmzv=up:reconnect} 2 up:standby 2024-03-04T16:11:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:49.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:49 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:50 smithi116 ceph-mon[94164]: pgmap v215: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 6 op/s 2024-03-04T16:11:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:50 smithi116 ceph-mon[94164]: mds.? [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] up:rejoin 2024-03-04T16:11:50.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:50 smithi116 ceph-mon[94164]: mds.? [v2:172.21.15.152:6826/1659417498,v1:172.21.15.152:6827/1659417498] up:boot 2024-03-04T16:11:50.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:50 smithi116 ceph-mon[94164]: fsmap cephfs:1/1 {0=cephfs.smithi116.pjrmzv=up:rejoin} 3 up:standby 2024-03-04T16:11:50.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:50 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:11:50.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:50 smithi116 ceph-mon[94164]: daemon mds.cephfs.smithi116.pjrmzv is now active in filesystem cephfs as rank 0 2024-03-04T16:11:50.498 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:50 smithi152 ceph-mon[82845]: pgmap v215: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 6 op/s 2024-03-04T16:11:50.498 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:50 smithi152 ceph-mon[82845]: mds.? [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] up:rejoin 2024-03-04T16:11:50.498 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:50 smithi152 ceph-mon[82845]: mds.? [v2:172.21.15.152:6826/1659417498,v1:172.21.15.152:6827/1659417498] up:boot 2024-03-04T16:11:50.498 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:50 smithi152 ceph-mon[82845]: fsmap cephfs:1/1 {0=cephfs.smithi116.pjrmzv=up:rejoin} 3 up:standby 2024-03-04T16:11:50.498 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:50 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:11:50.498 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:50 smithi152 ceph-mon[82845]: daemon mds.cephfs.smithi116.pjrmzv is now active in filesystem cephfs as rank 0 2024-03-04T16:11:51.422 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:51 smithi116 ceph-mon[94164]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T16:11:51.422 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:51 smithi116 ceph-mon[94164]: Cluster is now healthy 2024-03-04T16:11:51.422 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:51 smithi116 ceph-mon[94164]: mds.? [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] up:active 2024-03-04T16:11:51.422 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:51 smithi116 ceph-mon[94164]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 3 up:standby 2024-03-04T16:11:51.558 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:51 smithi152 ceph-mon[82845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T16:11:51.558 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:51 smithi152 ceph-mon[82845]: Cluster is now healthy 2024-03-04T16:11:51.558 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:51 smithi152 ceph-mon[82845]: mds.? [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] up:active 2024-03-04T16:11:51.558 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:51 smithi152 ceph-mon[82845]: fsmap cephfs:1 {0=cephfs.smithi116.pjrmzv=up:active} 3 up:standby 2024-03-04T16:11:51.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.980+0000 7fe654c66700 1 -- 172.21.15.116:0/1995339873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe650101610 msgr2=0x7fe650103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:51.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.980+0000 7fe654c66700 1 --2- 172.21.15.116:0/1995339873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe650101610 0x7fe650103a00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe640009910 tx=0x7fe640009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:51.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.980+0000 7fe654c66700 1 -- 172.21.15.116:0/1995339873 shutdown_connections 2024-03-04T16:11:51.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.980+0000 7fe654c66700 1 --2- 172.21.15.116:0/1995339873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe650103f40 0x7fe650106330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:51.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.980+0000 7fe654c66700 1 --2- 172.21.15.116:0/1995339873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe650101610 0x7fe650103a00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:51.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.980+0000 7fe654c66700 1 -- 172.21.15.116:0/1995339873 >> 172.21.15.116:0/1995339873 conn(0x7fe6500fb020 msgr2=0x7fe6500fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:51.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.980+0000 7fe654c66700 1 -- 172.21.15.116:0/1995339873 shutdown_connections 2024-03-04T16:11:51.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.980+0000 7fe654c66700 1 -- 172.21.15.116:0/1995339873 wait complete. 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.981+0000 7fe654c66700 1 Processor -- start 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.981+0000 7fe654c66700 1 -- start start 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.981+0000 7fe654c66700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe650101610 0x7fe6501962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.981+0000 7fe654c66700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe650103f40 0x7fe6501967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.981+0000 7fe654c66700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe650196e00 con 0x7fe650103f40 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.981+0000 7fe654c66700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe650196f40 con 0x7fe650101610 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe64dd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe650103f40 0x7fe6501967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe64dd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe650103f40 0x7fe6501967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53660/0 (socket says 172.21.15.116:53660) 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe64dd9b700 1 -- 172.21.15.116:0/103695913 learned_addr learned my addr 172.21.15.116:0/103695913 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe64e59c700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe650101610 0x7fe6501962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe64dd9b700 1 -- 172.21.15.116:0/103695913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe650101610 msgr2=0x7fe6501962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe64dd9b700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe650101610 0x7fe6501962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe64dd9b700 1 -- 172.21.15.116:0/103695913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6400095f0 con 0x7fe650103f40 2024-03-04T16:11:51.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe64dd9b700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe650103f40 0x7fe6501967e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe64400d8d0 tx=0x7fe64400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:51.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe63f7fe700 1 -- 172.21.15.116:0/103695913 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6440098a0 con 0x7fe650103f40 2024-03-04T16:11:51.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe63f7fe700 1 -- 172.21.15.116:0/103695913 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe644010460 con 0x7fe650103f40 2024-03-04T16:11:51.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe654c66700 1 -- 172.21.15.116:0/103695913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65019b9f0 con 0x7fe650103f40 2024-03-04T16:11:51.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe63f7fe700 1 -- 172.21.15.116:0/103695913 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe64400f5f0 con 0x7fe650103f40 2024-03-04T16:11:51.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.982+0000 7fe654c66700 1 -- 172.21.15.116:0/103695913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65019bf40 con 0x7fe650103f40 2024-03-04T16:11:51.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.983+0000 7fe654c66700 1 -- 172.21.15.116:0/103695913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6501904f0 con 0x7fe650103f40 2024-03-04T16:11:51.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.984+0000 7fe63f7fe700 1 -- 172.21.15.116:0/103695913 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fe644009a00 con 0x7fe650103f40 2024-03-04T16:11:51.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.984+0000 7fe63f7fe700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe63806d8a0 0x7fe63806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:51.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.984+0000 7fe63f7fe700 1 -- 172.21.15.116:0/103695913 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fe64408d440 con 0x7fe650103f40 2024-03-04T16:11:51.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.984+0000 7fe64e59c700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe63806d8a0 0x7fe63806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:51.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.985+0000 7fe64e59c700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe63806d8a0 0x7fe63806fd60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe640006010 tx=0x7fe64000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:51.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:51.989+0000 7fe63f7fe700 1 -- 172.21.15.116:0/103695913 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe64405a6d0 con 0x7fe650103f40 2024-03-04T16:11:52.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.202+0000 7fe654c66700 1 -- 172.21.15.116:0/103695913 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6500611d0 con 0x7fe63806d8a0 2024-03-04T16:11:52.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.203+0000 7fe63f7fe700 1 -- 172.21.15.116:0/103695913 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe6500611d0 con 0x7fe63806d8a0 2024-03-04T16:11:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.205+0000 7fe654c66700 1 -- 172.21.15.116:0/103695913 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe63806d8a0 msgr2=0x7fe63806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.205+0000 7fe654c66700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe63806d8a0 0x7fe63806fd60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe640006010 tx=0x7fe64000b540 comp rx=0 tx=0).stop 2024-03-04T16:11:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.205+0000 7fe654c66700 1 -- 172.21.15.116:0/103695913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe650103f40 msgr2=0x7fe6501967e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.205+0000 7fe654c66700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe650103f40 0x7fe6501967e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe64400d8d0 tx=0x7fe64400dbe0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.205+0000 7fe654c66700 1 -- 172.21.15.116:0/103695913 shutdown_connections 2024-03-04T16:11:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.205+0000 7fe654c66700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe63806d8a0 0x7fe63806fd60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.205+0000 7fe654c66700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe650101610 0x7fe6501962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.205+0000 7fe654c66700 1 --2- 172.21.15.116:0/103695913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe650103f40 0x7fe6501967e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.205+0000 7fe654c66700 1 -- 172.21.15.116:0/103695913 >> 172.21.15.116:0/103695913 conn(0x7fe6500fb020 msgr2=0x7fe6500fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.206+0000 7fe654c66700 1 -- 172.21.15.116:0/103695913 shutdown_connections 2024-03-04T16:11:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.206+0000 7fe654c66700 1 -- 172.21.15.116:0/103695913 wait complete. 2024-03-04T16:11:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:11:52.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.313+0000 7f8e564e4700 1 -- 172.21.15.116:0/3493676983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e50101670 msgr2=0x7f8e50103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.313+0000 7f8e564e4700 1 --2- 172.21.15.116:0/3493676983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e50101670 0x7f8e50103a60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8e40009910 tx=0x7f8e40009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:52.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 -- 172.21.15.116:0/3493676983 shutdown_connections 2024-03-04T16:11:52.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 --2- 172.21.15.116:0/3493676983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e50103fa0 0x7f8e50106390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 --2- 172.21.15.116:0/3493676983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e50101670 0x7f8e50103a60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 -- 172.21.15.116:0/3493676983 >> 172.21.15.116:0/3493676983 conn(0x7f8e500fb080 msgr2=0x7f8e500fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:52.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 -- 172.21.15.116:0/3493676983 shutdown_connections 2024-03-04T16:11:52.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 -- 172.21.15.116:0/3493676983 wait complete. 2024-03-04T16:11:52.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 Processor -- start 2024-03-04T16:11:52.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 -- start start 2024-03-04T16:11:52.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e50101670 0x7f8e50196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:52.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e50103fa0 0x7f8e50196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:52.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e50196e70 con 0x7f8e50101670 2024-03-04T16:11:52.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.314+0000 7f8e564e4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e50196fb0 con 0x7f8e50103fa0 2024-03-04T16:11:52.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e50101670 0x7f8e50196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:52.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e50101670 0x7f8e50196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53684/0 (socket says 172.21.15.116:53684) 2024-03-04T16:11:52.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4ffff700 1 -- 172.21.15.116:0/253984302 learned_addr learned my addr 172.21.15.116:0/253984302 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:52.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4f7fe700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e50103fa0 0x7f8e50196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:52.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4ffff700 1 -- 172.21.15.116:0/253984302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e50103fa0 msgr2=0x7f8e50196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4ffff700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e50103fa0 0x7f8e50196850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4ffff700 1 -- 172.21.15.116:0/253984302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e400095f0 con 0x7f8e50101670 2024-03-04T16:11:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4ffff700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e50101670 0x7f8e50196310 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000c00 tx=0x7f8e4000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4d7fa700 1 -- 172.21.15.116:0/253984302 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e4001d070 con 0x7f8e50101670 2024-03-04T16:11:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4d7fa700 1 -- 172.21.15.116:0/253984302 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8e40005dc0 con 0x7f8e50101670 2024-03-04T16:11:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e564e4700 1 -- 172.21.15.116:0/253984302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5019ba00 con 0x7f8e50101670 2024-03-04T16:11:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e4d7fa700 1 -- 172.21.15.116:0/253984302 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e400226a0 con 0x7f8e50101670 2024-03-04T16:11:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.315+0000 7f8e564e4700 1 -- 172.21.15.116:0/253984302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5019bef0 con 0x7f8e50101670 2024-03-04T16:11:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.316+0000 7f8e564e4700 1 -- 172.21.15.116:0/253984302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e50190550 con 0x7f8e50101670 2024-03-04T16:11:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.317+0000 7f8e4d7fa700 1 -- 172.21.15.116:0/253984302 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f8e40017400 con 0x7f8e50101670 2024-03-04T16:11:52.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.317+0000 7f8e4d7fa700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8e3806d8a0 0x7f8e3806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:52.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.317+0000 7f8e4d7fa700 1 -- 172.21.15.116:0/253984302 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8e4008eb20 con 0x7f8e50101670 2024-03-04T16:11:52.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.317+0000 7f8e4f7fe700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8e3806d8a0 0x7f8e3806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:52.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.318+0000 7f8e4f7fe700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8e3806d8a0 0x7f8e3806fd60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009c60 tx=0x7f8e44009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:52.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.322+0000 7f8e4d7fa700 1 -- 172.21.15.116:0/253984302 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8e4005bdb0 con 0x7f8e50101670 2024-03-04T16:11:52.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.536+0000 7f8e564e4700 1 -- 172.21.15.116:0/253984302 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e500611d0 con 0x7f8e3806d8a0 2024-03-04T16:11:52.537 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:52 smithi116 ceph-mon[94164]: pgmap v216: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 85 B/s wr, 6 op/s 2024-03-04T16:11:52.537 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:52.537 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:52.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.537+0000 7f8e4d7fa700 1 -- 172.21.15.116:0/253984302 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8e500611d0 con 0x7f8e3806d8a0 2024-03-04T16:11:52.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 -- 172.21.15.116:0/253984302 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8e3806d8a0 msgr2=0x7f8e3806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8e3806d8a0 0x7f8e3806fd60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009c60 tx=0x7f8e44009260 comp rx=0 tx=0).stop 2024-03-04T16:11:52.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 -- 172.21.15.116:0/253984302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e50101670 msgr2=0x7f8e50196310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e50101670 0x7f8e50196310 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000c00 tx=0x7f8e4000f690 comp rx=0 tx=0).stop 2024-03-04T16:11:52.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 -- 172.21.15.116:0/253984302 shutdown_connections 2024-03-04T16:11:52.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8e3806d8a0 0x7f8e3806fd60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e50101670 0x7f8e50196310 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 --2- 172.21.15.116:0/253984302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e50103fa0 0x7f8e50196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 -- 172.21.15.116:0/253984302 >> 172.21.15.116:0/253984302 conn(0x7f8e500fb080 msgr2=0x7f8e500fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:52.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 -- 172.21.15.116:0/253984302 shutdown_connections 2024-03-04T16:11:52.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.540+0000 7f8e564e4700 1 -- 172.21.15.116:0/253984302 wait complete. 2024-03-04T16:11:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:52 smithi152 ceph-mon[82845]: pgmap v216: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 85 B/s wr, 6 op/s 2024-03-04T16:11:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:52.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.643+0000 7f2d338a4700 1 -- 172.21.15.116:0/2169770507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c101db0 msgr2=0x7f2d2c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.643+0000 7f2d338a4700 1 --2- 172.21.15.116:0/2169770507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c101db0 0x7f2d2c102210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c009910 tx=0x7f2d1c009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:52.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.643+0000 7f2d338a4700 1 -- 172.21.15.116:0/2169770507 shutdown_connections 2024-03-04T16:11:52.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.643+0000 7f2d338a4700 1 --2- 172.21.15.116:0/2169770507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c101db0 0x7f2d2c102210 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.643+0000 7f2d338a4700 1 --2- 172.21.15.116:0/2169770507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d2c100bb0 0x7f2d2c100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.643+0000 7f2d338a4700 1 -- 172.21.15.116:0/2169770507 >> 172.21.15.116:0/2169770507 conn(0x7f2d2c0fc110 msgr2=0x7f2d2c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.643+0000 7f2d338a4700 1 -- 172.21.15.116:0/2169770507 shutdown_connections 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.643+0000 7f2d338a4700 1 -- 172.21.15.116:0/2169770507 wait complete. 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d338a4700 1 Processor -- start 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d338a4700 1 -- start start 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d338a4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c100bb0 0x7f2d2c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d338a4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d2c101db0 0x7f2d2c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d338a4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0736c0 con 0x7f2d2c100bb0 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d338a4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c073800 con 0x7f2d2c101db0 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d31640700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c100bb0 0x7f2d2c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d31640700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c100bb0 0x7f2d2c074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53710/0 (socket says 172.21.15.116:53710) 2024-03-04T16:11:52.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d31640700 1 -- 172.21.15.116:0/1496658166 learned_addr learned my addr 172.21.15.116:0/1496658166 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:52.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d30e3f700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d2c101db0 0x7f2d2c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:52.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d31640700 1 -- 172.21.15.116:0/1496658166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d2c101db0 msgr2=0x7f2d2c073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d31640700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d2c101db0 0x7f2d2c073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d31640700 1 -- 172.21.15.116:0/1496658166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c0095f0 con 0x7f2d2c100bb0 2024-03-04T16:11:52.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.644+0000 7f2d31640700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c100bb0 0x7f2d2c074b30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800d900 tx=0x7f2d2800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:52.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.645+0000 7f2d227fc700 1 -- 172.21.15.116:0/1496658166 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d2800b920 con 0x7f2d2c100bb0 2024-03-04T16:11:52.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.645+0000 7f2d227fc700 1 -- 172.21.15.116:0/1496658166 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2d28010460 con 0x7f2d2c100bb0 2024-03-04T16:11:52.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.645+0000 7f2d227fc700 1 -- 172.21.15.116:0/1496658166 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d2800f590 con 0x7f2d2c100bb0 2024-03-04T16:11:52.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.645+0000 7f2d338a4700 1 -- 172.21.15.116:0/1496658166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c073ae0 con 0x7f2d2c100bb0 2024-03-04T16:11:52.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.645+0000 7f2d338a4700 1 -- 172.21.15.116:0/1496658166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c19e3c0 con 0x7f2d2c100bb0 2024-03-04T16:11:52.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.646+0000 7f2d338a4700 1 -- 172.21.15.116:0/1496658166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c109710 con 0x7f2d2c100bb0 2024-03-04T16:11:52.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.646+0000 7f2d227fc700 1 -- 172.21.15.116:0/1496658166 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f2d280105d0 con 0x7f2d2c100bb0 2024-03-04T16:11:52.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.647+0000 7f2d227fc700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2d1806d8a0 0x7f2d1806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:52.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.647+0000 7f2d30e3f700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2d1806d8a0 0x7f2d1806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:52.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.647+0000 7f2d227fc700 1 -- 172.21.15.116:0/1496658166 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f2d2808d130 con 0x7f2d2c100bb0 2024-03-04T16:11:52.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.647+0000 7f2d30e3f700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2d1806d8a0 0x7f2d1806fd60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c006010 tx=0x7f2d1c00b560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:52.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.652+0000 7f2d227fc700 1 -- 172.21.15.116:0/1496658166 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2d2805a440 con 0x7f2d2c100bb0 2024-03-04T16:11:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.856+0000 7f2d338a4700 1 -- 172.21.15.116:0/1496658166 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d2c106700 con 0x7f2d1806d8a0 2024-03-04T16:11:52.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.867+0000 7f2d227fc700 1 -- 172.21.15.116:0/1496658166 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2d2c106700 con 0x7f2d1806d8a0 2024-03-04T16:11:52.868 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:11:52.868 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (14m) 46s ago 16m 25.5M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:11:52.868 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (16m) 46s ago 16m 8674k - 18.2.1 5be31c24972a 9f3c889f1b79 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (15m) 1s ago 15m 8850k - 18.2.1 5be31c24972a c698a13a09ef 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (4m) 46s ago 16m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (4m) 1s ago 15m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (14m) 46s ago 16m 87.8M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (52s) 46s ago 10m 12.2M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9d5061a093c6 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (72s) 46s ago 10m 14.8M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4bf8fc434069 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (5s) 1s ago 10m 11.3M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3abb2cd19d94 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (26s) 1s ago 10m 15.7M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4f752bc549b2 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (7m) 46s ago 17m 507M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (6m) 1s ago 15m 451M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (6m) 46s ago 18m 51.0M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (5m) 1s ago 15m 40.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (16m) 46s ago 16m 15.6M - 1.5.0 0da6a335fe13 619dce10e0de 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (15m) 1s ago 15m 16.5M - 1.5.0 0da6a335fe13 fa7026047732 2024-03-04T16:11:52.869 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (4m) 46s ago 14m 92.0M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:11:52.870 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (3m) 46s ago 14m 104M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:11:52.870 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (3m) 46s ago 13m 85.0M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:11:52.870 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (2m) 1s ago 13m 121M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6b01e97c8f9b 2024-03-04T16:11:52.870 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (2m) 1s ago 12m 112M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 5ecc30e5a86c 2024-03-04T16:11:52.870 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (90s) 1s ago 12m 101M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 1083700b476d 2024-03-04T16:11:52.870 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (6m) 46s ago 15m 73.6M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 -- 172.21.15.116:0/1496658166 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2d1806d8a0 msgr2=0x7f2d1806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2d1806d8a0 0x7f2d1806fd60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c006010 tx=0x7f2d1c00b560 comp rx=0 tx=0).stop 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 -- 172.21.15.116:0/1496658166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c100bb0 msgr2=0x7f2d2c074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c100bb0 0x7f2d2c074b30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800d900 tx=0x7f2d2800dc10 comp rx=0 tx=0).stop 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 -- 172.21.15.116:0/1496658166 shutdown_connections 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f2d1806d8a0 0x7f2d1806fd60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c100bb0 0x7f2d2c074b30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 --2- 172.21.15.116:0/1496658166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d2c101db0 0x7f2d2c073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 -- 172.21.15.116:0/1496658166 >> 172.21.15.116:0/1496658166 conn(0x7f2d2c0fc110 msgr2=0x7f2d2c104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 -- 172.21.15.116:0/1496658166 shutdown_connections 2024-03-04T16:11:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.870+0000 7f2d338a4700 1 -- 172.21.15.116:0/1496658166 wait complete. 2024-03-04T16:11:52.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.971+0000 7f3e815a1700 1 -- 172.21.15.116:0/1981720695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e7c074d60 msgr2=0x7f3e7c0731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.971+0000 7f3e815a1700 1 --2- 172.21.15.116:0/1981720695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e7c074d60 0x7f3e7c0731c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009910 tx=0x7f3e70009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:52.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.971+0000 7f3e815a1700 1 -- 172.21.15.116:0/1981720695 shutdown_connections 2024-03-04T16:11:52.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.971+0000 7f3e815a1700 1 --2- 172.21.15.116:0/1981720695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e7c073790 0x7f3e7c073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.971+0000 7f3e815a1700 1 --2- 172.21.15.116:0/1981720695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e7c074d60 0x7f3e7c0731c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.971+0000 7f3e815a1700 1 -- 172.21.15.116:0/1981720695 >> 172.21.15.116:0/1981720695 conn(0x7f3e7c0fbfd0 msgr2=0x7f3e7c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:52.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.971+0000 7f3e815a1700 1 -- 172.21.15.116:0/1981720695 shutdown_connections 2024-03-04T16:11:52.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.971+0000 7f3e815a1700 1 -- 172.21.15.116:0/1981720695 wait complete. 2024-03-04T16:11:52.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.972+0000 7f3e815a1700 1 Processor -- start 2024-03-04T16:11:52.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.972+0000 7f3e815a1700 1 -- start start 2024-03-04T16:11:52.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.972+0000 7f3e815a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e7c073790 0x7f3e7c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.972+0000 7f3e815a1700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e7c074d60 0x7f3e7c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.972+0000 7f3e815a1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c198ee0 con 0x7f3e7c073790 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.972+0000 7f3e815a1700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c199020 con 0x7f3e7c074d60 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.972+0000 7f3e7bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e7c073790 0x7f3e7c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e7bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e7c073790 0x7f3e7c198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53728/0 (socket says 172.21.15.116:53728) 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e7bfff700 1 -- 172.21.15.116:0/1578034405 learned_addr learned my addr 172.21.15.116:0/1578034405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e7b7fe700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e7c074d60 0x7f3e7c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e7bfff700 1 -- 172.21.15.116:0/1578034405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e7c074d60 msgr2=0x7f3e7c1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e7bfff700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e7c074d60 0x7f3e7c1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e7bfff700 1 -- 172.21.15.116:0/1578034405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e700095f0 con 0x7f3e7c073790 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e7bfff700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e7c073790 0x7f3e7c198380 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000bd70 tx=0x7f3e7000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e797fa700 1 -- 172.21.15.116:0/1578034405 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e7001d070 con 0x7f3e7c073790 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e797fa700 1 -- 172.21.15.116:0/1578034405 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3e70005dc0 con 0x7f3e7c073790 2024-03-04T16:11:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e815a1700 1 -- 172.21.15.116:0/1578034405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7c19da70 con 0x7f3e7c073790 2024-03-04T16:11:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e797fa700 1 -- 172.21.15.116:0/1578034405 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e70021410 con 0x7f3e7c073790 2024-03-04T16:11:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.973+0000 7f3e815a1700 1 -- 172.21.15.116:0/1578034405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7c19df60 con 0x7f3e7c073790 2024-03-04T16:11:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.974+0000 7f3e815a1700 1 -- 172.21.15.116:0/1578034405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7c10b780 con 0x7f3e7c073790 2024-03-04T16:11:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.974+0000 7f3e797fa700 1 -- 172.21.15.116:0/1578034405 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f3e7000f7d0 con 0x7f3e7c073790 2024-03-04T16:11:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.975+0000 7f3e797fa700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3e6406d8a0 0x7f3e6406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.975+0000 7f3e797fa700 1 -- 172.21.15.116:0/1578034405 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f3e7008e870 con 0x7f3e7c073790 2024-03-04T16:11:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.975+0000 7f3e7b7fe700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3e6406d8a0 0x7f3e6406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.975+0000 7f3e7b7fe700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3e6406d8a0 0x7f3e6406fd60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c007c60 tx=0x7f3e6c0073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:52.979+0000 7f3e797fa700 1 -- 172.21.15.116:0/1578034405 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3e7005bb00 con 0x7f3e7c073790 2024-03-04T16:11:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.257+0000 7f3e815a1700 1 -- 172.21.15.116:0/1578034405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3e7c19e980 con 0x7f3e7c073790 2024-03-04T16:11:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.258+0000 7f3e797fa700 1 -- 172.21.15.116:0/1578034405 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f3e7000f460 con 0x7f3e7c073790 2024-03-04T16:11:53.258 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:11:53.258 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:11:53.259 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 14 2024-03-04T16:11:53.260 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:11:53.260 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:11:53.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 -- 172.21.15.116:0/1578034405 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3e6406d8a0 msgr2=0x7f3e6406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:53.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3e6406d8a0 0x7f3e6406fd60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c007c60 tx=0x7f3e6c0073d0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 -- 172.21.15.116:0/1578034405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e7c073790 msgr2=0x7f3e7c198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e7c073790 0x7f3e7c198380 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000bd70 tx=0x7f3e7000bda0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 -- 172.21.15.116:0/1578034405 shutdown_connections 2024-03-04T16:11:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f3e6406d8a0 0x7f3e6406fd60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e7c073790 0x7f3e7c198380 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 --2- 172.21.15.116:0/1578034405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e7c074d60 0x7f3e7c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 -- 172.21.15.116:0/1578034405 >> 172.21.15.116:0/1578034405 conn(0x7f3e7c0fbfd0 msgr2=0x7f3e7c106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 -- 172.21.15.116:0/1578034405 shutdown_connections 2024-03-04T16:11:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.260+0000 7f3e815a1700 1 -- 172.21.15.116:0/1578034405 wait complete. 2024-03-04T16:11:53.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.365+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/2104998193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c00ffab0 msgr2=0x7fb8c00ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:53.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.365+0000 7fb8c4bf6700 1 --2- 172.21.15.116:0/2104998193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c00ffab0 0x7fb8c00ffed0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009910 tx=0x7fb8b0009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:53.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.365+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/2104998193 shutdown_connections 2024-03-04T16:11:53.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.365+0000 7fb8c4bf6700 1 --2- 172.21.15.116:0/2104998193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8c0100cb0 0x7fb8c0101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.365+0000 7fb8c4bf6700 1 --2- 172.21.15.116:0/2104998193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c00ffab0 0x7fb8c00ffed0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.365+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/2104998193 >> 172.21.15.116:0/2104998193 conn(0x7fb8c00fb050 msgr2=0x7fb8c00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:53.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.365+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/2104998193 shutdown_connections 2024-03-04T16:11:53.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.365+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/2104998193 wait complete. 2024-03-04T16:11:53.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.366+0000 7fb8c4bf6700 1 Processor -- start 2024-03-04T16:11:53.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.366+0000 7fb8c4bf6700 1 -- start start 2024-03-04T16:11:53.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.366+0000 7fb8c4bf6700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8c00ffab0 0x7fb8c0196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:53.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.366+0000 7fb8c4bf6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c0100cb0 0x7fb8c01968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:53.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.366+0000 7fb8c4bf6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c0196ed0 con 0x7fb8c0100cb0 2024-03-04T16:11:53.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.366+0000 7fb8c4bf6700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c0197010 con 0x7fb8c00ffab0 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.366+0000 7fb8bdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c0100cb0 0x7fb8c01968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.366+0000 7fb8bdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c0100cb0 0x7fb8c01968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53758/0 (socket says 172.21.15.116:53758) 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.366+0000 7fb8bdd9b700 1 -- 172.21.15.116:0/1138314924 learned_addr learned my addr 172.21.15.116:0/1138314924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.367+0000 7fb8be59c700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8c00ffab0 0x7fb8c0196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.367+0000 7fb8bdd9b700 1 -- 172.21.15.116:0/1138314924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8c00ffab0 msgr2=0x7fb8c0196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.367+0000 7fb8bdd9b700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8c00ffab0 0x7fb8c0196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.367+0000 7fb8bdd9b700 1 -- 172.21.15.116:0/1138314924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b00095f0 con 0x7fb8c0100cb0 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.367+0000 7fb8bdd9b700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c0100cb0 0x7fb8c01968b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb8b400e920 tx=0x7fb8b400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.367+0000 7fb8af7fe700 1 -- 172.21.15.116:0/1138314924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8b4009790 con 0x7fb8c0100cb0 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.367+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/1138314924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c019bac0 con 0x7fb8c0100cb0 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.367+0000 7fb8af7fe700 1 -- 172.21.15.116:0/1138314924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb8b400cba0 con 0x7fb8c0100cb0 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.367+0000 7fb8af7fe700 1 -- 172.21.15.116:0/1138314924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8b4018630 con 0x7fb8c0100cb0 2024-03-04T16:11:53.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.367+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/1138314924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c019c010 con 0x7fb8c0100cb0 2024-03-04T16:11:53.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.368+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/1138314924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c0108610 con 0x7fb8c0100cb0 2024-03-04T16:11:53.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.369+0000 7fb8af7fe700 1 -- 172.21.15.116:0/1138314924 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fb8b400cd10 con 0x7fb8c0100cb0 2024-03-04T16:11:53.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.369+0000 7fb8af7fe700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8a806d8a0 0x7fb8a806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:53.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.369+0000 7fb8af7fe700 1 -- 172.21.15.116:0/1138314924 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fb8b4014070 con 0x7fb8c0100cb0 2024-03-04T16:11:53.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.369+0000 7fb8be59c700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8a806d8a0 0x7fb8a806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:53.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.370+0000 7fb8be59c700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8a806d8a0 0x7fb8a806fd60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000b6f0 tx=0x7fb8b0009f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:53.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.375+0000 7fb8af7fe700 1 -- 172.21.15.116:0/1138314924 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb8b4052d20 con 0x7fb8c0100cb0 2024-03-04T16:11:53.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:53 smithi152 ceph-mon[82845]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:53.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:53.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:53.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:53 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1578034405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.610+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/1138314924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb8c019c620 con 0x7fb8c0100cb0 2024-03-04T16:11:53.611 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:53 smithi116 ceph-mon[94164]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:53.611 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:53.611 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:53.611 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:53.611 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:53 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1578034405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:53.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.610+0000 7fb8af7fe700 1 -- 172.21.15.116:0/1138314924 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fb8b405a8b0 con 0x7fb8c0100cb0 2024-03-04T16:11:53.611 INFO:teuthology.orchestra.run.smithi116.stdout:e29 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 29 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:11:49.225261+0000 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:11:53.612 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 84 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=34400} 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:11:53.613 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{0:34400} state up:active seq 20 join_fscid=1 addr [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6828/786823228,v1:172.21.15.116:6829/786823228] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.152:6824/3736721533,v1:172.21.15.152:6825/3736721533] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:11:53.614 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.152:6826/1659417498,v1:172.21.15.152:6827/1659417498] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:11:53.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.612+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/1138314924 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8a806d8a0 msgr2=0x7fb8a806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:53.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.612+0000 7fb8c4bf6700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8a806d8a0 0x7fb8a806fd60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000b6f0 tx=0x7fb8b0009f90 comp rx=0 tx=0).stop 2024-03-04T16:11:53.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.612+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/1138314924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c0100cb0 msgr2=0x7fb8c01968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:53.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.612+0000 7fb8c4bf6700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c0100cb0 0x7fb8c01968b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb8b400e920 tx=0x7fb8b400ece0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.612+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/1138314924 shutdown_connections 2024-03-04T16:11:53.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.612+0000 7fb8c4bf6700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8a806d8a0 0x7fb8a806fd60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.612+0000 7fb8c4bf6700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8c00ffab0 0x7fb8c0196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.612+0000 7fb8c4bf6700 1 --2- 172.21.15.116:0/1138314924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c0100cb0 0x7fb8c01968b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.612+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/1138314924 >> 172.21.15.116:0/1138314924 conn(0x7fb8c00fb050 msgr2=0x7fb8c0068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:53.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.613+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/1138314924 shutdown_connections 2024-03-04T16:11:53.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.613+0000 7fb8c4bf6700 1 -- 172.21.15.116:0/1138314924 wait complete. 2024-03-04T16:11:53.616 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-04T16:11:53.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.715+0000 7f37ee697700 1 -- 172.21.15.116:0/1279176327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100be0 msgr2=0x7f37e8101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:53.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.715+0000 7f37ee697700 1 --2- 172.21.15.116:0/1279176327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100be0 0x7f37e8101000 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f37d8009910 tx=0x7f37d8009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:53.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.716+0000 7f37ee697700 1 -- 172.21.15.116:0/1279176327 shutdown_connections 2024-03-04T16:11:53.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.716+0000 7f37ee697700 1 --2- 172.21.15.116:0/1279176327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e8101de0 0x7f37e8102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.716+0000 7f37ee697700 1 --2- 172.21.15.116:0/1279176327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100be0 0x7f37e8101000 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.716+0000 7f37ee697700 1 -- 172.21.15.116:0/1279176327 >> 172.21.15.116:0/1279176327 conn(0x7f37e80fc140 msgr2=0x7f37e80fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:53.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.716+0000 7f37ee697700 1 -- 172.21.15.116:0/1279176327 shutdown_connections 2024-03-04T16:11:53.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.716+0000 7f37ee697700 1 -- 172.21.15.116:0/1279176327 wait complete. 2024-03-04T16:11:53.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.716+0000 7f37ee697700 1 Processor -- start 2024-03-04T16:11:53.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.716+0000 7f37ee697700 1 -- start start 2024-03-04T16:11:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.716+0000 7f37ee697700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100be0 0x7f37e8196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.716+0000 7f37ee697700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e8101de0 0x7f37e81968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37ee697700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e8196ef0 con 0x7f37e8100be0 2024-03-04T16:11:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37ee697700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e8197030 con 0x7f37e8101de0 2024-03-04T16:11:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100be0 0x7f37e8196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100be0 0x7f37e8196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53780/0 (socket says 172.21.15.116:53780) 2024-03-04T16:11:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e7fff700 1 -- 172.21.15.116:0/2568140345 learned_addr learned my addr 172.21.15.116:0/2568140345 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e77fe700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e8101de0 0x7f37e81968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e7fff700 1 -- 172.21.15.116:0/2568140345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e8101de0 msgr2=0x7f37e81968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e7fff700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e8101de0 0x7f37e81968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e7fff700 1 -- 172.21.15.116:0/2568140345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d80095f0 con 0x7f37e8100be0 2024-03-04T16:11:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e7fff700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100be0 0x7f37e8196390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f37d8000c00 tx=0x7f37d800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e57fa700 1 -- 172.21.15.116:0/2568140345 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37d801d070 con 0x7f37e8100be0 2024-03-04T16:11:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e57fa700 1 -- 172.21.15.116:0/2568140345 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f37d8005dc0 con 0x7f37e8100be0 2024-03-04T16:11:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37ee697700 1 -- 172.21.15.116:0/2568140345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e819ba80 con 0x7f37e8100be0 2024-03-04T16:11:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37e57fa700 1 -- 172.21.15.116:0/2568140345 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37d8022470 con 0x7f37e8100be0 2024-03-04T16:11:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.717+0000 7f37ee697700 1 -- 172.21.15.116:0/2568140345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e819bf70 con 0x7f37e8100be0 2024-03-04T16:11:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.718+0000 7f37ee697700 1 -- 172.21.15.116:0/2568140345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e8109740 con 0x7f37e8100be0 2024-03-04T16:11:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.719+0000 7f37e57fa700 1 -- 172.21.15.116:0/2568140345 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f37d8003810 con 0x7f37e8100be0 2024-03-04T16:11:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.719+0000 7f37e57fa700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f37d006d8a0 0x7f37d006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.719+0000 7f37e77fe700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f37d006d8a0 0x7f37d006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.719+0000 7f37e57fa700 1 -- 172.21.15.116:0/2568140345 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f37d808ea80 con 0x7f37e8100be0 2024-03-04T16:11:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.720+0000 7f37e77fe700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f37d006d8a0 0x7f37d006fd60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009c60 tx=0x7f37dc009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.724+0000 7f37e57fa700 1 -- 172.21.15.116:0/2568140345 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f37d805bd10 con 0x7f37e8100be0 2024-03-04T16:11:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.940+0000 7f37ee697700 1 -- 172.21.15.116:0/2568140345 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f37e819c670 con 0x7f37d006d8a0 2024-03-04T16:11:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.941+0000 7f37e57fa700 1 -- 172.21.15.116:0/2568140345 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f37e819c670 con 0x7f37d006d8a0 2024-03-04T16:11:53.942 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:11:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:11:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:11:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:11:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:11:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:11:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "mds", 2024-03-04T16:11:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:11:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "osd", 2024-03-04T16:11:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:11:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:11:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "16/23 daemons upgraded", 2024-03-04T16:11:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mds daemons", 2024-03-04T16:11:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:11:53.943 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:11:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.944+0000 7f37ee697700 1 -- 172.21.15.116:0/2568140345 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f37d006d8a0 msgr2=0x7f37d006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.944+0000 7f37ee697700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f37d006d8a0 0x7f37d006fd60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009c60 tx=0x7f37dc009260 comp rx=0 tx=0).stop 2024-03-04T16:11:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.944+0000 7f37ee697700 1 -- 172.21.15.116:0/2568140345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100be0 msgr2=0x7f37e8196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.945+0000 7f37ee697700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100be0 0x7f37e8196390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f37d8000c00 tx=0x7f37d800f690 comp rx=0 tx=0).stop 2024-03-04T16:11:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.945+0000 7f37ee697700 1 -- 172.21.15.116:0/2568140345 shutdown_connections 2024-03-04T16:11:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.945+0000 7f37ee697700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f37d006d8a0 0x7f37d006fd60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.945+0000 7f37ee697700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100be0 0x7f37e8196390 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.945+0000 7f37ee697700 1 --2- 172.21.15.116:0/2568140345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e8101de0 0x7f37e81968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.945+0000 7f37ee697700 1 -- 172.21.15.116:0/2568140345 >> 172.21.15.116:0/2568140345 conn(0x7f37e80fc140 msgr2=0x7f37e8105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.945+0000 7f37ee697700 1 -- 172.21.15.116:0/2568140345 shutdown_connections 2024-03-04T16:11:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:53.945+0000 7f37ee697700 1 -- 172.21.15.116:0/2568140345 wait complete. 2024-03-04T16:11:54.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.048+0000 7fbaa0884700 1 -- 172.21.15.116:0/28467912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba9c0ffab0 msgr2=0x7fba9c0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:54.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.048+0000 7fbaa0884700 1 --2- 172.21.15.116:0/28467912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba9c0ffab0 0x7fba9c0ffed0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009910 tx=0x7fba8c009c20 comp rx=0 tx=0).stop 2024-03-04T16:11:54.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.048+0000 7fbaa0884700 1 -- 172.21.15.116:0/28467912 shutdown_connections 2024-03-04T16:11:54.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.048+0000 7fbaa0884700 1 --2- 172.21.15.116:0/28467912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba9c100cb0 0x7fba9c101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:54.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.048+0000 7fbaa0884700 1 --2- 172.21.15.116:0/28467912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba9c0ffab0 0x7fba9c0ffed0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:54.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.048+0000 7fbaa0884700 1 -- 172.21.15.116:0/28467912 >> 172.21.15.116:0/28467912 conn(0x7fba9c0fb030 msgr2=0x7fba9c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:54.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.048+0000 7fbaa0884700 1 -- 172.21.15.116:0/28467912 shutdown_connections 2024-03-04T16:11:54.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.048+0000 7fbaa0884700 1 -- 172.21.15.116:0/28467912 wait complete. 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fbaa0884700 1 Processor -- start 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fbaa0884700 1 -- start start 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fbaa0884700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba9c0ffab0 0x7fba9c195240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fbaa0884700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba9c100cb0 0x7fba9c195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fbaa0884700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c195da0 con 0x7fba9c0ffab0 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fbaa0884700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c195ee0 con 0x7fba9c100cb0 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fba9a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba9c0ffab0 0x7fba9c195240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fba9a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba9c0ffab0 0x7fba9c195240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53790/0 (socket says 172.21.15.116:53790) 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fba9a59c700 1 -- 172.21.15.116:0/2283579425 learned_addr learned my addr 172.21.15.116:0/2283579425 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fba99d9b700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba9c100cb0 0x7fba9c195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:54.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fba9a59c700 1 -- 172.21.15.116:0/2283579425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba9c100cb0 msgr2=0x7fba9c195780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:54.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fba9a59c700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba9c100cb0 0x7fba9c195780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:54.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.049+0000 7fba9a59c700 1 -- 172.21.15.116:0/2283579425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c0095f0 con 0x7fba9c0ffab0 2024-03-04T16:11:54.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.050+0000 7fba9a59c700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba9c0ffab0 0x7fba9c195240 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00bda0 tx=0x7fba8c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:54.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.050+0000 7fba8b7fe700 1 -- 172.21.15.116:0/2283579425 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8c01f070 con 0x7fba9c0ffab0 2024-03-04T16:11:54.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.050+0000 7fbaa0884700 1 -- 172.21.15.116:0/2283579425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c068410 con 0x7fba9c0ffab0 2024-03-04T16:11:54.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.050+0000 7fba8b7fe700 1 -- 172.21.15.116:0/2283579425 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fba8c019610 con 0x7fba9c0ffab0 2024-03-04T16:11:54.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.050+0000 7fba8b7fe700 1 -- 172.21.15.116:0/2283579425 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8c00f670 con 0x7fba9c0ffab0 2024-03-04T16:11:54.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.050+0000 7fbaa0884700 1 -- 172.21.15.116:0/2283579425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c068900 con 0x7fba9c0ffab0 2024-03-04T16:11:54.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.051+0000 7fbaa0884700 1 -- 172.21.15.116:0/2283579425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9c108610 con 0x7fba9c0ffab0 2024-03-04T16:11:54.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.051+0000 7fba8b7fe700 1 -- 172.21.15.116:0/2283579425 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fba8c003a60 con 0x7fba9c0ffab0 2024-03-04T16:11:54.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.052+0000 7fba8b7fe700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fba8406d8a0 0x7fba8406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:11:54.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.052+0000 7fba8b7fe700 1 -- 172.21.15.116:0/2283579425 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fba8c090a10 con 0x7fba9c0ffab0 2024-03-04T16:11:54.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.052+0000 7fba99d9b700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fba8406d8a0 0x7fba8406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:11:54.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.052+0000 7fba99d9b700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fba8406d8a0 0x7fba8406fd60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fba90005950 tx=0x7fba9000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:11:54.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.056+0000 7fba8b7fe700 1 -- 172.21.15.116:0/2283579425 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba8c05dd20 con 0x7fba9c0ffab0 2024-03-04T16:11:54.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.333+0000 7fbaa0884700 1 -- 172.21.15.116:0/2283579425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fba9c1087d0 con 0x7fba9c0ffab0 2024-03-04T16:11:54.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.333+0000 7fba8b7fe700 1 -- 172.21.15.116:0/2283579425 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fba8c05d8b0 con 0x7fba9c0ffab0 2024-03-04T16:11:54.334 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.335+0000 7fbaa0884700 1 -- 172.21.15.116:0/2283579425 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fba8406d8a0 msgr2=0x7fba8406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.335+0000 7fbaa0884700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fba8406d8a0 0x7fba8406fd60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fba90005950 tx=0x7fba9000b410 comp rx=0 tx=0).stop 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.335+0000 7fbaa0884700 1 -- 172.21.15.116:0/2283579425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba9c0ffab0 msgr2=0x7fba9c195240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.335+0000 7fbaa0884700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba9c0ffab0 0x7fba9c195240 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00bda0 tx=0x7fba8c00bdd0 comp rx=0 tx=0).stop 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.335+0000 7fbaa0884700 1 -- 172.21.15.116:0/2283579425 shutdown_connections 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.335+0000 7fbaa0884700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fba8406d8a0 0x7fba8406fd60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.335+0000 7fbaa0884700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba9c0ffab0 0x7fba9c195240 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.335+0000 7fbaa0884700 1 --2- 172.21.15.116:0/2283579425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba9c100cb0 0x7fba9c195780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.335+0000 7fbaa0884700 1 -- 172.21.15.116:0/2283579425 >> 172.21.15.116:0/2283579425 conn(0x7fba9c0fb030 msgr2=0x7fba9c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.336+0000 7fbaa0884700 1 -- 172.21.15.116:0/2283579425 shutdown_connections 2024-03-04T16:11:54.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:11:54.336+0000 7fbaa0884700 1 -- 172.21.15.116:0/2283579425 wait complete. 2024-03-04T16:11:54.513 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:54 smithi152 ceph-mon[82845]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:54.514 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:54 smithi152 ceph-mon[82845]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:54.514 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:54 smithi152 ceph-mon[82845]: pgmap v217: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 12 MiB/s rd, 85 B/s wr, 5 op/s 2024-03-04T16:11:54.514 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:54 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1138314924' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:11:54.514 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:54 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2283579425' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:11:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:54 smithi116 ceph-mon[94164]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:54 smithi116 ceph-mon[94164]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:54.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:54 smithi116 ceph-mon[94164]: pgmap v217: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 12 MiB/s rd, 85 B/s wr, 5 op/s 2024-03-04T16:11:54.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:54 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1138314924' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:11:54.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:54 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2283579425' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:11:55.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:55 smithi152 ceph-mon[82845]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:55.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:55 smithi116 ceph-mon[94164]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: pgmap v218: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi116.gkenwy"}]': finished 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi116.pjrmzv"}]': finished 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi152.kqoybf"}]': finished 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi152.tlpshy"}]': finished 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:56 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: pgmap v218: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi116.gkenwy"}]: dispatch 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi116.gkenwy"}]': finished 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi116.pjrmzv"}]: dispatch 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi116.pjrmzv"}]': finished 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi152.kqoybf"}]: dispatch 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi152.kqoybf"}]': finished 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi152.tlpshy"}]: dispatch 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi152.tlpshy"}]': finished 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:56.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:56 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:57.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:57 smithi116 ceph-mon[94164]: Upgrade: Setting container_image for all mds 2024-03-04T16:11:57.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:57 smithi116 ceph-mon[94164]: Upgrade: Setting container_image for all rgw 2024-03-04T16:11:57.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:57 smithi116 ceph-mon[94164]: Upgrade: Setting container_image for all rbd-mirror 2024-03-04T16:11:57.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:57 smithi152 ceph-mon[82845]: Upgrade: Setting container_image for all mds 2024-03-04T16:11:57.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:57 smithi152 ceph-mon[82845]: Upgrade: Setting container_image for all rgw 2024-03-04T16:11:57.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:57 smithi152 ceph-mon[82845]: Upgrade: Setting container_image for all rbd-mirror 2024-03-04T16:11:58.646 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:58 smithi116 ceph-mon[94164]: pgmap v219: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-03-04T16:11:58.646 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:58 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:58.646 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:58 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:11:58.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:58 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:58.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:58 smithi152 ceph-mon[82845]: pgmap v219: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-03-04T16:11:58.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:58 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:11:58.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:58 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi116", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:11:58.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:58 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:59.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:59 smithi116 ceph-mon[94164]: Upgrade: Updating ceph-exporter.smithi116 (1/2) 2024-03-04T16:11:59.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:11:59 smithi116 ceph-mon[94164]: Deploying daemon ceph-exporter.smithi116 on smithi116 2024-03-04T16:11:59.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:59 smithi152 ceph-mon[82845]: Upgrade: Updating ceph-exporter.smithi116 (1/2) 2024-03-04T16:11:59.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:11:59 smithi152 ceph-mon[82845]: Deploying daemon ceph-exporter.smithi116 on smithi116 2024-03-04T16:12:00.575 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:00 smithi116 ceph-mon[94164]: pgmap v220: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 5 op/s 2024-03-04T16:12:00.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:00 smithi152 ceph-mon[82845]: pgmap v220: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 5 op/s 2024-03-04T16:12:02.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:02 smithi116 ceph-mon[94164]: pgmap v221: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 686 KiB/s rd, 4.3 KiB/s wr, 5 op/s 2024-03-04T16:12:02.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:02 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:02.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:02 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:02.746 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:02 smithi152 ceph-mon[82845]: pgmap v221: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 686 KiB/s rd, 4.3 KiB/s wr, 5 op/s 2024-03-04T16:12:02.746 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:02 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:02.747 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:02 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:04 smithi116 ceph-mon[94164]: pgmap v222: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-03-04T16:12:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:04 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:04 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:12:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:04 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:04.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:04 smithi152 ceph-mon[82845]: pgmap v222: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-03-04T16:12:04.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:04 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:04.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:04 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi152", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-04T16:12:04.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:04 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:05.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:05 smithi152 ceph-mon[82845]: Upgrade: Updating ceph-exporter.smithi152 (2/2) 2024-03-04T16:12:05.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:05 smithi152 ceph-mon[82845]: Deploying daemon ceph-exporter.smithi152 on smithi152 2024-03-04T16:12:05.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:05 smithi116 ceph-mon[94164]: Upgrade: Updating ceph-exporter.smithi152 (2/2) 2024-03-04T16:12:05.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:05 smithi116 ceph-mon[94164]: Deploying daemon ceph-exporter.smithi152 on smithi152 2024-03-04T16:12:06.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:06 smithi152 ceph-mon[82845]: pgmap v223: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-04T16:12:06.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:06 smithi116 ceph-mon[94164]: pgmap v223: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-04T16:12:08.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:08 smithi116 ceph-mon[94164]: pgmap v224: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-04T16:12:08.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:08.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:08.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:08.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:08.472 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:08 smithi152 ceph-mon[82845]: pgmap v224: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-04T16:12:08.473 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:08.473 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:08.473 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:08.473 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:10 smithi152 ceph-mon[82845]: pgmap v225: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:10 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:10 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:10.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:10 smithi116 ceph-mon[94164]: pgmap v225: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:10.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:10.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:10 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:12.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:12.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:11 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:12.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:11 smithi152 ceph-mon[82845]: pgmap v226: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:12.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:12.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:11 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:12.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:11 smithi116 ceph-mon[94164]: pgmap v226: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:13.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:12 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:13.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:12 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:13.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:12 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:13.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:12 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:14.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:14.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:13 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:14.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:13 smithi116 ceph-mon[94164]: pgmap v227: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:14.270 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:14.271 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:13 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:14.271 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:13 smithi152 ceph-mon[82845]: pgmap v227: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:15.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:15.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:15 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:15.643 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:15.643 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:15 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:16.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:16 smithi152 ceph-mon[82845]: pgmap v228: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:16 smithi116 ceph-mon[94164]: pgmap v228: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:17.490 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.490 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.490 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:17.490 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:17.490 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.491 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:17.491 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.491 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.544 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.544 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.545 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:17.545 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:17.545 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.545 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:17.545 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.545 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.545 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: Upgrade: Setting container_image for all ceph-exporter 2024-03-04T16:12:17.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi116"}]: dispatch 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi116"}]': finished 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi152"}]: dispatch 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi152"}]': finished 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: Upgrade: Setting container_image for all iscsi 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: Upgrade: Setting container_image for all nfs 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: Upgrade: Setting container_image for all nvmeof 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:17 smithi116 ceph-mon[94164]: pgmap v229: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:17.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: Upgrade: Setting container_image for all ceph-exporter 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi116"}]: dispatch 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi116"}]': finished 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi152"}]: dispatch 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi152"}]': finished 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: Upgrade: Setting container_image for all iscsi 2024-03-04T16:12:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: Upgrade: Setting container_image for all nfs 2024-03-04T16:12:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: Upgrade: Setting container_image for all nvmeof 2024-03-04T16:12:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:17.859 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:17 smithi152 ceph-mon[82845]: pgmap v229: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:18.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:18 smithi116 ceph-mon[94164]: Upgrade: Updating node-exporter.smithi116 (1/2) 2024-03-04T16:12:18.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:18 smithi116 ceph-mon[94164]: Deploying daemon node-exporter.smithi116 on smithi116 2024-03-04T16:12:19.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:18 smithi152 ceph-mon[82845]: Upgrade: Updating node-exporter.smithi116 (1/2) 2024-03-04T16:12:19.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:18 smithi152 ceph-mon[82845]: Deploying daemon node-exporter.smithi116 on smithi116 2024-03-04T16:12:19.904 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:19 smithi116 ceph-mon[94164]: pgmap v230: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:20.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:19 smithi152 ceph-mon[82845]: pgmap v230: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:22.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:21 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:22.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:21 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:22.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:21 smithi152 ceph-mon[82845]: pgmap v231: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:22.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:21 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:22.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:21 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:22.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:21 smithi116 ceph-mon[94164]: pgmap v231: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:23.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:24.456 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:24 smithi116 ceph-mon[94164]: Upgrade: Updating node-exporter.smithi152 (2/2) 2024-03-04T16:12:24.456 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:24 smithi116 ceph-mon[94164]: Deploying daemon node-exporter.smithi152 on smithi152 2024-03-04T16:12:24.456 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:24 smithi116 ceph-mon[94164]: pgmap v232: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.454+0000 7f2432529700 1 -- 172.21.15.116:0/2952691358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c101dd0 msgr2=0x7f242c102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.454+0000 7f2432529700 1 --2- 172.21.15.116:0/2952691358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c101dd0 0x7f242c102230 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2420009910 tx=0x7f2420009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.454+0000 7f2432529700 1 -- 172.21.15.116:0/2952691358 shutdown_connections 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.454+0000 7f2432529700 1 --2- 172.21.15.116:0/2952691358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c101dd0 0x7f242c102230 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.454+0000 7f2432529700 1 --2- 172.21.15.116:0/2952691358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f242c100bd0 0x7f242c100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.454+0000 7f2432529700 1 -- 172.21.15.116:0/2952691358 >> 172.21.15.116:0/2952691358 conn(0x7f242c0fc190 msgr2=0x7f242c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.454+0000 7f2432529700 1 -- 172.21.15.116:0/2952691358 shutdown_connections 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.454+0000 7f2432529700 1 -- 172.21.15.116:0/2952691358 wait complete. 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.455+0000 7f2432529700 1 Processor -- start 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.455+0000 7f2432529700 1 -- start start 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.455+0000 7f2432529700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c100bd0 0x7f242c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.455+0000 7f2432529700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f242c101dd0 0x7f242c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.455+0000 7f2432529700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c0736c0 con 0x7f242c100bd0 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.455+0000 7f2432529700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c073800 con 0x7f242c101dd0 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f242bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c100bd0 0x7f242c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f242bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c100bd0 0x7f242c074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44564/0 (socket says 172.21.15.116:44564) 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f242bfff700 1 -- 172.21.15.116:0/3123884690 learned_addr learned my addr 172.21.15.116:0/3123884690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f242b7fe700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f242c101dd0 0x7f242c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f242bfff700 1 -- 172.21.15.116:0/3123884690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f242c101dd0 msgr2=0x7f242c073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f242bfff700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f242c101dd0 0x7f242c073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f242bfff700 1 -- 172.21.15.116:0/3123884690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24200095f0 con 0x7f242c100bd0 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f242bfff700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c100bd0 0x7f242c074b30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f241c00e8f0 tx=0x7f241c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f24297fa700 1 -- 172.21.15.116:0/3123884690 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f241c009790 con 0x7f242c100bd0 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f2432529700 1 -- 172.21.15.116:0/3123884690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c073ae0 con 0x7f242c100bd0 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f24297fa700 1 -- 172.21.15.116:0/3123884690 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f241c00c810 con 0x7f242c100bd0 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f24297fa700 1 -- 172.21.15.116:0/3123884690 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f241c003d20 con 0x7f242c100bd0 2024-03-04T16:12:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.456+0000 7f2432529700 1 -- 172.21.15.116:0/3123884690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c19e3b0 con 0x7f242c100bd0 2024-03-04T16:12:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.457+0000 7f2432529700 1 -- 172.21.15.116:0/3123884690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c109730 con 0x7f242c100bd0 2024-03-04T16:12:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.458+0000 7f24297fa700 1 -- 172.21.15.116:0/3123884690 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f241c0041f0 con 0x7f242c100bd0 2024-03-04T16:12:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.458+0000 7f24297fa700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f241406d8a0 0x7f241406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.458+0000 7f24297fa700 1 -- 172.21.15.116:0/3123884690 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f241c014070 con 0x7f242c100bd0 2024-03-04T16:12:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.458+0000 7f242b7fe700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f241406d8a0 0x7f241406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.459+0000 7f242b7fe700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f241406d8a0 0x7f241406fd60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2420009fd0 tx=0x7f2420005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:24.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.463+0000 7f24297fa700 1 -- 172.21.15.116:0/3123884690 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f241c05aae0 con 0x7f242c100bd0 2024-03-04T16:12:24.467 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:24 smithi152 ceph-mon[82845]: Upgrade: Updating node-exporter.smithi152 (2/2) 2024-03-04T16:12:24.467 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:24 smithi152 ceph-mon[82845]: Deploying daemon node-exporter.smithi152 on smithi152 2024-03-04T16:12:24.467 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:24 smithi152 ceph-mon[82845]: pgmap v232: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.677+0000 7f2432529700 1 -- 172.21.15.116:0/3123884690 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f242c19e960 con 0x7f241406d8a0 2024-03-04T16:12:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.678+0000 7f24297fa700 1 -- 172.21.15.116:0/3123884690 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+466 (secure 0 0 0) 0x7f242c19e960 con 0x7f241406d8a0 2024-03-04T16:12:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 -- 172.21.15.116:0/3123884690 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f241406d8a0 msgr2=0x7f241406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f241406d8a0 0x7f241406fd60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2420009fd0 tx=0x7f2420005940 comp rx=0 tx=0).stop 2024-03-04T16:12:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 -- 172.21.15.116:0/3123884690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c100bd0 msgr2=0x7f242c074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c100bd0 0x7f242c074b30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f241c00e8f0 tx=0x7f241c00ec00 comp rx=0 tx=0).stop 2024-03-04T16:12:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 -- 172.21.15.116:0/3123884690 shutdown_connections 2024-03-04T16:12:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f241406d8a0 0x7f241406fd60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c100bd0 0x7f242c074b30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 --2- 172.21.15.116:0/3123884690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f242c101dd0 0x7f242c073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 -- 172.21.15.116:0/3123884690 >> 172.21.15.116:0/3123884690 conn(0x7f242c0fc190 msgr2=0x7f242c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 -- 172.21.15.116:0/3123884690 shutdown_connections 2024-03-04T16:12:24.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.680+0000 7f2432529700 1 -- 172.21.15.116:0/3123884690 wait complete. 2024-03-04T16:12:24.692 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:12:24.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.787+0000 7f88f07ce700 1 -- 172.21.15.116:0/1948264477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80feec0 msgr2=0x7f88e80ff2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.787+0000 7f88f07ce700 1 --2- 172.21.15.116:0/1948264477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80feec0 0x7f88e80ff2e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009910 tx=0x7f88e4009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.787+0000 7f88f07ce700 1 -- 172.21.15.116:0/1948264477 shutdown_connections 2024-03-04T16:12:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.787+0000 7f88f07ce700 1 --2- 172.21.15.116:0/1948264477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88e8101ea0 0x7f88e8102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.787+0000 7f88f07ce700 1 --2- 172.21.15.116:0/1948264477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80feec0 0x7f88e80ff2e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.787+0000 7f88f07ce700 1 -- 172.21.15.116:0/1948264477 >> 172.21.15.116:0/1948264477 conn(0x7f88e80fa800 msgr2=0x7f88e80fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.787+0000 7f88f07ce700 1 -- 172.21.15.116:0/1948264477 shutdown_connections 2024-03-04T16:12:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.787+0000 7f88f07ce700 1 -- 172.21.15.116:0/1948264477 wait complete. 2024-03-04T16:12:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88f07ce700 1 Processor -- start 2024-03-04T16:12:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88f07ce700 1 -- start start 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88f07ce700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80feec0 0x7f88e8196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88f07ce700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88e8101ea0 0x7f88e81968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88f07ce700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e8196ee0 con 0x7f88e80feec0 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88f07ce700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e8197020 con 0x7f88e8101ea0 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88ee56a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80feec0 0x7f88e8196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88ee56a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80feec0 0x7f88e8196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53310/0 (socket says 172.21.15.116:53310) 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88ee56a700 1 -- 172.21.15.116:0/2810346744 learned_addr learned my addr 172.21.15.116:0/2810346744 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88edd69700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88e8101ea0 0x7f88e81968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88ee56a700 1 -- 172.21.15.116:0/2810346744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88e8101ea0 msgr2=0x7f88e81968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88ee56a700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88e8101ea0 0x7f88e81968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.788+0000 7f88ee56a700 1 -- 172.21.15.116:0/2810346744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e40095f0 con 0x7f88e80feec0 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.789+0000 7f88ee56a700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80feec0 0x7f88e8196380 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f88e400bda0 tx=0x7f88e400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.789+0000 7f88df7fe700 1 -- 172.21.15.116:0/2810346744 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88e401d070 con 0x7f88e80feec0 2024-03-04T16:12:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.789+0000 7f88df7fe700 1 -- 172.21.15.116:0/2810346744 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f88e400f460 con 0x7f88e80feec0 2024-03-04T16:12:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.789+0000 7f88f07ce700 1 -- 172.21.15.116:0/2810346744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e819ba70 con 0x7f88e80feec0 2024-03-04T16:12:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.789+0000 7f88df7fe700 1 -- 172.21.15.116:0/2810346744 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88e4021620 con 0x7f88e80feec0 2024-03-04T16:12:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.789+0000 7f88f07ce700 1 -- 172.21.15.116:0/2810346744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e819bf60 con 0x7f88e80feec0 2024-03-04T16:12:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.790+0000 7f88f07ce700 1 -- 172.21.15.116:0/2810346744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e8109760 con 0x7f88e80feec0 2024-03-04T16:12:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.790+0000 7f88df7fe700 1 -- 172.21.15.116:0/2810346744 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f88e400f5d0 con 0x7f88e80feec0 2024-03-04T16:12:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.791+0000 7f88df7fe700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f88d406d8a0 0x7f88d406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.791+0000 7f88edd69700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f88d406d8a0 0x7f88d406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.791+0000 7f88df7fe700 1 -- 172.21.15.116:0/2810346744 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f88e408e860 con 0x7f88e80feec0 2024-03-04T16:12:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.791+0000 7f88edd69700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f88d406d8a0 0x7f88d406fd60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f88d8005950 tx=0x7f88d800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:24.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:24.796+0000 7f88df7fe700 1 -- 172.21.15.116:0/2810346744 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f88e405baf0 con 0x7f88e80feec0 2024-03-04T16:12:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.005+0000 7f88f07ce700 1 -- 172.21.15.116:0/2810346744 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f88e819c630 con 0x7f88d406d8a0 2024-03-04T16:12:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.006+0000 7f88df7fe700 1 -- 172.21.15.116:0/2810346744 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+466 (secure 0 0 0) 0x7f88e819c630 con 0x7f88d406d8a0 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 -- 172.21.15.116:0/2810346744 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f88d406d8a0 msgr2=0x7f88d406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f88d406d8a0 0x7f88d406fd60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f88d8005950 tx=0x7f88d800b410 comp rx=0 tx=0).stop 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 -- 172.21.15.116:0/2810346744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80feec0 msgr2=0x7f88e8196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80feec0 0x7f88e8196380 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f88e400bda0 tx=0x7f88e400bdd0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 -- 172.21.15.116:0/2810346744 shutdown_connections 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f88d406d8a0 0x7f88d406fd60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80feec0 0x7f88e8196380 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 --2- 172.21.15.116:0/2810346744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88e8101ea0 0x7f88e81968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 -- 172.21.15.116:0/2810346744 >> 172.21.15.116:0/2810346744 conn(0x7f88e80fa800 msgr2=0x7f88e81050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 -- 172.21.15.116:0/2810346744 shutdown_connections 2024-03-04T16:12:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.008+0000 7f88f07ce700 1 -- 172.21.15.116:0/2810346744 wait complete. 2024-03-04T16:12:25.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.112+0000 7f7834503700 1 -- 172.21.15.116:0/1044720933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f782c0ffab0 msgr2=0x7f782c0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.112+0000 7f7834503700 1 --2- 172.21.15.116:0/1044720933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f782c0ffab0 0x7f782c0ffed0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f782400b0a0 tx=0x7f782400b3b0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.112+0000 7f7834503700 1 -- 172.21.15.116:0/1044720933 shutdown_connections 2024-03-04T16:12:25.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.112+0000 7f7834503700 1 --2- 172.21.15.116:0/1044720933 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f782c100cb0 0x7f782c101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.112+0000 7f7834503700 1 --2- 172.21.15.116:0/1044720933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f782c0ffab0 0x7f782c0ffed0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.112+0000 7f7834503700 1 -- 172.21.15.116:0/1044720933 >> 172.21.15.116:0/1044720933 conn(0x7f782c0fb030 msgr2=0x7f782c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:25.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.112+0000 7f7834503700 1 -- 172.21.15.116:0/1044720933 shutdown_connections 2024-03-04T16:12:25.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.112+0000 7f7834503700 1 -- 172.21.15.116:0/1044720933 wait complete. 2024-03-04T16:12:25.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.112+0000 7f7834503700 1 Processor -- start 2024-03-04T16:12:25.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.112+0000 7f7834503700 1 -- start start 2024-03-04T16:12:25.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f7834503700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f782c0ffab0 0x7f782c196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f7834503700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f782c100cb0 0x7f782c1968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f7834503700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c196ec0 con 0x7f782c0ffab0 2024-03-04T16:12:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f7834503700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c197000 con 0x7f782c100cb0 2024-03-04T16:12:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f783229f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f782c0ffab0 0x7f782c196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f783229f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f782c0ffab0 0x7f782c196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53334/0 (socket says 172.21.15.116:53334) 2024-03-04T16:12:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f783229f700 1 -- 172.21.15.116:0/1823816830 learned_addr learned my addr 172.21.15.116:0/1823816830 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f7831a9e700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f782c100cb0 0x7f782c1968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f783229f700 1 -- 172.21.15.116:0/1823816830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f782c100cb0 msgr2=0x7f782c1968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f783229f700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f782c100cb0 0x7f782c1968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f783229f700 1 -- 172.21.15.116:0/1823816830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7824009d00 con 0x7f782c0ffab0 2024-03-04T16:12:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f783229f700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f782c0ffab0 0x7f782c196360 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f78240077b0 tx=0x7f7824009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f782b7fe700 1 -- 172.21.15.116:0/1823816830 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f782400e040 con 0x7f782c0ffab0 2024-03-04T16:12:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f782b7fe700 1 -- 172.21.15.116:0/1823816830 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7824003de0 con 0x7f782c0ffab0 2024-03-04T16:12:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.113+0000 7f782b7fe700 1 -- 172.21.15.116:0/1823816830 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f782401d7f0 con 0x7f782c0ffab0 2024-03-04T16:12:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.114+0000 7f7834503700 1 -- 172.21.15.116:0/1823816830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782c19ba50 con 0x7f782c0ffab0 2024-03-04T16:12:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.114+0000 7f7834503700 1 -- 172.21.15.116:0/1823816830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c19bfc0 con 0x7f782c0ffab0 2024-03-04T16:12:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.115+0000 7f7834503700 1 -- 172.21.15.116:0/1823816830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c108610 con 0x7f782c0ffab0 2024-03-04T16:12:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.115+0000 7f782b7fe700 1 -- 172.21.15.116:0/1823816830 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f7824004780 con 0x7f782c0ffab0 2024-03-04T16:12:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.115+0000 7f782b7fe700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f781406d8a0 0x7f781406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.115+0000 7f782b7fe700 1 -- 172.21.15.116:0/1823816830 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f782408eef0 con 0x7f782c0ffab0 2024-03-04T16:12:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.116+0000 7f7831a9e700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f781406d8a0 0x7f781406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:25.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.116+0000 7f7831a9e700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f781406d8a0 0x7f781406fd60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7820005950 tx=0x7f782000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:25.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.120+0000 7f782b7fe700 1 -- 172.21.15.116:0/1823816830 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f782405c240 con 0x7f782c0ffab0 2024-03-04T16:12:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.337+0000 7f7834503700 1 -- 172.21.15.116:0/1823816830 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f782c19c710 con 0x7f781406d8a0 2024-03-04T16:12:25.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.348+0000 7f782b7fe700 1 -- 172.21.15.116:0/1823816830 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f782c19c710 con 0x7f781406d8a0 2024-03-04T16:12:25.349 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:12:25.349 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (15m) 13s ago 16m 25.5M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:12:25.349 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (23s) 13s ago 17m 7323k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c67439dd3802 2024-03-04T16:12:25.349 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (18s) 15s ago 16m 6929k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6d482a1a1fd4 2024-03-04T16:12:25.349 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (5m) 13s ago 17m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:12:25.349 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (5m) 15s ago 16m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:12:25.349 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (15m) 13s ago 16m 88.6M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:12:25.349 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (85s) 13s ago 11m 13.5M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9d5061a093c6 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (104s) 13s ago 11m 86.5M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4bf8fc434069 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (37s) 15s ago 11m 13.5M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3abb2cd19d94 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (58s) 15s ago 11m 15.8M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4f752bc549b2 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (7m) 13s ago 18m 508M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (7m) 15s ago 15m 451M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (6m) 13s ago 18m 54.0M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (6m) 15s ago 15m 40.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 starting - - - - 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 starting - - - - 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (4m) 13s ago 15m 97.5M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (4m) 13s ago 14m 115M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (3m) 13s ago 14m 93.8M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (3m) 15s ago 13m 121M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6b01e97c8f9b 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (2m) 15s ago 13m 112M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 5ecc30e5a86c 2024-03-04T16:12:25.350 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (2m) 15s ago 12m 102M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 1083700b476d 2024-03-04T16:12:25.351 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (7m) 13s ago 16m 73.7M - 2.43.0 a07b618ecd1d f96367a91175 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 -- 172.21.15.116:0/1823816830 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f781406d8a0 msgr2=0x7f781406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f781406d8a0 0x7f781406fd60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7820005950 tx=0x7f782000b410 comp rx=0 tx=0).stop 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 -- 172.21.15.116:0/1823816830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f782c0ffab0 msgr2=0x7f782c196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f782c0ffab0 0x7f782c196360 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f78240077b0 tx=0x7f7824009760 comp rx=0 tx=0).stop 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 -- 172.21.15.116:0/1823816830 shutdown_connections 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f781406d8a0 0x7f781406fd60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f782c0ffab0 0x7f782c196360 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 --2- 172.21.15.116:0/1823816830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f782c100cb0 0x7f782c1968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 -- 172.21.15.116:0/1823816830 >> 172.21.15.116:0/1823816830 conn(0x7f782c0fb030 msgr2=0x7f782c068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 -- 172.21.15.116:0/1823816830 shutdown_connections 2024-03-04T16:12:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.351+0000 7f7834503700 1 -- 172.21.15.116:0/1823816830 wait complete. 2024-03-04T16:12:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.458+0000 7f804b90f700 1 -- 172.21.15.116:0/3080974532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80440ffab0 msgr2=0x7f80440ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.458+0000 7f804b90f700 1 --2- 172.21.15.116:0/3080974532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80440ffab0 0x7f80440ffed0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8040009910 tx=0x7f8040009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.458+0000 7f804b90f700 1 -- 172.21.15.116:0/3080974532 shutdown_connections 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.458+0000 7f804b90f700 1 --2- 172.21.15.116:0/3080974532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8044100cb0 0x7f8044101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.458+0000 7f804b90f700 1 --2- 172.21.15.116:0/3080974532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80440ffab0 0x7f80440ffed0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.458+0000 7f804b90f700 1 -- 172.21.15.116:0/3080974532 >> 172.21.15.116:0/3080974532 conn(0x7f80440fb030 msgr2=0x7f80440fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.458+0000 7f804b90f700 1 -- 172.21.15.116:0/3080974532 shutdown_connections 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.458+0000 7f804b90f700 1 -- 172.21.15.116:0/3080974532 wait complete. 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f804b90f700 1 Processor -- start 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f804b90f700 1 -- start start 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f804b90f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80440ffab0 0x7f8044103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f804b90f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8044100cb0 0x7f8044102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f804b90f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80441025d0 con 0x7f80440ffab0 2024-03-04T16:12:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f804b90f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8044102710 con 0x7f8044100cb0 2024-03-04T16:12:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f80496ab700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80440ffab0 0x7f8044103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f8048eaa700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8044100cb0 0x7f8044102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f80496ab700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80440ffab0 0x7f8044103a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53352/0 (socket says 172.21.15.116:53352) 2024-03-04T16:12:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f80496ab700 1 -- 172.21.15.116:0/3992790251 learned_addr learned my addr 172.21.15.116:0/3992790251 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f8048eaa700 1 -- 172.21.15.116:0/3992790251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80440ffab0 msgr2=0x7f8044103a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.459+0000 7f8048eaa700 1 --2- 172.21.15.116:0/3992790251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80440ffab0 0x7f8044103a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.460+0000 7f8048eaa700 1 -- 172.21.15.116:0/3992790251 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80400095f0 con 0x7f8044100cb0 2024-03-04T16:12:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.460+0000 7f8048eaa700 1 --2- 172.21.15.116:0/3992790251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8044100cb0 0x7f8044102090 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f803400d8d0 tx=0x7f803400dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.460+0000 7f803a7fc700 1 -- 172.21.15.116:0/3992790251 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f803400f860 con 0x7f8044100cb0 2024-03-04T16:12:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.460+0000 7f804b90f700 1 -- 172.21.15.116:0/3992790251 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80441029f0 con 0x7f8044100cb0 2024-03-04T16:12:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.460+0000 7f803a7fc700 1 -- 172.21.15.116:0/3992790251 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f803400fe70 con 0x7f8044100cb0 2024-03-04T16:12:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.460+0000 7f804b90f700 1 -- 172.21.15.116:0/3992790251 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f804419e380 con 0x7f8044100cb0 2024-03-04T16:12:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.460+0000 7f803a7fc700 1 -- 172.21.15.116:0/3992790251 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f803400e5e0 con 0x7f8044100cb0 2024-03-04T16:12:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.461+0000 7f804b90f700 1 -- 172.21.15.116:0/3992790251 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8044108610 con 0x7f8044100cb0 2024-03-04T16:12:25.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.462+0000 7f803a7fc700 1 -- 172.21.15.116:0/3992790251 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f8034010460 con 0x7f8044100cb0 2024-03-04T16:12:25.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.462+0000 7f803a7fc700 1 --2- 172.21.15.116:0/3992790251 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f803006d8a0 0x7f803006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:25.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.462+0000 7f803a7fc700 1 -- 172.21.15.116:0/3992790251 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f803408d0a0 con 0x7f8044100cb0 2024-03-04T16:12:25.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.462+0000 7f80496ab700 1 --2- 172.21.15.116:0/3992790251 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f803006d8a0 0x7f803006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:25.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.463+0000 7f80496ab700 1 --2- 172.21.15.116:0/3992790251 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f803006d8a0 0x7f803006fd60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8040006010 tx=0x7f804000b560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:25.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.467+0000 7f803a7fc700 1 -- 172.21.15.116:0/3992790251 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f803405a3b0 con 0x7f8044100cb0 2024-03-04T16:12:25.766 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:12:25.766 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 14 2024-03-04T16:12:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:12:25.768 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:12:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.764+0000 7f804b90f700 1 -- 172.21.15.116:0/3992790251 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f804419ec50 con 0x7f8044100cb0 2024-03-04T16:12:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.765+0000 7f803a7fc700 1 -- 172.21.15.116:0/3992790251 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f8034059f40 con 0x7f8044100cb0 2024-03-04T16:12:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.767+0000 7f804b90f700 1 -- 172.21.15.116:0/3992790251 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f803006d8a0 msgr2=0x7f803006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.767+0000 7f804b90f700 1 --2- 172.21.15.116:0/3992790251 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f803006d8a0 0x7f803006fd60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8040006010 tx=0x7f804000b560 comp rx=0 tx=0).stop 2024-03-04T16:12:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.767+0000 7f804b90f700 1 -- 172.21.15.116:0/3992790251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8044100cb0 msgr2=0x7f8044102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.767+0000 7f804b90f700 1 --2- 172.21.15.116:0/3992790251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8044100cb0 0x7f8044102090 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f803400d8d0 tx=0x7f803400dbe0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.767+0000 7f804b90f700 1 -- 172.21.15.116:0/3992790251 shutdown_connections 2024-03-04T16:12:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.767+0000 7f804b90f700 1 --2- 172.21.15.116:0/3992790251 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f803006d8a0 0x7f803006fd60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.767+0000 7f804b90f700 1 --2- 172.21.15.116:0/3992790251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80440ffab0 0x7f8044103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.767+0000 7f804b90f700 1 --2- 172.21.15.116:0/3992790251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8044100cb0 0x7f8044102090 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.767+0000 7f804b90f700 1 -- 172.21.15.116:0/3992790251 >> 172.21.15.116:0/3992790251 conn(0x7f80440fb030 msgr2=0x7f80440683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.768+0000 7f804b90f700 1 -- 172.21.15.116:0/3992790251 shutdown_connections 2024-03-04T16:12:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.768+0000 7f804b90f700 1 -- 172.21.15.116:0/3992790251 wait complete. 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.870+0000 7fad16fb6700 1 -- 172.21.15.116:0/3458875657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad10101d80 msgr2=0x7fad101021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.870+0000 7fad16fb6700 1 --2- 172.21.15.116:0/3458875657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad10101d80 0x7fad101021e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7facfc009910 tx=0x7facfc009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.870+0000 7fad16fb6700 1 -- 172.21.15.116:0/3458875657 shutdown_connections 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.870+0000 7fad16fb6700 1 --2- 172.21.15.116:0/3458875657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad10101d80 0x7fad101021e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.870+0000 7fad16fb6700 1 --2- 172.21.15.116:0/3458875657 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad10100b80 0x7fad10100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.870+0000 7fad16fb6700 1 -- 172.21.15.116:0/3458875657 >> 172.21.15.116:0/3458875657 conn(0x7fad100fc140 msgr2=0x7fad100fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad16fb6700 1 -- 172.21.15.116:0/3458875657 shutdown_connections 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad16fb6700 1 -- 172.21.15.116:0/3458875657 wait complete. 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad16fb6700 1 Processor -- start 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad16fb6700 1 -- start start 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad16fb6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad10100b80 0x7fad10196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad16fb6700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad10101d80 0x7fad10196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad16fb6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10196e70 con 0x7fad10100b80 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad16fb6700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10196fb0 con 0x7fad10101d80 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad14d52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad10100b80 0x7fad10196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad14d52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad10100b80 0x7fad10196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53380/0 (socket says 172.21.15.116:53380) 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.871+0000 7fad14d52700 1 -- 172.21.15.116:0/322889381 learned_addr learned my addr 172.21.15.116:0/322889381 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.872+0000 7fad0ffff700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad10101d80 0x7fad10196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.872+0000 7fad14d52700 1 -- 172.21.15.116:0/322889381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad10101d80 msgr2=0x7fad10196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.872+0000 7fad14d52700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad10101d80 0x7fad10196850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.872+0000 7fad14d52700 1 -- 172.21.15.116:0/322889381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc0095f0 con 0x7fad10100b80 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.872+0000 7fad14d52700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad10100b80 0x7fad10196310 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fad0400d900 tx=0x7fad0400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.872+0000 7fad0dffb700 1 -- 172.21.15.116:0/322889381 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad040041f0 con 0x7fad10100b80 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.872+0000 7fad0dffb700 1 -- 172.21.15.116:0/322889381 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fad04004350 con 0x7fad10100b80 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.872+0000 7fad16fb6700 1 -- 172.21.15.116:0/322889381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1019ba60 con 0x7fad10100b80 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.872+0000 7fad0dffb700 1 -- 172.21.15.116:0/322889381 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad04003d20 con 0x7fad10100b80 2024-03-04T16:12:25.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.872+0000 7fad16fb6700 1 -- 172.21.15.116:0/322889381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1019bfb0 con 0x7fad10100b80 2024-03-04T16:12:25.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.873+0000 7fad16fb6700 1 -- 172.21.15.116:0/322889381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad101096e0 con 0x7fad10100b80 2024-03-04T16:12:25.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.874+0000 7fad0dffb700 1 -- 172.21.15.116:0/322889381 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fad04009730 con 0x7fad10100b80 2024-03-04T16:12:25.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.874+0000 7fad0dffb700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad0006d8a0 0x7fad0006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:25.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.874+0000 7fad0dffb700 1 -- 172.21.15.116:0/322889381 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fad0408d440 con 0x7fad10100b80 2024-03-04T16:12:25.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.874+0000 7fad0ffff700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad0006d8a0 0x7fad0006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:25.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.875+0000 7fad0ffff700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad0006d8a0 0x7fad0006fd60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7facfc006010 tx=0x7facfc00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:25.879+0000 7fad0dffb700 1 -- 172.21.15.116:0/322889381 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fad0405a6d0 con 0x7fad10100b80 2024-03-04T16:12:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.120+0000 7fad16fb6700 1 -- 172.21.15.116:0/322889381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fad1019ca80 con 0x7fad10100b80 2024-03-04T16:12:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.121+0000 7fad0dffb700 1 -- 172.21.15.116:0/322889381 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fad04003e80 con 0x7fad10100b80 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:e29 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 29 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:11:49.225261+0000 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:12:26.122 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 84 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=34400} 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:12:26.123 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{0:34400} state up:active seq 20 join_fscid=1 addr [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6828/786823228,v1:172.21.15.116:6829/786823228] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.152:6824/3736721533,v1:172.21.15.152:6825/3736721533] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:12:26.124 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.152:6826/1659417498,v1:172.21.15.152:6827/1659417498] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:12:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 -- 172.21.15.116:0/322889381 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad0006d8a0 msgr2=0x7fad0006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad0006d8a0 0x7fad0006fd60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7facfc006010 tx=0x7facfc00b540 comp rx=0 tx=0).stop 2024-03-04T16:12:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 -- 172.21.15.116:0/322889381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad10100b80 msgr2=0x7fad10196310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad10100b80 0x7fad10196310 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fad0400d900 tx=0x7fad0400dc10 comp rx=0 tx=0).stop 2024-03-04T16:12:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 -- 172.21.15.116:0/322889381 shutdown_connections 2024-03-04T16:12:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fad0006d8a0 0x7fad0006fd60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad10100b80 0x7fad10196310 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 --2- 172.21.15.116:0/322889381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad10101d80 0x7fad10196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 -- 172.21.15.116:0/322889381 >> 172.21.15.116:0/322889381 conn(0x7fad100fc140 msgr2=0x7fad10104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 -- 172.21.15.116:0/322889381 shutdown_connections 2024-03-04T16:12:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.124+0000 7fad16fb6700 1 -- 172.21.15.116:0/322889381 wait complete. 2024-03-04T16:12:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-04T16:12:26.226 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:26 smithi152 ceph-mon[82845]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:26.226 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:26 smithi152 ceph-mon[82845]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:26.226 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:26 smithi152 ceph-mon[82845]: pgmap v233: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:26.226 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:26 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:26.226 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:26 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:26.226 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:26 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:26.226 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:26 smithi152 ceph-mon[82845]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:26.226 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:26 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3992790251' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:26.226 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:26 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/322889381' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:12:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.230+0000 7ff00a852700 1 -- 172.21.15.116:0/1875589154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004104030 msgr2=0x7ff004106420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.230+0000 7ff00a852700 1 --2- 172.21.15.116:0/1875589154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004104030 0x7ff004106420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feff8009910 tx=0x7feff8009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.231+0000 7ff00a852700 1 -- 172.21.15.116:0/1875589154 shutdown_connections 2024-03-04T16:12:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.231+0000 7ff00a852700 1 --2- 172.21.15.116:0/1875589154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004104030 0x7ff004106420 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.231+0000 7ff00a852700 1 --2- 172.21.15.116:0/1875589154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff004101700 0x7ff004103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.231+0000 7ff00a852700 1 -- 172.21.15.116:0/1875589154 >> 172.21.15.116:0/1875589154 conn(0x7ff0040fb030 msgr2=0x7ff0040fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.231+0000 7ff00a852700 1 -- 172.21.15.116:0/1875589154 shutdown_connections 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.231+0000 7ff00a852700 1 -- 172.21.15.116:0/1875589154 wait complete. 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.231+0000 7ff00a852700 1 Processor -- start 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.231+0000 7ff00a852700 1 -- start start 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.231+0000 7ff00a852700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff004101700 0x7ff004198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff00a852700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004104030 0x7ff004198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff00a852700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff004199080 con 0x7ff004104030 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff00a852700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0041991c0 con 0x7ff004101700 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff0037fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004104030 0x7ff004198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff0037fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004104030 0x7ff004198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53410/0 (socket says 172.21.15.116:53410) 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff0037fe700 1 -- 172.21.15.116:0/982745582 learned_addr learned my addr 172.21.15.116:0/982745582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff003fff700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff004101700 0x7ff004198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff0037fe700 1 -- 172.21.15.116:0/982745582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff004101700 msgr2=0x7ff004198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff0037fe700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff004101700 0x7ff004198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff0037fe700 1 -- 172.21.15.116:0/982745582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff80095f0 con 0x7ff004104030 2024-03-04T16:12:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff0037fe700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004104030 0x7ff004198a60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7feff800bd70 tx=0x7feff800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff0017fa700 1 -- 172.21.15.116:0/982745582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feff801d070 con 0x7ff004104030 2024-03-04T16:12:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff00a852700 1 -- 172.21.15.116:0/982745582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00419dc10 con 0x7ff004104030 2024-03-04T16:12:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff0017fa700 1 -- 172.21.15.116:0/982745582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feff8005940 con 0x7ff004104030 2024-03-04T16:12:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff0017fa700 1 -- 172.21.15.116:0/982745582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feff8021430 con 0x7ff004104030 2024-03-04T16:12:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.232+0000 7ff00a852700 1 -- 172.21.15.116:0/982745582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00419e100 con 0x7ff004104030 2024-03-04T16:12:26.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.233+0000 7ff00a852700 1 -- 172.21.15.116:0/982745582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0040fcc20 con 0x7ff004104030 2024-03-04T16:12:26.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.234+0000 7ff0017fa700 1 -- 172.21.15.116:0/982745582 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7feff800f860 con 0x7ff004104030 2024-03-04T16:12:26.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.234+0000 7ff0017fa700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefec06d8a0 0x7fefec06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:26.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.234+0000 7ff0017fa700 1 -- 172.21.15.116:0/982745582 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7feff808e9a0 con 0x7ff004104030 2024-03-04T16:12:26.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.234+0000 7ff003fff700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefec06d8a0 0x7fefec06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:26.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.235+0000 7ff003fff700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefec06d8a0 0x7fefec06fd60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feff4005950 tx=0x7feff400a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.239+0000 7ff0017fa700 1 -- 172.21.15.116:0/982745582 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feff805bc30 con 0x7ff004104030 2024-03-04T16:12:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.465+0000 7ff00a852700 1 -- 172.21.15.116:0/982745582 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0040611d0 con 0x7fefec06d8a0 2024-03-04T16:12:26.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:26 smithi116 ceph-mon[94164]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:26.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:26 smithi116 ceph-mon[94164]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:26.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:26 smithi116 ceph-mon[94164]: pgmap v233: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:26.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:26 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:26.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:26 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:26.467 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:26 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:26.467 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:26 smithi116 ceph-mon[94164]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:26.467 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:26 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3992790251' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:26.467 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:26 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/322889381' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:12:26.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.467+0000 7ff0017fa700 1 -- 172.21.15.116:0/982745582 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+466 (secure 0 0 0) 0x7ff0040611d0 con 0x7fefec06d8a0 2024-03-04T16:12:26.467 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:12:26.467 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "mds", 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph-exporter", 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "osd", 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "18/23 daemons upgraded", 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:12:26.468 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:12:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 -- 172.21.15.116:0/982745582 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefec06d8a0 msgr2=0x7fefec06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefec06d8a0 0x7fefec06fd60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feff4005950 tx=0x7feff400a300 comp rx=0 tx=0).stop 2024-03-04T16:12:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 -- 172.21.15.116:0/982745582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004104030 msgr2=0x7ff004198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004104030 0x7ff004198a60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7feff800bd70 tx=0x7feff800bda0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 -- 172.21.15.116:0/982745582 shutdown_connections 2024-03-04T16:12:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fefec06d8a0 0x7fefec06fd60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff004101700 0x7ff004198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 --2- 172.21.15.116:0/982745582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004104030 0x7ff004198a60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 -- 172.21.15.116:0/982745582 >> 172.21.15.116:0/982745582 conn(0x7ff0040fb030 msgr2=0x7ff0040ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 -- 172.21.15.116:0/982745582 shutdown_connections 2024-03-04T16:12:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.469+0000 7ff00a852700 1 -- 172.21.15.116:0/982745582 wait complete. 2024-03-04T16:12:26.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.583+0000 7fb7d3fff700 1 -- 172.21.15.116:0/554198698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7cc0054b0 con 0x7fb7dc100cb0 2024-03-04T16:12:26.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.583+0000 7fb7e3562700 1 -- 172.21.15.116:0/554198698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7dc100cb0 msgr2=0x7fb7dc101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:26.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.583+0000 7fb7e3562700 1 --2- 172.21.15.116:0/554198698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7dc100cb0 0x7fb7dc101110 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009910 tx=0x7fb7cc009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:26.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.583+0000 7fb7e3562700 1 -- 172.21.15.116:0/554198698 shutdown_connections 2024-03-04T16:12:26.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.583+0000 7fb7e3562700 1 --2- 172.21.15.116:0/554198698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7dc100cb0 0x7fb7dc101110 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.583+0000 7fb7e3562700 1 --2- 172.21.15.116:0/554198698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7dc0ffab0 0x7fb7dc0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.583+0000 7fb7e3562700 1 -- 172.21.15.116:0/554198698 >> 172.21.15.116:0/554198698 conn(0x7fb7dc0fb050 msgr2=0x7fb7dc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:26.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.583+0000 7fb7e3562700 1 -- 172.21.15.116:0/554198698 shutdown_connections 2024-03-04T16:12:26.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.583+0000 7fb7e3562700 1 -- 172.21.15.116:0/554198698 wait complete. 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e3562700 1 Processor -- start 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e3562700 1 -- start start 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e3562700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7dc0ffab0 0x7fb7dc069800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e3562700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7dc100cb0 0x7fb7dc19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e3562700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc19b690 con 0x7fb7dc100cb0 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e3562700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc19b7d0 con 0x7fb7dc0ffab0 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e0afd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7dc100cb0 0x7fb7dc19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e0afd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7dc100cb0 0x7fb7dc19b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53434/0 (socket says 172.21.15.116:53434) 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e0afd700 1 -- 172.21.15.116:0/491185318 learned_addr learned my addr 172.21.15.116:0/491185318 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e12fe700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7dc0ffab0 0x7fb7dc069800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e0afd700 1 -- 172.21.15.116:0/491185318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7dc0ffab0 msgr2=0x7fb7dc069800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e0afd700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7dc0ffab0 0x7fb7dc069800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.584+0000 7fb7e0afd700 1 -- 172.21.15.116:0/491185318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc0095f0 con 0x7fb7dc100cb0 2024-03-04T16:12:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.585+0000 7fb7e0afd700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7dc100cb0 0x7fb7dc19b0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc005b40 tx=0x7fb7cc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.585+0000 7fb7d27fc700 1 -- 172.21.15.116:0/491185318 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7cc01d070 con 0x7fb7dc100cb0 2024-03-04T16:12:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.585+0000 7fb7e3562700 1 -- 172.21.15.116:0/491185318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7dc19ba50 con 0x7fb7dc100cb0 2024-03-04T16:12:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.585+0000 7fb7d27fc700 1 -- 172.21.15.116:0/491185318 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb7cc00f460 con 0x7fb7dc100cb0 2024-03-04T16:12:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.585+0000 7fb7d27fc700 1 -- 172.21.15.116:0/491185318 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7cc021600 con 0x7fb7dc100cb0 2024-03-04T16:12:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.585+0000 7fb7e3562700 1 -- 172.21.15.116:0/491185318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7dc19bf40 con 0x7fb7dc100cb0 2024-03-04T16:12:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.586+0000 7fb7e3562700 1 -- 172.21.15.116:0/491185318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7dc108610 con 0x7fb7dc100cb0 2024-03-04T16:12:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.586+0000 7fb7d27fc700 1 -- 172.21.15.116:0/491185318 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fb7cc00f5d0 con 0x7fb7dc100cb0 2024-03-04T16:12:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.587+0000 7fb7d27fc700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb7c806d8a0 0x7fb7c806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.587+0000 7fb7d27fc700 1 -- 172.21.15.116:0/491185318 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fb7cc08e8a0 con 0x7fb7dc100cb0 2024-03-04T16:12:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.587+0000 7fb7e12fe700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb7c806d8a0 0x7fb7c806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.587+0000 7fb7e12fe700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb7c806d8a0 0x7fb7c806fd60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009550 tx=0x7fb7d40091b0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.592+0000 7fb7d27fc700 1 -- 172.21.15.116:0/491185318 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb7cc05bb30 con 0x7fb7dc100cb0 2024-03-04T16:12:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.883+0000 7fb7e3562700 1 -- 172.21.15.116:0/491185318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb7dc19c640 con 0x7fb7dc100cb0 2024-03-04T16:12:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.883+0000 7fb7d27fc700 1 -- 172.21.15.116:0/491185318 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fb7cc05bb30 con 0x7fb7dc100cb0 2024-03-04T16:12:26.884 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:12:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.885+0000 7fb7e3562700 1 -- 172.21.15.116:0/491185318 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb7c806d8a0 msgr2=0x7fb7c806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.885+0000 7fb7e3562700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb7c806d8a0 0x7fb7c806fd60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009550 tx=0x7fb7d40091b0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.885+0000 7fb7e3562700 1 -- 172.21.15.116:0/491185318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7dc100cb0 msgr2=0x7fb7dc19b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.885+0000 7fb7e3562700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7dc100cb0 0x7fb7dc19b0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc005b40 tx=0x7fb7cc00bd70 comp rx=0 tx=0).stop 2024-03-04T16:12:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.885+0000 7fb7e3562700 1 -- 172.21.15.116:0/491185318 shutdown_connections 2024-03-04T16:12:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.885+0000 7fb7e3562700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb7c806d8a0 0x7fb7c806fd60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.885+0000 7fb7e3562700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7dc0ffab0 0x7fb7dc069800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.885+0000 7fb7e3562700 1 --2- 172.21.15.116:0/491185318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7dc100cb0 0x7fb7dc19b0c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.885+0000 7fb7e3562700 1 -- 172.21.15.116:0/491185318 >> 172.21.15.116:0/491185318 conn(0x7fb7dc0fb050 msgr2=0x7fb7dc103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.885+0000 7fb7e3562700 1 -- 172.21.15.116:0/491185318 shutdown_connections 2024-03-04T16:12:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:26.886+0000 7fb7e3562700 1 -- 172.21.15.116:0/491185318 wait complete. 2024-03-04T16:12:27.333 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:27 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/491185318' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:12:27.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:27 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/491185318' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:12:28.445 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:28 smithi116 ceph-mon[94164]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:28.445 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:28 smithi116 ceph-mon[94164]: pgmap v234: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:28.445 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:28.445 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:28 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:28.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:28 smithi152 ceph-mon[82845]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:28.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:28 smithi152 ceph-mon[82845]: pgmap v234: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:28.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:28.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:28 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:29.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:29 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:29.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:29 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:29.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:29 smithi152 ceph-mon[82845]: pgmap v235: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:29.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:29 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:29.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:29 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:29.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:29 smithi116 ceph-mon[94164]: pgmap v235: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:30 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:30 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:31.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:30 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:31.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:30 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:32.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:32.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:32.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:31 smithi116 ceph-mon[94164]: pgmap v236: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:32.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:32.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:32.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:31 smithi152 ceph-mon[82845]: pgmap v236: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:33.113 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:33 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:33.113 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:33 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:33.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:33 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:33.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:33 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:34.200 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:34 smithi116 ceph-mon[94164]: pgmap v237: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:34.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:34 smithi152 ceph-mon[82845]: pgmap v237: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:35.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:35 smithi116 ceph-mon[94164]: pgmap v238: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:36.109 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:35 smithi152 ceph-mon[82845]: pgmap v238: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:38.466 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:38 smithi116 ceph-mon[94164]: Upgrade: Updating prometheus.smithi116 2024-03-04T16:12:38.467 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:38 smithi116 ceph-mon[94164]: Deploying daemon prometheus.smithi116 on smithi116 2024-03-04T16:12:38.467 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:38 smithi116 ceph-mon[94164]: pgmap v239: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:38.467 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:38 smithi152 ceph-mon[82845]: Upgrade: Updating prometheus.smithi116 2024-03-04T16:12:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:38 smithi152 ceph-mon[82845]: Deploying daemon prometheus.smithi116 on smithi116 2024-03-04T16:12:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:38 smithi152 ceph-mon[82845]: pgmap v239: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:38.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:40.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:40 smithi116 ceph-mon[94164]: pgmap v240: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:40.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:40 smithi152 ceph-mon[82845]: pgmap v240: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:42 smithi116 ceph-mon[94164]: pgmap v241: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:42.475 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:42 smithi152 ceph-mon[82845]: pgmap v241: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:43.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:43 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:43.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:43 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:43.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:43 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:43 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:43 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:43 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:44.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:44 smithi116 ceph-mon[94164]: pgmap v242: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:44.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:44 smithi152 ceph-mon[82845]: pgmap v242: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:46.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:46 smithi116 ceph-mon[94164]: pgmap v243: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:46.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:46 smithi152 ceph-mon[82845]: pgmap v243: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:47.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:47 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:47.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:47 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:47.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:47 smithi116 ceph-mon[94164]: pgmap v244: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:48.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:47 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:48.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:47 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:48.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:47 smithi152 ceph-mon[82845]: pgmap v244: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:48.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:48 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:48.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:48 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:49.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:48 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:49.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:48 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:49.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:49 smithi116 ceph-mon[94164]: pgmap v245: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:50.107 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:49 smithi152 ceph-mon[82845]: pgmap v245: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: pgmap v246: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:12:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:12:52.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:52 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.573 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: pgmap v246: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:52.573 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.574 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:52.575 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:52 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:53.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:53.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:54.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:54 smithi116 ceph-mon[94164]: Upgrade: Updating alertmanager.smithi116 2024-03-04T16:12:54.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:54 smithi116 ceph-mon[94164]: Deploying daemon alertmanager.smithi116 on smithi116 2024-03-04T16:12:54.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:54 smithi116 ceph-mon[94164]: pgmap v247: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:54.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:54 smithi152 ceph-mon[82845]: Upgrade: Updating alertmanager.smithi116 2024-03-04T16:12:54.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:54 smithi152 ceph-mon[82845]: Deploying daemon alertmanager.smithi116 on smithi116 2024-03-04T16:12:54.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:54 smithi152 ceph-mon[82845]: pgmap v247: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:56 smithi116 ceph-mon[94164]: pgmap v248: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:56.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:56 smithi152 ceph-mon[82845]: pgmap v248: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.995+0000 7f26a4489700 1 -- 172.21.15.116:0/2349479811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f269c1016d0 msgr2=0x7f269c103ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.995+0000 7f26a4489700 1 --2- 172.21.15.116:0/2349479811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f269c1016d0 0x7f269c103ac0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2698009910 tx=0x7f2698009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.995+0000 7f26a4489700 1 -- 172.21.15.116:0/2349479811 shutdown_connections 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.995+0000 7f26a4489700 1 --2- 172.21.15.116:0/2349479811 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f269c104000 0x7f269c1063f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.995+0000 7f26a4489700 1 --2- 172.21.15.116:0/2349479811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f269c1016d0 0x7f269c103ac0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.995+0000 7f26a4489700 1 -- 172.21.15.116:0/2349479811 >> 172.21.15.116:0/2349479811 conn(0x7f269c0fb080 msgr2=0x7f269c0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.995+0000 7f26a4489700 1 -- 172.21.15.116:0/2349479811 shutdown_connections 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.995+0000 7f26a4489700 1 -- 172.21.15.116:0/2349479811 wait complete. 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.995+0000 7f26a4489700 1 Processor -- start 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a4489700 1 -- start start 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a4489700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f269c1016d0 0x7f269c101030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a4489700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f269c104000 0x7f269c0ff680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a4489700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0ffbc0 con 0x7f269c104000 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a4489700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0ffd00 con 0x7f269c1016d0 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a1a24700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f269c104000 0x7f269c0ff680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a1a24700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f269c104000 0x7f269c0ff680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36238/0 (socket says 172.21.15.116:36238) 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a1a24700 1 -- 172.21.15.116:0/2201874153 learned_addr learned my addr 172.21.15.116:0/2201874153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a2225700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f269c1016d0 0x7f269c101030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a1a24700 1 -- 172.21.15.116:0/2201874153 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f269c1016d0 msgr2=0x7f269c101030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a1a24700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f269c1016d0 0x7f269c101030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a1a24700 1 -- 172.21.15.116:0/2201874153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26980095f0 con 0x7f269c104000 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26a1a24700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f269c104000 0x7f269c0ff680 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f268c00c8a0 tx=0x7f268c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26937fe700 1 -- 172.21.15.116:0/2201874153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268c0078e0 con 0x7f269c104000 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.996+0000 7f26937fe700 1 -- 172.21.15.116:0/2201874153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f268c00f450 con 0x7f269c104000 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.997+0000 7f26a4489700 1 -- 172.21.15.116:0/2201874153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c0fffe0 con 0x7f269c104000 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.997+0000 7f26937fe700 1 -- 172.21.15.116:0/2201874153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268c00e5e0 con 0x7f269c104000 2024-03-04T16:12:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.997+0000 7f26a4489700 1 -- 172.21.15.116:0/2201874153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c19e3c0 con 0x7f269c104000 2024-03-04T16:12:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.997+0000 7f26a4489700 1 -- 172.21.15.116:0/2201874153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c190550 con 0x7f269c104000 2024-03-04T16:12:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.998+0000 7f26937fe700 1 -- 172.21.15.116:0/2201874153 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f268c00f5c0 con 0x7f269c104000 2024-03-04T16:12:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.998+0000 7f26937fe700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f268806d8a0 0x7f268806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.999+0000 7f26937fe700 1 -- 172.21.15.116:0/2201874153 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f268c08d580 con 0x7f269c104000 2024-03-04T16:12:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.999+0000 7f26a2225700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f268806d8a0 0x7f268806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:56.999+0000 7f26a2225700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f268806d8a0 0x7f268806fd60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2698006010 tx=0x7f269800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.003+0000 7f26937fe700 1 -- 172.21.15.116:0/2201874153 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f268c05a810 con 0x7f269c104000 2024-03-04T16:12:57.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.217+0000 7f26a4489700 1 -- 172.21.15.116:0/2201874153 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f269c0611d0 con 0x7f268806d8a0 2024-03-04T16:12:57.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.218+0000 7f26937fe700 1 -- 172.21.15.116:0/2201874153 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+465 (secure 0 0 0) 0x7f269c0611d0 con 0x7f268806d8a0 2024-03-04T16:12:57.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 -- 172.21.15.116:0/2201874153 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f268806d8a0 msgr2=0x7f268806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:57.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f268806d8a0 0x7f268806fd60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2698006010 tx=0x7f269800b540 comp rx=0 tx=0).stop 2024-03-04T16:12:57.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 -- 172.21.15.116:0/2201874153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f269c104000 msgr2=0x7f269c0ff680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:57.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f269c104000 0x7f269c0ff680 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f268c00c8a0 tx=0x7f268c00cbb0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 -- 172.21.15.116:0/2201874153 shutdown_connections 2024-03-04T16:12:57.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f268806d8a0 0x7f268806fd60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f269c1016d0 0x7f269c101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 --2- 172.21.15.116:0/2201874153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f269c104000 0x7f269c0ff680 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 -- 172.21.15.116:0/2201874153 >> 172.21.15.116:0/2201874153 conn(0x7f269c0fb080 msgr2=0x7f269c0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:57.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 -- 172.21.15.116:0/2201874153 shutdown_connections 2024-03-04T16:12:57.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.220+0000 7f26a4489700 1 -- 172.21.15.116:0/2201874153 wait complete. 2024-03-04T16:12:57.231 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:12:57.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.338+0000 7fed7a93b700 1 -- 172.21.15.116:0/3504735801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74100cb0 msgr2=0x7fed74101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:57.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.338+0000 7fed7a93b700 1 --2- 172.21.15.116:0/3504735801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74100cb0 0x7fed74101110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fed68009910 tx=0x7fed68009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:57.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.338+0000 7fed7a93b700 1 -- 172.21.15.116:0/3504735801 shutdown_connections 2024-03-04T16:12:57.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.338+0000 7fed7a93b700 1 --2- 172.21.15.116:0/3504735801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74100cb0 0x7fed74101110 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.338+0000 7fed7a93b700 1 --2- 172.21.15.116:0/3504735801 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed740ffab0 0x7fed740ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.338+0000 7fed7a93b700 1 -- 172.21.15.116:0/3504735801 >> 172.21.15.116:0/3504735801 conn(0x7fed740fb050 msgr2=0x7fed740fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:57.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.339+0000 7fed7a93b700 1 -- 172.21.15.116:0/3504735801 shutdown_connections 2024-03-04T16:12:57.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.339+0000 7fed7a93b700 1 -- 172.21.15.116:0/3504735801 wait complete. 2024-03-04T16:12:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.339+0000 7fed7a93b700 1 Processor -- start 2024-03-04T16:12:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.339+0000 7fed7a93b700 1 -- start start 2024-03-04T16:12:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed7a93b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed740ffab0 0x7fed74069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed7a93b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74100cb0 0x7fed7419b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed7a93b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7419b680 con 0x7fed74100cb0 2024-03-04T16:12:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed7a93b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7419b7c0 con 0x7fed740ffab0 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed737fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74100cb0 0x7fed7419b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed737fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74100cb0 0x7fed7419b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36256/0 (socket says 172.21.15.116:36256) 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed737fe700 1 -- 172.21.15.116:0/71699550 learned_addr learned my addr 172.21.15.116:0/71699550 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed73fff700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed740ffab0 0x7fed74069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed737fe700 1 -- 172.21.15.116:0/71699550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed740ffab0 msgr2=0x7fed74069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed737fe700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed740ffab0 0x7fed74069860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed737fe700 1 -- 172.21.15.116:0/71699550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed680095f0 con 0x7fed74100cb0 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.340+0000 7fed737fe700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74100cb0 0x7fed7419b060 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fed68006010 tx=0x7fed6800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.341+0000 7fed717fa700 1 -- 172.21.15.116:0/71699550 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed6801d070 con 0x7fed74100cb0 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.341+0000 7fed717fa700 1 -- 172.21.15.116:0/71699550 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fed6800f460 con 0x7fed74100cb0 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.341+0000 7fed717fa700 1 -- 172.21.15.116:0/71699550 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed68021640 con 0x7fed74100cb0 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.341+0000 7fed7a93b700 1 -- 172.21.15.116:0/71699550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7419ba40 con 0x7fed74100cb0 2024-03-04T16:12:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.341+0000 7fed7a93b700 1 -- 172.21.15.116:0/71699550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7419bf30 con 0x7fed74100cb0 2024-03-04T16:12:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.342+0000 7fed7a93b700 1 -- 172.21.15.116:0/71699550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed74108610 con 0x7fed74100cb0 2024-03-04T16:12:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.342+0000 7fed717fa700 1 -- 172.21.15.116:0/71699550 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fed6800fab0 con 0x7fed74100cb0 2024-03-04T16:12:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.343+0000 7fed717fa700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed5c06d8a0 0x7fed5c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.343+0000 7fed717fa700 1 -- 172.21.15.116:0/71699550 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fed6808eb90 con 0x7fed74100cb0 2024-03-04T16:12:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.343+0000 7fed73fff700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed5c06d8a0 0x7fed5c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.343+0000 7fed73fff700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed5c06d8a0 0x7fed5c06fd60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed74100b10 tx=0x7fed6400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.347+0000 7fed717fa700 1 -- 172.21.15.116:0/71699550 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fed6805c080 con 0x7fed74100cb0 2024-03-04T16:12:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.587+0000 7fed7a93b700 1 -- 172.21.15.116:0/71699550 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed74105600 con 0x7fed5c06d8a0 2024-03-04T16:12:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.589+0000 7fed717fa700 1 -- 172.21.15.116:0/71699550 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+465 (secure 0 0 0) 0x7fed74105600 con 0x7fed5c06d8a0 2024-03-04T16:12:57.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 -- 172.21.15.116:0/71699550 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed5c06d8a0 msgr2=0x7fed5c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:57.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed5c06d8a0 0x7fed5c06fd60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed74100b10 tx=0x7fed6400b410 comp rx=0 tx=0).stop 2024-03-04T16:12:57.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 -- 172.21.15.116:0/71699550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74100cb0 msgr2=0x7fed7419b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74100cb0 0x7fed7419b060 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fed68006010 tx=0x7fed6800bd70 comp rx=0 tx=0).stop 2024-03-04T16:12:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 -- 172.21.15.116:0/71699550 shutdown_connections 2024-03-04T16:12:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fed5c06d8a0 0x7fed5c06fd60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed740ffab0 0x7fed74069860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 --2- 172.21.15.116:0/71699550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74100cb0 0x7fed7419b060 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 -- 172.21.15.116:0/71699550 >> 172.21.15.116:0/71699550 conn(0x7fed740fb050 msgr2=0x7fed74103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 -- 172.21.15.116:0/71699550 shutdown_connections 2024-03-04T16:12:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.591+0000 7fed7a93b700 1 -- 172.21.15.116:0/71699550 wait complete. 2024-03-04T16:12:57.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.695+0000 7f1ed4372700 1 -- 172.21.15.116:0/480459386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ecc0ffab0 msgr2=0x7f1ecc0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.695+0000 7f1ed4372700 1 --2- 172.21.15.116:0/480459386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ecc0ffab0 0x7f1ecc0ffed0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009910 tx=0x7f1ec8009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.695+0000 7f1ed4372700 1 -- 172.21.15.116:0/480459386 shutdown_connections 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.695+0000 7f1ed4372700 1 --2- 172.21.15.116:0/480459386 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ecc100cb0 0x7f1ecc101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.695+0000 7f1ed4372700 1 --2- 172.21.15.116:0/480459386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ecc0ffab0 0x7f1ecc0ffed0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.695+0000 7f1ed4372700 1 -- 172.21.15.116:0/480459386 >> 172.21.15.116:0/480459386 conn(0x7f1ecc0fb030 msgr2=0x7f1ecc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.695+0000 7f1ed4372700 1 -- 172.21.15.116:0/480459386 shutdown_connections 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.695+0000 7f1ed4372700 1 -- 172.21.15.116:0/480459386 wait complete. 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed4372700 1 Processor -- start 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed4372700 1 -- start start 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed4372700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ecc0ffab0 0x7f1ecc068f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed4372700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ecc100cb0 0x7f1ecc19b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed4372700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc19b6f0 con 0x7f1ecc100cb0 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed4372700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc19b860 con 0x7f1ecc0ffab0 2024-03-04T16:12:57.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed190d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ecc100cb0 0x7f1ecc19b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed190d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ecc100cb0 0x7f1ecc19b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36268/0 (socket says 172.21.15.116:36268) 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed190d700 1 -- 172.21.15.116:0/3444788431 learned_addr learned my addr 172.21.15.116:0/3444788431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed210e700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ecc0ffab0 0x7f1ecc068f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed190d700 1 -- 172.21.15.116:0/3444788431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ecc0ffab0 msgr2=0x7f1ecc068f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed190d700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ecc0ffab0 0x7f1ecc068f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed190d700 1 -- 172.21.15.116:0/3444788431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80095f0 con 0x7f1ecc100cb0 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.696+0000 7f1ed190d700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ecc100cb0 0x7f1ecc19b120 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc00b4b0 tx=0x7f1ebc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.697+0000 7f1ec37fe700 1 -- 172.21.15.116:0/3444788431 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ebc011800 con 0x7f1ecc100cb0 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.697+0000 7f1ed4372700 1 -- 172.21.15.116:0/3444788431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ecc19bb40 con 0x7f1ecc100cb0 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.697+0000 7f1ed4372700 1 -- 172.21.15.116:0/3444788431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ecc19c110 con 0x7f1ecc100cb0 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.697+0000 7f1ec37fe700 1 -- 172.21.15.116:0/3444788431 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1ebc011e10 con 0x7f1ecc100cb0 2024-03-04T16:12:57.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.697+0000 7f1ec37fe700 1 -- 172.21.15.116:0/3444788431 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ebc0193b0 con 0x7f1ecc100cb0 2024-03-04T16:12:57.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.697+0000 7f1ed4372700 1 -- 172.21.15.116:0/3444788431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb0005320 con 0x7f1ecc100cb0 2024-03-04T16:12:57.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.698+0000 7f1ec37fe700 1 -- 172.21.15.116:0/3444788431 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f1ebc011960 con 0x7f1ecc100cb0 2024-03-04T16:12:57.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.698+0000 7f1ec37fe700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1eb806d8a0 0x7f1eb806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:57.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.699+0000 7f1ec37fe700 1 -- 172.21.15.116:0/3444788431 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f1ebc08cfa0 con 0x7f1ecc100cb0 2024-03-04T16:12:57.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.699+0000 7f1ed210e700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1eb806d8a0 0x7f1eb806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:57.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.699+0000 7f1ed210e700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1eb806d8a0 0x7f1eb806fd60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80059d0 tx=0x7f1ec8005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:57.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.703+0000 7f1ec37fe700 1 -- 172.21.15.116:0/3444788431 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1ebc0c0d20 con 0x7f1ecc100cb0 2024-03-04T16:12:57.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.947+0000 7f1ed4372700 1 -- 172.21.15.116:0/3444788431 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1eb0000bf0 con 0x7f1eb806d8a0 2024-03-04T16:12:57.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.957+0000 7f1ec37fe700 1 -- 172.21.15.116:0/3444788431 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f1eb0000bf0 con 0x7f1eb806d8a0 2024-03-04T16:12:57.958 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:12:57.958 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (16m) 11s ago 17m 25.5M - 0.25.0 c8568f914cd2 e4e251136180 2024-03-04T16:12:57.958 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (56s) 11s ago 17m 7864k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c67439dd3802 2024-03-04T16:12:57.958 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (51s) 30s ago 16m 7541k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6d482a1a1fd4 2024-03-04T16:12:57.958 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (5m) 11s ago 17m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:12:57.958 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (5m) 30s ago 16m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:12:57.958 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (15m) 11s ago 17m 88.9M - 9.4.7 954c08fa6188 c80831e8834c 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (117s) 11s ago 11m 13.8M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9d5061a093c6 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (2m) 11s ago 11m 86.6M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4bf8fc434069 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (70s) 30s ago 11m 13.7M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3abb2cd19d94 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (91s) 30s ago 11m 16.0M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4f752bc549b2 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (8m) 11s ago 19m 511M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (7m) 30s ago 16m 451M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (7m) 11s ago 19m 54.7M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (6m) 30s ago 16m 42.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (37s) 11s ago 17m 8530k - 1.5.0 0da6a335fe13 6a5a1f49ec33 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (33s) 30s ago 16m 3573k - 1.5.0 0da6a335fe13 d929fe9ef4b1 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (5m) 11s ago 15m 99.1M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (4m) 11s ago 15m 116M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (4m) 11s ago 14m 95.7M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (3m) 30s ago 14m 121M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6b01e97c8f9b 2024-03-04T16:12:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (3m) 30s ago 13m 112M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 5ecc30e5a86c 2024-03-04T16:12:57.960 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (2m) 30s ago 13m 102M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 1083700b476d 2024-03-04T16:12:57.960 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (16s) 11s ago 16m 40.3M - 2.43.0 a07b618ecd1d 0d85379f4e6b 2024-03-04T16:12:57.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.959+0000 7f1ed4372700 1 -- 172.21.15.116:0/3444788431 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1eb806d8a0 msgr2=0x7f1eb806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:57.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.959+0000 7f1ed4372700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1eb806d8a0 0x7f1eb806fd60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80059d0 tx=0x7f1ec8005960 comp rx=0 tx=0).stop 2024-03-04T16:12:57.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.959+0000 7f1ed4372700 1 -- 172.21.15.116:0/3444788431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ecc100cb0 msgr2=0x7f1ecc19b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:57.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.959+0000 7f1ed4372700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ecc100cb0 0x7f1ecc19b120 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc00b4b0 tx=0x7f1ebc00b870 comp rx=0 tx=0).stop 2024-03-04T16:12:57.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.959+0000 7f1ed4372700 1 -- 172.21.15.116:0/3444788431 shutdown_connections 2024-03-04T16:12:57.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.959+0000 7f1ed4372700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1eb806d8a0 0x7f1eb806fd60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.959+0000 7f1ed4372700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ecc0ffab0 0x7f1ecc068f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.959+0000 7f1ed4372700 1 --2- 172.21.15.116:0/3444788431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ecc100cb0 0x7f1ecc19b120 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:57.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.959+0000 7f1ed4372700 1 -- 172.21.15.116:0/3444788431 >> 172.21.15.116:0/3444788431 conn(0x7f1ecc0fb030 msgr2=0x7f1ecc103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:57.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.960+0000 7f1ed4372700 1 -- 172.21.15.116:0/3444788431 shutdown_connections 2024-03-04T16:12:57.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:57.960+0000 7f1ed4372700 1 -- 172.21.15.116:0/3444788431 wait complete. 2024-03-04T16:12:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.063+0000 7fc2ff732700 1 -- 172.21.15.116:0/153205033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f80ffab0 msgr2=0x7fc2f80ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.063+0000 7fc2ff732700 1 --2- 172.21.15.116:0/153205033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f80ffab0 0x7fc2f80ffed0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4009910 tx=0x7fc2f4009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.063+0000 7fc2ff732700 1 -- 172.21.15.116:0/153205033 shutdown_connections 2024-03-04T16:12:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.063+0000 7fc2ff732700 1 --2- 172.21.15.116:0/153205033 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2f8100cb0 0x7fc2f8101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.063+0000 7fc2ff732700 1 --2- 172.21.15.116:0/153205033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f80ffab0 0x7fc2f80ffed0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.063+0000 7fc2ff732700 1 -- 172.21.15.116:0/153205033 >> 172.21.15.116:0/153205033 conn(0x7fc2f80fb050 msgr2=0x7fc2f80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.063+0000 7fc2ff732700 1 -- 172.21.15.116:0/153205033 shutdown_connections 2024-03-04T16:12:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.063+0000 7fc2ff732700 1 -- 172.21.15.116:0/153205033 wait complete. 2024-03-04T16:12:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2ff732700 1 Processor -- start 2024-03-04T16:12:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2ff732700 1 -- start start 2024-03-04T16:12:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2ff732700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2f80ffab0 0x7fc2f8103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2ff732700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f8100cb0 0x7fc2f8102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2ff732700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f81025d0 con 0x7fc2f8100cb0 2024-03-04T16:12:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2ff732700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f8102710 con 0x7fc2f80ffab0 2024-03-04T16:12:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2fcccd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f8100cb0 0x7fc2f8102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2fcccd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f8100cb0 0x7fc2f8102090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36274/0 (socket says 172.21.15.116:36274) 2024-03-04T16:12:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2fcccd700 1 -- 172.21.15.116:0/3315369207 learned_addr learned my addr 172.21.15.116:0/3315369207 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2fd4ce700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2f80ffab0 0x7fc2f8103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2fcccd700 1 -- 172.21.15.116:0/3315369207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2f80ffab0 msgr2=0x7fc2f8103a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2fcccd700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2f80ffab0 0x7fc2f8103a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.064+0000 7fc2fcccd700 1 -- 172.21.15.116:0/3315369207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f40095f0 con 0x7fc2f8100cb0 2024-03-04T16:12:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.065+0000 7fc2fcccd700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f8100cb0 0x7fc2f8102090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800d8d0 tx=0x7fc2e800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.065+0000 7fc2ee7fc700 1 -- 172.21.15.116:0/3315369207 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e8009940 con 0x7fc2f8100cb0 2024-03-04T16:12:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.065+0000 7fc2ee7fc700 1 -- 172.21.15.116:0/3315369207 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc2e8010460 con 0x7fc2f8100cb0 2024-03-04T16:12:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.065+0000 7fc2ff732700 1 -- 172.21.15.116:0/3315369207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f81029f0 con 0x7fc2f8100cb0 2024-03-04T16:12:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.065+0000 7fc2ee7fc700 1 -- 172.21.15.116:0/3315369207 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e800f5f0 con 0x7fc2f8100cb0 2024-03-04T16:12:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.065+0000 7fc2ff732700 1 -- 172.21.15.116:0/3315369207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f819e3d0 con 0x7fc2f8100cb0 2024-03-04T16:12:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.066+0000 7fc2ff732700 1 -- 172.21.15.116:0/3315369207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f8108610 con 0x7fc2f8100cb0 2024-03-04T16:12:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.066+0000 7fc2ee7fc700 1 -- 172.21.15.116:0/3315369207 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fc2e8009aa0 con 0x7fc2f8100cb0 2024-03-04T16:12:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.067+0000 7fc2ee7fc700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc2e406d6a0 0x7fc2e406fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.067+0000 7fc2ee7fc700 1 -- 172.21.15.116:0/3315369207 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc2e808cfc0 con 0x7fc2f8100cb0 2024-03-04T16:12:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.067+0000 7fc2fd4ce700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc2e406d6a0 0x7fc2e406fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.067+0000 7fc2fd4ce700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc2e406d6a0 0x7fc2e406fb60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400b5c0 tx=0x7fc2f4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.072+0000 7fc2ee7fc700 1 -- 172.21.15.116:0/3315369207 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc2e805a250 con 0x7fc2f8100cb0 2024-03-04T16:12:58.368 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:12:58.368 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:12:58.368 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:12:58.369 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:12:58.370 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 14 2024-03-04T16:12:58.370 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:12:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:12:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.366+0000 7fc2ff732700 1 -- 172.21.15.116:0/3315369207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc2f819e980 con 0x7fc2f8100cb0 2024-03-04T16:12:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.367+0000 7fc2ee7fc700 1 -- 172.21.15.116:0/3315369207 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fc2e8059de0 con 0x7fc2f8100cb0 2024-03-04T16:12:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.369+0000 7fc2ff732700 1 -- 172.21.15.116:0/3315369207 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc2e406d6a0 msgr2=0x7fc2e406fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.369+0000 7fc2ff732700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc2e406d6a0 0x7fc2e406fb60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400b5c0 tx=0x7fc2f4005fd0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.369+0000 7fc2ff732700 1 -- 172.21.15.116:0/3315369207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f8100cb0 msgr2=0x7fc2f8102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.369+0000 7fc2ff732700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f8100cb0 0x7fc2f8102090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800d8d0 tx=0x7fc2e800dc90 comp rx=0 tx=0).stop 2024-03-04T16:12:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.369+0000 7fc2ff732700 1 -- 172.21.15.116:0/3315369207 shutdown_connections 2024-03-04T16:12:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.369+0000 7fc2ff732700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc2e406d6a0 0x7fc2e406fb60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.370+0000 7fc2ff732700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2f80ffab0 0x7fc2f8103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.370+0000 7fc2ff732700 1 --2- 172.21.15.116:0/3315369207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f8100cb0 0x7fc2f8102090 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.370+0000 7fc2ff732700 1 -- 172.21.15.116:0/3315369207 >> 172.21.15.116:0/3315369207 conn(0x7fc2f80fb050 msgr2=0x7fc2f8068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.370+0000 7fc2ff732700 1 -- 172.21.15.116:0/3315369207 shutdown_connections 2024-03-04T16:12:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.370+0000 7fc2ff732700 1 -- 172.21.15.116:0/3315369207 wait complete. 2024-03-04T16:12:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:58 smithi116 ceph-mon[94164]: pgmap v249: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:58 smithi116 ceph-mon[94164]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.476+0000 7fdcd21f2700 1 -- 172.21.15.116:0/3891874972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdccc103fd0 msgr2=0x7fdccc1063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.476+0000 7fdcd21f2700 1 --2- 172.21.15.116:0/3891874972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdccc103fd0 0x7fdccc1063c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009910 tx=0x7fdcb4009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.476+0000 7fdcd21f2700 1 -- 172.21.15.116:0/3891874972 shutdown_connections 2024-03-04T16:12:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.476+0000 7fdcd21f2700 1 --2- 172.21.15.116:0/3891874972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdccc103fd0 0x7fdccc1063c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.476+0000 7fdcd21f2700 1 --2- 172.21.15.116:0/3891874972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdccc1016a0 0x7fdccc103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.476+0000 7fdcd21f2700 1 -- 172.21.15.116:0/3891874972 >> 172.21.15.116:0/3891874972 conn(0x7fdccc0fb030 msgr2=0x7fdccc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.476+0000 7fdcd21f2700 1 -- 172.21.15.116:0/3891874972 shutdown_connections 2024-03-04T16:12:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.476+0000 7fdcd21f2700 1 -- 172.21.15.116:0/3891874972 wait complete. 2024-03-04T16:12:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.477+0000 7fdcd21f2700 1 Processor -- start 2024-03-04T16:12:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.477+0000 7fdcd21f2700 1 -- start start 2024-03-04T16:12:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.477+0000 7fdcd21f2700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdccc1016a0 0x7fdccc196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.477+0000 7fdcd21f2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdccc103fd0 0x7fdccc196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.477+0000 7fdcd21f2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc196e90 con 0x7fdccc103fd0 2024-03-04T16:12:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.477+0000 7fdcd21f2700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc196fd0 con 0x7fdccc1016a0 2024-03-04T16:12:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.477+0000 7fdccaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdccc103fd0 0x7fdccc196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.477+0000 7fdccaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdccc103fd0 0x7fdccc196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36288/0 (socket says 172.21.15.116:36288) 2024-03-04T16:12:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.477+0000 7fdccaffd700 1 -- 172.21.15.116:0/2396598938 learned_addr learned my addr 172.21.15.116:0/2396598938 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.478+0000 7fdccb7fe700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdccc1016a0 0x7fdccc196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.478+0000 7fdccaffd700 1 -- 172.21.15.116:0/2396598938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdccc1016a0 msgr2=0x7fdccc196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.478+0000 7fdccaffd700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdccc1016a0 0x7fdccc196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.478+0000 7fdccaffd700 1 -- 172.21.15.116:0/2396598938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb40095f0 con 0x7fdccc103fd0 2024-03-04T16:12:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.478+0000 7fdccaffd700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdccc103fd0 0x7fdccc196870 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400bd70 tx=0x7fdcb400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.478+0000 7fdcc8ff9700 1 -- 172.21.15.116:0/2396598938 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcb401d070 con 0x7fdccc103fd0 2024-03-04T16:12:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.478+0000 7fdcc8ff9700 1 -- 172.21.15.116:0/2396598938 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdcb4005940 con 0x7fdccc103fd0 2024-03-04T16:12:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.478+0000 7fdcc8ff9700 1 -- 172.21.15.116:0/2396598938 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcb4021430 con 0x7fdccc103fd0 2024-03-04T16:12:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.478+0000 7fdcd21f2700 1 -- 172.21.15.116:0/2396598938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc19ba20 con 0x7fdccc103fd0 2024-03-04T16:12:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.478+0000 7fdcd21f2700 1 -- 172.21.15.116:0/2396598938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdccc19bf10 con 0x7fdccc103fd0 2024-03-04T16:12:58.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.479+0000 7fdcd21f2700 1 -- 172.21.15.116:0/2396598938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdccc190520 con 0x7fdccc103fd0 2024-03-04T16:12:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.480+0000 7fdcc8ff9700 1 -- 172.21.15.116:0/2396598938 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fdcb400f860 con 0x7fdccc103fd0 2024-03-04T16:12:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.480+0000 7fdcc8ff9700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdcb806d8a0 0x7fdcb806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.480+0000 7fdcc8ff9700 1 -- 172.21.15.116:0/2396598938 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fdcb408e8c0 con 0x7fdccc103fd0 2024-03-04T16:12:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.480+0000 7fdccb7fe700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdcb806d8a0 0x7fdcb806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.481+0000 7fdccb7fe700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdcb806d8a0 0x7fdcb806fd60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4005950 tx=0x7fdcc400a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:58.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.485+0000 7fdcc8ff9700 1 -- 172.21.15.116:0/2396598938 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdcb405bb50 con 0x7fdccc103fd0 2024-03-04T16:12:58.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:58 smithi152 ceph-mon[82845]: pgmap v249: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:12:58.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:58 smithi152 ceph-mon[82845]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:58.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.736+0000 7fdcd21f2700 1 -- 172.21.15.116:0/2396598938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdccc02cc70 con 0x7fdccc103fd0 2024-03-04T16:12:58.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.736+0000 7fdcc8ff9700 1 -- 172.21.15.116:0/2396598938 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fdcb405b6e0 con 0x7fdccc103fd0 2024-03-04T16:12:58.737 INFO:teuthology.orchestra.run.smithi116.stdout:e29 2024-03-04T16:12:58.737 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:12:58.737 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:12:58.737 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:12:58.737 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:12:58.737 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:12:58.737 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:12:58.737 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 29 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:11:49.225261+0000 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:12:58.738 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 84 2024-03-04T16:12:58.739 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T16:12:58.739 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:12:58.739 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:12:58.739 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=34400} 2024-03-04T16:12:58.739 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:12:58.739 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:12:58.739 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:12:58.739 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:12:58.740 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:12:58.740 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:12:58.740 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:12:58.740 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:12:58.740 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:12:58.740 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{0:34400} state up:active seq 20 join_fscid=1 addr [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:12:58.740 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:12:58.740 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:12:58.740 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:12:58.740 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:12:58.741 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6828/786823228,v1:172.21.15.116:6829/786823228] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:12:58.741 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.152:6824/3736721533,v1:172.21.15.152:6825/3736721533] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:12:58.741 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.152:6826/1659417498,v1:172.21.15.152:6827/1659417498] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:12:58.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 -- 172.21.15.116:0/2396598938 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdcb806d8a0 msgr2=0x7fdcb806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:58.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdcb806d8a0 0x7fdcb806fd60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4005950 tx=0x7fdcc400a300 comp rx=0 tx=0).stop 2024-03-04T16:12:58.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 -- 172.21.15.116:0/2396598938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdccc103fd0 msgr2=0x7fdccc196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:58.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdccc103fd0 0x7fdccc196870 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400bd70 tx=0x7fdcb400bda0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 -- 172.21.15.116:0/2396598938 shutdown_connections 2024-03-04T16:12:58.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdcb806d8a0 0x7fdcb806fd60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdccc1016a0 0x7fdccc196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 --2- 172.21.15.116:0/2396598938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdccc103fd0 0x7fdccc196870 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 -- 172.21.15.116:0/2396598938 >> 172.21.15.116:0/2396598938 conn(0x7fdccc0fb030 msgr2=0x7fdccc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:58.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 -- 172.21.15.116:0/2396598938 shutdown_connections 2024-03-04T16:12:58.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.738+0000 7fdcd21f2700 1 -- 172.21.15.116:0/2396598938 wait complete. 2024-03-04T16:12:58.743 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-04T16:12:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.848+0000 7fef61754700 1 -- 172.21.15.116:0/1733654330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c074d40 msgr2=0x7fef5c0731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.848+0000 7fef61754700 1 --2- 172.21.15.116:0/1733654330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c074d40 0x7fef5c0731a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009910 tx=0x7fef4c009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.848+0000 7fef61754700 1 -- 172.21.15.116:0/1733654330 shutdown_connections 2024-03-04T16:12:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.848+0000 7fef61754700 1 --2- 172.21.15.116:0/1733654330 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef5c073770 0x7fef5c073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.848+0000 7fef61754700 1 --2- 172.21.15.116:0/1733654330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c074d40 0x7fef5c0731a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.848+0000 7fef61754700 1 -- 172.21.15.116:0/1733654330 >> 172.21.15.116:0/1733654330 conn(0x7fef5c0fc180 msgr2=0x7fef5c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.848+0000 7fef61754700 1 -- 172.21.15.116:0/1733654330 shutdown_connections 2024-03-04T16:12:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.848+0000 7fef61754700 1 -- 172.21.15.116:0/1733654330 wait complete. 2024-03-04T16:12:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.848+0000 7fef61754700 1 Processor -- start 2024-03-04T16:12:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef61754700 1 -- start start 2024-03-04T16:12:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef61754700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef5c073770 0x7fef5c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef61754700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c074d40 0x7fef5c198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef61754700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c1990b0 con 0x7fef5c074d40 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef61754700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c1991f0 con 0x7fef5c073770 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef5a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c074d40 0x7fef5c198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef5a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c074d40 0x7fef5c198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36302/0 (socket says 172.21.15.116:36302) 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef5a7fc700 1 -- 172.21.15.116:0/2851103430 learned_addr learned my addr 172.21.15.116:0/2851103430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef5affd700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef5c073770 0x7fef5c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef5a7fc700 1 -- 172.21.15.116:0/2851103430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef5c073770 msgr2=0x7fef5c198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef5a7fc700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef5c073770 0x7fef5c198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef5a7fc700 1 -- 172.21.15.116:0/2851103430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c0095f0 con 0x7fef5c074d40 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.849+0000 7fef5a7fc700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c074d40 0x7fef5c198a90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fef5000d8d0 tx=0x7fef5000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.850+0000 7fef43fff700 1 -- 172.21.15.116:0/2851103430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef50009940 con 0x7fef5c074d40 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.850+0000 7fef43fff700 1 -- 172.21.15.116:0/2851103430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fef50010460 con 0x7fef5c074d40 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.850+0000 7fef43fff700 1 -- 172.21.15.116:0/2851103430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef5000f5f0 con 0x7fef5c074d40 2024-03-04T16:12:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.850+0000 7fef61754700 1 -- 172.21.15.116:0/2851103430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5c19dca0 con 0x7fef5c074d40 2024-03-04T16:12:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.850+0000 7fef61754700 1 -- 172.21.15.116:0/2851103430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c19e1a0 con 0x7fef5c074d40 2024-03-04T16:12:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.851+0000 7fef61754700 1 -- 172.21.15.116:0/2851103430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5c10b950 con 0x7fef5c074d40 2024-03-04T16:12:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.851+0000 7fef43fff700 1 -- 172.21.15.116:0/2851103430 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fef500105d0 con 0x7fef5c074d40 2024-03-04T16:12:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.852+0000 7fef43fff700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fef4406d8a0 0x7fef4406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.852+0000 7fef43fff700 1 -- 172.21.15.116:0/2851103430 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fef5008d560 con 0x7fef5c074d40 2024-03-04T16:12:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.852+0000 7fef5affd700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fef4406d8a0 0x7fef4406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.852+0000 7fef5affd700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fef4406d8a0 0x7fef4406fd60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fef4c005b40 tx=0x7fef4c005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:58.858+0000 7fef43fff700 1 -- 172.21.15.116:0/2851103430 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fef5005a7f0 con 0x7fef5c074d40 2024-03-04T16:12:59.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.086+0000 7fef61754700 1 -- 172.21.15.116:0/2851103430 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef5c108180 con 0x7fef4406d8a0 2024-03-04T16:12:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.087+0000 7fef43fff700 1 -- 172.21.15.116:0/2851103430 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+465 (secure 0 0 0) 0x7fef5c108180 con 0x7fef4406d8a0 2024-03-04T16:12:59.088 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:12:59.088 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:12:59.088 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:12:59.088 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:12:59.088 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:12:59.088 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:12:59.088 INFO:teuthology.orchestra.run.smithi116.stdout: "mds", 2024-03-04T16:12:59.089 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph-exporter", 2024-03-04T16:12:59.089 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:12:59.089 INFO:teuthology.orchestra.run.smithi116.stdout: "osd", 2024-03-04T16:12:59.089 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:12:59.089 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:12:59.089 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "18/23 daemons upgraded", 2024-03-04T16:12:59.089 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading alertmanager daemons", 2024-03-04T16:12:59.089 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:12:59.089 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.089+0000 7fef61754700 1 -- 172.21.15.116:0/2851103430 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fef4406d8a0 msgr2=0x7fef4406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.089+0000 7fef61754700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fef4406d8a0 0x7fef4406fd60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fef4c005b40 tx=0x7fef4c005a90 comp rx=0 tx=0).stop 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.089+0000 7fef61754700 1 -- 172.21.15.116:0/2851103430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c074d40 msgr2=0x7fef5c198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.089+0000 7fef61754700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c074d40 0x7fef5c198a90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fef5000d8d0 tx=0x7fef5000dc90 comp rx=0 tx=0).stop 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.089+0000 7fef61754700 1 -- 172.21.15.116:0/2851103430 shutdown_connections 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.089+0000 7fef61754700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fef4406d8a0 0x7fef4406fd60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.090+0000 7fef61754700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef5c073770 0x7fef5c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.090+0000 7fef61754700 1 --2- 172.21.15.116:0/2851103430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c074d40 0x7fef5c198a90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.090+0000 7fef61754700 1 -- 172.21.15.116:0/2851103430 >> 172.21.15.116:0/2851103430 conn(0x7fef5c0fc180 msgr2=0x7fef5c106a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.090+0000 7fef61754700 1 -- 172.21.15.116:0/2851103430 shutdown_connections 2024-03-04T16:12:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.090+0000 7fef61754700 1 -- 172.21.15.116:0/2851103430 wait complete. 2024-03-04T16:12:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.198+0000 7fade7832700 1 -- 172.21.15.116:0/4218519620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fade0101de0 msgr2=0x7fade0102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.198+0000 7fade7832700 1 --2- 172.21.15.116:0/4218519620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fade0101de0 0x7fade0102240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009910 tx=0x7fadd0009c20 comp rx=0 tx=0).stop 2024-03-04T16:12:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.198+0000 7fade7832700 1 -- 172.21.15.116:0/4218519620 shutdown_connections 2024-03-04T16:12:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.198+0000 7fade7832700 1 --2- 172.21.15.116:0/4218519620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fade0101de0 0x7fade0102240 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.198+0000 7fade7832700 1 --2- 172.21.15.116:0/4218519620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0100be0 0x7fade0101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.198+0000 7fade7832700 1 -- 172.21.15.116:0/4218519620 >> 172.21.15.116:0/4218519620 conn(0x7fade00fc140 msgr2=0x7fade00fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade7832700 1 -- 172.21.15.116:0/4218519620 shutdown_connections 2024-03-04T16:12:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade7832700 1 -- 172.21.15.116:0/4218519620 wait complete. 2024-03-04T16:12:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade7832700 1 Processor -- start 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade7832700 1 -- start start 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade7832700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fade0100be0 0x7fade0196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade7832700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0101de0 0x7fade01968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade7832700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade0196ef0 con 0x7fade0100be0 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade7832700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade0197030 con 0x7fade0101de0 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade55ce700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fade0100be0 0x7fade0196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade55ce700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fade0100be0 0x7fade0196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36320/0 (socket says 172.21.15.116:36320) 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.199+0000 7fade55ce700 1 -- 172.21.15.116:0/1312813787 learned_addr learned my addr 172.21.15.116:0/1312813787 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.200+0000 7fade4dcd700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0101de0 0x7fade01968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.200+0000 7fade55ce700 1 -- 172.21.15.116:0/1312813787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0101de0 msgr2=0x7fade01968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.200+0000 7fade55ce700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0101de0 0x7fade01968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.200+0000 7fade55ce700 1 -- 172.21.15.116:0/1312813787 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd00095f0 con 0x7fade0100be0 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.200+0000 7fade55ce700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fade0100be0 0x7fade0196390 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faddc00d900 tx=0x7faddc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.200+0000 7fadd67fc700 1 -- 172.21.15.116:0/1312813787 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faddc0041f0 con 0x7fade0100be0 2024-03-04T16:12:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.200+0000 7fadd67fc700 1 -- 172.21.15.116:0/1312813787 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faddc004350 con 0x7fade0100be0 2024-03-04T16:12:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.200+0000 7fade7832700 1 -- 172.21.15.116:0/1312813787 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade019bae0 con 0x7fade0100be0 2024-03-04T16:12:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.200+0000 7fadd67fc700 1 -- 172.21.15.116:0/1312813787 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faddc003d20 con 0x7fade0100be0 2024-03-04T16:12:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.200+0000 7fade7832700 1 -- 172.21.15.116:0/1312813787 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade019c030 con 0x7fade0100be0 2024-03-04T16:12:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.201+0000 7fade7832700 1 -- 172.21.15.116:0/1312813787 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade0109740 con 0x7fade0100be0 2024-03-04T16:12:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.202+0000 7fadd67fc700 1 -- 172.21.15.116:0/1312813787 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7faddc010460 con 0x7fade0100be0 2024-03-04T16:12:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.202+0000 7fadd67fc700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fadcc06d8a0 0x7fadcc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:12:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.202+0000 7fadd67fc700 1 -- 172.21.15.116:0/1312813787 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7faddc08d4d0 con 0x7fade0100be0 2024-03-04T16:12:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.202+0000 7fade4dcd700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fadcc06d8a0 0x7fadcc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:12:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.202+0000 7fade4dcd700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fadcc06d8a0 0x7fadcc06fd60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fadd0006010 tx=0x7fadd000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:12:59.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.211+0000 7fadd67fc700 1 -- 172.21.15.116:0/1312813787 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faddc05a7d0 con 0x7fade0100be0 2024-03-04T16:12:59.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:59 smithi116 ceph-mon[94164]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:59.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:59 smithi116 ceph-mon[94164]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:59.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:59 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3315369207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:59 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2396598938' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:12:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:59 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:59 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:12:59 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:59.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.503+0000 7fade7832700 1 -- 172.21.15.116:0/1312813787 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fade0109930 con 0x7fade0100be0 2024-03-04T16:12:59.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.503+0000 7fadd67fc700 1 -- 172.21.15.116:0/1312813787 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7faddc05a360 con 0x7fade0100be0 2024-03-04T16:12:59.504 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:12:59.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 -- 172.21.15.116:0/1312813787 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fadcc06d8a0 msgr2=0x7fadcc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:59.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fadcc06d8a0 0x7fadcc06fd60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fadd0006010 tx=0x7fadd000b540 comp rx=0 tx=0).stop 2024-03-04T16:12:59.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 -- 172.21.15.116:0/1312813787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fade0100be0 msgr2=0x7fade0196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:12:59.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fade0100be0 0x7fade0196390 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faddc00d900 tx=0x7faddc00dc10 comp rx=0 tx=0).stop 2024-03-04T16:12:59.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 -- 172.21.15.116:0/1312813787 shutdown_connections 2024-03-04T16:12:59.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fadcc06d8a0 0x7fadcc06fd60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:59.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fade0100be0 0x7fade0196390 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faddc00d900 tx=0x7faddc00dc10 comp rx=0 tx=0).stop 2024-03-04T16:12:59.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 --2- 172.21.15.116:0/1312813787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0101de0 0x7fade01968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:12:59.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 -- 172.21.15.116:0/1312813787 >> 172.21.15.116:0/1312813787 conn(0x7fade00fc140 msgr2=0x7fade0105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:12:59.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 -- 172.21.15.116:0/1312813787 shutdown_connections 2024-03-04T16:12:59.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:12:59.506+0000 7fade7832700 1 -- 172.21.15.116:0/1312813787 wait complete. 2024-03-04T16:12:59.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:59 smithi152 ceph-mon[82845]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:59.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:59 smithi152 ceph-mon[82845]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:59.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:59 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3315369207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:59.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:59 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2396598938' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:12:59.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:59 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:59.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:59 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:12:59.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:12:59 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:00 smithi116 ceph-mon[94164]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:00 smithi116 ceph-mon[94164]: pgmap v250: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:00 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1312813787' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:13:00.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:00 smithi152 ceph-mon[82845]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:00.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:00 smithi152 ceph-mon[82845]: pgmap v250: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:00.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:00 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1312813787' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:13:02.405 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:02 smithi116 ceph-mon[94164]: pgmap v251: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:02.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:02 smithi152 ceph-mon[82845]: pgmap v251: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:04 smithi116 ceph-mon[94164]: pgmap v252: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:04 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:04 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:04.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:04 smithi152 ceph-mon[82845]: pgmap v252: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:04.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:04 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:04.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:04 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:05.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:05 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:05.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:05 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:05.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:05 smithi116 ceph-mon[94164]: pgmap v253: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:05.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:05 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:05.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:05 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:05.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:05 smithi152 ceph-mon[82845]: pgmap v253: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:08 smithi116 ceph-mon[94164]: pgmap v254: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:08 smithi152 ceph-mon[82845]: pgmap v254: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:09.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:13:09.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:09 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:13:09.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:13:09.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:09.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:09 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:10.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:10 smithi116 ceph-mon[94164]: pgmap v255: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:10 smithi152 ceph-mon[82845]: pgmap v255: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:11 smithi116 ceph-mon[94164]: Upgrade: Updating grafana.smithi116 2024-03-04T16:13:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:11 smithi116 ceph-mon[94164]: Deploying daemon grafana.smithi116 on smithi116 2024-03-04T16:13:11.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:11 smithi152 ceph-mon[82845]: Upgrade: Updating grafana.smithi116 2024-03-04T16:13:11.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:11 smithi152 ceph-mon[82845]: Deploying daemon grafana.smithi116 on smithi116 2024-03-04T16:13:12.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:12 smithi116 ceph-mon[94164]: pgmap v256: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:12.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:12 smithi152 ceph-mon[82845]: pgmap v256: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:14 smithi116 ceph-mon[94164]: pgmap v257: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:14.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:14 smithi152 ceph-mon[82845]: pgmap v257: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:16 smithi116 ceph-mon[94164]: pgmap v258: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:16.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:16 smithi152 ceph-mon[82845]: pgmap v258: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:18 smithi116 ceph-mon[94164]: pgmap v259: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:18.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:18 smithi152 ceph-mon[82845]: pgmap v259: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:20 smithi116 ceph-mon[94164]: pgmap v260: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:20.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:20 smithi152 ceph-mon[82845]: pgmap v260: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:22.450 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:22 smithi116 ceph-mon[94164]: pgmap v261: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:22.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:22 smithi152 ceph-mon[82845]: pgmap v261: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:23.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:24.293 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:24 smithi116 ceph-mon[94164]: pgmap v262: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:24.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:24 smithi152 ceph-mon[82845]: pgmap v262: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:26.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:26 smithi116 ceph-mon[94164]: pgmap v263: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:26.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:26 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:26.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:26 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:26.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:26 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:26 smithi152 ceph-mon[82845]: pgmap v263: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:26 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:26 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:26 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:28.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:28 smithi116 ceph-mon[94164]: pgmap v264: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:28.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:28 smithi152 ceph-mon[82845]: pgmap v264: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.615+0000 7fce29e92700 1 -- 172.21.15.116:0/662592557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce240ffab0 msgr2=0x7fce240ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.615+0000 7fce29e92700 1 --2- 172.21.15.116:0/662592557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce240ffab0 0x7fce240ffed0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fce18009910 tx=0x7fce18009c20 comp rx=0 tx=0).stop 2024-03-04T16:13:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.615+0000 7fce29e92700 1 -- 172.21.15.116:0/662592557 shutdown_connections 2024-03-04T16:13:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.615+0000 7fce29e92700 1 --2- 172.21.15.116:0/662592557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce24100cb0 0x7fce24101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.615+0000 7fce29e92700 1 --2- 172.21.15.116:0/662592557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce240ffab0 0x7fce240ffed0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.615+0000 7fce29e92700 1 -- 172.21.15.116:0/662592557 >> 172.21.15.116:0/662592557 conn(0x7fce240fb050 msgr2=0x7fce240fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.615+0000 7fce29e92700 1 -- 172.21.15.116:0/662592557 shutdown_connections 2024-03-04T16:13:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.615+0000 7fce29e92700 1 -- 172.21.15.116:0/662592557 wait complete. 2024-03-04T16:13:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.616+0000 7fce29e92700 1 Processor -- start 2024-03-04T16:13:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.616+0000 7fce29e92700 1 -- start start 2024-03-04T16:13:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.616+0000 7fce29e92700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce240ffab0 0x7fce24198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.616+0000 7fce29e92700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24100cb0 0x7fce24198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.616+0000 7fce29e92700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce241990e0 con 0x7fce24100cb0 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.616+0000 7fce29e92700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce24199220 con 0x7fce240ffab0 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.616+0000 7fce22ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24100cb0 0x7fce24198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.616+0000 7fce22ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24100cb0 0x7fce24198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59848/0 (socket says 172.21.15.116:59848) 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.616+0000 7fce22ffd700 1 -- 172.21.15.116:0/2538188419 learned_addr learned my addr 172.21.15.116:0/2538188419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.617+0000 7fce237fe700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce240ffab0 0x7fce24198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.617+0000 7fce22ffd700 1 -- 172.21.15.116:0/2538188419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce240ffab0 msgr2=0x7fce24198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.617+0000 7fce22ffd700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce240ffab0 0x7fce24198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.617+0000 7fce22ffd700 1 -- 172.21.15.116:0/2538188419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce180095f0 con 0x7fce24100cb0 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.617+0000 7fce22ffd700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24100cb0 0x7fce24198ac0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fce1400d8d0 tx=0x7fce1400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.617+0000 7fce20ff9700 1 -- 172.21.15.116:0/2538188419 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce140098a0 con 0x7fce24100cb0 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.617+0000 7fce29e92700 1 -- 172.21.15.116:0/2538188419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2419dcd0 con 0x7fce24100cb0 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.617+0000 7fce20ff9700 1 -- 172.21.15.116:0/2538188419 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fce14010460 con 0x7fce24100cb0 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.617+0000 7fce20ff9700 1 -- 172.21.15.116:0/2538188419 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce1400f5f0 con 0x7fce24100cb0 2024-03-04T16:13:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.617+0000 7fce29e92700 1 -- 172.21.15.116:0/2538188419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2419e220 con 0x7fce24100cb0 2024-03-04T16:13:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.618+0000 7fce29e92700 1 -- 172.21.15.116:0/2538188419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce24108610 con 0x7fce24100cb0 2024-03-04T16:13:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.619+0000 7fce20ff9700 1 -- 172.21.15.116:0/2538188419 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fce1400f820 con 0x7fce24100cb0 2024-03-04T16:13:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.619+0000 7fce20ff9700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fce0c06d8a0 0x7fce0c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.619+0000 7fce20ff9700 1 -- 172.21.15.116:0/2538188419 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fce1408d470 con 0x7fce24100cb0 2024-03-04T16:13:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.619+0000 7fce237fe700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fce0c06d8a0 0x7fce0c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.620+0000 7fce237fe700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fce0c06d8a0 0x7fce0c06fd60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fce18006010 tx=0x7fce1800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.624+0000 7fce20ff9700 1 -- 172.21.15.116:0/2538188419 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fce1405a700 con 0x7fce24100cb0 2024-03-04T16:13:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.853+0000 7fce29e92700 1 -- 172.21.15.116:0/2538188419 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce24069390 con 0x7fce0c06d8a0 2024-03-04T16:13:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.855+0000 7fce20ff9700 1 -- 172.21.15.116:0/2538188419 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+460 (secure 0 0 0) 0x7fce24069390 con 0x7fce0c06d8a0 2024-03-04T16:13:29.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.856+0000 7fce29e92700 1 -- 172.21.15.116:0/2538188419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fce0c06d8a0 msgr2=0x7fce0c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:29.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.856+0000 7fce29e92700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fce0c06d8a0 0x7fce0c06fd60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fce18006010 tx=0x7fce1800b540 comp rx=0 tx=0).stop 2024-03-04T16:13:29.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.856+0000 7fce29e92700 1 -- 172.21.15.116:0/2538188419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24100cb0 msgr2=0x7fce24198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:29.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.856+0000 7fce29e92700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24100cb0 0x7fce24198ac0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fce1400d8d0 tx=0x7fce1400dbe0 comp rx=0 tx=0).stop 2024-03-04T16:13:29.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.857+0000 7fce29e92700 1 -- 172.21.15.116:0/2538188419 shutdown_connections 2024-03-04T16:13:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.857+0000 7fce29e92700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fce0c06d8a0 0x7fce0c06fd60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.857+0000 7fce29e92700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce240ffab0 0x7fce24198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.857+0000 7fce29e92700 1 --2- 172.21.15.116:0/2538188419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24100cb0 0x7fce24198ac0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.857+0000 7fce29e92700 1 -- 172.21.15.116:0/2538188419 >> 172.21.15.116:0/2538188419 conn(0x7fce240fb050 msgr2=0x7fce24068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.857+0000 7fce29e92700 1 -- 172.21.15.116:0/2538188419 shutdown_connections 2024-03-04T16:13:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.857+0000 7fce29e92700 1 -- 172.21.15.116:0/2538188419 wait complete. 2024-03-04T16:13:29.868 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:13:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.967+0000 7f8fad17f700 1 -- 172.21.15.116:0/3007262407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa8103f70 msgr2=0x7f8fa8106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.967+0000 7f8fad17f700 1 --2- 172.21.15.116:0/3007262407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa8103f70 0x7f8fa8106360 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009910 tx=0x7f8f9c009c20 comp rx=0 tx=0).stop 2024-03-04T16:13:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.967+0000 7f8fad17f700 1 -- 172.21.15.116:0/3007262407 shutdown_connections 2024-03-04T16:13:29.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.967+0000 7f8fad17f700 1 --2- 172.21.15.116:0/3007262407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa8103f70 0x7f8fa8106360 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:29.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.968+0000 7f8fad17f700 1 --2- 172.21.15.116:0/3007262407 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fa8101640 0x7f8fa8103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:29.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.968+0000 7f8fad17f700 1 -- 172.21.15.116:0/3007262407 >> 172.21.15.116:0/3007262407 conn(0x7f8fa80fb050 msgr2=0x7f8fa80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:29.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.968+0000 7f8fad17f700 1 -- 172.21.15.116:0/3007262407 shutdown_connections 2024-03-04T16:13:29.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.968+0000 7f8fad17f700 1 -- 172.21.15.116:0/3007262407 wait complete. 2024-03-04T16:13:29.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.968+0000 7f8fad17f700 1 Processor -- start 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fad17f700 1 -- start start 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fad17f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa8101640 0x7f8fa8196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fad17f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fa8103f70 0x7f8fa8196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fad17f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa8196df0 con 0x7f8fa8101640 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fad17f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa8196f30 con 0x7f8fa8103f70 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fa6d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa8101640 0x7f8fa8196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fa6d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa8101640 0x7f8fa8196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59880/0 (socket says 172.21.15.116:59880) 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fa6d9d700 1 -- 172.21.15.116:0/3649498527 learned_addr learned my addr 172.21.15.116:0/3649498527 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fa659c700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fa8103f70 0x7f8fa8196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fa6d9d700 1 -- 172.21.15.116:0/3649498527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fa8103f70 msgr2=0x7f8fa8196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fa6d9d700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fa8103f70 0x7f8fa8196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fa6d9d700 1 -- 172.21.15.116:0/3649498527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c0095f0 con 0x7f8fa8101640 2024-03-04T16:13:29.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.969+0000 7f8fa6d9d700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa8101640 0x7f8fa8196320 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800c930 tx=0x7f8f9800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:29.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.970+0000 7f8f8ffff700 1 -- 172.21.15.116:0/3649498527 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f98007ad0 con 0x7f8fa8101640 2024-03-04T16:13:29.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.970+0000 7f8f8ffff700 1 -- 172.21.15.116:0/3649498527 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8f9800ce80 con 0x7f8fa8101640 2024-03-04T16:13:29.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.970+0000 7f8fad17f700 1 -- 172.21.15.116:0/3649498527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa819b9f0 con 0x7f8fa8101640 2024-03-04T16:13:29.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.970+0000 7f8f8ffff700 1 -- 172.21.15.116:0/3649498527 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f98018630 con 0x7f8fa8101640 2024-03-04T16:13:29.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.970+0000 7f8fad17f700 1 -- 172.21.15.116:0/3649498527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa819bf10 con 0x7f8fa8101640 2024-03-04T16:13:29.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.971+0000 7f8fad17f700 1 -- 172.21.15.116:0/3649498527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa8190520 con 0x7f8fa8101640 2024-03-04T16:13:29.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.971+0000 7f8f8ffff700 1 -- 172.21.15.116:0/3649498527 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f8f98007c30 con 0x7f8fa8101640 2024-03-04T16:13:29.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.972+0000 7f8f8ffff700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8f9006d8a0 0x7f8f9006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:29.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.972+0000 7f8fa659c700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8f9006d8a0 0x7f8f9006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:29.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.972+0000 7f8f8ffff700 1 -- 172.21.15.116:0/3649498527 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8f9808d580 con 0x7f8fa8101640 2024-03-04T16:13:29.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.972+0000 7f8fa659c700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8f9006d8a0 0x7f8f9006fd60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00b5c0 tx=0x7f8f9c0052a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:29.976+0000 7f8f8ffff700 1 -- 172.21.15.116:0/3649498527 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8f9805a810 con 0x7f8fa8101640 2024-03-04T16:13:30.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.194+0000 7f8fad17f700 1 -- 172.21.15.116:0/3649498527 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fa80611d0 con 0x7f8f9006d8a0 2024-03-04T16:13:30.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.195+0000 7f8f8ffff700 1 -- 172.21.15.116:0/3649498527 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+460 (secure 0 0 0) 0x7f8fa80611d0 con 0x7f8f9006d8a0 2024-03-04T16:13:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.197+0000 7f8fad17f700 1 -- 172.21.15.116:0/3649498527 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8f9006d8a0 msgr2=0x7f8f9006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.197+0000 7f8fad17f700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8f9006d8a0 0x7f8f9006fd60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00b5c0 tx=0x7f8f9c0052a0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.197+0000 7f8fad17f700 1 -- 172.21.15.116:0/3649498527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa8101640 msgr2=0x7f8fa8196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.197+0000 7f8fad17f700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa8101640 0x7f8fa8196320 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800c930 tx=0x7f8f9800cc40 comp rx=0 tx=0).stop 2024-03-04T16:13:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.197+0000 7f8fad17f700 1 -- 172.21.15.116:0/3649498527 shutdown_connections 2024-03-04T16:13:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.197+0000 7f8fad17f700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f8f9006d8a0 0x7f8f9006fd60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.197+0000 7f8fad17f700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa8101640 0x7f8fa8196320 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.197+0000 7f8fad17f700 1 --2- 172.21.15.116:0/3649498527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fa8103f70 0x7f8fa8196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.197+0000 7f8fad17f700 1 -- 172.21.15.116:0/3649498527 >> 172.21.15.116:0/3649498527 conn(0x7f8fa80fb050 msgr2=0x7f8fa80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.197+0000 7f8fad17f700 1 -- 172.21.15.116:0/3649498527 shutdown_connections 2024-03-04T16:13:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.198+0000 7f8fad17f700 1 -- 172.21.15.116:0/3649498527 wait complete. 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.302+0000 7fdbdc897700 1 -- 172.21.15.116:0/2205680632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd8100cc0 msgr2=0x7fdbd8101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.302+0000 7fdbdc897700 1 --2- 172.21.15.116:0/2205680632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd8100cc0 0x7fdbd8101120 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8009910 tx=0x7fdbc8009c20 comp rx=0 tx=0).stop 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.302+0000 7fdbdc897700 1 -- 172.21.15.116:0/2205680632 shutdown_connections 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.302+0000 7fdbdc897700 1 --2- 172.21.15.116:0/2205680632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd8100cc0 0x7fdbd8101120 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.302+0000 7fdbdc897700 1 --2- 172.21.15.116:0/2205680632 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbd80ffac0 0x7fdbd80ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.302+0000 7fdbdc897700 1 -- 172.21.15.116:0/2205680632 >> 172.21.15.116:0/2205680632 conn(0x7fdbd80fb060 msgr2=0x7fdbd80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.302+0000 7fdbdc897700 1 -- 172.21.15.116:0/2205680632 shutdown_connections 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.302+0000 7fdbdc897700 1 -- 172.21.15.116:0/2205680632 wait complete. 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.303+0000 7fdbdc897700 1 Processor -- start 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.303+0000 7fdbdc897700 1 -- start start 2024-03-04T16:13:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.303+0000 7fdbdc897700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd80ffac0 0x7fdbd8069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.303+0000 7fdbdc897700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbd8100cc0 0x7fdbd819b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.303+0000 7fdbdc897700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd819b720 con 0x7fdbd80ffac0 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.303+0000 7fdbdc897700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd819b890 con 0x7fdbd8100cc0 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.303+0000 7fdbd659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd80ffac0 0x7fdbd8069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.303+0000 7fdbd659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd80ffac0 0x7fdbd8069930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59884/0 (socket says 172.21.15.116:59884) 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.303+0000 7fdbd659c700 1 -- 172.21.15.116:0/2293668419 learned_addr learned my addr 172.21.15.116:0/2293668419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.304+0000 7fdbd5d9b700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbd8100cc0 0x7fdbd819b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.304+0000 7fdbd659c700 1 -- 172.21.15.116:0/2293668419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbd8100cc0 msgr2=0x7fdbd819b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.304+0000 7fdbd659c700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbd8100cc0 0x7fdbd819b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.304+0000 7fdbd659c700 1 -- 172.21.15.116:0/2293668419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc80095f0 con 0x7fdbd80ffac0 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.304+0000 7fdbd659c700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd80ffac0 0x7fdbd8069930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00b480 tx=0x7fdbcc00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.304+0000 7fdbc77fe700 1 -- 172.21.15.116:0/2293668419 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbcc011800 con 0x7fdbd80ffac0 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.304+0000 7fdbc77fe700 1 -- 172.21.15.116:0/2293668419 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdbcc011e10 con 0x7fdbd80ffac0 2024-03-04T16:13:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.304+0000 7fdbdc897700 1 -- 172.21.15.116:0/2293668419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd819bb70 con 0x7fdbd80ffac0 2024-03-04T16:13:30.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.304+0000 7fdbc77fe700 1 -- 172.21.15.116:0/2293668419 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbcc010ea0 con 0x7fdbd80ffac0 2024-03-04T16:13:30.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.304+0000 7fdbdc897700 1 -- 172.21.15.116:0/2293668419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd819c140 con 0x7fdbd80ffac0 2024-03-04T16:13:30.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.305+0000 7fdbdc897700 1 -- 172.21.15.116:0/2293668419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd81086c0 con 0x7fdbd80ffac0 2024-03-04T16:13:30.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.306+0000 7fdbc77fe700 1 -- 172.21.15.116:0/2293668419 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fdbcc010380 con 0x7fdbd80ffac0 2024-03-04T16:13:30.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.306+0000 7fdbc77fe700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdbc006d8a0 0x7fdbc006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:30.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.306+0000 7fdbd5d9b700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdbc006d8a0 0x7fdbc006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:30.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.306+0000 7fdbc77fe700 1 -- 172.21.15.116:0/2293668419 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fdbcc08d0e0 con 0x7fdbd80ffac0 2024-03-04T16:13:30.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.307+0000 7fdbd5d9b700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdbc006d8a0 0x7fdbc006fd60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8069e90 tx=0x7fdbc800b560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.311+0000 7fdbc77fe700 1 -- 172.21.15.116:0/2293668419 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdbcc05a370 con 0x7fdbd80ffac0 2024-03-04T16:13:30.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:30 smithi116 ceph-mon[94164]: pgmap v265: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:30.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.544+0000 7fdbdc897700 1 -- 172.21.15.116:0/2293668419 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbd8105610 con 0x7fdbc006d8a0 2024-03-04T16:13:30.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.555+0000 7fdbc77fe700 1 -- 172.21.15.116:0/2293668419 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fdbd8105610 con 0x7fdbc006d8a0 2024-03-04T16:13:30.556 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:13:30.556 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (32s) 27s ago 18m 14.7M - 0.25.0 c8568f914cd2 5a4a3998d5d9 2024-03-04T16:13:30.556 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (89s) 27s ago 18m 7898k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c67439dd3802 2024-03-04T16:13:30.556 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (83s) 62s ago 17m 7541k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6d482a1a1fd4 2024-03-04T16:13:30.556 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (6m) 27s ago 18m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:13:30.556 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (6m) 62s ago 17m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:13:30.556 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 starting - - - - 2024-03-04T16:13:30.556 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (2m) 27s ago 12m 13.9M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9d5061a093c6 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (2m) 27s ago 12m 86.6M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4bf8fc434069 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (102s) 62s ago 12m 13.7M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3abb2cd19d94 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (2m) 62s ago 12m 16.0M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4f752bc549b2 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (9m) 27s ago 19m 511M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (8m) 62s ago 17m 451M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (7m) 27s ago 19m 55.7M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (7m) 62s ago 16m 42.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (70s) 27s ago 18m 8556k - 1.5.0 0da6a335fe13 6a5a1f49ec33 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (65s) 62s ago 17m 3573k - 1.5.0 0da6a335fe13 d929fe9ef4b1 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (6m) 27s ago 16m 97.9M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:13:30.557 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (5m) 27s ago 15m 115M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:13:30.558 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (4m) 27s ago 15m 94.5M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:13:30.558 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (4m) 62s ago 14m 121M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6b01e97c8f9b 2024-03-04T16:13:30.558 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (3m) 62s ago 14m 112M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 5ecc30e5a86c 2024-03-04T16:13:30.558 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (3m) 62s ago 13m 102M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 1083700b476d 2024-03-04T16:13:30.558 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (48s) 27s ago 17m 48.8M - 2.43.0 a07b618ecd1d 0d85379f4e6b 2024-03-04T16:13:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 -- 172.21.15.116:0/2293668419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdbc006d8a0 msgr2=0x7fdbc006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdbc006d8a0 0x7fdbc006fd60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8069e90 tx=0x7fdbc800b560 comp rx=0 tx=0).stop 2024-03-04T16:13:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 -- 172.21.15.116:0/2293668419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd80ffac0 msgr2=0x7fdbd8069930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd80ffac0 0x7fdbd8069930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00b480 tx=0x7fdbcc00b790 comp rx=0 tx=0).stop 2024-03-04T16:13:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 -- 172.21.15.116:0/2293668419 shutdown_connections 2024-03-04T16:13:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdbc006d8a0 0x7fdbc006fd60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd80ffac0 0x7fdbd8069930 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 --2- 172.21.15.116:0/2293668419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbd8100cc0 0x7fdbd819b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 -- 172.21.15.116:0/2293668419 >> 172.21.15.116:0/2293668419 conn(0x7fdbd80fb060 msgr2=0x7fdbd8103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 -- 172.21.15.116:0/2293668419 shutdown_connections 2024-03-04T16:13:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.557+0000 7fdbdc897700 1 -- 172.21.15.116:0/2293668419 wait complete. 2024-03-04T16:13:30.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:30 smithi152 ceph-mon[82845]: pgmap v265: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.668+0000 7f58511a0700 1 -- 172.21.15.116:0/4230663794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f584c103f70 msgr2=0x7f584c106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.668+0000 7f58511a0700 1 --2- 172.21.15.116:0/4230663794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f584c103f70 0x7f584c106360 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5840009910 tx=0x7f5840009c20 comp rx=0 tx=0).stop 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.669+0000 7f58511a0700 1 -- 172.21.15.116:0/4230663794 shutdown_connections 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.669+0000 7f58511a0700 1 --2- 172.21.15.116:0/4230663794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f584c103f70 0x7f584c106360 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.669+0000 7f58511a0700 1 --2- 172.21.15.116:0/4230663794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f584c101640 0x7f584c103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.669+0000 7f58511a0700 1 -- 172.21.15.116:0/4230663794 >> 172.21.15.116:0/4230663794 conn(0x7f584c0fb030 msgr2=0x7f584c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.669+0000 7f58511a0700 1 -- 172.21.15.116:0/4230663794 shutdown_connections 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.669+0000 7f58511a0700 1 -- 172.21.15.116:0/4230663794 wait complete. 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.669+0000 7f58511a0700 1 Processor -- start 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f58511a0700 1 -- start start 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f58511a0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f584c101640 0x7f584c196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f58511a0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f584c103f70 0x7f584c196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f58511a0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c196e80 con 0x7f584c101640 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f58511a0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c196fc0 con 0x7f584c103f70 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f584ad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f584c101640 0x7f584c196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f584ad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f584c101640 0x7f584c196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59902/0 (socket says 172.21.15.116:59902) 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f584ad9d700 1 -- 172.21.15.116:0/2355720458 learned_addr learned my addr 172.21.15.116:0/2355720458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f584a59c700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f584c103f70 0x7f584c196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f584ad9d700 1 -- 172.21.15.116:0/2355720458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f584c103f70 msgr2=0x7f584c196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f584ad9d700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f584c103f70 0x7f584c196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.670+0000 7f584ad9d700 1 -- 172.21.15.116:0/2355720458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400095f0 con 0x7f584c101640 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.671+0000 7f584ad9d700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f584c101640 0x7f584c196320 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f583c00c8f0 tx=0x7f583c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.671+0000 7f5833fff700 1 -- 172.21.15.116:0/2355720458 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f583c004a00 con 0x7f584c101640 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.671+0000 7f5833fff700 1 -- 172.21.15.116:0/2355720458 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f583c0054d0 con 0x7f584c101640 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.671+0000 7f58511a0700 1 -- 172.21.15.116:0/2355720458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584c19ba70 con 0x7f584c101640 2024-03-04T16:13:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.671+0000 7f5833fff700 1 -- 172.21.15.116:0/2355720458 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f583c007ce0 con 0x7f584c101640 2024-03-04T16:13:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.671+0000 7f58511a0700 1 -- 172.21.15.116:0/2355720458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584c19bfc0 con 0x7f584c101640 2024-03-04T16:13:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.672+0000 7f58511a0700 1 -- 172.21.15.116:0/2355720458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584c190540 con 0x7f584c101640 2024-03-04T16:13:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.673+0000 7f5833fff700 1 -- 172.21.15.116:0/2355720458 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f583c005020 con 0x7f584c101640 2024-03-04T16:13:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.673+0000 7f5833fff700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f583406d8a0 0x7f583406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.673+0000 7f5833fff700 1 -- 172.21.15.116:0/2355720458 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f583c013070 con 0x7f584c101640 2024-03-04T16:13:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.673+0000 7f584a59c700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f583406d8a0 0x7f583406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.674+0000 7f584a59c700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f583406d8a0 0x7f583406fd60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5840006010 tx=0x7f584000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.678+0000 7f5833fff700 1 -- 172.21.15.116:0/2355720458 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f583c05a620 con 0x7f584c101640 2024-03-04T16:13:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.957+0000 7f58511a0700 1 -- 172.21.15.116:0/2355720458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f584c0619a0 con 0x7f584c101640 2024-03-04T16:13:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.957+0000 7f5833fff700 1 -- 172.21.15.116:0/2355720458 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f583c007e60 con 0x7f584c101640 2024-03-04T16:13:30.958 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:13:30.958 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:13:30.958 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:13:30.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:13:30.958 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:13:30.958 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:13:30.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:13:30.958 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:13:30.959 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:13:30.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:13:30.959 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:13:30.959 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:13:30.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:13:30.959 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:13:30.959 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 14 2024-03-04T16:13:30.959 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:13:30.959 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:13:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 -- 172.21.15.116:0/2355720458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f583406d8a0 msgr2=0x7f583406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f583406d8a0 0x7f583406fd60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5840006010 tx=0x7f584000b540 comp rx=0 tx=0).stop 2024-03-04T16:13:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 -- 172.21.15.116:0/2355720458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f584c101640 msgr2=0x7f584c196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f584c101640 0x7f584c196320 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f583c00c8f0 tx=0x7f583c00cc00 comp rx=0 tx=0).stop 2024-03-04T16:13:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 -- 172.21.15.116:0/2355720458 shutdown_connections 2024-03-04T16:13:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f583406d8a0 0x7f583406fd60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f584c101640 0x7f584c196320 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 --2- 172.21.15.116:0/2355720458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f584c103f70 0x7f584c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 -- 172.21.15.116:0/2355720458 >> 172.21.15.116:0/2355720458 conn(0x7f584c0fb030 msgr2=0x7f584c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 -- 172.21.15.116:0/2355720458 shutdown_connections 2024-03-04T16:13:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:30.961+0000 7f58511a0700 1 -- 172.21.15.116:0/2355720458 wait complete. 2024-03-04T16:13:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 -- 172.21.15.116:0/531686887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0101640 msgr2=0x7fa0e0103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 --2- 172.21.15.116:0/531686887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0101640 0x7fa0e0103a30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009910 tx=0x7fa0dc009c20 comp rx=0 tx=0).stop 2024-03-04T16:13:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 -- 172.21.15.116:0/531686887 shutdown_connections 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 --2- 172.21.15.116:0/531686887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0e0103f70 0x7fa0e0106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 --2- 172.21.15.116:0/531686887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0101640 0x7fa0e0103a30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 -- 172.21.15.116:0/531686887 >> 172.21.15.116:0/531686887 conn(0x7fa0e00fb030 msgr2=0x7fa0e00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 -- 172.21.15.116:0/531686887 shutdown_connections 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 -- 172.21.15.116:0/531686887 wait complete. 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 Processor -- start 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 -- start start 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.071+0000 7fa0e83f0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0101640 0x7fa0e0196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e83f0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0e0103f70 0x7fa0e0196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e83f0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e0196dd0 con 0x7fa0e0101640 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e83f0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e0196f10 con 0x7fa0e0103f70 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e618c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0101640 0x7fa0e0196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e618c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0101640 0x7fa0e0196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59918/0 (socket says 172.21.15.116:59918) 2024-03-04T16:13:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e618c700 1 -- 172.21.15.116:0/4252092174 learned_addr learned my addr 172.21.15.116:0/4252092174 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e598b700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0e0103f70 0x7fa0e0196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e618c700 1 -- 172.21.15.116:0/4252092174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0e0103f70 msgr2=0x7fa0e0196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e618c700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0e0103f70 0x7fa0e0196840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e618c700 1 -- 172.21.15.116:0/4252092174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc0095f0 con 0x7fa0e0101640 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e598b700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0e0103f70 0x7fa0e0196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e618c700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0101640 0x7fa0e0196300 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00bd70 tx=0x7fa0dc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0d77fe700 1 -- 172.21.15.116:0/4252092174 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0dc01d070 con 0x7fa0e0101640 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0d77fe700 1 -- 172.21.15.116:0/4252092174 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa0dc022470 con 0x7fa0e0101640 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e83f0700 1 -- 172.21.15.116:0/4252092174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e019b970 con 0x7fa0e0101640 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0d77fe700 1 -- 172.21.15.116:0/4252092174 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0dc00f670 con 0x7fa0e0101640 2024-03-04T16:13:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.072+0000 7fa0e83f0700 1 -- 172.21.15.116:0/4252092174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e019be30 con 0x7fa0e0101640 2024-03-04T16:13:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.074+0000 7fa0e83f0700 1 -- 172.21.15.116:0/4252092174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e0190520 con 0x7fa0e0101640 2024-03-04T16:13:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.074+0000 7fa0d77fe700 1 -- 172.21.15.116:0/4252092174 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fa0dc022a50 con 0x7fa0e0101640 2024-03-04T16:13:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.074+0000 7fa0d77fe700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa0cc06d8a0 0x7fa0cc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.075+0000 7fa0e598b700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa0cc06d8a0 0x7fa0cc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.075+0000 7fa0d77fe700 1 -- 172.21.15.116:0/4252092174 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fa0dc08eca0 con 0x7fa0e0101640 2024-03-04T16:13:31.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.075+0000 7fa0e598b700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa0cc06d8a0 0x7fa0cc06fd60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0005fd0 tx=0x7fa0d0005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:31.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.079+0000 7fa0d77fe700 1 -- 172.21.15.116:0/4252092174 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa0dc05c180 con 0x7fa0e0101640 2024-03-04T16:13:31.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.367+0000 7fa0e83f0700 1 -- 172.21.15.116:0/4252092174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa0e002cc70 con 0x7fa0e0101640 2024-03-04T16:13:31.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.368+0000 7fa0d77fe700 1 -- 172.21.15.116:0/4252092174 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fa0dc027070 con 0x7fa0e0101640 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:e29 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 29 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:11:49.225261+0000 2024-03-04T16:13:31.369 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 84 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=34400} 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:13:31.370 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{0:34400} state up:active seq 20 join_fscid=1 addr [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6828/786823228,v1:172.21.15.116:6829/786823228] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.152:6824/3736721533,v1:172.21.15.152:6825/3736721533] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:13:31.371 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.152:6826/1659417498,v1:172.21.15.152:6827/1659417498] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:13:31.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.370+0000 7fa0e83f0700 1 -- 172.21.15.116:0/4252092174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa0cc06d8a0 msgr2=0x7fa0cc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.370+0000 7fa0e83f0700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa0cc06d8a0 0x7fa0cc06fd60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0005fd0 tx=0x7fa0d0005e20 comp rx=0 tx=0).stop 2024-03-04T16:13:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.370+0000 7fa0e83f0700 1 -- 172.21.15.116:0/4252092174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0101640 msgr2=0x7fa0e0196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.370+0000 7fa0e83f0700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0101640 0x7fa0e0196300 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00bd70 tx=0x7fa0dc00be50 comp rx=0 tx=0).stop 2024-03-04T16:13:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.370+0000 7fa0e83f0700 1 -- 172.21.15.116:0/4252092174 shutdown_connections 2024-03-04T16:13:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.370+0000 7fa0e83f0700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa0cc06d8a0 0x7fa0cc06fd60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.370+0000 7fa0e83f0700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0101640 0x7fa0e0196300 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.370+0000 7fa0e83f0700 1 --2- 172.21.15.116:0/4252092174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0e0103f70 0x7fa0e0196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.370+0000 7fa0e83f0700 1 -- 172.21.15.116:0/4252092174 >> 172.21.15.116:0/4252092174 conn(0x7fa0e00fb030 msgr2=0x7fa0e00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.371+0000 7fa0e83f0700 1 -- 172.21.15.116:0/4252092174 shutdown_connections 2024-03-04T16:13:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.371+0000 7fa0e83f0700 1 -- 172.21.15.116:0/4252092174 wait complete. 2024-03-04T16:13:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-04T16:13:31.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:31 smithi116 ceph-mon[94164]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:31.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:31 smithi116 ceph-mon[94164]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:31.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:31.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:31 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:31.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:31 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2355720458' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:31.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.479+0000 7f58d4e28700 1 -- 172.21.15.116:0/2702795732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d0104010 msgr2=0x7f58d0106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:31.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.479+0000 7f58d4e28700 1 --2- 172.21.15.116:0/2702795732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d0104010 0x7f58d0106400 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009910 tx=0x7f58c4009c20 comp rx=0 tx=0).stop 2024-03-04T16:13:31.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.480+0000 7f58d4e28700 1 -- 172.21.15.116:0/2702795732 shutdown_connections 2024-03-04T16:13:31.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.480+0000 7f58d4e28700 1 --2- 172.21.15.116:0/2702795732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d0104010 0x7f58d0106400 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.480+0000 7f58d4e28700 1 --2- 172.21.15.116:0/2702795732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58d01016e0 0x7f58d0103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.480+0000 7f58d4e28700 1 -- 172.21.15.116:0/2702795732 >> 172.21.15.116:0/2702795732 conn(0x7f58d00fb030 msgr2=0x7f58d00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:31.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.480+0000 7f58d4e28700 1 -- 172.21.15.116:0/2702795732 shutdown_connections 2024-03-04T16:13:31.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.480+0000 7f58d4e28700 1 -- 172.21.15.116:0/2702795732 wait complete. 2024-03-04T16:13:31.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.480+0000 7f58d4e28700 1 Processor -- start 2024-03-04T16:13:31.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.480+0000 7f58d4e28700 1 -- start start 2024-03-04T16:13:31.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58d4e28700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d01016e0 0x7f58d0101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:31.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58d4e28700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58d0104010 0x7f58d00ff690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:31.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58d4e28700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00ffbd0 con 0x7f58d01016e0 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58d4e28700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00ffd10 con 0x7f58d0104010 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58ce59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d01016e0 0x7f58d0101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58ce59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d01016e0 0x7f58d0101040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59934/0 (socket says 172.21.15.116:59934) 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58ce59c700 1 -- 172.21.15.116:0/3422148625 learned_addr learned my addr 172.21.15.116:0/3422148625 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58cdd9b700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58d0104010 0x7f58d00ff690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58ce59c700 1 -- 172.21.15.116:0/3422148625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58d0104010 msgr2=0x7f58d00ff690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58ce59c700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58d0104010 0x7f58d00ff690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58ce59c700 1 -- 172.21.15.116:0/3422148625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c40095f0 con 0x7f58d01016e0 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58ce59c700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d01016e0 0x7f58d0101040 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f58c000d900 tx=0x7f58c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58bf7fe700 1 -- 172.21.15.116:0/3422148625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58c0004a00 con 0x7f58d01016e0 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58d4e28700 1 -- 172.21.15.116:0/3422148625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d00ffff0 con 0x7f58d01016e0 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58bf7fe700 1 -- 172.21.15.116:0/3422148625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f58c00054d0 con 0x7f58d01016e0 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.481+0000 7f58bf7fe700 1 -- 172.21.15.116:0/3422148625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58c0009d60 con 0x7f58d01016e0 2024-03-04T16:13:31.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.482+0000 7f58d4e28700 1 -- 172.21.15.116:0/3422148625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d019e390 con 0x7f58d01016e0 2024-03-04T16:13:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.482+0000 7f58d4e28700 1 -- 172.21.15.116:0/3422148625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d0190550 con 0x7f58d01016e0 2024-03-04T16:13:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.484+0000 7f58bf7fe700 1 -- 172.21.15.116:0/3422148625 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f58c0005000 con 0x7f58d01016e0 2024-03-04T16:13:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.484+0000 7f58bf7fe700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f58b806d8a0 0x7f58b806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.484+0000 7f58bf7fe700 1 -- 172.21.15.116:0/3422148625 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f58c008da60 con 0x7f58d01016e0 2024-03-04T16:13:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.484+0000 7f58cdd9b700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f58b806d8a0 0x7f58b806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.485+0000 7f58cdd9b700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f58b806d8a0 0x7f58b806fd60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f58c4005b40 tx=0x7f58c4005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.488+0000 7f58bf7fe700 1 -- 172.21.15.116:0/3422148625 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f58c005ae30 con 0x7f58d01016e0 2024-03-04T16:13:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:31 smithi152 ceph-mon[82845]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:31 smithi152 ceph-mon[82845]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:31 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:31.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:31 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2355720458' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.709+0000 7f58d4e28700 1 -- 172.21.15.116:0/3422148625 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f58d00611d0 con 0x7f58b806d8a0 2024-03-04T16:13:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.710+0000 7f58bf7fe700 1 -- 172.21.15.116:0/3422148625 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+460 (secure 0 0 0) 0x7f58d00611d0 con 0x7f58b806d8a0 2024-03-04T16:13:31.710 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:13:31.710 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "mds", 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph-exporter", 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "osd", 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "18/23 daemons upgraded", 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading grafana daemons", 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:13:31.711 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:13:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 -- 172.21.15.116:0/3422148625 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f58b806d8a0 msgr2=0x7f58b806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f58b806d8a0 0x7f58b806fd60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f58c4005b40 tx=0x7f58c4005a90 comp rx=0 tx=0).stop 2024-03-04T16:13:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 -- 172.21.15.116:0/3422148625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d01016e0 msgr2=0x7f58d0101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d01016e0 0x7f58d0101040 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f58c000d900 tx=0x7f58c000dcc0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 -- 172.21.15.116:0/3422148625 shutdown_connections 2024-03-04T16:13:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f58b806d8a0 0x7f58b806fd60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d01016e0 0x7f58d0101040 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 --2- 172.21.15.116:0/3422148625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58d0104010 0x7f58d00ff690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 -- 172.21.15.116:0/3422148625 >> 172.21.15.116:0/3422148625 conn(0x7f58d00fb030 msgr2=0x7f58d00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 -- 172.21.15.116:0/3422148625 shutdown_connections 2024-03-04T16:13:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.712+0000 7f58d4e28700 1 -- 172.21.15.116:0/3422148625 wait complete. 2024-03-04T16:13:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.836+0000 7f016315c700 1 -- 172.21.15.116:0/4248267986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0ffab0 msgr2=0x7f015c0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.836+0000 7f016315c700 1 --2- 172.21.15.116:0/4248267986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0ffab0 0x7f015c0ffed0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0150009910 tx=0x7f0150009c20 comp rx=0 tx=0).stop 2024-03-04T16:13:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.836+0000 7f016315c700 1 -- 172.21.15.116:0/4248267986 shutdown_connections 2024-03-04T16:13:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.836+0000 7f016315c700 1 --2- 172.21.15.116:0/4248267986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f015c100cb0 0x7f015c101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.836+0000 7f016315c700 1 --2- 172.21.15.116:0/4248267986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0ffab0 0x7f015c0ffed0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.836+0000 7f016315c700 1 -- 172.21.15.116:0/4248267986 >> 172.21.15.116:0/4248267986 conn(0x7f015c0fb050 msgr2=0x7f015c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.836+0000 7f016315c700 1 -- 172.21.15.116:0/4248267986 shutdown_connections 2024-03-04T16:13:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.836+0000 7f016315c700 1 -- 172.21.15.116:0/4248267986 wait complete. 2024-03-04T16:13:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.836+0000 7f016315c700 1 Processor -- start 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f016315c700 1 -- start start 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f016315c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f015c0ffab0 0x7f015c1963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f016315c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c100cb0 0x7f015c196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f016315c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c196f20 con 0x7f015c100cb0 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f016315c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c197060 con 0x7f015c0ffab0 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f015bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c100cb0 0x7f015c196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f015bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c100cb0 0x7f015c196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59960/0 (socket says 172.21.15.116:59960) 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f015bfff700 1 -- 172.21.15.116:0/2764954630 learned_addr learned my addr 172.21.15.116:0/2764954630 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f0160ef8700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f015c0ffab0 0x7f015c1963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f015bfff700 1 -- 172.21.15.116:0/2764954630 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f015c0ffab0 msgr2=0x7f015c1963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f015bfff700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f015c0ffab0 0x7f015c1963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f015bfff700 1 -- 172.21.15.116:0/2764954630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01500095f0 con 0x7f015c100cb0 2024-03-04T16:13:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.837+0000 7f015bfff700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c100cb0 0x7f015c196900 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f014800b4b0 tx=0x7f014800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.838+0000 7f0159ffb700 1 -- 172.21.15.116:0/2764954630 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0148011800 con 0x7f015c100cb0 2024-03-04T16:13:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.838+0000 7f0159ffb700 1 -- 172.21.15.116:0/2764954630 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0148011e10 con 0x7f015c100cb0 2024-03-04T16:13:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.838+0000 7f0159ffb700 1 -- 172.21.15.116:0/2764954630 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f014800f2e0 con 0x7f015c100cb0 2024-03-04T16:13:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.838+0000 7f016315c700 1 -- 172.21.15.116:0/2764954630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c19bb10 con 0x7f015c100cb0 2024-03-04T16:13:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.838+0000 7f016315c700 1 -- 172.21.15.116:0/2764954630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c19c0e0 con 0x7f015c100cb0 2024-03-04T16:13:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.839+0000 7f016315c700 1 -- 172.21.15.116:0/2764954630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c108690 con 0x7f015c100cb0 2024-03-04T16:13:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.839+0000 7f0159ffb700 1 -- 172.21.15.116:0/2764954630 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f0148010380 con 0x7f015c100cb0 2024-03-04T16:13:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.840+0000 7f0159ffb700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f014c06d8a0 0x7f014c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:13:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.840+0000 7f0159ffb700 1 -- 172.21.15.116:0/2764954630 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f014808d1d0 con 0x7f015c100cb0 2024-03-04T16:13:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.840+0000 7f0160ef8700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f014c06d8a0 0x7f014c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:13:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.840+0000 7f0160ef8700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f014c06d8a0 0x7f014c06fd60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0150006010 tx=0x7f0150005920 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:13:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:31.845+0000 7f0159ffb700 1 -- 172.21.15.116:0/2764954630 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f014805a460 con 0x7f015c100cb0 2024-03-04T16:13:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.130+0000 7f016315c700 1 -- 172.21.15.116:0/2764954630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f015c108880 con 0x7f015c100cb0 2024-03-04T16:13:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.131+0000 7f0159ffb700 1 -- 172.21.15.116:0/2764954630 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f0148059ff0 con 0x7f015c100cb0 2024-03-04T16:13:32.131 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:13:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 -- 172.21.15.116:0/2764954630 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f014c06d8a0 msgr2=0x7f014c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f014c06d8a0 0x7f014c06fd60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0150006010 tx=0x7f0150005920 comp rx=0 tx=0).stop 2024-03-04T16:13:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 -- 172.21.15.116:0/2764954630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c100cb0 msgr2=0x7f015c196900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:13:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c100cb0 0x7f015c196900 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f014800b4b0 tx=0x7f014800b870 comp rx=0 tx=0).stop 2024-03-04T16:13:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 -- 172.21.15.116:0/2764954630 shutdown_connections 2024-03-04T16:13:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f014c06d8a0 0x7f014c06fd60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f015c0ffab0 0x7f015c1963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 --2- 172.21.15.116:0/2764954630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c100cb0 0x7f015c196900 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:13:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 -- 172.21.15.116:0/2764954630 >> 172.21.15.116:0/2764954630 conn(0x7f015c0fb050 msgr2=0x7f015c068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:13:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 -- 172.21.15.116:0/2764954630 shutdown_connections 2024-03-04T16:13:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:13:32.133+0000 7f016315c700 1 -- 172.21.15.116:0/2764954630 wait complete. 2024-03-04T16:13:32.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:32 smithi116 ceph-mon[94164]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:32.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:32 smithi116 ceph-mon[94164]: pgmap v266: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:32.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:32 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/4252092174' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:13:32.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:32.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:32 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:32.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:32 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2764954630' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:13:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:32 smithi152 ceph-mon[82845]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:32 smithi152 ceph-mon[82845]: pgmap v266: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:32 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/4252092174' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:13:32.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:32.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:32 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:32.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:32 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2764954630' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:13:33.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:33 smithi116 ceph-mon[94164]: from='client.34576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:33.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:33 smithi152 ceph-mon[82845]: from='client.34576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:34 smithi116 ceph-mon[94164]: pgmap v267: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:34.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:34 smithi152 ceph-mon[82845]: pgmap v267: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: pgmap v268: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:13:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:36.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-04T16:13:36.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-04T16:13:36.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-04T16:13:36.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-04T16:13:36.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-04T16:13:36.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:36 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:36.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: pgmap v268: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:36.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:36.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:36.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:36.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:36.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:36.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:13:36.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:36.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.729 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.729 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.729 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.729 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.729 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.729 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:36.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:36.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-04T16:13:36.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-04T16:13:36.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-04T16:13:36.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-04T16:13:36.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-04T16:13:36.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-04T16:13:36.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-04T16:13:36.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-04T16:13:36.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-04T16:13:36.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-04T16:13:36.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-04T16:13:36.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-04T16:13:36.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-04T16:13:36.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-04T16:13:36.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-04T16:13:36.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-04T16:13:36.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-04T16:13:36.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-04T16:13:36.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-04T16:13:36.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-04T16:13:36.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-04T16:13:36.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-04T16:13:36.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-04T16:13:36.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-04T16:13:36.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:36 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:37 smithi152 ceph-mon[82845]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:13:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:37 smithi152 ceph-mon[82845]: Upgrade: Finalizing container_image settings 2024-03-04T16:13:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:37 smithi152 ceph-mon[82845]: Upgrade: Complete! 2024-03-04T16:13:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:37.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:37 smithi116 ceph-mon[94164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:13:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:37 smithi116 ceph-mon[94164]: Upgrade: Finalizing container_image settings 2024-03-04T16:13:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:37 smithi116 ceph-mon[94164]: Upgrade: Complete! 2024-03-04T16:13:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:37.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:38 smithi116 ceph-mon[94164]: pgmap v269: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:38 smithi152 ceph-mon[82845]: pgmap v269: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:13:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:40 smithi116 ceph-mon[94164]: pgmap v270: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:40 smithi152 ceph-mon[82845]: pgmap v270: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:42.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:42 smithi116 ceph-mon[94164]: pgmap v271: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:42 smithi152 ceph-mon[82845]: pgmap v271: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:44 smithi116 ceph-mon[94164]: pgmap v272: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:44.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:44 smithi152 ceph-mon[82845]: pgmap v272: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:46 smithi116 ceph-mon[94164]: pgmap v273: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:46.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:46 smithi152 ceph-mon[82845]: pgmap v273: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:47.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:47 smithi152 ceph-mon[82845]: pgmap v274: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:47.904 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:47 smithi116 ceph-mon[94164]: pgmap v274: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:50 smithi116 ceph-mon[94164]: pgmap v275: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:50.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:50 smithi152 ceph-mon[82845]: pgmap v275: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:52.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:52 smithi116 ceph-mon[94164]: pgmap v276: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:52 smithi152 ceph-mon[82845]: pgmap v276: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:53.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:53.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:54.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:54 smithi116 ceph-mon[94164]: pgmap v277: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:54.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:54 smithi152 ceph-mon[82845]: pgmap v277: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:56 smithi116 ceph-mon[94164]: pgmap v278: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:56.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:56 smithi152 ceph-mon[82845]: pgmap v278: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:13:58 smithi116 ceph-mon[94164]: pgmap v279: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:13:58.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:13:58 smithi152 ceph-mon[82845]: pgmap v279: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:00 smithi116 ceph-mon[94164]: pgmap v280: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:00.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:00 smithi152 ceph-mon[82845]: pgmap v280: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.253+0000 7fb7908b5700 1 -- 172.21.15.116:0/1130794395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb78c100cb0 msgr2=0x7fb78c101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.253+0000 7fb7908b5700 1 --2- 172.21.15.116:0/1130794395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb78c100cb0 0x7fb78c101110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb780009910 tx=0x7fb780009c20 comp rx=0 tx=0).stop 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.253+0000 7fb7908b5700 1 -- 172.21.15.116:0/1130794395 shutdown_connections 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.253+0000 7fb7908b5700 1 --2- 172.21.15.116:0/1130794395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb78c100cb0 0x7fb78c101110 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.253+0000 7fb7908b5700 1 --2- 172.21.15.116:0/1130794395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb78c0ffab0 0x7fb78c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.253+0000 7fb7908b5700 1 -- 172.21.15.116:0/1130794395 >> 172.21.15.116:0/1130794395 conn(0x7fb78c0fb030 msgr2=0x7fb78c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.253+0000 7fb7908b5700 1 -- 172.21.15.116:0/1130794395 shutdown_connections 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.253+0000 7fb7908b5700 1 -- 172.21.15.116:0/1130794395 wait complete. 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb7908b5700 1 Processor -- start 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb7908b5700 1 -- start start 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb7908b5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb78c0ffab0 0x7fb78c196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:02.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb7908b5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb78c100cb0 0x7fb78c1968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb7908b5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c196ed0 con 0x7fb78c0ffab0 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb7908b5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c197010 con 0x7fb78c100cb0 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb78a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb78c0ffab0 0x7fb78c196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb78a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb78c0ffab0 0x7fb78c196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51790/0 (socket says 172.21.15.116:51790) 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb78a59c700 1 -- 172.21.15.116:0/659306573 learned_addr learned my addr 172.21.15.116:0/659306573 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb789d9b700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb78c100cb0 0x7fb78c1968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb78a59c700 1 -- 172.21.15.116:0/659306573 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb78c100cb0 msgr2=0x7fb78c1968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb78a59c700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb78c100cb0 0x7fb78c1968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb78a59c700 1 -- 172.21.15.116:0/659306573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7800095f0 con 0x7fb78c0ffab0 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.254+0000 7fb78a59c700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb78c0ffab0 0x7fb78c196370 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00c8f0 tx=0x7fb77c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.255+0000 7fb77b7fe700 1 -- 172.21.15.116:0/659306573 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb77c0078e0 con 0x7fb78c0ffab0 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.255+0000 7fb77b7fe700 1 -- 172.21.15.116:0/659306573 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb77c00f450 con 0x7fb78c0ffab0 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.255+0000 7fb7908b5700 1 -- 172.21.15.116:0/659306573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c19bac0 con 0x7fb78c0ffab0 2024-03-04T16:14:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.255+0000 7fb77b7fe700 1 -- 172.21.15.116:0/659306573 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb77c018610 con 0x7fb78c0ffab0 2024-03-04T16:14:02.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.255+0000 7fb7908b5700 1 -- 172.21.15.116:0/659306573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c19c040 con 0x7fb78c0ffab0 2024-03-04T16:14:02.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.256+0000 7fb7908b5700 1 -- 172.21.15.116:0/659306573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c108610 con 0x7fb78c0ffab0 2024-03-04T16:14:02.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.256+0000 7fb77b7fe700 1 -- 172.21.15.116:0/659306573 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fb77c007a40 con 0x7fb78c0ffab0 2024-03-04T16:14:02.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.257+0000 7fb77b7fe700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb77406d8a0 0x7fb77406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:02.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.257+0000 7fb77b7fe700 1 -- 172.21.15.116:0/659306573 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fb77c08d400 con 0x7fb78c0ffab0 2024-03-04T16:14:02.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.257+0000 7fb789d9b700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb77406d8a0 0x7fb77406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:02.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.257+0000 7fb789d9b700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb77406d8a0 0x7fb77406fd60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb780006010 tx=0x7fb78000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:14:02.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.261+0000 7fb77b7fe700 1 -- 172.21.15.116:0/659306573 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb77c05a690 con 0x7fb78c0ffab0 2024-03-04T16:14:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.472+0000 7fb7908b5700 1 -- 172.21.15.116:0/659306573 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb78c068b20 con 0x7fb77406d8a0 2024-03-04T16:14:02.472 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:02 smithi116 ceph-mon[94164]: pgmap v281: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.473+0000 7fb77b7fe700 1 -- 172.21.15.116:0/659306573 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fb78c068b20 con 0x7fb77406d8a0 2024-03-04T16:14:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.475+0000 7fb7908b5700 1 -- 172.21.15.116:0/659306573 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb77406d8a0 msgr2=0x7fb77406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.475+0000 7fb7908b5700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb77406d8a0 0x7fb77406fd60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb780006010 tx=0x7fb78000b540 comp rx=0 tx=0).stop 2024-03-04T16:14:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.475+0000 7fb7908b5700 1 -- 172.21.15.116:0/659306573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb78c0ffab0 msgr2=0x7fb78c196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.475+0000 7fb7908b5700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb78c0ffab0 0x7fb78c196370 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00c8f0 tx=0x7fb77c00cc00 comp rx=0 tx=0).stop 2024-03-04T16:14:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.475+0000 7fb7908b5700 1 -- 172.21.15.116:0/659306573 shutdown_connections 2024-03-04T16:14:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.475+0000 7fb7908b5700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb77406d8a0 0x7fb77406fd60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.475+0000 7fb7908b5700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb78c0ffab0 0x7fb78c196370 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.475+0000 7fb7908b5700 1 --2- 172.21.15.116:0/659306573 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb78c100cb0 0x7fb78c1968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.475+0000 7fb7908b5700 1 -- 172.21.15.116:0/659306573 >> 172.21.15.116:0/659306573 conn(0x7fb78c0fb030 msgr2=0x7fb78c068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:14:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.476+0000 7fb7908b5700 1 -- 172.21.15.116:0/659306573 shutdown_connections 2024-03-04T16:14:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:02.476+0000 7fb7908b5700 1 -- 172.21.15.116:0/659306573 wait complete. 2024-03-04T16:14:02.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:02 smithi152 ceph-mon[82845]: pgmap v281: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:03.138 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch ps' 2024-03-04T16:14:03.349 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:14:04.455 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:04 smithi116 ceph-mon[94164]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:04.455 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:04 smithi116 ceph-mon[94164]: pgmap v282: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:04 smithi152 ceph-mon[82845]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:04.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:04 smithi152 ceph-mon[82845]: pgmap v282: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.791+0000 7f18cf73d700 1 -- 172.21.15.116:0/3357884070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8101750 msgr2=0x7f18c81057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.791+0000 7f18cf73d700 1 --2- 172.21.15.116:0/3357884070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8101750 0x7f18c81057a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009910 tx=0x7f18c4009c20 comp rx=0 tx=0).stop 2024-03-04T16:14:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.791+0000 7f18cf73d700 1 -- 172.21.15.116:0/3357884070 shutdown_connections 2024-03-04T16:14:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.791+0000 7f18cf73d700 1 --2- 172.21.15.116:0/3357884070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8101750 0x7f18c81057a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.791+0000 7f18cf73d700 1 --2- 172.21.15.116:0/3357884070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18c8100da0 0x7f18c8101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.791+0000 7f18cf73d700 1 -- 172.21.15.116:0/3357884070 >> 172.21.15.116:0/3357884070 conn(0x7f18c80fc630 msgr2=0x7f18c80fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:14:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.791+0000 7f18cf73d700 1 -- 172.21.15.116:0/3357884070 shutdown_connections 2024-03-04T16:14:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.791+0000 7f18cf73d700 1 -- 172.21.15.116:0/3357884070 wait complete. 2024-03-04T16:14:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cf73d700 1 Processor -- start 2024-03-04T16:14:04.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cf73d700 1 -- start start 2024-03-04T16:14:04.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cf73d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8100da0 0x7f18c8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:04.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cf73d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18c8101750 0x7f18c80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:04.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cf73d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c8073b10 con 0x7f18c8100da0 2024-03-04T16:14:04.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cf73d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c8073c50 con 0x7f18c8101750 2024-03-04T16:14:04.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cd4d9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8100da0 0x7f18c8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:04.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cd4d9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8100da0 0x7f18c8073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37570/0 (socket says 172.21.15.116:37570) 2024-03-04T16:14:04.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cd4d9700 1 -- 172.21.15.116:0/2779404259 learned_addr learned my addr 172.21.15.116:0/2779404259 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:14:04.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cccd8700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18c8101750 0x7f18c80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:04.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cd4d9700 1 -- 172.21.15.116:0/2779404259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18c8101750 msgr2=0x7f18c80735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:04.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cd4d9700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18c8101750 0x7f18c80735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:04.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cd4d9700 1 -- 172.21.15.116:0/2779404259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c40095f0 con 0x7f18c8100da0 2024-03-04T16:14:04.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.792+0000 7f18cccd8700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18c8101750 0x7f18c80735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:14:04.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.793+0000 7f18cd4d9700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8100da0 0x7f18c8073090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f18b800b4b0 tx=0x7f18b800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:14:04.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.793+0000 7f18be7fc700 1 -- 172.21.15.116:0/2779404259 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18b8011800 con 0x7f18c8100da0 2024-03-04T16:14:04.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.793+0000 7f18be7fc700 1 -- 172.21.15.116:0/2779404259 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18b8011e20 con 0x7f18c8100da0 2024-03-04T16:14:04.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.793+0000 7f18cf73d700 1 -- 172.21.15.116:0/2779404259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c81a2aa0 con 0x7f18c8100da0 2024-03-04T16:14:04.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.793+0000 7f18be7fc700 1 -- 172.21.15.116:0/2779404259 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18b800f2e0 con 0x7f18c8100da0 2024-03-04T16:14:04.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.793+0000 7f18cf73d700 1 -- 172.21.15.116:0/2779404259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c81a2eb0 con 0x7f18c8100da0 2024-03-04T16:14:04.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.794+0000 7f18cf73d700 1 -- 172.21.15.116:0/2779404259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c81090d0 con 0x7f18c8100da0 2024-03-04T16:14:04.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.794+0000 7f18be7fc700 1 -- 172.21.15.116:0/2779404259 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f18b8010380 con 0x7f18c8100da0 2024-03-04T16:14:04.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.795+0000 7f18be7fc700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f18b406d8a0 0x7f18b406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:04.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.795+0000 7f18be7fc700 1 -- 172.21.15.116:0/2779404259 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f18b808d050 con 0x7f18c8100da0 2024-03-04T16:14:04.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.795+0000 7f18cccd8700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f18b406d8a0 0x7f18b406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:04.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.795+0000 7f18cccd8700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f18b406d8a0 0x7f18b406fd60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f18c8074880 tx=0x7f18c401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:14:04.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:04.799+0000 7f18be7fc700 1 -- 172.21.15.116:0/2779404259 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f18b8056830 con 0x7f18c8100da0 2024-03-04T16:14:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.001+0000 7f18cf73d700 1 -- 172.21.15.116:0/2779404259 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f18c80611d0 con 0x7f18b406d8a0 2024-03-04T16:14:05.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.012+0000 7f18be7fc700 1 -- 172.21.15.116:0/2779404259 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f18c80611d0 con 0x7f18b406d8a0 2024-03-04T16:14:05.012 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:14:05.012 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (66s) 34s ago 18m 14.7M - 0.25.0 c8568f914cd2 5a4a3998d5d9 2024-03-04T16:14:05.012 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (2m) 34s ago 18m 7990k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c67439dd3802 2024-03-04T16:14:05.012 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (118s) 97s ago 17m 7541k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6d482a1a1fd4 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (7m) 34s ago 18m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (6m) 97s ago 17m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (39s) 34s ago 18m 57.3M - 9.4.12 a0ea5e3c46d9 363309c073af 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (3m) 34s ago 12m 14.2M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9d5061a093c6 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (3m) 34s ago 13m 86.6M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4bf8fc434069 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (2m) 97s ago 12m 13.7M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3abb2cd19d94 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (2m) 97s ago 12m 16.0M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4f752bc549b2 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (9m) 34s ago 20m 512M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (8m) 97s ago 17m 451M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (8m) 34s ago 20m 56.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (7m) 97s ago 17m 42.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (104s) 34s ago 18m 8228k - 1.5.0 0da6a335fe13 6a5a1f49ec33 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (100s) 97s ago 17m 3573k - 1.5.0 0da6a335fe13 d929fe9ef4b1 2024-03-04T16:14:05.013 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (6m) 34s ago 16m 98.0M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:14:05.014 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (6m) 34s ago 16m 115M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:14:05.014 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (5m) 34s ago 15m 94.5M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:14:05.014 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (4m) 97s ago 15m 121M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6b01e97c8f9b 2024-03-04T16:14:05.014 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (4m) 97s ago 14m 112M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 5ecc30e5a86c 2024-03-04T16:14:05.014 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (3m) 97s ago 14m 102M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 1083700b476d 2024-03-04T16:14:05.014 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (83s) 34s ago 18m 51.3M - 2.43.0 a07b618ecd1d 0d85379f4e6b 2024-03-04T16:14:05.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 -- 172.21.15.116:0/2779404259 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f18b406d8a0 msgr2=0x7f18b406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:05.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f18b406d8a0 0x7f18b406fd60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f18c8074880 tx=0x7f18c401a040 comp rx=0 tx=0).stop 2024-03-04T16:14:05.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 -- 172.21.15.116:0/2779404259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8100da0 msgr2=0x7f18c8073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:05.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8100da0 0x7f18c8073090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f18b800b4b0 tx=0x7f18b800b870 comp rx=0 tx=0).stop 2024-03-04T16:14:05.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 -- 172.21.15.116:0/2779404259 shutdown_connections 2024-03-04T16:14:05.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f18b406d8a0 0x7f18b406fd60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:05.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8100da0 0x7f18c8073090 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:05.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 --2- 172.21.15.116:0/2779404259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18c8101750 0x7f18c80735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:05.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 -- 172.21.15.116:0/2779404259 >> 172.21.15.116:0/2779404259 conn(0x7f18c80fc630 msgr2=0x7f18c80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:14:05.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 -- 172.21.15.116:0/2779404259 shutdown_connections 2024-03-04T16:14:05.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:05.014+0000 7f18cf73d700 1 -- 172.21.15.116:0/2779404259 wait complete. 2024-03-04T16:14:05.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch upgrade status' 2024-03-04T16:14:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:14:06.253 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:06 smithi116 ceph-mon[94164]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:06.253 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:06 smithi116 ceph-mon[94164]: pgmap v283: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:06.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:06 smithi152 ceph-mon[82845]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:06.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:06 smithi152 ceph-mon[82845]: pgmap v283: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.557+0000 7fbbde332700 1 -- 172.21.15.116:0/1293432801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd8101d00 msgr2=0x7fbbd81020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.557+0000 7fbbde332700 1 --2- 172.21.15.116:0/1293432801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd8101d00 0x7fbbd81020e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009910 tx=0x7fbbc8009c20 comp rx=0 tx=0).stop 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.557+0000 7fbbde332700 1 -- 172.21.15.116:0/1293432801 shutdown_connections 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.557+0000 7fbbde332700 1 --2- 172.21.15.116:0/1293432801 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbd8102620 0x7fbbd810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.557+0000 7fbbde332700 1 --2- 172.21.15.116:0/1293432801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd8101d00 0x7fbbd81020e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.557+0000 7fbbde332700 1 -- 172.21.15.116:0/1293432801 >> 172.21.15.116:0/1293432801 conn(0x7fbbd80fb570 msgr2=0x7fbbd80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.557+0000 7fbbde332700 1 -- 172.21.15.116:0/1293432801 shutdown_connections 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.557+0000 7fbbde332700 1 -- 172.21.15.116:0/1293432801 wait complete. 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbde332700 1 Processor -- start 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbde332700 1 -- start start 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbde332700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd8101d00 0x7fbbd819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbde332700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbd8102620 0x7fbbd819cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbde332700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd819d230 con 0x7fbbd8101d00 2024-03-04T16:14:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbde332700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd81966e0 con 0x7fbbd8102620 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbd7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd8101d00 0x7fbbd819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbd7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd8101d00 0x7fbbd819c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37598/0 (socket says 172.21.15.116:37598) 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbd7fff700 1 -- 172.21.15.116:0/3825046506 learned_addr learned my addr 172.21.15.116:0/3825046506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbd77fe700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbd8102620 0x7fbbd819cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbd7fff700 1 -- 172.21.15.116:0/3825046506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbd8102620 msgr2=0x7fbbd819cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbd7fff700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbd8102620 0x7fbbd819cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbd7fff700 1 -- 172.21.15.116:0/3825046506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc80095f0 con 0x7fbbd8101d00 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.558+0000 7fbbd77fe700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbd8102620 0x7fbbd819cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.559+0000 7fbbd7fff700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd8101d00 0x7fbbd819c660 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8006010 tx=0x7fbbc800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.559+0000 7fbbd57fa700 1 -- 172.21.15.116:0/3825046506 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbc801d070 con 0x7fbbd8101d00 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.559+0000 7fbbd57fa700 1 -- 172.21.15.116:0/3825046506 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbc8004fb0 con 0x7fbbd8101d00 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.559+0000 7fbbde332700 1 -- 172.21.15.116:0/3825046506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd8196960 con 0x7fbbd8101d00 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.559+0000 7fbbd57fa700 1 -- 172.21.15.116:0/3825046506 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbc8022ea0 con 0x7fbbd8101d00 2024-03-04T16:14:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.559+0000 7fbbde332700 1 -- 172.21.15.116:0/3825046506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd8196e50 con 0x7fbbd8101d00 2024-03-04T16:14:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.560+0000 7fbbde332700 1 -- 172.21.15.116:0/3825046506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd8108210 con 0x7fbbd8101d00 2024-03-04T16:14:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.560+0000 7fbbd57fa700 1 -- 172.21.15.116:0/3825046506 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fbbc8022470 con 0x7fbbd8101d00 2024-03-04T16:14:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.561+0000 7fbbd57fa700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbbc006d8a0 0x7fbbc006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.561+0000 7fbbd57fa700 1 -- 172.21.15.116:0/3825046506 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fbbc808ec70 con 0x7fbbd8101d00 2024-03-04T16:14:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.561+0000 7fbbd77fe700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbbc006d8a0 0x7fbbc006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.561+0000 7fbbd77fe700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbbc006d8a0 0x7fbbc006fd60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8197b50 tx=0x7fbbcc008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:14:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.565+0000 7fbbd57fa700 1 -- 172.21.15.116:0/3825046506 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbbc805c150 con 0x7fbbd8101d00 2024-03-04T16:14:07.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.780+0000 7fbbde332700 1 -- 172.21.15.116:0/3825046506 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbd81970c0 con 0x7fbbc006d8a0 2024-03-04T16:14:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.781+0000 7fbbd57fa700 1 -- 172.21.15.116:0/3825046506 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fbbd81970c0 con 0x7fbbc006d8a0 2024-03-04T16:14:07.782 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:14:07.782 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": null, 2024-03-04T16:14:07.782 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": false, 2024-03-04T16:14:07.782 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "", 2024-03-04T16:14:07.782 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2024-03-04T16:14:07.782 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": null, 2024-03-04T16:14:07.782 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2024-03-04T16:14:07.782 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2024-03-04T16:14:07.782 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:14:07.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.783+0000 7fbbde332700 1 -- 172.21.15.116:0/3825046506 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbbc006d8a0 msgr2=0x7fbbc006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:07.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.783+0000 7fbbde332700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbbc006d8a0 0x7fbbc006fd60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8197b50 tx=0x7fbbcc008040 comp rx=0 tx=0).stop 2024-03-04T16:14:07.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.783+0000 7fbbde332700 1 -- 172.21.15.116:0/3825046506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd8101d00 msgr2=0x7fbbd819c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:07.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.783+0000 7fbbde332700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd8101d00 0x7fbbd819c660 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8006010 tx=0x7fbbc800be20 comp rx=0 tx=0).stop 2024-03-04T16:14:07.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.784+0000 7fbbde332700 1 -- 172.21.15.116:0/3825046506 shutdown_connections 2024-03-04T16:14:07.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.784+0000 7fbbde332700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fbbc006d8a0 0x7fbbc006fd60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:07.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.784+0000 7fbbde332700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd8101d00 0x7fbbd819c660 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:07.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.784+0000 7fbbde332700 1 --2- 172.21.15.116:0/3825046506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbd8102620 0x7fbbd819cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:07.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.784+0000 7fbbde332700 1 -- 172.21.15.116:0/3825046506 >> 172.21.15.116:0/3825046506 conn(0x7fbbd80fb570 msgr2=0x7fbbd8105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:14:07.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.784+0000 7fbbde332700 1 -- 172.21.15.116:0/3825046506 shutdown_connections 2024-03-04T16:14:07.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:07.784+0000 7fbbde332700 1 -- 172.21.15.116:0/3825046506 wait complete. 2024-03-04T16:14:08.451 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:08 smithi116 ceph-mon[94164]: pgmap v284: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:08.451 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:08.472 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph health detail' 2024-03-04T16:14:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:08 smithi152 ceph-mon[82845]: pgmap v284: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:08.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:14:09.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:09 smithi116 ceph-mon[94164]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:09.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:09 smithi152 ceph-mon[82845]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.370+0000 7f69a93f5700 1 -- 172.21.15.116:0/3578411626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40ffa40 msgr2=0x7f69a40ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.370+0000 7f69a93f5700 1 --2- 172.21.15.116:0/3578411626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40ffa40 0x7f69a40ffec0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6998009910 tx=0x7f6998009c20 comp rx=0 tx=0).stop 2024-03-04T16:14:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.370+0000 7f69a93f5700 1 -- 172.21.15.116:0/3578411626 shutdown_connections 2024-03-04T16:14:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.370+0000 7f69a93f5700 1 --2- 172.21.15.116:0/3578411626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40ffa40 0x7f69a40ffec0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.370+0000 7f69a93f5700 1 --2- 172.21.15.116:0/3578411626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a4105a40 0x7f69a4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.370+0000 7f69a93f5700 1 -- 172.21.15.116:0/3578411626 >> 172.21.15.116:0/3578411626 conn(0x7f69a40fb550 msgr2=0x7f69a40fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:14:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.370+0000 7f69a93f5700 1 -- 172.21.15.116:0/3578411626 shutdown_connections 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.370+0000 7f69a93f5700 1 -- 172.21.15.116:0/3578411626 wait complete. 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.370+0000 7f69a93f5700 1 Processor -- start 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a93f5700 1 -- start start 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a93f5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40ffa40 0x7f69a419b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a93f5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a4105a40 0x7f69a419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a93f5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a41955d0 con 0x7f69a40ffa40 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a93f5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a4195740 con 0x7f69a4105a40 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40ffa40 0x7f69a419b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40ffa40 0x7f69a419b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37628/0 (socket says 172.21.15.116:37628) 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a2ffd700 1 -- 172.21.15.116:0/689225181 learned_addr learned my addr 172.21.15.116:0/689225181 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a2ffd700 1 -- 172.21.15.116:0/689225181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a4105a40 msgr2=0x7f69a419bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a2ffd700 1 --2- 172.21.15.116:0/689225181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a4105a40 0x7f69a419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a2ffd700 1 -- 172.21.15.116:0/689225181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69980095f0 con 0x7f69a40ffa40 2024-03-04T16:14:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f69a2ffd700 1 --2- 172.21.15.116:0/689225181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40ffa40 0x7f69a419b5e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f699400b4d0 tx=0x7f699400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:14:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.371+0000 7f698bfff700 1 -- 172.21.15.116:0/689225181 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6994011800 con 0x7f69a40ffa40 2024-03-04T16:14:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.372+0000 7f698bfff700 1 -- 172.21.15.116:0/689225181 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6994011e20 con 0x7f69a40ffa40 2024-03-04T16:14:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.372+0000 7f69a93f5700 1 -- 172.21.15.116:0/689225181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a4195a20 con 0x7f69a40ffa40 2024-03-04T16:14:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.372+0000 7f698bfff700 1 -- 172.21.15.116:0/689225181 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f699400f2e0 con 0x7f69a40ffa40 2024-03-04T16:14:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.372+0000 7f69a93f5700 1 -- 172.21.15.116:0/689225181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a4195f70 con 0x7f69a40ffa40 2024-03-04T16:14:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.372+0000 7f69a93f5700 1 -- 172.21.15.116:0/689225181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a4108060 con 0x7f69a40ffa40 2024-03-04T16:14:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.373+0000 7f698bfff700 1 -- 172.21.15.116:0/689225181 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f6994010380 con 0x7f69a40ffa40 2024-03-04T16:14:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.373+0000 7f698bfff700 1 --2- 172.21.15.116:0/689225181 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f698c06d8a0 0x7f698c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.373+0000 7f698bfff700 1 -- 172.21.15.116:0/689225181 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f699408d080 con 0x7f69a40ffa40 2024-03-04T16:14:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.374+0000 7f69a27fc700 1 --2- 172.21.15.116:0/689225181 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f698c06d8a0 0x7f698c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.374+0000 7f69a27fc700 1 --2- 172.21.15.116:0/689225181 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f698c06d8a0 0x7f698c06fd60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6998009fd0 tx=0x7f6998005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:14:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.378+0000 7f698bfff700 1 -- 172.21.15.116:0/689225181 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6994056860 con 0x7f69a40ffa40 2024-03-04T16:14:10.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:10 smithi116 ceph-mon[94164]: pgmap v285: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:10.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:10 smithi152 ceph-mon[82845]: pgmap v285: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:10.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.651+0000 7f69a93f5700 1 -- 172.21.15.116:0/689225181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f69a41962a0 con 0x7f69a40ffa40 2024-03-04T16:14:10.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.651+0000 7f698bfff700 1 -- 172.21.15.116:0/689225181 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f6994059f20 con 0x7f69a40ffa40 2024-03-04T16:14:10.652 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2024-03-04T16:14:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.653+0000 7f69a93f5700 1 -- 172.21.15.116:0/689225181 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f698c06d8a0 msgr2=0x7f698c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.653+0000 7f69a93f5700 1 --2- 172.21.15.116:0/689225181 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f698c06d8a0 0x7f698c06fd60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6998009fd0 tx=0x7f6998005fb0 comp rx=0 tx=0).stop 2024-03-04T16:14:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.653+0000 7f69a93f5700 1 -- 172.21.15.116:0/689225181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40ffa40 msgr2=0x7f69a419b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.653+0000 7f69a93f5700 1 --2- 172.21.15.116:0/689225181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40ffa40 0x7f69a419b5e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f699400b4d0 tx=0x7f699400b890 comp rx=0 tx=0).stop 2024-03-04T16:14:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.653+0000 7f69a93f5700 1 -- 172.21.15.116:0/689225181 shutdown_connections 2024-03-04T16:14:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.653+0000 7f69a93f5700 1 --2- 172.21.15.116:0/689225181 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f698c06d8a0 0x7f698c06fd60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.653+0000 7f69a93f5700 1 --2- 172.21.15.116:0/689225181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40ffa40 0x7f69a419b5e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.653+0000 7f69a93f5700 1 --2- 172.21.15.116:0/689225181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69a4105a40 0x7f69a419bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.653+0000 7f69a93f5700 1 -- 172.21.15.116:0/689225181 >> 172.21.15.116:0/689225181 conn(0x7f69a40fb550 msgr2=0x7f69a40fc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:14:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.654+0000 7f69a93f5700 1 -- 172.21.15.116:0/689225181 shutdown_connections 2024-03-04T16:14:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:10.654+0000 7f69a93f5700 1 -- 172.21.15.116:0/689225181 wait complete. 2024-03-04T16:14:11.287 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph versions' 2024-03-04T16:14:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:11 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/689225181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:14:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:14:11.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:11 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/689225181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:14:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:12 smithi116 ceph-mon[94164]: pgmap v286: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:12.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:12 smithi152 ceph-mon[82845]: pgmap v286: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:13.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.067+0000 7faac3536700 1 -- 172.21.15.116:0/698046115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faabc102630 msgr2=0x7faabc10ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:13.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.067+0000 7faac3536700 1 --2- 172.21.15.116:0/698046115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faabc102630 0x7faabc10ab20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faaac009910 tx=0x7faaac009c20 comp rx=0 tx=0).stop 2024-03-04T16:14:13.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.067+0000 7faac3536700 1 -- 172.21.15.116:0/698046115 shutdown_connections 2024-03-04T16:14:13.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.067+0000 7faac3536700 1 --2- 172.21.15.116:0/698046115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faabc102630 0x7faabc10ab20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:13.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.067+0000 7faac3536700 1 --2- 172.21.15.116:0/698046115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc101c80 0x7faabc102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.067+0000 7faac3536700 1 -- 172.21.15.116:0/698046115 >> 172.21.15.116:0/698046115 conn(0x7faabc0fb540 msgr2=0x7faabc0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.067+0000 7faac3536700 1 -- 172.21.15.116:0/698046115 shutdown_connections 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.067+0000 7faac3536700 1 -- 172.21.15.116:0/698046115 wait complete. 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac3536700 1 Processor -- start 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac3536700 1 -- start start 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac3536700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc101c80 0x7faabc19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac3536700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faabc102630 0x7faabc19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac3536700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc19f2e0 con 0x7faabc102630 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac3536700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc198790 con 0x7faabc101c80 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac0ad1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faabc102630 0x7faabc19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac0ad1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faabc102630 0x7faabc19ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37644/0 (socket says 172.21.15.116:37644) 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac0ad1700 1 -- 172.21.15.116:0/210312281 learned_addr learned my addr 172.21.15.116:0/210312281 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:14:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac12d2700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc101c80 0x7faabc19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:13.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac0ad1700 1 -- 172.21.15.116:0/210312281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc101c80 msgr2=0x7faabc19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:13.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac0ad1700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc101c80 0x7faabc19e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:13.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac0ad1700 1 -- 172.21.15.116:0/210312281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac0095f0 con 0x7faabc102630 2024-03-04T16:14:13.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.068+0000 7faac0ad1700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faabc102630 0x7faabc19ec50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faaac0092e0 tx=0x7faaac00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:14:13.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.069+0000 7faab27fc700 1 -- 172.21.15.116:0/210312281 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaac01d070 con 0x7faabc102630 2024-03-04T16:14:13.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.069+0000 7faab27fc700 1 -- 172.21.15.116:0/210312281 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faaac005dc0 con 0x7faabc102630 2024-03-04T16:14:13.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.069+0000 7faac3536700 1 -- 172.21.15.116:0/210312281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faabc198a10 con 0x7faabc102630 2024-03-04T16:14:13.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.069+0000 7faab27fc700 1 -- 172.21.15.116:0/210312281 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaac017400 con 0x7faabc102630 2024-03-04T16:14:13.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.069+0000 7faac3536700 1 -- 172.21.15.116:0/210312281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faabc198f00 con 0x7faabc102630 2024-03-04T16:14:13.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.070+0000 7faac3536700 1 -- 172.21.15.116:0/210312281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faabc10a680 con 0x7faabc102630 2024-03-04T16:14:13.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.070+0000 7faab27fc700 1 -- 172.21.15.116:0/210312281 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7faaac00f6f0 con 0x7faabc102630 2024-03-04T16:14:13.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.070+0000 7faab27fc700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faaa806d8a0 0x7faaa806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:14:13.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.070+0000 7faab27fc700 1 -- 172.21.15.116:0/210312281 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7faaac08e900 con 0x7faabc102630 2024-03-04T16:14:13.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.071+0000 7faac12d2700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faaa806d8a0 0x7faaa806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:14:13.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.071+0000 7faac12d2700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faaa806d8a0 0x7faaa806fd60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faab4009550 tx=0x7faab40091b0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:14:13.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.075+0000 7faab27fc700 1 -- 172.21.15.116:0/210312281 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faaac05bb90 con 0x7faabc102630 2024-03-04T16:14:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.363+0000 7faac3536700 1 -- 172.21.15.116:0/210312281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faabc04ea90 con 0x7faabc102630 2024-03-04T16:14:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.363+0000 7faab27fc700 1 -- 172.21.15.116:0/210312281 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7faaac05b720 con 0x7faabc102630 2024-03-04T16:14:13.364 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:14:13.364 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:14:13.365 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:14:13.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:14:13.365 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:14:13.365 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:14:13.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:14:13.365 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:14:13.365 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:14:13.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:14:13.366 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:14:13.366 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:14:13.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:14:13.366 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:14:13.366 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 14 2024-03-04T16:14:13.366 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:14:13.366 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:14:13.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 -- 172.21.15.116:0/210312281 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faaa806d8a0 msgr2=0x7faaa806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:13.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faaa806d8a0 0x7faaa806fd60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faab4009550 tx=0x7faab40091b0 comp rx=0 tx=0).stop 2024-03-04T16:14:13.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 -- 172.21.15.116:0/210312281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faabc102630 msgr2=0x7faabc19ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:14:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faabc102630 0x7faabc19ec50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faaac0092e0 tx=0x7faaac00bd70 comp rx=0 tx=0).stop 2024-03-04T16:14:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 -- 172.21.15.116:0/210312281 shutdown_connections 2024-03-04T16:14:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7faaa806d8a0 0x7faaa806fd60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc101c80 0x7faabc19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 --2- 172.21.15.116:0/210312281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faabc102630 0x7faabc19ec50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:14:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 -- 172.21.15.116:0/210312281 >> 172.21.15.116:0/210312281 conn(0x7faabc0fb540 msgr2=0x7faabc100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:14:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 -- 172.21.15.116:0/210312281 shutdown_connections 2024-03-04T16:14:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:14:13.366+0000 7faac3536700 1 -- 172.21.15.116:0/210312281 wait complete. 2024-03-04T16:14:14.069 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-04T16:14:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:14:14.399 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:14 smithi116 ceph-mon[94164]: pgmap v287: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:14.399 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:14 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/210312281' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:14.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:14 smithi152 ceph-mon[82845]: pgmap v287: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:14.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:14 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/210312281' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:15.916 INFO:teuthology.orchestra.run.smithi116.stdout:wait for servicemap items w/ changing names to refresh 2024-03-04T16:14:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:16 smithi116 ceph-mon[94164]: pgmap v288: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:16.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'sleep 60' 2024-03-04T16:14:16.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:16 smithi152 ceph-mon[82845]: pgmap v288: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:14:18.313 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:18 smithi116 ceph-mon[94164]: pgmap v289: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:18.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:18 smithi152 ceph-mon[82845]: pgmap v289: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:20.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:20 smithi152 ceph-mon[82845]: pgmap v290: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:20 smithi116 ceph-mon[94164]: pgmap v290: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:22.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:22 smithi152 ceph-mon[82845]: pgmap v291: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:22 smithi116 ceph-mon[94164]: pgmap v291: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:23.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:24.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:24 smithi152 ceph-mon[82845]: pgmap v292: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:24.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:24 smithi116 ceph-mon[94164]: pgmap v292: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:26.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:26 smithi152 ceph-mon[82845]: pgmap v293: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:26.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:26 smithi116 ceph-mon[94164]: pgmap v293: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:28.507 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:28 smithi152 ceph-mon[82845]: pgmap v294: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:28 smithi116 ceph-mon[94164]: pgmap v294: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:30.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:30 smithi152 ceph-mon[82845]: pgmap v295: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:30.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:30 smithi116 ceph-mon[94164]: pgmap v295: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:32.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:32 smithi152 ceph-mon[82845]: pgmap v296: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:32.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:32 smithi116 ceph-mon[94164]: pgmap v296: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:34.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:34 smithi152 ceph-mon[82845]: pgmap v297: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:34.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:34 smithi116 ceph-mon[94164]: pgmap v297: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:36.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:36 smithi152 ceph-mon[82845]: pgmap v298: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:36.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:36 smithi116 ceph-mon[94164]: pgmap v298: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:37.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:14:37.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:37.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:37.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:37.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:14:38.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:38 smithi152 ceph-mon[82845]: pgmap v299: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:38.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:38 smithi116 ceph-mon[94164]: pgmap v299: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:40.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:40 smithi152 ceph-mon[82845]: pgmap v300: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:40 smithi116 ceph-mon[94164]: pgmap v300: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:42.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:42 smithi152 ceph-mon[82845]: pgmap v301: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:42.645 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:42 smithi116 ceph-mon[94164]: pgmap v301: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:44.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:44 smithi152 ceph-mon[82845]: pgmap v302: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:44 smithi116 ceph-mon[94164]: pgmap v302: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:46.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:46 smithi152 ceph-mon[82845]: pgmap v303: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:46 smithi116 ceph-mon[94164]: pgmap v303: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:48.550 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:48 smithi116 ceph-mon[94164]: pgmap v304: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:48.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:48 smithi152 ceph-mon[82845]: pgmap v304: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:50.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:50 smithi152 ceph-mon[82845]: pgmap v305: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:50 smithi116 ceph-mon[94164]: pgmap v305: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:52.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:52 smithi152 ceph-mon[82845]: pgmap v306: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:52 smithi116 ceph-mon[94164]: pgmap v306: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:53.603 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:53.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:54.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:54 smithi152 ceph-mon[82845]: pgmap v307: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:54 smithi116 ceph-mon[94164]: pgmap v307: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:56.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:56 smithi116 ceph-mon[94164]: pgmap v308: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:56.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:56 smithi152 ceph-mon[82845]: pgmap v308: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:14:58 smithi116 ceph-mon[94164]: pgmap v309: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:14:58.804 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:14:58 smithi152 ceph-mon[82845]: pgmap v309: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:00 smithi116 ceph-mon[94164]: pgmap v310: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:00.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:00 smithi152 ceph-mon[82845]: pgmap v310: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:02.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:02 smithi116 ceph-mon[94164]: pgmap v311: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:02.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:02 smithi152 ceph-mon[82845]: pgmap v311: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:04.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:04 smithi116 ceph-mon[94164]: pgmap v312: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:04.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:04 smithi152 ceph-mon[82845]: pgmap v312: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:06.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:06 smithi116 ceph-mon[94164]: pgmap v313: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:06.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:06 smithi152 ceph-mon[82845]: pgmap v313: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:08.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:08 smithi116 ceph-mon[94164]: pgmap v314: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:08.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:08 smithi152 ceph-mon[82845]: pgmap v314: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:10.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:10 smithi116 ceph-mon[94164]: pgmap v315: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:10.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:10 smithi152 ceph-mon[82845]: pgmap v315: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:12.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:12 smithi116 ceph-mon[94164]: pgmap v316: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:12.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:12 smithi152 ceph-mon[82845]: pgmap v316: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:14.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:14 smithi116 ceph-mon[94164]: pgmap v317: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:14.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:14 smithi152 ceph-mon[82845]: pgmap v317: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:16 smithi116 ceph-mon[94164]: pgmap v318: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:16.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:16 smithi152 ceph-mon[82845]: pgmap v318: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:18.674 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:18 smithi116 ceph-mon[94164]: pgmap v319: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:18.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:18 smithi152 ceph-mon[82845]: pgmap v319: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:19.006 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch ps' 2024-03-04T16:15:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:15:20.720 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:20 smithi116 ceph-mon[94164]: pgmap v320: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:20.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:20 smithi152 ceph-mon[82845]: pgmap v320: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:20.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.942+0000 7f1af5861700 1 -- 172.21.15.116:0/3335846424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af0105a40 msgr2=0x7f1af0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:20.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.942+0000 7f1af5861700 1 --2- 172.21.15.116:0/3335846424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af0105a40 0x7f1af0105e20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800b0a0 tx=0x7f1ae800b3b0 comp rx=0 tx=0).stop 2024-03-04T16:15:20.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.942+0000 7f1af5861700 1 -- 172.21.15.116:0/3335846424 shutdown_connections 2024-03-04T16:15:20.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.942+0000 7f1af5861700 1 --2- 172.21.15.116:0/3335846424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af00ffbd0 0x7f1af0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:20.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.942+0000 7f1af5861700 1 --2- 172.21.15.116:0/3335846424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af0105a40 0x7f1af0105e20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:20.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.942+0000 7f1af5861700 1 -- 172.21.15.116:0/3335846424 >> 172.21.15.116:0/3335846424 conn(0x7f1af00755a0 msgr2=0x7f1af00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:20.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.942+0000 7f1af5861700 1 -- 172.21.15.116:0/3335846424 shutdown_connections 2024-03-04T16:15:20.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.942+0000 7f1af5861700 1 -- 172.21.15.116:0/3335846424 wait complete. 2024-03-04T16:15:20.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.943+0000 7f1af5861700 1 Processor -- start 2024-03-04T16:15:20.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.943+0000 7f1af5861700 1 -- start start 2024-03-04T16:15:20.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.943+0000 7f1af5861700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af00ffbd0 0x7f1af01975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:20.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.943+0000 7f1af5861700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af0105a40 0x7f1af0197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:20.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.943+0000 7f1af5861700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af0198120 con 0x7f1af0105a40 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.943+0000 7f1af5861700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af0198290 con 0x7f1af00ffbd0 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.943+0000 7f1aee7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af0105a40 0x7f1af0197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.943+0000 7f1aee7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af0105a40 0x7f1af0197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59532/0 (socket says 172.21.15.116:59532) 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.943+0000 7f1aee7fc700 1 -- 172.21.15.116:0/1042127144 learned_addr learned my addr 172.21.15.116:0/1042127144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.944+0000 7f1aee7fc700 1 -- 172.21.15.116:0/1042127144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af00ffbd0 msgr2=0x7f1af01975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.944+0000 7f1aeeffd700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af00ffbd0 0x7f1af01975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.944+0000 7f1aee7fc700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af00ffbd0 0x7f1af01975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.944+0000 7f1aee7fc700 1 -- 172.21.15.116:0/1042127144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae8009d00 con 0x7f1af0105a40 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.944+0000 7f1aee7fc700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af0105a40 0x7f1af0197b00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800d8d0 tx=0x7f1ad800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.944+0000 7f1af485f700 1 -- 172.21.15.116:0/1042127144 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ad8009940 con 0x7f1af0105a40 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.944+0000 7f1af485f700 1 -- 172.21.15.116:0/1042127144 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ad8010460 con 0x7f1af0105a40 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.944+0000 7f1af485f700 1 -- 172.21.15.116:0/1042127144 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ad800f5f0 con 0x7f1af0105a40 2024-03-04T16:15:20.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.944+0000 7f1af5861700 1 -- 172.21.15.116:0/1042127144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af01a0630 con 0x7f1af0105a40 2024-03-04T16:15:20.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.944+0000 7f1af5861700 1 -- 172.21.15.116:0/1042127144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af01a0ad0 con 0x7f1af0105a40 2024-03-04T16:15:20.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.945+0000 7f1af5861700 1 -- 172.21.15.116:0/1042127144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af0108040 con 0x7f1af0105a40 2024-03-04T16:15:20.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.946+0000 7f1af485f700 1 -- 172.21.15.116:0/1042127144 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f1ad80105d0 con 0x7f1af0105a40 2024-03-04T16:15:20.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.946+0000 7f1af485f700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1adc06d8a0 0x7f1adc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:20.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.946+0000 7f1af485f700 1 -- 172.21.15.116:0/1042127144 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f1ad808d580 con 0x7f1af0105a40 2024-03-04T16:15:20.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.946+0000 7f1aeeffd700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1adc06d8a0 0x7f1adc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:20.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.946+0000 7f1aeeffd700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1adc06d8a0 0x7f1adc06fd60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8015040 tx=0x7f1ae8000f40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:20.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:20.951+0000 7f1af485f700 1 -- 172.21.15.116:0/1042127144 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1ad8056ce0 con 0x7f1af0105a40 2024-03-04T16:15:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.154+0000 7f1af5861700 1 -- 172.21.15.116:0/1042127144 --> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1af00611d0 con 0x7f1adc06d8a0 2024-03-04T16:15:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.165+0000 7f1af485f700 1 -- 172.21.15.116:0/1042127144 <== mgr.34102 v2:172.21.15.116:6800/2954839814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f1af00611d0 con 0x7f1adc06d8a0 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.smithi116 smithi116 *:9093,9094 running (2m) 110s ago 19m 14.7M - 0.25.0 c8568f914cd2 5a4a3998d5d9 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi116 smithi116 running (3m) 110s ago 20m 7990k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c67439dd3802 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-exporter.smithi152 smithi152 running (3m) 2m ago 19m 7541k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6d482a1a1fd4 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi116 smithi116 running (8m) 110s ago 20m 6660k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 17f658584c45 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:crash.smithi152 smithi152 running (8m) 2m ago 18m 6665k - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b979344d8f40 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.smithi116 smithi116 *:3000 running (115s) 110s ago 19m 57.3M - 9.4.12 a0ea5e3c46d9 363309c073af 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.gkenwy smithi116 running (4m) 110s ago 14m 14.2M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9d5061a093c6 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi116.pjrmzv smithi116 running (4m) 110s ago 14m 86.6M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4bf8fc434069 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.kqoybf smithi152 running (3m) 2m ago 14m 13.7M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3abb2cd19d94 2024-03-04T16:15:21.166 INFO:teuthology.orchestra.run.smithi116.stdout:mds.cephfs.smithi152.tlpshy smithi152 running (3m) 2m ago 14m 16.0M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4f752bc549b2 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi116.jwdhop smithi116 *:8443,9283,8765 running (10m) 110s ago 21m 512M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b6a1402dd35e 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.smithi152.aawieo smithi152 *:8443,9283,8765 running (10m) 2m ago 18m 451M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9dba8ef5538f 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi116 smithi116 running (9m) 110s ago 21m 56.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ecbaf85d8a2 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:mon.smithi152 smithi152 running (9m) 2m ago 18m 42.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 9129fd088663 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi116 smithi116 *:9100 running (3m) 110s ago 20m 8228k - 1.5.0 0da6a335fe13 6a5a1f49ec33 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.smithi152 smithi152 *:9100 running (2m) 2m ago 18m 3573k - 1.5.0 0da6a335fe13 d929fe9ef4b1 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (7m) 110s ago 18m 98.0M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 98c52b3190c9 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (7m) 110s ago 17m 115M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3e403d77dd72 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (6m) 110s ago 17m 94.5M 1216M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 43927e90a69f 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi152 running (6m) 2m ago 16m 121M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6b01e97c8f9b 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi152 running (5m) 2m ago 16m 112M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 5ecc30e5a86c 2024-03-04T16:15:21.167 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (4m) 2m ago 15m 102M 2240M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 1083700b476d 2024-03-04T16:15:21.168 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.smithi116 smithi116 *:9095 running (2m) 110s ago 19m 51.3M - 2.43.0 a07b618ecd1d 0d85379f4e6b 2024-03-04T16:15:21.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 -- 172.21.15.116:0/1042127144 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1adc06d8a0 msgr2=0x7f1adc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:21.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1adc06d8a0 0x7f1adc06fd60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8015040 tx=0x7f1ae8000f40 comp rx=0 tx=0).stop 2024-03-04T16:15:21.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 -- 172.21.15.116:0/1042127144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af0105a40 msgr2=0x7f1af0197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:21.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af0105a40 0x7f1af0197b00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800d8d0 tx=0x7f1ad800dc90 comp rx=0 tx=0).stop 2024-03-04T16:15:21.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 -- 172.21.15.116:0/1042127144 shutdown_connections 2024-03-04T16:15:21.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f1adc06d8a0 0x7f1adc06fd60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:21.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af00ffbd0 0x7f1af01975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:21.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 --2- 172.21.15.116:0/1042127144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af0105a40 0x7f1af0197b00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:21.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 -- 172.21.15.116:0/1042127144 >> 172.21.15.116:0/1042127144 conn(0x7f1af00755a0 msgr2=0x7f1af00fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:21.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 -- 172.21.15.116:0/1042127144 shutdown_connections 2024-03-04T16:15:21.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:21.167+0000 7f1af5861700 1 -- 172.21.15.116:0/1042127144 wait complete. 2024-03-04T16:15:21.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:21 smithi116 ceph-mon[94164]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:21.546 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:21 smithi116 ceph-mon[94164]: pgmap v321: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:21.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:21 smithi152 ceph-mon[82845]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:21.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:21 smithi152 ceph-mon[82845]: pgmap v321: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:21.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph versions' 2024-03-04T16:15:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:15:23.417 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:23.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.631+0000 7f5a974ae700 1 -- 172.21.15.116:0/4220847221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a90073a50 msgr2=0x7f5a90111630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.631+0000 7f5a974ae700 1 --2- 172.21.15.116:0/4220847221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a90073a50 0x7f5a90111630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009910 tx=0x7f5a80009c20 comp rx=0 tx=0).stop 2024-03-04T16:15:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.631+0000 7f5a974ae700 1 -- 172.21.15.116:0/4220847221 shutdown_connections 2024-03-04T16:15:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.631+0000 7f5a974ae700 1 --2- 172.21.15.116:0/4220847221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a90073a50 0x7f5a90111630 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.631+0000 7f5a974ae700 1 --2- 172.21.15.116:0/4220847221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a90073130 0x7f5a90073510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.631+0000 7f5a974ae700 1 -- 172.21.15.116:0/4220847221 >> 172.21.15.116:0/4220847221 conn(0x7f5a900fc650 msgr2=0x7f5a900fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.631+0000 7f5a974ae700 1 -- 172.21.15.116:0/4220847221 shutdown_connections 2024-03-04T16:15:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.631+0000 7f5a974ae700 1 -- 172.21.15.116:0/4220847221 wait complete. 2024-03-04T16:15:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.631+0000 7f5a974ae700 1 Processor -- start 2024-03-04T16:15:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.631+0000 7f5a974ae700 1 -- start start 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a974ae700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a90073130 0x7f5a901a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a974ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a90073a50 0x7f5a901a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a974ae700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a901a3680 con 0x7f5a90073a50 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a974ae700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9019cb30 con 0x7f5a90073130 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a94a49700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a90073a50 0x7f5a901a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a94a49700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a90073a50 0x7f5a901a2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59552/0 (socket says 172.21.15.116:59552) 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a94a49700 1 -- 172.21.15.116:0/2202536077 learned_addr learned my addr 172.21.15.116:0/2202536077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a9524a700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a90073130 0x7f5a901a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a94a49700 1 -- 172.21.15.116:0/2202536077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a90073130 msgr2=0x7f5a901a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a94a49700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a90073130 0x7f5a901a2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a94a49700 1 -- 172.21.15.116:0/2202536077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a800095f0 con 0x7f5a90073a50 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a94a49700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a90073a50 0x7f5a901a2ff0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5a80006010 tx=0x7f5a8000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a867fc700 1 -- 172.21.15.116:0/2202536077 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a8001d070 con 0x7f5a90073a50 2024-03-04T16:15:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a867fc700 1 -- 172.21.15.116:0/2202536077 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a80005dc0 con 0x7f5a90073a50 2024-03-04T16:15:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a974ae700 1 -- 172.21.15.116:0/2202536077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9019ce10 con 0x7f5a90073a50 2024-03-04T16:15:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a867fc700 1 -- 172.21.15.116:0/2202536077 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a80017400 con 0x7f5a90073a50 2024-03-04T16:15:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.632+0000 7f5a974ae700 1 -- 172.21.15.116:0/2202536077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9019d360 con 0x7f5a90073a50 2024-03-04T16:15:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.633+0000 7f5a974ae700 1 -- 172.21.15.116:0/2202536077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9010edb0 con 0x7f5a90073a50 2024-03-04T16:15:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.634+0000 7f5a867fc700 1 -- 172.21.15.116:0/2202536077 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f5a80003a60 con 0x7f5a90073a50 2024-03-04T16:15:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.634+0000 7f5a867fc700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5a7c06d8a0 0x7f5a7c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.634+0000 7f5a9524a700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5a7c06d8a0 0x7f5a7c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.634+0000 7f5a867fc700 1 -- 172.21.15.116:0/2202536077 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f5a8008e9a0 con 0x7f5a90073a50 2024-03-04T16:15:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.635+0000 7f5a9524a700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5a7c06d8a0 0x7f5a7c06fd60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c005950 tx=0x7f5a8c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.639+0000 7f5a867fc700 1 -- 172.21.15.116:0/2202536077 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5a8005bc30 con 0x7f5a90073a50 2024-03-04T16:15:23.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.910+0000 7f5a974ae700 1 -- 172.21.15.116:0/2202536077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5a9004ea90 con 0x7f5a90073a50 2024-03-04T16:15:23.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.910+0000 7f5a867fc700 1 -- 172.21.15.116:0/2202536077 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f5a8005b7c0 con 0x7f5a90073a50 2024-03-04T16:15:23.911 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T16:15:23.911 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2024-03-04T16:15:23.911 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:15:23.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:15:23.911 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2024-03-04T16:15:23.911 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:15:23.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:15:23.912 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2024-03-04T16:15:23.912 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:15:23.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:15:23.912 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-04T16:15:23.912 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:15:23.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T16:15:23.912 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2024-03-04T16:15:23.912 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 14 2024-03-04T16:15:23.912 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T16:15:23.912 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T16:15:23.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 -- 172.21.15.116:0/2202536077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5a7c06d8a0 msgr2=0x7f5a7c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:23.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5a7c06d8a0 0x7f5a7c06fd60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c005950 tx=0x7f5a8c00b410 comp rx=0 tx=0).stop 2024-03-04T16:15:23.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 -- 172.21.15.116:0/2202536077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a90073a50 msgr2=0x7f5a901a2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:23.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a90073a50 0x7f5a901a2ff0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5a80006010 tx=0x7f5a8000bd70 comp rx=0 tx=0).stop 2024-03-04T16:15:23.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 -- 172.21.15.116:0/2202536077 shutdown_connections 2024-03-04T16:15:23.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f5a7c06d8a0 0x7f5a7c06fd60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:23.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a90073130 0x7f5a901a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:23.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 --2- 172.21.15.116:0/2202536077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a90073a50 0x7f5a901a2ff0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:23.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 -- 172.21.15.116:0/2202536077 >> 172.21.15.116:0/2202536077 conn(0x7f5a900fc650 msgr2=0x7f5a90103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:23.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 -- 172.21.15.116:0/2202536077 shutdown_connections 2024-03-04T16:15:23.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:23.913+0000 7f5a974ae700 1 -- 172.21.15.116:0/2202536077 wait complete. 2024-03-04T16:15:24.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:24 smithi116 ceph-mon[94164]: pgmap v322: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:24.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:24 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2202536077' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:24 smithi152 ceph-mon[82845]: pgmap v322: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:24.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:24 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2202536077' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:24.558 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-04T16:15:24.770 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:15:26.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.385+0000 7f5843a08700 1 -- 172.21.15.116:0/3525508249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583c100da0 msgr2=0x7f583c101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:26.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.385+0000 7f5843a08700 1 --2- 172.21.15.116:0/3525508249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583c100da0 0x7f583c101180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5838009910 tx=0x7f5838009c20 comp rx=0 tx=0).stop 2024-03-04T16:15:26.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.385+0000 7f5843a08700 1 -- 172.21.15.116:0/3525508249 shutdown_connections 2024-03-04T16:15:26.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.385+0000 7f5843a08700 1 --2- 172.21.15.116:0/3525508249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f583c101750 0x7f583c1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:26.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.385+0000 7f5843a08700 1 --2- 172.21.15.116:0/3525508249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583c100da0 0x7f583c101180 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:26.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.385+0000 7f5843a08700 1 -- 172.21.15.116:0/3525508249 >> 172.21.15.116:0/3525508249 conn(0x7f583c0fc650 msgr2=0x7f583c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:26.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.386+0000 7f5843a08700 1 -- 172.21.15.116:0/3525508249 shutdown_connections 2024-03-04T16:15:26.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.386+0000 7f5843a08700 1 -- 172.21.15.116:0/3525508249 wait complete. 2024-03-04T16:15:26.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.386+0000 7f5843a08700 1 Processor -- start 2024-03-04T16:15:26.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.386+0000 7f5843a08700 1 -- start start 2024-03-04T16:15:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.386+0000 7f5843a08700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f583c100da0 0x7f583c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.386+0000 7f5843a08700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583c101750 0x7f583c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.386+0000 7f5843a08700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c073b10 con 0x7f583c101750 2024-03-04T16:15:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.386+0000 7f5843a08700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c073c50 con 0x7f583c100da0 2024-03-04T16:15:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.386+0000 7f5840fa3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583c101750 0x7f583c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f5840fa3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583c101750 0x7f583c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47956/0 (socket says 172.21.15.116:47956) 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f5840fa3700 1 -- 172.21.15.116:0/734546464 learned_addr learned my addr 172.21.15.116:0/734546464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f58417a4700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f583c100da0 0x7f583c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f5840fa3700 1 -- 172.21.15.116:0/734546464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f583c100da0 msgr2=0x7f583c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f5840fa3700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f583c100da0 0x7f583c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f5840fa3700 1 -- 172.21.15.116:0/734546464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58380095f0 con 0x7f583c101750 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f5840fa3700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583c101750 0x7f583c0735d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f582c00b480 tx=0x7f582c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f58327fc700 1 -- 172.21.15.116:0/734546464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f582c011800 con 0x7f583c101750 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f58327fc700 1 -- 172.21.15.116:0/734546464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f582c011e20 con 0x7f583c101750 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f5843a08700 1 -- 172.21.15.116:0/734546464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c1a2a20 con 0x7f583c101750 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f58327fc700 1 -- 172.21.15.116:0/734546464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f582c010ea0 con 0x7f583c101750 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.387+0000 7f5843a08700 1 -- 172.21.15.116:0/734546464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c1a2e30 con 0x7f583c101750 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.388+0000 7f5843a08700 1 -- 172.21.15.116:0/734546464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c109140 con 0x7f583c101750 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.389+0000 7f58327fc700 1 -- 172.21.15.116:0/734546464 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f582c010380 con 0x7f583c101750 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.389+0000 7f58327fc700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f582806d8a0 0x7f582806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.389+0000 7f58327fc700 1 -- 172.21.15.116:0/734546464 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f582c08cfc0 con 0x7f583c101750 2024-03-04T16:15:26.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.389+0000 7f58417a4700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f582806d8a0 0x7f582806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:26.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.389+0000 7f58417a4700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f582806d8a0 0x7f582806fd60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5838006010 tx=0x7f583801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:26.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.394+0000 7f58327fc700 1 -- 172.21.15.116:0/734546464 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f582c0567a0 con 0x7f583c101750 2024-03-04T16:15:26.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:26 smithi116 ceph-mon[94164]: pgmap v323: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:26.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.669+0000 7f5843a08700 1 -- 172.21.15.116:0/734546464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f583c073e10 con 0x7f583c101750 2024-03-04T16:15:26.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.669+0000 7f58327fc700 1 -- 172.21.15.116:0/734546464 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f582c059e60 con 0x7f583c101750 2024-03-04T16:15:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.671+0000 7f5843a08700 1 -- 172.21.15.116:0/734546464 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f582806d8a0 msgr2=0x7f582806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.671+0000 7f5843a08700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f582806d8a0 0x7f582806fd60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5838006010 tx=0x7f583801a040 comp rx=0 tx=0).stop 2024-03-04T16:15:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.671+0000 7f5843a08700 1 -- 172.21.15.116:0/734546464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583c101750 msgr2=0x7f583c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.671+0000 7f5843a08700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583c101750 0x7f583c0735d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f582c00b480 tx=0x7f582c00b790 comp rx=0 tx=0).stop 2024-03-04T16:15:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.672+0000 7f5843a08700 1 -- 172.21.15.116:0/734546464 shutdown_connections 2024-03-04T16:15:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.672+0000 7f5843a08700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f582806d8a0 0x7f582806fd60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.672+0000 7f5843a08700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f583c100da0 0x7f583c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.672+0000 7f5843a08700 1 --2- 172.21.15.116:0/734546464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583c101750 0x7f583c0735d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.672+0000 7f5843a08700 1 -- 172.21.15.116:0/734546464 >> 172.21.15.116:0/734546464 conn(0x7f583c0fc650 msgr2=0x7f583c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.672+0000 7f5843a08700 1 -- 172.21.15.116:0/734546464 shutdown_connections 2024-03-04T16:15:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:26.672+0000 7f5843a08700 1 -- 172.21.15.116:0/734546464 wait complete. 2024-03-04T16:15:26.682 INFO:teuthology.orchestra.run.smithi116.stdout:true 2024-03-04T16:15:26.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:26 smithi152 ceph-mon[82845]: pgmap v323: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:27.344 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-04T16:15:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:15:27.682 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:27 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/734546464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:27.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:27 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/734546464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:28.676 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:28 smithi116 ceph-mon[94164]: pgmap v324: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:28.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:28 smithi152 ceph-mon[82845]: pgmap v324: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.238+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3620101536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1c4101e20 msgr2=0x7fa1c40ff570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.238+0000 7fa1cb40b700 1 --2- 172.21.15.116:0/3620101536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1c4101e20 0x7fa1c40ff570 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4009910 tx=0x7fa1b4009c20 comp rx=0 tx=0).stop 2024-03-04T16:15:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.238+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3620101536 shutdown_connections 2024-03-04T16:15:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.238+0000 7fa1cb40b700 1 --2- 172.21.15.116:0/3620101536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1c4101e20 0x7fa1c40ff570 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.238+0000 7fa1cb40b700 1 --2- 172.21.15.116:0/3620101536 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1c4101a40 0x7fa1c40ff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.238+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3620101536 >> 172.21.15.116:0/3620101536 conn(0x7fa1c4075e30 msgr2=0x7fa1c40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.238+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3620101536 shutdown_connections 2024-03-04T16:15:29.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.238+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3620101536 wait complete. 2024-03-04T16:15:29.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1cb40b700 1 Processor -- start 2024-03-04T16:15:29.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1cb40b700 1 -- start start 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1cb40b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1c4101a40 0x7fa1c419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1cb40b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1c4101e20 0x7fa1c419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1cb40b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c419f210 con 0x7fa1c4101e20 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1cb40b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c4198750 con 0x7fa1c4101a40 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1c89a6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1c4101e20 0x7fa1c419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1c89a6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1c4101e20 0x7fa1c419ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47974/0 (socket says 172.21.15.116:47974) 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1c89a6700 1 -- 172.21.15.116:0/3209976746 learned_addr learned my addr 172.21.15.116:0/3209976746 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1c91a7700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1c4101a40 0x7fa1c419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1c89a6700 1 -- 172.21.15.116:0/3209976746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1c4101a40 msgr2=0x7fa1c419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1c89a6700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1c4101a40 0x7fa1c419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1c89a6700 1 -- 172.21.15.116:0/3209976746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b40095f0 con 0x7fa1c4101e20 2024-03-04T16:15:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.239+0000 7fa1c91a7700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1c4101a40 0x7fa1c419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.240+0000 7fa1c89a6700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1c4101e20 0x7fa1c419ec10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4006010 tx=0x7fa1b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.240+0000 7fa1ba7fc700 1 -- 172.21.15.116:0/3209976746 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1b401d070 con 0x7fa1c4101e20 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.240+0000 7fa1ba7fc700 1 -- 172.21.15.116:0/3209976746 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1b400f460 con 0x7fa1c4101e20 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.240+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3209976746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c41989d0 con 0x7fa1c4101e20 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.240+0000 7fa1ba7fc700 1 -- 172.21.15.116:0/3209976746 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1b4017610 con 0x7fa1c4101e20 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.240+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3209976746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c4198ec0 con 0x7fa1c4101e20 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.241+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3209976746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c404ea90 con 0x7fa1c4101e20 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.241+0000 7fa1ba7fc700 1 -- 172.21.15.116:0/3209976746 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fa1b400f5d0 con 0x7fa1c4101e20 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.241+0000 7fa1ba7fc700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa1b006d8a0 0x7fa1b006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.242+0000 7fa1ba7fc700 1 -- 172.21.15.116:0/3209976746 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fa1b408e9c0 con 0x7fa1c4101e20 2024-03-04T16:15:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.242+0000 7fa1c91a7700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa1b006d8a0 0x7fa1b006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:29.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.242+0000 7fa1c91a7700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa1b006d8a0 0x7fa1b006fd60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc005fd0 tx=0x7fa1bc005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.246+0000 7fa1ba7fc700 1 -- 172.21.15.116:0/3209976746 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa1b405bc90 con 0x7fa1c4101e20 2024-03-04T16:15:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.520+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3209976746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa1c4199c60 con 0x7fa1c4101e20 2024-03-04T16:15:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.520+0000 7fa1ba7fc700 1 -- 172.21.15.116:0/3209976746 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fa1b405b820 con 0x7fa1c4101e20 2024-03-04T16:15:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3209976746 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa1b006d8a0 msgr2=0x7fa1b006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa1b006d8a0 0x7fa1b006fd60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc005fd0 tx=0x7fa1bc005e20 comp rx=0 tx=0).stop 2024-03-04T16:15:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3209976746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1c4101e20 msgr2=0x7fa1c419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1c4101e20 0x7fa1c419ec10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4006010 tx=0x7fa1b400be20 comp rx=0 tx=0).stop 2024-03-04T16:15:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3209976746 shutdown_connections 2024-03-04T16:15:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fa1b006d8a0 0x7fa1b006fd60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1c4101a40 0x7fa1c419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 --2- 172.21.15.116:0/3209976746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1c4101e20 0x7fa1c419ec10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3209976746 >> 172.21.15.116:0/3209976746 conn(0x7fa1c4075e30 msgr2=0x7fa1c40fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3209976746 shutdown_connections 2024-03-04T16:15:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:29.522+0000 7fa1cb40b700 1 -- 172.21.15.116:0/3209976746 wait complete. 2024-03-04T16:15:29.533 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)" 2024-03-04T16:15:30.213 DEBUG:teuthology.parallel:result is None 2024-03-04T16:15:30.214 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:15:30.228 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:15:30.228 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4960653c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-04T16:15:30.439 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/mon.smithi116/config 2024-03-04T16:15:30.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:30 smithi116 ceph-mon[94164]: pgmap v325: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:30.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:30 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3209976746' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:30.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:30 smithi152 ceph-mon[82845]: pgmap v325: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:30.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:30 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3209976746' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.085+0000 7f320a471700 1 -- 172.21.15.116:0/193026843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3204073130 msgr2=0x7f3204073510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.085+0000 7f320a471700 1 --2- 172.21.15.116:0/193026843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3204073130 0x7f3204073510 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31f4009910 tx=0x7f31f4009c20 comp rx=0 tx=0).stop 2024-03-04T16:15:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.085+0000 7f320a471700 1 -- 172.21.15.116:0/193026843 shutdown_connections 2024-03-04T16:15:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.085+0000 7f320a471700 1 --2- 172.21.15.116:0/193026843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3204073a50 0x7f3204111640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.085+0000 7f320a471700 1 --2- 172.21.15.116:0/193026843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3204073130 0x7f3204073510 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.085+0000 7f320a471700 1 -- 172.21.15.116:0/193026843 >> 172.21.15.116:0/193026843 conn(0x7f32040fc640 msgr2=0x7f32040fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.085+0000 7f320a471700 1 -- 172.21.15.116:0/193026843 shutdown_connections 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.085+0000 7f320a471700 1 -- 172.21.15.116:0/193026843 wait complete. 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.085+0000 7f320a471700 1 Processor -- start 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.085+0000 7f320a471700 1 -- start start 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f320a471700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3204073130 0x7f320419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f320a471700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3204073a50 0x7f320419d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f320a471700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320419d8f0 con 0x7f3204073130 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f320a471700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32041a05d0 con 0x7f3204073a50 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f3203fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3204073130 0x7f320419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f3203fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3204073130 0x7f320419cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47986/0 (socket says 172.21.15.116:47986) 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f3203fff700 1 -- 172.21.15.116:0/2976615171 learned_addr learned my addr 172.21.15.116:0/2976615171 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f32037fe700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3204073a50 0x7f320419d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f3203fff700 1 -- 172.21.15.116:0/2976615171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3204073a50 msgr2=0x7f320419d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f3203fff700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3204073a50 0x7f320419d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f3203fff700 1 -- 172.21.15.116:0/2976615171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f40095f0 con 0x7f3204073130 2024-03-04T16:15:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f3203fff700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3204073130 0x7f320419cd20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f31f4006010 tx=0x7f31f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f32017fa700 1 -- 172.21.15.116:0/2976615171 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31f401d070 con 0x7f3204073130 2024-03-04T16:15:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f320a471700 1 -- 172.21.15.116:0/2976615171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32041a08b0 con 0x7f3204073130 2024-03-04T16:15:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.086+0000 7f32017fa700 1 -- 172.21.15.116:0/2976615171 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31f4004fb0 con 0x7f3204073130 2024-03-04T16:15:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.087+0000 7f32017fa700 1 -- 172.21.15.116:0/2976615171 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31f4022ea0 con 0x7f3204073130 2024-03-04T16:15:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.087+0000 7f320a471700 1 -- 172.21.15.116:0/2976615171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32041a0e00 con 0x7f3204073130 2024-03-04T16:15:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.088+0000 7f320a471700 1 -- 172.21.15.116:0/2976615171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320410edc0 con 0x7f3204073130 2024-03-04T16:15:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.088+0000 7f32017fa700 1 -- 172.21.15.116:0/2976615171 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f31f4022470 con 0x7f3204073130 2024-03-04T16:15:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.088+0000 7f32017fa700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f31ec06d8a0 0x7f31ec06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.089+0000 7f32017fa700 1 -- 172.21.15.116:0/2976615171 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f31f408ecf0 con 0x7f3204073130 2024-03-04T16:15:32.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.089+0000 7f32037fe700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f31ec06d8a0 0x7f31ec06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:32.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.089+0000 7f32037fe700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f31ec06d8a0 0x7f31ec06fd60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f31f8005fd0 tx=0x7f31f800a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:32.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.093+0000 7f32017fa700 1 -- 172.21.15.116:0/2976615171 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f31f4058620 con 0x7f3204073130 2024-03-04T16:15:32.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.337+0000 7f320a471700 1 -- 172.21.15.116:0/2976615171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f320404ea90 con 0x7f3204073130 2024-03-04T16:15:32.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.337+0000 7f32017fa700 1 -- 172.21.15.116:0/2976615171 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f31f4027770 con 0x7f3204073130 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout:e29 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 29 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T16:01:00.313772+0000 2024-03-04T16:15:32.338 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T16:11:49.225261+0000 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 60 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 84 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=34400} 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T16:15:32.339 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 1 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.pjrmzv{0:34400} state up:active seq 20 join_fscid=1 addr [v2:172.21.15.116:6826/487584748,v1:172.21.15.116:6827/487584748] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi116.gkenwy{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6828/786823228,v1:172.21.15.116:6829/786823228] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:15:32.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.tlpshy{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.152:6824/3736721533,v1:172.21.15.152:6825/3736721533] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:15:32.341 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.cephfs.smithi152.kqoybf{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.152:6826/1659417498,v1:172.21.15.152:6827/1659417498] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T16:15:32.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.339+0000 7f320a471700 1 -- 172.21.15.116:0/2976615171 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f31ec06d8a0 msgr2=0x7f31ec06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:32.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.339+0000 7f320a471700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f31ec06d8a0 0x7f31ec06fd60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f31f8005fd0 tx=0x7f31f800a560 comp rx=0 tx=0).stop 2024-03-04T16:15:32.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.339+0000 7f320a471700 1 -- 172.21.15.116:0/2976615171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3204073130 msgr2=0x7f320419cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.339+0000 7f320a471700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3204073130 0x7f320419cd20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f31f4006010 tx=0x7f31f400be20 comp rx=0 tx=0).stop 2024-03-04T16:15:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.339+0000 7f320a471700 1 -- 172.21.15.116:0/2976615171 shutdown_connections 2024-03-04T16:15:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.339+0000 7f320a471700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f31ec06d8a0 0x7f31ec06fd60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.339+0000 7f320a471700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3204073130 0x7f320419cd20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.339+0000 7f320a471700 1 --2- 172.21.15.116:0/2976615171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3204073a50 0x7f320419d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.339+0000 7f320a471700 1 -- 172.21.15.116:0/2976615171 >> 172.21.15.116:0/2976615171 conn(0x7f32040fc640 msgr2=0x7f3204103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.340+0000 7f320a471700 1 -- 172.21.15.116:0/2976615171 shutdown_connections 2024-03-04T16:15:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:32.340+0000 7f320a471700 1 -- 172.21.15.116:0/2976615171 wait complete. 2024-03-04T16:15:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-04T16:15:32.712 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:32 smithi116 ceph-mon[94164]: pgmap v326: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:32.712 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:32 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2976615171' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:15:32.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:32 smithi152 ceph-mon[82845]: pgmap v326: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:32.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:32 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2976615171' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T16:15:33.052 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-03-04T16:15:33.066 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T16:15:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.170+0000 7fcecad49640 1 -- 172.21.15.116:0/132073638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec4103f20 msgr2=0x7fcec4104300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.170+0000 7fcecad49640 1 --2- 172.21.15.116:0/132073638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec4103f20 0x7fcec4104300 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009880 tx=0x7fceb802bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.170+0000 7fcecad49640 1 -- 172.21.15.116:0/132073638 shutdown_connections 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.170+0000 7fcecad49640 1 --2- 172.21.15.116:0/132073638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcec40fefe0 0x7fcec40ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.170+0000 7fcecad49640 1 --2- 172.21.15.116:0/132073638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec4103f20 0x7fcec4104300 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.170+0000 7fcecad49640 1 -- 172.21.15.116:0/132073638 >> 172.21.15.116:0/132073638 conn(0x7fcec4075b40 msgr2=0x7fcec4075f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.170+0000 7fcecad49640 1 -- 172.21.15.116:0/132073638 shutdown_connections 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.170+0000 7fcecad49640 1 -- 172.21.15.116:0/132073638 wait complete. 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcecad49640 1 Processor -- start 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcecad49640 1 -- start start 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcecad49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec40fefe0 0x7fcec4196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcecad49640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcec4103f20 0x7fcec41970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcecad49640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec4197700 con 0x7fcec40fefe0 2024-03-04T16:15:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcecad49640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec4197840 con 0x7fcec4103f20 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec8ae5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec40fefe0 0x7fcec4196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec8ae5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec40fefe0 0x7fcec4196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47996/0 (socket says 172.21.15.116:47996) 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec8ae5640 1 -- 172.21.15.116:0/511975066 learned_addr learned my addr 172.21.15.116:0/511975066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec8ae5640 1 -- 172.21.15.116:0/511975066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcec4103f20 msgr2=0x7fcec41970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec3fff640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcec4103f20 0x7fcec41970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec8ae5640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcec4103f20 0x7fcec41970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec8ae5640 1 -- 172.21.15.116:0/511975066 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb8009560 con 0x7fcec40fefe0 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec3fff640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcec4103f20 0x7fcec41970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec8ae5640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec40fefe0 0x7fcec4196ba0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009230 tx=0x7fceb8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec1ffb640 1 -- 172.21.15.116:0/511975066 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceb80035f0 con 0x7fcec40fefe0 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcecad49640 1 -- 172.21.15.116:0/511975066 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec419fc10 con 0x7fcec40fefe0 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec1ffb640 1 -- 172.21.15.116:0/511975066 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fceb8003c10 con 0x7fcec40fefe0 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.171+0000 7fcec1ffb640 1 -- 172.21.15.116:0/511975066 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceb8034440 con 0x7fcec40fefe0 2024-03-04T16:15:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.172+0000 7fcecad49640 1 -- 172.21.15.116:0/511975066 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec41a0030 con 0x7fcec40fefe0 2024-03-04T16:15:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.173+0000 7fcecad49640 1 -- 172.21.15.116:0/511975066 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec404e040 con 0x7fcec40fefe0 2024-03-04T16:15:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.173+0000 7fcec1ffb640 1 -- 172.21.15.116:0/511975066 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fceb8004fd0 con 0x7fcec40fefe0 2024-03-04T16:15:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.173+0000 7fcec1ffb640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fceb406d8d0 0x7fceb406fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.173+0000 7fcec1ffb640 1 -- 172.21.15.116:0/511975066 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fceb80afa00 con 0x7fcec40fefe0 2024-03-04T16:15:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.174+0000 7fcec3fff640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fceb406d8d0 0x7fceb406fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.174+0000 7fcec3fff640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fceb406d8d0 0x7fceb406fd90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fceb0006fd0 tx=0x7fceb0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:33.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.178+0000 7fcec1ffb640 1 -- 172.21.15.116:0/511975066 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fceb8079160 con 0x7fcec40fefe0 2024-03-04T16:15:33.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.351+0000 7fcecad49640 1 -- 172.21.15.116:0/511975066 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcec41a0220 con 0x7fcec40fefe0 2024-03-04T16:15:33.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.352+0000 7fcec1ffb640 1 -- 172.21.15.116:0/511975066 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16947 (secure 0 0 0) 0x7fceb807c820 con 0x7fcec40fefe0 2024-03-04T16:15:33.353 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:33.353 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":84,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:53:48.465240+0000","modified":"2024-03-04T16:11:38.459407+0000","last_up_change":"2024-03-04T16:10:31.185124+0000","last_in_change":"2024-03-04T15:59:22.307720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:58:27.372335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-04T16:00:58.546762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-04T16:00:59.546717+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d12966f-954e-4a09-bbbd-5b0f6a04d039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6803","nonce":2502615728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6805","nonce":2502615728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6809","nonce":2502615728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6807","nonce":2502615728}]},"public_addr":"172.21.15.116:6803/2502615728","cluster_addr":"172.21.15.116:6805/2502615728","heartbeat_back_addr":"172.21.15.116:6809/2502615728","heartbeat_front_addr":"172.21.15.116:6807/2502615728","state":["exists","up"]},{"osd":1,"uuid":"f990720f-a19d-48c1-9545-81e511c2690b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6811","nonce":22677649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6813","nonce":22677649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6817","nonce":22677649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6815","nonce":22677649}]},"public_addr":"172.21.15.116:6811/22677649","cluster_addr":"172.21.15.116:6813/22677649","heartbeat_back_addr":"172.21.15.116:6817/22677649","heartbeat_front_addr":"172.21.15.116:6815/22677649","state":["exists","up"]},{"osd":2,"uuid":"fe3132c8-6b15-4a11-a884-7c6a09d42955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6819","nonce":1756826225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6821","nonce":1756826225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6825","nonce":1756826225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6823","nonce":1756826225}]},"public_addr":"172.21.15.116:6819/1756826225","cluster_addr":"172.21.15.116:6821/1756826225","heartbeat_back_addr":"172.21.15.116:6825/1756826225","heartbeat_front_addr":"172.21.15.116:6823/1756826225","state":["exists","up"]},{"osd":3,"uuid":"15930a9b-f721-4c30-9c5b-bef5ef69777c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6801","nonce":3040811013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6803","nonce":3040811013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6807","nonce":3040811013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6805","nonce":3040811013}]},"public_addr":"172.21.15.152:6801/3040811013","cluster_addr":"172.21.15.152:6803/3040811013","heartbeat_back_addr":"172.21.15.152:6807/3040811013","heartbeat_front_addr":"172.21.15.152:6805/3040811013","state":["exists","up"]},{"osd":4,"uuid":"8d201b6a-6a91-4a7e-984e-e8d7f0289793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6809","nonce":878265314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6811","nonce":878265314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6815","nonce":878265314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6813","nonce":878265314}]},"public_addr":"172.21.15.152:6809/878265314","cluster_addr":"172.21.15.152:6811/878265314","heartbeat_back_addr":"172.21.15.152:6815/878265314","heartbeat_front_addr":"172.21.15.152:6813/878265314","state":["exists","up"]},{"osd":5,"uuid":"c06ef0f5-6894-4e01-a693-bb129d248fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6817","nonce":1180242057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6819","nonce":1180242057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6823","nonce":1180242057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6821","nonce":1180242057}]},"public_addr":"172.21.15.152:6817/1180242057","cluster_addr":"172.21.15.152:6819/1180242057","heartbeat_back_addr":"172.21.15.152:6823/1180242057","heartbeat_front_addr":"172.21.15.152:6821/1180242057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:24.230215+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:52.709103+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:24.072974+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:47.684390+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:13.571654+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:39.874966+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.152:6827/1672192052":"2024-03-05T16:11:38.459377+0000","172.21.15.152:6826/1672192052":"2024-03-05T16:11:38.459377+0000","172.21.15.152:6825/287206196":"2024-03-05T16:11:14.444369+0000","172.21.15.152:6824/287206196":"2024-03-05T16:11:14.444369+0000","172.21.15.116:0/3594007286":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/4255448814":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/317318762":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/2770714257":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/3287532122":"2024-03-05T15:54:53.931186+0000","172.21.15.116:6801/2966163948":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/976694871":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6800/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6801/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6801/3356602598":"2024-03-05T16:03:33.250227+0000","172.21.15.116:6800/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3670178561":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6800/3356602598":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/2883878856":"2024-03-05T16:05:52.994136+0000","172.21.15.152:0/3493116756":"2024-03-05T16:04:41.923774+0000","172.21.15.116:6800/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.152:0/1831164264":"2024-03-05T16:04:41.923774+0000","172.21.15.116:6801/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:6827/114739798":"2024-03-05T16:01:15.387483+0000","172.21.15.116:6801/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/385581892":"2024-03-05T15:54:21.166941+0000","172.21.15.152:0/1293393009":"2024-03-05T16:04:41.923774+0000","172.21.15.152:6829/2881662614":"2024-03-05T16:04:41.923774+0000","172.21.15.152:0/1265936494":"2024-03-05T16:04:41.923774+0000","172.21.15.116:0/831649128":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3323890559":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/355050440":"2024-03-05T15:56:07.046998+0000","172.21.15.152:6828/2881662614":"2024-03-05T16:04:41.923774+0000","172.21.15.116:0/590312912":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/232576737":"2024-03-05T16:03:33.250227+0000","172.21.15.152:0/549268400":"2024-03-05T16:04:41.923774+0000","172.21.15.152:0/768114426":"2024-03-05T16:04:41.923774+0000","172.21.15.116:0/4142805454":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/1756904921":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/821201733":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6826/114739798":"2024-03-05T16:01:15.387483+0000","172.21.15.116:0/3213147898":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/2867113891":"2024-03-05T16:05:52.994136+0000","172.21.15.116:6800/2966163948":"2024-03-05T16:05:52.994136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:15:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.354+0000 7fcecad49640 1 -- 172.21.15.116:0/511975066 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fceb406d8d0 msgr2=0x7fceb406fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.354+0000 7fcecad49640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fceb406d8d0 0x7fceb406fd90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fceb0006fd0 tx=0x7fceb0008040 comp rx=0 tx=0).stop 2024-03-04T16:15:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.354+0000 7fcecad49640 1 -- 172.21.15.116:0/511975066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec40fefe0 msgr2=0x7fcec4196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.354+0000 7fcecad49640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec40fefe0 0x7fcec4196ba0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009230 tx=0x7fceb8004040 comp rx=0 tx=0).stop 2024-03-04T16:15:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.354+0000 7fcecad49640 1 -- 172.21.15.116:0/511975066 shutdown_connections 2024-03-04T16:15:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.354+0000 7fcecad49640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fceb406d8d0 0x7fceb406fd90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.354+0000 7fcecad49640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcec4103f20 0x7fcec41970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.354+0000 7fcecad49640 1 --2- 172.21.15.116:0/511975066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec40fefe0 0x7fcec4196ba0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.354+0000 7fcecad49640 1 -- 172.21.15.116:0/511975066 >> 172.21.15.116:0/511975066 conn(0x7fcec4075b40 msgr2=0x7fcec40fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.354+0000 7fcecad49640 1 -- 172.21.15.116:0/511975066 shutdown_connections 2024-03-04T16:15:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.355+0000 7fcecad49640 1 -- 172.21.15.116:0/511975066 wait complete. 2024-03-04T16:15:33.366 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T15:58:27.372335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-04T16:00:58.546762+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-04T16:00:59.546717+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T16:15:33.367 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T16:15:33.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.471+0000 7f7470950640 1 -- 172.21.15.116:0/2892651786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f746c0ff180 msgr2=0x7f746c0ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.471+0000 7f7470950640 1 --2- 172.21.15.116:0/2892651786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f746c0ff180 0x7f746c0ff560 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f745c009880 tx=0x7f745c02bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.471+0000 7f7470950640 1 -- 172.21.15.116:0/2892651786 shutdown_connections 2024-03-04T16:15:33.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.471+0000 7f7470950640 1 --2- 172.21.15.116:0/2892651786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f746c0ffb30 0x7f746c103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.471+0000 7f7470950640 1 --2- 172.21.15.116:0/2892651786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f746c0ff180 0x7f746c0ff560 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.471+0000 7f7470950640 1 -- 172.21.15.116:0/2892651786 >> 172.21.15.116:0/2892651786 conn(0x7f746c0faa00 msgr2=0x7f746c0fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.471+0000 7f7470950640 1 -- 172.21.15.116:0/2892651786 shutdown_connections 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.471+0000 7f7470950640 1 -- 172.21.15.116:0/2892651786 wait complete. 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f7470950640 1 Processor -- start 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f7470950640 1 -- start start 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f7470950640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f746c0ff180 0x7f746c196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f7470950640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f746c197220 0x7f746c197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f7470950640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746c19fd50 con 0x7f746c0ff180 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f7470950640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746c19fe90 con 0x7f746c197220 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f746a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f746c0ff180 0x7f746c196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f746a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f746c0ff180 0x7f746c196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48024/0 (socket says 172.21.15.116:48024) 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f746a59c640 1 -- 172.21.15.116:0/1056030107 learned_addr learned my addr 172.21.15.116:0/1056030107 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f746a59c640 1 -- 172.21.15.116:0/1056030107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f746c197220 msgr2=0x7f746c197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f746a59c640 1 --2- 172.21.15.116:0/1056030107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f746c197220 0x7f746c197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f746a59c640 1 -- 172.21.15.116:0/1056030107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c009560 con 0x7f746c0ff180 2024-03-04T16:15:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f746a59c640 1 --2- 172.21.15.116:0/1056030107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f746c0ff180 0x7f746c196ce0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f745c009430 tx=0x7f745c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.472+0000 7f74637fe640 1 -- 172.21.15.116:0/1056030107 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f745c0035f0 con 0x7f746c0ff180 2024-03-04T16:15:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.473+0000 7f7470950640 1 -- 172.21.15.116:0/1056030107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746c1a0090 con 0x7f746c0ff180 2024-03-04T16:15:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.473+0000 7f74637fe640 1 -- 172.21.15.116:0/1056030107 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f745c003c10 con 0x7f746c0ff180 2024-03-04T16:15:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.473+0000 7f74637fe640 1 -- 172.21.15.116:0/1056030107 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f745c034440 con 0x7f746c0ff180 2024-03-04T16:15:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.473+0000 7f7470950640 1 -- 172.21.15.116:0/1056030107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746c1a0540 con 0x7f746c0ff180 2024-03-04T16:15:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.473+0000 7f7470950640 1 -- 172.21.15.116:0/1056030107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746c04e040 con 0x7f746c0ff180 2024-03-04T16:15:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.474+0000 7f74637fe640 1 -- 172.21.15.116:0/1056030107 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f745c004fd0 con 0x7f746c0ff180 2024-03-04T16:15:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.474+0000 7f74637fe640 1 --2- 172.21.15.116:0/1056030107 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f745806d810 0x7f745806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.475+0000 7f74637fe640 1 -- 172.21.15.116:0/1056030107 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f745c0afa20 con 0x7f746c0ff180 2024-03-04T16:15:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.475+0000 7f7463fff640 1 --2- 172.21.15.116:0/1056030107 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f745806d810 0x7f745806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.475+0000 7f7463fff640 1 --2- 172.21.15.116:0/1056030107 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f745806d810 0x7f745806fcd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f746c069810 tx=0x7f7454006c90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.479+0000 7f74637fe640 1 -- 172.21.15.116:0/1056030107 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f745c079180 con 0x7f746c0ff180 2024-03-04T16:15:33.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.652+0000 7f7470950640 1 -- 172.21.15.116:0/1056030107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f746c0ffb60 con 0x7f746c0ff180 2024-03-04T16:15:33.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.652+0000 7f74637fe640 1 -- 172.21.15.116:0/1056030107 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f745c07c840 con 0x7f746c0ff180 2024-03-04T16:15:33.653 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T16:15:33.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.654+0000 7f7470950640 1 -- 172.21.15.116:0/1056030107 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f745806d810 msgr2=0x7f745806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.654+0000 7f7470950640 1 --2- 172.21.15.116:0/1056030107 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f745806d810 0x7f745806fcd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f746c069810 tx=0x7f7454006c90 comp rx=0 tx=0).stop 2024-03-04T16:15:33.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.654+0000 7f7470950640 1 -- 172.21.15.116:0/1056030107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f746c0ff180 msgr2=0x7f746c196ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.654+0000 7f7470950640 1 --2- 172.21.15.116:0/1056030107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f746c0ff180 0x7f746c196ce0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f745c009430 tx=0x7f745c004040 comp rx=0 tx=0).stop 2024-03-04T16:15:33.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.654+0000 7f7470950640 1 -- 172.21.15.116:0/1056030107 shutdown_connections 2024-03-04T16:15:33.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.654+0000 7f7470950640 1 --2- 172.21.15.116:0/1056030107 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f745806d810 0x7f745806fcd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.654+0000 7f7470950640 1 --2- 172.21.15.116:0/1056030107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f746c197220 0x7f746c197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.654+0000 7f7470950640 1 --2- 172.21.15.116:0/1056030107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f746c0ff180 0x7f746c196ce0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.654+0000 7f7470950640 1 -- 172.21.15.116:0/1056030107 >> 172.21.15.116:0/1056030107 conn(0x7f746c0faa00 msgr2=0x7f746c0fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:33.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.654+0000 7f7470950640 1 -- 172.21.15.116:0/1056030107 shutdown_connections 2024-03-04T16:15:33.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.655+0000 7f7470950640 1 -- 172.21.15.116:0/1056030107 wait complete. 2024-03-04T16:15:33.664 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:33 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/511975066' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:15:33.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-04T16:15:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.770+0000 7f04f953e640 1 -- 172.21.15.116:0/2932049403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f40ffb50 msgr2=0x7f04f4103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.770+0000 7f04f953e640 1 --2- 172.21.15.116:0/2932049403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f40ffb50 0x7f04f4103b90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009880 tx=0x7f04dc02bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.770+0000 7f04f953e640 1 -- 172.21.15.116:0/2932049403 shutdown_connections 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.770+0000 7f04f953e640 1 --2- 172.21.15.116:0/2932049403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f40ffb50 0x7f04f4103b90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.770+0000 7f04f953e640 1 --2- 172.21.15.116:0/2932049403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04f40ff1a0 0x7f04f40ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.770+0000 7f04f953e640 1 -- 172.21.15.116:0/2932049403 >> 172.21.15.116:0/2932049403 conn(0x7f04f40faa20 msgr2=0x7f04f40fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.770+0000 7f04f953e640 1 -- 172.21.15.116:0/2932049403 shutdown_connections 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.770+0000 7f04f953e640 1 -- 172.21.15.116:0/2932049403 wait complete. 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04f953e640 1 Processor -- start 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04f953e640 1 -- start start 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04f953e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04f40ff1a0 0x7f04f4196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04f953e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f4197120 0x7f04f4197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04f953e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f419fc50 con 0x7f04f4197120 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04f953e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f419fd90 con 0x7f04f40ff1a0 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f4197120 0x7f04f4197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f4197120 0x7f04f4197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48032/0 (socket says 172.21.15.116:48032) 2024-03-04T16:15:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04ebfff640 1 -- 172.21.15.116:0/1098416404 learned_addr learned my addr 172.21.15.116:0/1098416404 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04ebfff640 1 -- 172.21.15.116:0/1098416404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04f40ff1a0 msgr2=0x7f04f4196be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04ebfff640 1 --2- 172.21.15.116:0/1098416404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04f40ff1a0 0x7f04f4196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04ebfff640 1 -- 172.21.15.116:0/1098416404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc009560 con 0x7f04f4197120 2024-03-04T16:15:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.771+0000 7f04ebfff640 1 --2- 172.21.15.116:0/1098416404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f4197120 0x7f04f4197580 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f04dc036040 tx=0x7f04dc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.772+0000 7f04f0ff9640 1 -- 172.21.15.116:0/1098416404 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04dc0037c0 con 0x7f04f4197120 2024-03-04T16:15:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.772+0000 7f04f0ff9640 1 -- 172.21.15.116:0/1098416404 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04dc003de0 con 0x7f04f4197120 2024-03-04T16:15:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.772+0000 7f04f953e640 1 -- 172.21.15.116:0/1098416404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f419ff90 con 0x7f04f4197120 2024-03-04T16:15:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.772+0000 7f04f0ff9640 1 -- 172.21.15.116:0/1098416404 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04dc034440 con 0x7f04f4197120 2024-03-04T16:15:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.772+0000 7f04f953e640 1 -- 172.21.15.116:0/1098416404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f41a0410 con 0x7f04f4197120 2024-03-04T16:15:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.773+0000 7f04f953e640 1 -- 172.21.15.116:0/1098416404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f404e040 con 0x7f04f4197120 2024-03-04T16:15:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.773+0000 7f04f0ff9640 1 -- 172.21.15.116:0/1098416404 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f04dc004fb0 con 0x7f04f4197120 2024-03-04T16:15:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.773+0000 7f04f0ff9640 1 --2- 172.21.15.116:0/1098416404 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04e006d810 0x7f04e006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.774+0000 7f04f0ff9640 1 -- 172.21.15.116:0/1098416404 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f04dc0afa20 con 0x7f04f4197120 2024-03-04T16:15:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.774+0000 7f04f2ffd640 1 --2- 172.21.15.116:0/1098416404 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04e006d810 0x7f04e006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.774+0000 7f04f2ffd640 1 --2- 172.21.15.116:0/1098416404 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04e006d810 0x7f04e006fcd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f04e400a8f0 tx=0x7f04e4005bf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.778+0000 7f04f0ff9640 1 -- 172.21.15.116:0/1098416404 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f04dc079180 con 0x7f04f4197120 2024-03-04T16:15:33.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:33 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/511975066' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:15:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.950+0000 7f04f953e640 1 -- 172.21.15.116:0/1098416404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f04f41074a0 con 0x7f04f4197120 2024-03-04T16:15:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.950+0000 7f04f0ff9640 1 -- 172.21.15.116:0/1098416404 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f04dc07c840 con 0x7f04f4197120 2024-03-04T16:15:33.951 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T16:15:33.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 -- 172.21.15.116:0/1098416404 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04e006d810 msgr2=0x7f04e006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 --2- 172.21.15.116:0/1098416404 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04e006d810 0x7f04e006fcd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f04e400a8f0 tx=0x7f04e4005bf0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 -- 172.21.15.116:0/1098416404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f4197120 msgr2=0x7f04f4197580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 --2- 172.21.15.116:0/1098416404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f4197120 0x7f04f4197580 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f04dc036040 tx=0x7f04dc004270 comp rx=0 tx=0).stop 2024-03-04T16:15:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 -- 172.21.15.116:0/1098416404 shutdown_connections 2024-03-04T16:15:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 --2- 172.21.15.116:0/1098416404 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f04e006d810 0x7f04e006fcd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 --2- 172.21.15.116:0/1098416404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f4197120 0x7f04f4197580 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 --2- 172.21.15.116:0/1098416404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04f40ff1a0 0x7f04f4196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 -- 172.21.15.116:0/1098416404 >> 172.21.15.116:0/1098416404 conn(0x7f04f40faa20 msgr2=0x7f04f40fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 -- 172.21.15.116:0/1098416404 shutdown_connections 2024-03-04T16:15:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:33.952+0000 7f04f953e640 1 -- 172.21.15.116:0/1098416404 wait complete. 2024-03-04T16:15:33.963 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-04T16:15:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.068+0000 7fc3da377640 1 -- 172.21.15.116:0/4145674882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4067f70 msgr2=0x7fc3d4104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.068+0000 7fc3da377640 1 --2- 172.21.15.116:0/4145674882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4067f70 0x7fc3d4104c70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009880 tx=0x7fc3c402bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.068+0000 7fc3da377640 1 -- 172.21.15.116:0/4145674882 shutdown_connections 2024-03-04T16:15:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.068+0000 7fc3da377640 1 --2- 172.21.15.116:0/4145674882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4067f70 0x7fc3d4104c70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.069+0000 7fc3da377640 1 --2- 172.21.15.116:0/4145674882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc3d40675c0 0x7fc3d40679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.069+0000 7fc3da377640 1 -- 172.21.15.116:0/4145674882 >> 172.21.15.116:0/4145674882 conn(0x7fc3d4074a50 msgr2=0x7fc3d4074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.069+0000 7fc3da377640 1 -- 172.21.15.116:0/4145674882 shutdown_connections 2024-03-04T16:15:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.069+0000 7fc3da377640 1 -- 172.21.15.116:0/4145674882 wait complete. 2024-03-04T16:15:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.069+0000 7fc3da377640 1 Processor -- start 2024-03-04T16:15:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.069+0000 7fc3da377640 1 -- start start 2024-03-04T16:15:34.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.069+0000 7fc3da377640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d40675c0 0x7fc3d4195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:34.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.069+0000 7fc3da377640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc3d4196440 0x7fc3d419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.069+0000 7fc3da377640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d4196a40 con 0x7fc3d40675c0 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.069+0000 7fc3da377640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d4196bb0 con 0x7fc3d4196440 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d40675c0 0x7fc3d4195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d40675c0 0x7fc3d4195f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48048/0 (socket says 172.21.15.116:48048) 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d3fff640 1 -- 172.21.15.116:0/1326732174 learned_addr learned my addr 172.21.15.116:0/1326732174 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d37fe640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc3d4196440 0x7fc3d419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d3fff640 1 -- 172.21.15.116:0/1326732174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc3d4196440 msgr2=0x7fc3d419a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d3fff640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc3d4196440 0x7fc3d419a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d3fff640 1 -- 172.21.15.116:0/1326732174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c4009560 con 0x7fc3d40675c0 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d37fe640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc3d4196440 0x7fc3d419a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d3fff640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d40675c0 0x7fc3d4195f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800b450 tx=0x7fc3c800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d17fa640 1 -- 172.21.15.116:0/1326732174 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3c8011860 con 0x7fc3d40675c0 2024-03-04T16:15:34.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3da377640 1 -- 172.21.15.116:0/1326732174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d419aea0 con 0x7fc3d40675c0 2024-03-04T16:15:34.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d17fa640 1 -- 172.21.15.116:0/1326732174 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3c8011e80 con 0x7fc3d40675c0 2024-03-04T16:15:34.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3d17fa640 1 -- 172.21.15.116:0/1326732174 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3c800f570 con 0x7fc3d40675c0 2024-03-04T16:15:34.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.070+0000 7fc3da377640 1 -- 172.21.15.116:0/1326732174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d419b400 con 0x7fc3d40675c0 2024-03-04T16:15:34.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.071+0000 7fc3da377640 1 -- 172.21.15.116:0/1326732174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d404e040 con 0x7fc3d40675c0 2024-03-04T16:15:34.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.072+0000 7fc3d17fa640 1 -- 172.21.15.116:0/1326732174 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fc3c80103e0 con 0x7fc3d40675c0 2024-03-04T16:15:34.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.072+0000 7fc3d17fa640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc3bc06d810 0x7fc3bc06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:34.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.072+0000 7fc3d17fa640 1 -- 172.21.15.116:0/1326732174 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc3c808d530 con 0x7fc3d40675c0 2024-03-04T16:15:34.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.072+0000 7fc3d37fe640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc3bc06d810 0x7fc3bc06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:34.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.073+0000 7fc3d37fe640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc3bc06d810 0x7fc3bc06fcd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4006010 tx=0x7fc3c402cc00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.077+0000 7fc3d17fa640 1 -- 172.21.15.116:0/1326732174 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc3c8056c10 con 0x7fc3d40675c0 2024-03-04T16:15:34.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.249+0000 7fc3da377640 1 -- 172.21.15.116:0/1326732174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fc3d419b680 con 0x7fc3d40675c0 2024-03-04T16:15:34.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.249+0000 7fc3d17fa640 1 -- 172.21.15.116:0/1326732174 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc3c805a2d0 con 0x7fc3d40675c0 2024-03-04T16:15:34.250 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T16:15:34.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.251+0000 7fc3da377640 1 -- 172.21.15.116:0/1326732174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc3bc06d810 msgr2=0x7fc3bc06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.251+0000 7fc3da377640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc3bc06d810 0x7fc3bc06fcd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4006010 tx=0x7fc3c402cc00 comp rx=0 tx=0).stop 2024-03-04T16:15:34.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.251+0000 7fc3da377640 1 -- 172.21.15.116:0/1326732174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d40675c0 msgr2=0x7fc3d4195f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.251+0000 7fc3da377640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d40675c0 0x7fc3d4195f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800b450 tx=0x7fc3c800b910 comp rx=0 tx=0).stop 2024-03-04T16:15:34.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.251+0000 7fc3da377640 1 -- 172.21.15.116:0/1326732174 shutdown_connections 2024-03-04T16:15:34.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.251+0000 7fc3da377640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fc3bc06d810 0x7fc3bc06fcd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.252+0000 7fc3da377640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc3d4196440 0x7fc3d419a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.252+0000 7fc3da377640 1 --2- 172.21.15.116:0/1326732174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d40675c0 0x7fc3d4195f00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.252+0000 7fc3da377640 1 -- 172.21.15.116:0/1326732174 >> 172.21.15.116:0/1326732174 conn(0x7fc3d4074a50 msgr2=0x7fc3d40fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:34.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.252+0000 7fc3da377640 1 -- 172.21.15.116:0/1326732174 shutdown_connections 2024-03-04T16:15:34.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.252+0000 7fc3da377640 1 -- 172.21.15.116:0/1326732174 wait complete. 2024-03-04T16:15:34.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T16:15:34.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 -- 172.21.15.116:0/1815403355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c980ffac0 msgr2=0x7f4c98103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 --2- 172.21.15.116:0/1815403355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c980ffac0 0x7f4c98103b00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009880 tx=0x7f4c9402bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 -- 172.21.15.116:0/1815403355 shutdown_connections 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 --2- 172.21.15.116:0/1815403355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c980ffac0 0x7f4c98103b00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 --2- 172.21.15.116:0/1815403355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c980ff110 0x7f4c980ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 -- 172.21.15.116:0/1815403355 >> 172.21.15.116:0/1815403355 conn(0x7f4c980fa990 msgr2=0x7f4c980fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 -- 172.21.15.116:0/1815403355 shutdown_connections 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 -- 172.21.15.116:0/1815403355 wait complete. 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 Processor -- start 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 -- start start 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c980ff110 0x7f4c98100ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c981010e0 0x7f4c98101540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981a1e00 con 0x7f4c981010e0 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.364+0000 7f4c9f599640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981a1f40 con 0x7f4c980ff110 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c9cb34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c981010e0 0x7f4c98101540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c9cb34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c981010e0 0x7f4c98101540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48070/0 (socket says 172.21.15.116:48070) 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c9cb34640 1 -- 172.21.15.116:0/3129975975 learned_addr learned my addr 172.21.15.116:0/3129975975 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c9d335640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c980ff110 0x7f4c98100ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c9cb34640 1 -- 172.21.15.116:0/3129975975 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c980ff110 msgr2=0x7f4c98100ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c9cb34640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c980ff110 0x7f4c98100ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c9cb34640 1 -- 172.21.15.116:0/3129975975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c94009560 con 0x7f4c981010e0 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c9cb34640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c981010e0 0x7f4c98101540 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4c94036040 tx=0x7f4c94004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c8e7fc640 1 -- 172.21.15.116:0/3129975975 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c940037c0 con 0x7f4c981010e0 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c8e7fc640 1 -- 172.21.15.116:0/3129975975 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c94003de0 con 0x7f4c981010e0 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c9f599640 1 -- 172.21.15.116:0/3129975975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c981a2080 con 0x7f4c981010e0 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c8e7fc640 1 -- 172.21.15.116:0/3129975975 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c94034440 con 0x7f4c981010e0 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.365+0000 7f4c9f599640 1 -- 172.21.15.116:0/3129975975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c981a2530 con 0x7f4c981010e0 2024-03-04T16:15:34.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.366+0000 7f4c9f599640 1 -- 172.21.15.116:0/3129975975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9804e040 con 0x7f4c981010e0 2024-03-04T16:15:34.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.367+0000 7f4c8e7fc640 1 -- 172.21.15.116:0/3129975975 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f4c94004fb0 con 0x7f4c981010e0 2024-03-04T16:15:34.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.367+0000 7f4c8e7fc640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c8406d8d0 0x7f4c8406fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:34.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.367+0000 7f4c8e7fc640 1 -- 172.21.15.116:0/3129975975 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f4c940af990 con 0x7f4c981010e0 2024-03-04T16:15:34.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.367+0000 7f4c9d335640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c8406d8d0 0x7f4c8406fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:34.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.368+0000 7f4c9d335640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c8406d8d0 0x7f4c8406fd90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4c880058c0 tx=0x7f4c8800a2d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:34.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.372+0000 7f4c8e7fc640 1 -- 172.21.15.116:0/3129975975 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c940791a0 con 0x7f4c981010e0 2024-03-04T16:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.584+0000 7f4c9f599640 1 -- 172.21.15.116:0/3129975975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c981020d0 con 0x7f4c981010e0 2024-03-04T16:15:34.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.585+0000 7f4c8e7fc640 1 -- 172.21.15.116:0/3129975975 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5002 (secure 0 0 0) 0x7f4c9407c860 con 0x7f4c981010e0 2024-03-04T16:15:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:34.585 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":44331,"name":"cephfs.smithi152.tlpshy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6825/3736721533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3736721533},{"type":"v1","addr":"172.21.15.152:6825","nonce":3736721533}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":44335,"name":"cephfs.smithi152.kqoybf","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6827/1659417498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1659417498},{"type":"v1","addr":"172.21.15.152:6827","nonce":1659417498}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:49.225261+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34400":{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":0,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-04T16:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.586+0000 7f4c9f599640 1 -- 172.21.15.116:0/3129975975 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c8406d8d0 msgr2=0x7f4c8406fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.586+0000 7f4c9f599640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c8406d8d0 0x7f4c8406fd90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4c880058c0 tx=0x7f4c8800a2d0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.586+0000 7f4c9f599640 1 -- 172.21.15.116:0/3129975975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c981010e0 msgr2=0x7f4c98101540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.586+0000 7f4c9f599640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c981010e0 0x7f4c98101540 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4c94036040 tx=0x7f4c94004270 comp rx=0 tx=0).stop 2024-03-04T16:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.587+0000 7f4c9f599640 1 -- 172.21.15.116:0/3129975975 shutdown_connections 2024-03-04T16:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.587+0000 7f4c9f599640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4c8406d8d0 0x7f4c8406fd90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.587+0000 7f4c9f599640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c981010e0 0x7f4c98101540 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.587+0000 7f4c9f599640 1 --2- 172.21.15.116:0/3129975975 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c980ff110 0x7f4c98100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.587+0000 7f4c9f599640 1 -- 172.21.15.116:0/3129975975 >> 172.21.15.116:0/3129975975 conn(0x7f4c980fa990 msgr2=0x7f4c98103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:34.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.587+0000 7f4c9f599640 1 -- 172.21.15.116:0/3129975975 shutdown_connections 2024-03-04T16:15:34.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.587+0000 7f4c9f599640 1 -- 172.21.15.116:0/3129975975 wait complete. 2024-03-04T16:15:34.596 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:34 smithi116 ceph-mon[94164]: pgmap v327: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:34.597 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:34 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1056030107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:15:34.597 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:34 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1098416404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-04T16:15:34.597 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:34 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1326732174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-04T16:15:34.598 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 14, 'max_mds': 1, 'flags': 18} 2024-03-04T16:15:34.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 15 2024-03-04T16:15:34.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.700+0000 7f6f64470640 1 -- 172.21.15.116:0/360734753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5c0ff340 msgr2=0x7f6f5c104c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.700+0000 7f6f64470640 1 --2- 172.21.15.116:0/360734753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5c0ff340 0x7f6f5c104c20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009880 tx=0x7f6f4c02bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.700+0000 7f6f64470640 1 -- 172.21.15.116:0/360734753 shutdown_connections 2024-03-04T16:15:34.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.700+0000 7f6f64470640 1 --2- 172.21.15.116:0/360734753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5c0ff340 0x7f6f5c104c20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.700+0000 7f6f64470640 1 --2- 172.21.15.116:0/360734753 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f5c0fe990 0x7f6f5c0fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.700+0000 7f6f64470640 1 -- 172.21.15.116:0/360734753 >> 172.21.15.116:0/360734753 conn(0x7f6f5c075250 msgr2=0x7f6f5c075660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:34.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.700+0000 7f6f64470640 1 -- 172.21.15.116:0/360734753 shutdown_connections 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.700+0000 7f6f64470640 1 -- 172.21.15.116:0/360734753 wait complete. 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f64470640 1 Processor -- start 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f64470640 1 -- start start 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f64470640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5c0fe990 0x7f6f5c195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f64470640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f5c196320 0x7f6f5c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f64470640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c196920 con 0x7f6f5c0fe990 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f64470640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c196a90 con 0x7f6f5c196320 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f6220c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5c0fe990 0x7f6f5c195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f6220c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5c0fe990 0x7f6f5c195de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32798/0 (socket says 172.21.15.116:32798) 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f6220c640 1 -- 172.21.15.116:0/3771625719 learned_addr learned my addr 172.21.15.116:0/3771625719 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f61a0b640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f5c196320 0x7f6f5c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f6220c640 1 -- 172.21.15.116:0/3771625719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f5c196320 msgr2=0x7f6f5c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f6220c640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f5c196320 0x7f6f5c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f6220c640 1 -- 172.21.15.116:0/3771625719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c009560 con 0x7f6f5c0fe990 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f61a0b640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f5c196320 0x7f6f5c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.701+0000 7f6f6220c640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5c0fe990 0x7f6f5c195de0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800c840 tx=0x7f6f5800cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.702+0000 7f6f537fe640 1 -- 172.21.15.116:0/3771625719 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f580041f0 con 0x7f6f5c0fe990 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.702+0000 7f6f537fe640 1 -- 172.21.15.116:0/3771625719 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f58007e70 con 0x7f6f5c0fe990 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.702+0000 7f6f64470640 1 -- 172.21.15.116:0/3771625719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5c19ad80 con 0x7f6f5c0fe990 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.702+0000 7f6f537fe640 1 -- 172.21.15.116:0/3771625719 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f580049d0 con 0x7f6f5c0fe990 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.702+0000 7f6f64470640 1 -- 172.21.15.116:0/3771625719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5c19b2e0 con 0x7f6f5c0fe990 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.703+0000 7f6f64470640 1 -- 172.21.15.116:0/3771625719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5c04e040 con 0x7f6f5c0fe990 2024-03-04T16:15:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.703+0000 7f6f537fe640 1 -- 172.21.15.116:0/3771625719 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f6f580079b0 con 0x7f6f5c0fe990 2024-03-04T16:15:34.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.703+0000 7f6f537fe640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6f4806d810 0x7f6f4806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:34.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.703+0000 7f6f537fe640 1 -- 172.21.15.116:0/3771625719 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f6f5808d340 con 0x7f6f5c0fe990 2024-03-04T16:15:34.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.703+0000 7f6f61a0b640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6f4806d810 0x7f6f4806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:34.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.704+0000 7f6f61a0b640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6f4806d810 0x7f6f4806fcd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c005ec0 tx=0x7f6f4c005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.708+0000 7f6f537fe640 1 -- 172.21.15.116:0/3771625719 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6f58056b20 con 0x7f6f5c0fe990 2024-03-04T16:15:34.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:34 smithi152 ceph-mon[82845]: pgmap v327: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:34.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:34 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1056030107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:15:34.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:34 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1098416404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-04T16:15:34.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:34 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1326732174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-04T16:15:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.915+0000 7f6f64470640 1 -- 172.21.15.116:0/3771625719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 15, "format": "json"} v 0) v1 -- 0x7f6f5c1971f0 con 0x7f6f5c0fe990 2024-03-04T16:15:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.916+0000 7f6f537fe640 1 -- 172.21.15.116:0/3771625719 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 15, "format": "json"}]=0 dumped fsmap epoch 15 v29) v1 ==== 107+0+4052 (secure 0 0 0) 0x7f6f5805a1e0 con 0x7f6f5c0fe990 2024-03-04T16:15:34.917 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:34.917 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.116:6829/3470200572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3470200572},{"type":"v1","addr":"172.21.15.116:6829","nonce":3470200572}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:01:18.395823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24247":{"gid":24247,"name":"cephfs.smithi152.tlpshy","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.152:6825/287206196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":287206196},{"type":"v1","addr":"172.21.15.152:6825","nonce":287206196}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 15 2024-03-04T16:15:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 -- 172.21.15.116:0/3771625719 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6f4806d810 msgr2=0x7f6f4806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6f4806d810 0x7f6f4806fcd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c005ec0 tx=0x7f6f4c005fd0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 -- 172.21.15.116:0/3771625719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5c0fe990 msgr2=0x7f6f5c195de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5c0fe990 0x7f6f5c195de0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800c840 tx=0x7f6f5800cd00 comp rx=0 tx=0).stop 2024-03-04T16:15:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 -- 172.21.15.116:0/3771625719 shutdown_connections 2024-03-04T16:15:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6f4806d810 0x7f6f4806fcd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f5c196320 0x7f6f5c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 --2- 172.21.15.116:0/3771625719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5c0fe990 0x7f6f5c195de0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 -- 172.21.15.116:0/3771625719 >> 172.21.15.116:0/3771625719 conn(0x7f6f5c075250 msgr2=0x7f6f5c0fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 -- 172.21.15.116:0/3771625719 shutdown_connections 2024-03-04T16:15:34.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:34.918+0000 7f6f64470640 1 -- 172.21.15.116:0/3771625719 wait complete. 2024-03-04T16:15:34.930 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 16 2024-03-04T16:15:35.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.030+0000 7f9f25cb6640 1 -- 172.21.15.116:0/2273882428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20067c70 msgr2=0x7f9f2010c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.030+0000 7f9f25cb6640 1 --2- 172.21.15.116:0/2273882428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20067c70 0x7f9f2010c660 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009880 tx=0x7f9f0802bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.030+0000 7f9f25cb6640 1 -- 172.21.15.116:0/2273882428 shutdown_connections 2024-03-04T16:15:35.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.030+0000 7f9f25cb6640 1 --2- 172.21.15.116:0/2273882428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20067c70 0x7f9f2010c660 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.030+0000 7f9f25cb6640 1 --2- 172.21.15.116:0/2273882428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f20067350 0x7f9f20067730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.030+0000 7f9f25cb6640 1 -- 172.21.15.116:0/2273882428 >> 172.21.15.116:0/2273882428 conn(0x7f9f20074a50 msgr2=0x7f9f20074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:35.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.030+0000 7f9f25cb6640 1 -- 172.21.15.116:0/2273882428 shutdown_connections 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.031+0000 7f9f25cb6640 1 -- 172.21.15.116:0/2273882428 wait complete. 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.031+0000 7f9f25cb6640 1 Processor -- start 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.031+0000 7f9f25cb6640 1 -- start start 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.031+0000 7f9f25cb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20067350 0x7f9f20197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.031+0000 7f9f25cb6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f20067c70 0x7f9f201984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.031+0000 7f9f25cb6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f20198b70 con 0x7f9f20067350 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.031+0000 7f9f25cb6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2019c940 con 0x7f9f20067c70 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.031+0000 7f9f1f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20067350 0x7f9f20197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f1f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20067350 0x7f9f20197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32810/0 (socket says 172.21.15.116:32810) 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f1f7fe640 1 -- 172.21.15.116:0/233930795 learned_addr learned my addr 172.21.15.116:0/233930795 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f16ffd640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f20067c70 0x7f9f201984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f1f7fe640 1 -- 172.21.15.116:0/233930795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f20067c70 msgr2=0x7f9f201984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f1f7fe640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f20067c70 0x7f9f201984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f1f7fe640 1 -- 172.21.15.116:0/233930795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f08009560 con 0x7f9f20067350 2024-03-04T16:15:35.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f16ffd640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f20067c70 0x7f9f201984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:15:35.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f1f7fe640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20067350 0x7f9f20197fa0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000c810 tx=0x7f9f1000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:35.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f1d7fa640 1 -- 172.21.15.116:0/233930795 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f100041f0 con 0x7f9f20067350 2024-03-04T16:15:35.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f1d7fa640 1 -- 172.21.15.116:0/233930795 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f10007e70 con 0x7f9f20067350 2024-03-04T16:15:35.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f25cb6640 1 -- 172.21.15.116:0/233930795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2019cc20 con 0x7f9f20067350 2024-03-04T16:15:35.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f1d7fa640 1 -- 172.21.15.116:0/233930795 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f100049d0 con 0x7f9f20067350 2024-03-04T16:15:35.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.032+0000 7f9f25cb6640 1 -- 172.21.15.116:0/233930795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2019d180 con 0x7f9f20067350 2024-03-04T16:15:35.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.033+0000 7f9f25cb6640 1 -- 172.21.15.116:0/233930795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2004e040 con 0x7f9f20067350 2024-03-04T16:15:35.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.034+0000 7f9f1d7fa640 1 -- 172.21.15.116:0/233930795 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f9f100079b0 con 0x7f9f20067350 2024-03-04T16:15:35.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.034+0000 7f9f1d7fa640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9f0c06d810 0x7f9f0c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:35.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.034+0000 7f9f1d7fa640 1 -- 172.21.15.116:0/233930795 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f9f1008d3d0 con 0x7f9f20067350 2024-03-04T16:15:35.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.034+0000 7f9f16ffd640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9f0c06d810 0x7f9f0c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:35.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.034+0000 7f9f16ffd640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9f0c06d810 0x7f9f0c06fcd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9f08005ec0 tx=0x7f9f08005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:35.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.038+0000 7f9f1d7fa640 1 -- 172.21.15.116:0/233930795 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9f10056b30 con 0x7f9f20067350 2024-03-04T16:15:35.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.250+0000 7f9f25cb6640 1 -- 172.21.15.116:0/233930795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 16, "format": "json"} v 0) v1 -- 0x7f9f20109d40 con 0x7f9f20067350 2024-03-04T16:15:35.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.250+0000 7f9f1d7fa640 1 -- 172.21.15.116:0/233930795 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 16, "format": "json"}]=0 dumped fsmap epoch 16 v29) v1 ==== 107+0+4865 (secure 0 0 0) 0x7f9f1005a1f0 con 0x7f9f20067350 2024-03-04T16:15:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:35.251 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.116:6829/3470200572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3470200572},{"type":"v1","addr":"172.21.15.116:6829","nonce":3470200572}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:01:18.395823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24247":{"gid":24247,"name":"cephfs.smithi152.tlpshy","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.152:6825/287206196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":287206196},{"type":"v1","addr":"172.21.15.152:6825","nonce":287206196}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:35.251 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2024-03-04T16:15:35.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 -- 172.21.15.116:0/233930795 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9f0c06d810 msgr2=0x7f9f0c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9f0c06d810 0x7f9f0c06fcd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9f08005ec0 tx=0x7f9f08005fd0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 -- 172.21.15.116:0/233930795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20067350 msgr2=0x7f9f20197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20067350 0x7f9f20197fa0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000c810 tx=0x7f9f1000ccd0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 -- 172.21.15.116:0/233930795 shutdown_connections 2024-03-04T16:15:35.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9f0c06d810 0x7f9f0c06fcd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f20067c70 0x7f9f201984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 --2- 172.21.15.116:0/233930795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20067350 0x7f9f20197fa0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 -- 172.21.15.116:0/233930795 >> 172.21.15.116:0/233930795 conn(0x7f9f20074a50 msgr2=0x7f9f200fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:35.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 -- 172.21.15.116:0/233930795 shutdown_connections 2024-03-04T16:15:35.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.253+0000 7f9f25cb6640 1 -- 172.21.15.116:0/233930795 wait complete. 2024-03-04T16:15:35.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 17 2024-03-04T16:15:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.366+0000 7f6d8e651640 1 -- 172.21.15.116:0/1005751173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d880fe730 msgr2=0x7f6d880feb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.366+0000 7f6d8e651640 1 --2- 172.21.15.116:0/1005751173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d880fe730 0x7f6d880feb70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009880 tx=0x7f6d7c02bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.366+0000 7f6d8e651640 1 -- 172.21.15.116:0/1005751173 shutdown_connections 2024-03-04T16:15:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.366+0000 7f6d8e651640 1 --2- 172.21.15.116:0/1005751173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d880fe730 0x7f6d880feb70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.366+0000 7f6d8e651640 1 --2- 172.21.15.116:0/1005751173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d88106070 0x7f6d88106450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.366+0000 7f6d8e651640 1 -- 172.21.15.116:0/1005751173 >> 172.21.15.116:0/1005751173 conn(0x7f6d88074250 msgr2=0x7f6d88074660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d8e651640 1 -- 172.21.15.116:0/1005751173 shutdown_connections 2024-03-04T16:15:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d8e651640 1 -- 172.21.15.116:0/1005751173 wait complete. 2024-03-04T16:15:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d8e651640 1 Processor -- start 2024-03-04T16:15:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d8e651640 1 -- start start 2024-03-04T16:15:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d8e651640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d880fe730 0x7f6d8819bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d8e651640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d88106070 0x7f6d8819c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d8e651640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d88195b50 con 0x7f6d880fe730 2024-03-04T16:15:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d8e651640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d88195cc0 con 0x7f6d88106070 2024-03-04T16:15:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d880fe730 0x7f6d8819bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d880fe730 0x7f6d8819bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32830/0 (socket says 172.21.15.116:32830) 2024-03-04T16:15:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.367+0000 7f6d87fff640 1 -- 172.21.15.116:0/1313705799 learned_addr learned my addr 172.21.15.116:0/1313705799 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.368+0000 7f6d877fe640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d88106070 0x7f6d8819c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.368+0000 7f6d87fff640 1 -- 172.21.15.116:0/1313705799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d88106070 msgr2=0x7f6d8819c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.368+0000 7f6d87fff640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d88106070 0x7f6d8819c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.368+0000 7f6d87fff640 1 -- 172.21.15.116:0/1313705799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c009560 con 0x7f6d880fe730 2024-03-04T16:15:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.368+0000 7f6d87fff640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d880fe730 0x7f6d8819bae0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800c810 tx=0x7f6d7800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.368+0000 7f6d857fa640 1 -- 172.21.15.116:0/1313705799 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d780041f0 con 0x7f6d880fe730 2024-03-04T16:15:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.368+0000 7f6d857fa640 1 -- 172.21.15.116:0/1313705799 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d78007e70 con 0x7f6d880fe730 2024-03-04T16:15:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.368+0000 7f6d857fa640 1 -- 172.21.15.116:0/1313705799 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d780049d0 con 0x7f6d880fe730 2024-03-04T16:15:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.368+0000 7f6d8e651640 1 -- 172.21.15.116:0/1313705799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d88195fa0 con 0x7f6d880fe730 2024-03-04T16:15:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.368+0000 7f6d8e651640 1 -- 172.21.15.116:0/1313705799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d881964d0 con 0x7f6d880fe730 2024-03-04T16:15:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.369+0000 7f6d8e651640 1 -- 172.21.15.116:0/1313705799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8804e040 con 0x7f6d880fe730 2024-03-04T16:15:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.370+0000 7f6d857fa640 1 -- 172.21.15.116:0/1313705799 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f6d780079b0 con 0x7f6d880fe730 2024-03-04T16:15:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.370+0000 7f6d857fa640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6d7006d810 0x7f6d7006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.370+0000 7f6d857fa640 1 -- 172.21.15.116:0/1313705799 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f6d7808d3f0 con 0x7f6d880fe730 2024-03-04T16:15:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.370+0000 7f6d877fe640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6d7006d810 0x7f6d7006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.371+0000 7f6d877fe640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6d7006d810 0x7f6d7006fcd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6d88196fc0 tx=0x7f6d7c000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.375+0000 7f6d857fa640 1 -- 172.21.15.116:0/1313705799 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6d78056b50 con 0x7f6d880fe730 2024-03-04T16:15:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.584+0000 7f6d8e651640 1 -- 172.21.15.116:0/1313705799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 17, "format": "json"} v 0) v1 -- 0x7f6d88196ff0 con 0x7f6d880fe730 2024-03-04T16:15:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.584+0000 7f6d857fa640 1 -- 172.21.15.116:0/1313705799 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 17, "format": "json"}]=0 dumped fsmap epoch 17 v29) v1 ==== 107+0+4083 (secure 0 0 0) 0x7f6d7805a210 con 0x7f6d880fe730 2024-03-04T16:15:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:35.585 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:01:18.395823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24247":{"gid":24247,"name":"cephfs.smithi152.tlpshy","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.152:6825/287206196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":287206196},{"type":"v1","addr":"172.21.15.152:6825","nonce":287206196}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:35.585 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 17 2024-03-04T16:15:35.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 -- 172.21.15.116:0/1313705799 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6d7006d810 msgr2=0x7f6d7006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6d7006d810 0x7f6d7006fcd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6d88196fc0 tx=0x7f6d7c000bc0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 -- 172.21.15.116:0/1313705799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d880fe730 msgr2=0x7f6d8819bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d880fe730 0x7f6d8819bae0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800c810 tx=0x7f6d7800ccd0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 -- 172.21.15.116:0/1313705799 shutdown_connections 2024-03-04T16:15:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6d7006d810 0x7f6d7006fcd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d88106070 0x7f6d8819c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 --2- 172.21.15.116:0/1313705799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d880fe730 0x7f6d8819bae0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 -- 172.21.15.116:0/1313705799 >> 172.21.15.116:0/1313705799 conn(0x7f6d88074250 msgr2=0x7f6d880fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 -- 172.21.15.116:0/1313705799 shutdown_connections 2024-03-04T16:15:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.586+0000 7f6d8e651640 1 -- 172.21.15.116:0/1313705799 wait complete. 2024-03-04T16:15:35.596 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:35 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3129975975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T16:15:35.597 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:35 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3771625719' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 15, "format": "json"}]: dispatch 2024-03-04T16:15:35.597 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:35 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/233930795' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-03-04T16:15:35.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 18 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.699+0000 7f84c805c640 1 -- 172.21.15.116:0/2723746181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c0072f60 msgr2=0x7f84c0110b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.699+0000 7f84c805c640 1 --2- 172.21.15.116:0/2723746181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c0072f60 0x7f84c0110b00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009880 tx=0x7f84b002bd70 comp rx=0 tx=0).stop 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.699+0000 7f84c805c640 1 -- 172.21.15.116:0/2723746181 shutdown_connections 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.699+0000 7f84c805c640 1 --2- 172.21.15.116:0/2723746181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c0072f60 0x7f84c0110b00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.699+0000 7f84c805c640 1 --2- 172.21.15.116:0/2723746181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84c0072640 0x7f84c0072a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.699+0000 7f84c805c640 1 -- 172.21.15.116:0/2723746181 >> 172.21.15.116:0/2723746181 conn(0x7f84c00fbb20 msgr2=0x7f84c00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.699+0000 7f84c805c640 1 -- 172.21.15.116:0/2723746181 shutdown_connections 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.700+0000 7f84c805c640 1 -- 172.21.15.116:0/2723746181 wait complete. 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.700+0000 7f84c805c640 1 Processor -- start 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.700+0000 7f84c805c640 1 -- start start 2024-03-04T16:15:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.700+0000 7f84c805c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c0072640 0x7f84c019c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.700+0000 7f84c805c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84c0072f60 0x7f84c019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.700+0000 7f84c805c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c019cf20 con 0x7f84c0072640 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.700+0000 7f84c805c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c01a0cf0 con 0x7f84c0072f60 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.700+0000 7f84c5df8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c0072640 0x7f84c019c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.700+0000 7f84c5df8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c0072640 0x7f84c019c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32842/0 (socket says 172.21.15.116:32842) 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.700+0000 7f84c5df8640 1 -- 172.21.15.116:0/3017601777 learned_addr learned my addr 172.21.15.116:0/3017601777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.701+0000 7f84c5df8640 1 -- 172.21.15.116:0/3017601777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84c0072f60 msgr2=0x7f84c019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.701+0000 7f84c5df8640 1 --2- 172.21.15.116:0/3017601777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84c0072f60 0x7f84c019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.701+0000 7f84c5df8640 1 -- 172.21.15.116:0/3017601777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b0009560 con 0x7f84c0072640 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.701+0000 7f84c5df8640 1 --2- 172.21.15.116:0/3017601777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c0072640 0x7f84c019c350 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00d6f0 tx=0x7f84bc00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.701+0000 7f84b6ffd640 1 -- 172.21.15.116:0/3017601777 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84bc009a30 con 0x7f84c0072640 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.701+0000 7f84b6ffd640 1 -- 172.21.15.116:0/3017601777 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84bc0054e0 con 0x7f84c0072640 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.701+0000 7f84c805c640 1 -- 172.21.15.116:0/3017601777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c01a0fd0 con 0x7f84c0072640 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.701+0000 7f84b6ffd640 1 -- 172.21.15.116:0/3017601777 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84bc010460 con 0x7f84c0072640 2024-03-04T16:15:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.701+0000 7f84c805c640 1 -- 172.21.15.116:0/3017601777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c01a1530 con 0x7f84c0072640 2024-03-04T16:15:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.702+0000 7f84c805c640 1 -- 172.21.15.116:0/3017601777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c004e040 con 0x7f84c0072640 2024-03-04T16:15:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.702+0000 7f84b6ffd640 1 -- 172.21.15.116:0/3017601777 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f84bc009b90 con 0x7f84c0072640 2024-03-04T16:15:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.703+0000 7f84b6ffd640 1 --2- 172.21.15.116:0/3017601777 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f84ac06d810 0x7f84ac06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.703+0000 7f84b6ffd640 1 -- 172.21.15.116:0/3017601777 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f84bc08d100 con 0x7f84c0072640 2024-03-04T16:15:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.703+0000 7f84c55f7640 1 --2- 172.21.15.116:0/3017601777 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f84ac06d810 0x7f84ac06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.703+0000 7f84c55f7640 1 --2- 172.21.15.116:0/3017601777 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f84ac06d810 0x7f84ac06fcd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f84b0036040 tx=0x7f84b0000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.707+0000 7f84b6ffd640 1 -- 172.21.15.116:0/3017601777 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f84bc056860 con 0x7f84c0072640 2024-03-04T16:15:35.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:35 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3129975975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T16:15:35.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:35 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3771625719' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 15, "format": "json"}]: dispatch 2024-03-04T16:15:35.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:35 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/233930795' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-03-04T16:15:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.915+0000 7f84c805c640 1 -- 172.21.15.116:0/3017601777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 18, "format": "json"} v 0) v1 -- 0x7f84c010e1f0 con 0x7f84c0072640 2024-03-04T16:15:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.916+0000 7f84b6ffd640 1 -- 172.21.15.116:0/3017601777 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 18, "format": "json"}]=0 dumped fsmap epoch 18 v29) v1 ==== 107+0+4896 (secure 0 0 0) 0x7f84bc059f20 con 0x7f84c0072640 2024-03-04T16:15:35.917 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:35.917 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:01:18.395823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24247":{"gid":24247,"name":"cephfs.smithi152.tlpshy","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.152:6825/287206196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":287206196},{"type":"v1","addr":"172.21.15.152:6825","nonce":287206196}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:35.917 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 18 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 -- 172.21.15.116:0/3017601777 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f84ac06d810 msgr2=0x7f84ac06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 --2- 172.21.15.116:0/3017601777 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f84ac06d810 0x7f84ac06fcd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f84b0036040 tx=0x7f84b0000bc0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 -- 172.21.15.116:0/3017601777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c0072640 msgr2=0x7f84c019c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 --2- 172.21.15.116:0/3017601777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c0072640 0x7f84c019c350 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00d6f0 tx=0x7f84bc00dbb0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 -- 172.21.15.116:0/3017601777 shutdown_connections 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 --2- 172.21.15.116:0/3017601777 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f84ac06d810 0x7f84ac06fcd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 --2- 172.21.15.116:0/3017601777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84c0072f60 0x7f84c019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 --2- 172.21.15.116:0/3017601777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c0072640 0x7f84c019c350 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 -- 172.21.15.116:0/3017601777 >> 172.21.15.116:0/3017601777 conn(0x7f84c00fbb20 msgr2=0x7f84c0102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 -- 172.21.15.116:0/3017601777 shutdown_connections 2024-03-04T16:15:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:35.918+0000 7f84c805c640 1 -- 172.21.15.116:0/3017601777 wait complete. 2024-03-04T16:15:35.929 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-03-04T16:15:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.031+0000 7f1613041640 1 -- 172.21.15.116:0/1426498944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160c100460 msgr2=0x7f160c103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.031+0000 7f1613041640 1 --2- 172.21.15.116:0/1426498944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160c100460 0x7f160c103c10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009880 tx=0x7f15f802bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.031+0000 7f1613041640 1 -- 172.21.15.116:0/1426498944 shutdown_connections 2024-03-04T16:15:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.031+0000 7f1613041640 1 --2- 172.21.15.116:0/1426498944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160c100460 0x7f160c103c10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.031+0000 7f1613041640 1 --2- 172.21.15.116:0/1426498944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f160c0ffab0 0x7f160c0ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.031+0000 7f1613041640 1 -- 172.21.15.116:0/1426498944 >> 172.21.15.116:0/1426498944 conn(0x7f160c074250 msgr2=0x7f160c074660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.031+0000 7f1613041640 1 -- 172.21.15.116:0/1426498944 shutdown_connections 2024-03-04T16:15:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.031+0000 7f1613041640 1 -- 172.21.15.116:0/1426498944 wait complete. 2024-03-04T16:15:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f1613041640 1 Processor -- start 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f1613041640 1 -- start start 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f1613041640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f160c0ffab0 0x7f160c195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f1613041640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160c196340 0x7f160c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f1613041640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c196940 con 0x7f160c196340 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f1613041640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c196ab0 con 0x7f160c0ffab0 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f160bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160c196340 0x7f160c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f160bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160c196340 0x7f160c19a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32860/0 (socket says 172.21.15.116:32860) 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f160bfff640 1 -- 172.21.15.116:0/4037991077 learned_addr learned my addr 172.21.15.116:0/4037991077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f1610ddd640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f160c0ffab0 0x7f160c195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.032+0000 7f160bfff640 1 -- 172.21.15.116:0/4037991077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f160c0ffab0 msgr2=0x7f160c195e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.033+0000 7f160bfff640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f160c0ffab0 0x7f160c195e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.033+0000 7f160bfff640 1 -- 172.21.15.116:0/4037991077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f8009560 con 0x7f160c196340 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.033+0000 7f1610ddd640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f160c0ffab0 0x7f160c195e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.033+0000 7f160bfff640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160c196340 0x7f160c19a7a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f15f8036040 tx=0x7f15f802c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:36.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.033+0000 7f1609ffb640 1 -- 172.21.15.116:0/4037991077 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15f80035f0 con 0x7f160c196340 2024-03-04T16:15:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.033+0000 7f1609ffb640 1 -- 172.21.15.116:0/4037991077 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15f8003c10 con 0x7f160c196340 2024-03-04T16:15:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.033+0000 7f1609ffb640 1 -- 172.21.15.116:0/4037991077 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15f8034440 con 0x7f160c196340 2024-03-04T16:15:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.033+0000 7f1613041640 1 -- 172.21.15.116:0/4037991077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160c19ad40 con 0x7f160c196340 2024-03-04T16:15:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.033+0000 7f1613041640 1 -- 172.21.15.116:0/4037991077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f160c19b2a0 con 0x7f160c196340 2024-03-04T16:15:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.034+0000 7f1613041640 1 -- 172.21.15.116:0/4037991077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160c04e040 con 0x7f160c196340 2024-03-04T16:15:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.035+0000 7f1609ffb640 1 -- 172.21.15.116:0/4037991077 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f15f8003750 con 0x7f160c196340 2024-03-04T16:15:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.035+0000 7f1609ffb640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f15fc06d810 0x7f15fc06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.035+0000 7f1610ddd640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f15fc06d810 0x7f15fc06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.035+0000 7f1609ffb640 1 -- 172.21.15.116:0/4037991077 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f15f80af930 con 0x7f160c196340 2024-03-04T16:15:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.035+0000 7f1610ddd640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f15fc06d810 0x7f15fc06fcd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1600009b40 tx=0x7f1600005e50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:36.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.040+0000 7f1609ffb640 1 -- 172.21.15.116:0/4037991077 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f15f8079090 con 0x7f160c196340 2024-03-04T16:15:36.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.249+0000 7f1613041640 1 -- 172.21.15.116:0/4037991077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7f160c19b5e0 con 0x7f160c196340 2024-03-04T16:15:36.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.250+0000 7f1609ffb640 1 -- 172.21.15.116:0/4037991077 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v29) v1 ==== 107+0+4106 (secure 0 0 0) 0x7f15f8081e60 con 0x7f160c196340 2024-03-04T16:15:36.251 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:36.251 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:14.444381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 -- 172.21.15.116:0/4037991077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f15fc06d810 msgr2=0x7f15fc06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f15fc06d810 0x7f15fc06fcd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1600009b40 tx=0x7f1600005e50 comp rx=0 tx=0).stop 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 -- 172.21.15.116:0/4037991077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160c196340 msgr2=0x7f160c19a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160c196340 0x7f160c19a7a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f15f8036040 tx=0x7f15f802c850 comp rx=0 tx=0).stop 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 -- 172.21.15.116:0/4037991077 shutdown_connections 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f15fc06d810 0x7f15fc06fcd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160c196340 0x7f160c19a7a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 --2- 172.21.15.116:0/4037991077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f160c0ffab0 0x7f160c195e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 -- 172.21.15.116:0/4037991077 >> 172.21.15.116:0/4037991077 conn(0x7f160c074250 msgr2=0x7f160c0fc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 -- 172.21.15.116:0/4037991077 shutdown_connections 2024-03-04T16:15:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.252+0000 7f1613041640 1 -- 172.21.15.116:0/4037991077 wait complete. 2024-03-04T16:15:36.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-03-04T16:15:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.365+0000 7f4a33ed2640 1 -- 172.21.15.116:0/4140686234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2c072f70 msgr2=0x7f4a2c10c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.365+0000 7f4a33ed2640 1 --2- 172.21.15.116:0/4140686234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2c072f70 0x7f4a2c10c3f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4a20009880 tx=0x7f4a2002bd70 comp rx=0 tx=0).stop 2024-03-04T16:15:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.365+0000 7f4a33ed2640 1 -- 172.21.15.116:0/4140686234 shutdown_connections 2024-03-04T16:15:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.365+0000 7f4a33ed2640 1 --2- 172.21.15.116:0/4140686234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2c072f70 0x7f4a2c10c3f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.365+0000 7f4a33ed2640 1 --2- 172.21.15.116:0/4140686234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a2c072650 0x7f4a2c072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.365+0000 7f4a33ed2640 1 -- 172.21.15.116:0/4140686234 >> 172.21.15.116:0/4140686234 conn(0x7f4a2c0fbb80 msgr2=0x7f4a2c0fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a33ed2640 1 -- 172.21.15.116:0/4140686234 shutdown_connections 2024-03-04T16:15:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a33ed2640 1 -- 172.21.15.116:0/4140686234 wait complete. 2024-03-04T16:15:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a33ed2640 1 Processor -- start 2024-03-04T16:15:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a33ed2640 1 -- start start 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a33ed2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a2c072650 0x7f4a2c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a33ed2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2c072f70 0x7f4a2c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a33ed2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c198b20 con 0x7f4a2c072f70 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a33ed2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c19c8f0 con 0x7f4a2c072650 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a3146d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2c072f70 0x7f4a2c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a3146d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2c072f70 0x7f4a2c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32882/0 (socket says 172.21.15.116:32882) 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.366+0000 7f4a3146d640 1 -- 172.21.15.116:0/3995229244 learned_addr learned my addr 172.21.15.116:0/3995229244 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.367+0000 7f4a31c6e640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a2c072650 0x7f4a2c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.367+0000 7f4a3146d640 1 -- 172.21.15.116:0/3995229244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a2c072650 msgr2=0x7f4a2c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.367+0000 7f4a3146d640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a2c072650 0x7f4a2c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.367+0000 7f4a3146d640 1 -- 172.21.15.116:0/3995229244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a20009560 con 0x7f4a2c072f70 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.367+0000 7f4a3146d640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2c072f70 0x7f4a2c198490 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4a20009230 tx=0x7f4a20004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.367+0000 7f4a2affd640 1 -- 172.21.15.116:0/3995229244 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a200037c0 con 0x7f4a2c072f70 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.367+0000 7f4a33ed2640 1 -- 172.21.15.116:0/3995229244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c19cb70 con 0x7f4a2c072f70 2024-03-04T16:15:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.367+0000 7f4a2affd640 1 -- 172.21.15.116:0/3995229244 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a20004aa0 con 0x7f4a2c072f70 2024-03-04T16:15:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.367+0000 7f4a2affd640 1 -- 172.21.15.116:0/3995229244 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a2003e450 con 0x7f4a2c072f70 2024-03-04T16:15:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.367+0000 7f4a33ed2640 1 -- 172.21.15.116:0/3995229244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c19d0d0 con 0x7f4a2c072f70 2024-03-04T16:15:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.368+0000 7f4a33ed2640 1 -- 172.21.15.116:0/3995229244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c04e040 con 0x7f4a2c072f70 2024-03-04T16:15:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.369+0000 7f4a2affd640 1 -- 172.21.15.116:0/3995229244 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f4a20005060 con 0x7f4a2c072f70 2024-03-04T16:15:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.369+0000 7f4a2affd640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4a1406d810 0x7f4a1406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.369+0000 7f4a2affd640 1 -- 172.21.15.116:0/3995229244 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f4a200b03e0 con 0x7f4a2c072f70 2024-03-04T16:15:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.369+0000 7f4a31c6e640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4a1406d810 0x7f4a1406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.369+0000 7f4a31c6e640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4a1406d810 0x7f4a1406fcd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c005c40 tx=0x7f4a1c00a390 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.373+0000 7f4a2affd640 1 -- 172.21.15.116:0/3995229244 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a20079b40 con 0x7f4a2c072f70 2024-03-04T16:15:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.583+0000 7f4a33ed2640 1 -- 172.21.15.116:0/3995229244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7f4a2c10f510 con 0x7f4a2c072f70 2024-03-04T16:15:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.584+0000 7f4a2affd640 1 -- 172.21.15.116:0/3995229244 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v29) v1 ==== 107+0+4117 (secure 0 0 0) 0x7f4a2007d200 con 0x7f4a2c072f70 2024-03-04T16:15:36.585 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:36.585 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:14.449915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24257":{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":0,"incarnation":20,"state":"up:replay","state_seq":2,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:36.585 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 20 2024-03-04T16:15:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 -- 172.21.15.116:0/3995229244 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4a1406d810 msgr2=0x7f4a1406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4a1406d810 0x7f4a1406fcd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c005c40 tx=0x7f4a1c00a390 comp rx=0 tx=0).stop 2024-03-04T16:15:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 -- 172.21.15.116:0/3995229244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2c072f70 msgr2=0x7f4a2c198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2c072f70 0x7f4a2c198490 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4a20009230 tx=0x7f4a20004270 comp rx=0 tx=0).stop 2024-03-04T16:15:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 -- 172.21.15.116:0/3995229244 shutdown_connections 2024-03-04T16:15:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f4a1406d810 0x7f4a1406fcd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2c072f70 0x7f4a2c198490 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 --2- 172.21.15.116:0/3995229244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a2c072650 0x7f4a2c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 -- 172.21.15.116:0/3995229244 >> 172.21.15.116:0/3995229244 conn(0x7f4a2c0fbb80 msgr2=0x7f4a2c106c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:36.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 -- 172.21.15.116:0/3995229244 shutdown_connections 2024-03-04T16:15:36.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.586+0000 7f4a33ed2640 1 -- 172.21.15.116:0/3995229244 wait complete. 2024-03-04T16:15:36.596 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:36 smithi116 ceph-mon[94164]: pgmap v328: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:36.596 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:36 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1313705799' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-03-04T16:15:36.596 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:36 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3017601777' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-03-04T16:15:36.597 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:36 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/4037991077' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-03-04T16:15:36.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-03-04T16:15:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.699+0000 7f17c409d640 1 -- 172.21.15.116:0/3092514748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17bc0ff260 msgr2=0x7f17bc0ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.699+0000 7f17c409d640 1 --2- 172.21.15.116:0/3092514748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17bc0ff260 0x7f17bc0ff640 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f17ac009c20 tx=0x7f17ac02c1b0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.699+0000 7f17c409d640 1 -- 172.21.15.116:0/3092514748 shutdown_connections 2024-03-04T16:15:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.699+0000 7f17c409d640 1 --2- 172.21.15.116:0/3092514748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17bc0ffc10 0x7f17bc103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.699+0000 7f17c409d640 1 --2- 172.21.15.116:0/3092514748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17bc0ff260 0x7f17bc0ff640 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.699+0000 7f17c409d640 1 -- 172.21.15.116:0/3092514748 >> 172.21.15.116:0/3092514748 conn(0x7f17bc074a50 msgr2=0x7f17bc074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c409d640 1 -- 172.21.15.116:0/3092514748 shutdown_connections 2024-03-04T16:15:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c409d640 1 -- 172.21.15.116:0/3092514748 wait complete. 2024-03-04T16:15:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c409d640 1 Processor -- start 2024-03-04T16:15:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c409d640 1 -- start start 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c409d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17bc0ff260 0x7f17bc195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c409d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17bc1962f0 0x7f17bc19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c409d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc1968f0 con 0x7f17bc0ff260 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c409d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc196a60 con 0x7f17bc1962f0 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c1e39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17bc0ff260 0x7f17bc195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c1e39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17bc0ff260 0x7f17bc195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32904/0 (socket says 172.21.15.116:32904) 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.700+0000 7f17c1e39640 1 -- 172.21.15.116:0/2826441595 learned_addr learned my addr 172.21.15.116:0/2826441595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.701+0000 7f17c1e39640 1 -- 172.21.15.116:0/2826441595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17bc1962f0 msgr2=0x7f17bc19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.701+0000 7f17c1e39640 1 --2- 172.21.15.116:0/2826441595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17bc1962f0 0x7f17bc19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.701+0000 7f17c1e39640 1 -- 172.21.15.116:0/2826441595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac009900 con 0x7f17bc0ff260 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.701+0000 7f17c1e39640 1 --2- 172.21.15.116:0/2826441595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17bc0ff260 0x7f17bc195db0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f17ac006010 tx=0x7f17ac032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.701+0000 7f17baffd640 1 -- 172.21.15.116:0/2826441595 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17ac034820 con 0x7f17bc0ff260 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.701+0000 7f17baffd640 1 -- 172.21.15.116:0/2826441595 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17ac003e30 con 0x7f17bc0ff260 2024-03-04T16:15:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.701+0000 7f17baffd640 1 -- 172.21.15.116:0/2826441595 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17ac004830 con 0x7f17bc0ff260 2024-03-04T16:15:36.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.701+0000 7f17c409d640 1 -- 172.21.15.116:0/2826441595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17bc19acf0 con 0x7f17bc0ff260 2024-03-04T16:15:36.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.701+0000 7f17c409d640 1 -- 172.21.15.116:0/2826441595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17bc19b250 con 0x7f17bc0ff260 2024-03-04T16:15:36.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.702+0000 7f17c409d640 1 -- 172.21.15.116:0/2826441595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17bc04e040 con 0x7f17bc0ff260 2024-03-04T16:15:36.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.702+0000 7f17baffd640 1 -- 172.21.15.116:0/2826441595 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f17ac03b040 con 0x7f17bc0ff260 2024-03-04T16:15:36.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.703+0000 7f17baffd640 1 --2- 172.21.15.116:0/2826441595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f17a406d810 0x7f17a406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:36.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.703+0000 7f17baffd640 1 -- 172.21.15.116:0/2826441595 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f17ac0b03e0 con 0x7f17bc0ff260 2024-03-04T16:15:36.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.703+0000 7f17c1638640 1 --2- 172.21.15.116:0/2826441595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f17a406d810 0x7f17a406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:36.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.703+0000 7f17c1638640 1 --2- 172.21.15.116:0/2826441595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f17a406d810 0x7f17a406fcd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f17b0005fd0 tx=0x7f17b0005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.707+0000 7f17baffd640 1 -- 172.21.15.116:0/2826441595 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f17ac079ac0 con 0x7f17bc0ff260 2024-03-04T16:15:36.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:36 smithi152 ceph-mon[82845]: pgmap v328: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:36.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:36 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1313705799' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-03-04T16:15:36.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:36 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3017601777' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-03-04T16:15:36.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:36 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/4037991077' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-03-04T16:15:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.915+0000 7f17c409d640 1 -- 172.21.15.116:0/2826441595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7f17bc0ffc10 con 0x7f17bc0ff260 2024-03-04T16:15:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.916+0000 7f17baffd640 1 -- 172.21.15.116:0/2826441595 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v29) v1 ==== 107+0+4122 (secure 0 0 0) 0x7f17ac07d180 con 0x7f17bc0ff260 2024-03-04T16:15:36.916 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:36.916 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:23.420528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24257":{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":0,"incarnation":20,"state":"up:reconnect","state_seq":154,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 21 2024-03-04T16:15:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 -- 172.21.15.116:0/2826441595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f17a406d810 msgr2=0x7f17a406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 --2- 172.21.15.116:0/2826441595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f17a406d810 0x7f17a406fcd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f17b0005fd0 tx=0x7f17b0005850 comp rx=0 tx=0).stop 2024-03-04T16:15:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 -- 172.21.15.116:0/2826441595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17bc0ff260 msgr2=0x7f17bc195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 --2- 172.21.15.116:0/2826441595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17bc0ff260 0x7f17bc195db0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f17ac006010 tx=0x7f17ac032b60 comp rx=0 tx=0).stop 2024-03-04T16:15:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 -- 172.21.15.116:0/2826441595 shutdown_connections 2024-03-04T16:15:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 --2- 172.21.15.116:0/2826441595 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f17a406d810 0x7f17a406fcd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 --2- 172.21.15.116:0/2826441595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17bc1962f0 0x7f17bc19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 --2- 172.21.15.116:0/2826441595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17bc0ff260 0x7f17bc195db0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 -- 172.21.15.116:0/2826441595 >> 172.21.15.116:0/2826441595 conn(0x7f17bc074a50 msgr2=0x7f17bc1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 -- 172.21.15.116:0/2826441595 shutdown_connections 2024-03-04T16:15:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:36.918+0000 7f17c409d640 1 -- 172.21.15.116:0/2826441595 wait complete. 2024-03-04T16:15:36.929 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-03-04T16:15:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.030+0000 7fdfd79dd640 1 -- 172.21.15.116:0/2798753508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd00fe730 msgr2=0x7fdfd00feb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.030+0000 7fdfd79dd640 1 --2- 172.21.15.116:0/2798753508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd00fe730 0x7fdfd00feb70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000a990 tx=0x7fdfc002d010 comp rx=0 tx=0).stop 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.030+0000 7fdfd79dd640 1 -- 172.21.15.116:0/2798753508 shutdown_connections 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.030+0000 7fdfd79dd640 1 --2- 172.21.15.116:0/2798753508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd00fe730 0x7fdfd00feb70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.030+0000 7fdfd79dd640 1 --2- 172.21.15.116:0/2798753508 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfd01046f0 0x7fdfd0104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.030+0000 7fdfd79dd640 1 -- 172.21.15.116:0/2798753508 >> 172.21.15.116:0/2798753508 conn(0x7fdfd0074250 msgr2=0x7fdfd0074660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.030+0000 7fdfd79dd640 1 -- 172.21.15.116:0/2798753508 shutdown_connections 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.030+0000 7fdfd79dd640 1 -- 172.21.15.116:0/2798753508 wait complete. 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd79dd640 1 Processor -- start 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd79dd640 1 -- start start 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd79dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd00fe730 0x7fdfd0195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd79dd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfd01046f0 0x7fdfd01962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd79dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd0196980 con 0x7fdfd00fe730 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd79dd640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd019a750 con 0x7fdfd01046f0 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd5779640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd00fe730 0x7fdfd0195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd5779640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd00fe730 0x7fdfd0195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32918/0 (socket says 172.21.15.116:32918) 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd5779640 1 -- 172.21.15.116:0/1863118695 learned_addr learned my addr 172.21.15.116:0/1863118695 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd5779640 1 -- 172.21.15.116:0/1863118695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfd01046f0 msgr2=0x7fdfd01962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd5779640 1 --2- 172.21.15.116:0/1863118695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfd01046f0 0x7fdfd01962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd5779640 1 -- 172.21.15.116:0/1863118695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc000a670 con 0x7fdfd00fe730 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.031+0000 7fdfd5779640 1 --2- 172.21.15.116:0/1863118695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd00fe730 0x7fdfd0195db0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00d6f0 tx=0x7fdfcc00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.032+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1863118695 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfcc009a30 con 0x7fdfd00fe730 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.032+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1863118695 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfcc0054e0 con 0x7fdfd00fe730 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.032+0000 7fdfd79dd640 1 -- 172.21.15.116:0/1863118695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd019aa30 con 0x7fdfd00fe730 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.032+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1863118695 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfcc010460 con 0x7fdfd00fe730 2024-03-04T16:15:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.032+0000 7fdfd79dd640 1 -- 172.21.15.116:0/1863118695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd019af90 con 0x7fdfd00fe730 2024-03-04T16:15:37.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.033+0000 7fdfd79dd640 1 -- 172.21.15.116:0/1863118695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd004e040 con 0x7fdfd00fe730 2024-03-04T16:15:37.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.033+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1863118695 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fdfcc009b90 con 0x7fdfd00fe730 2024-03-04T16:15:37.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.034+0000 7fdfc67fc640 1 --2- 172.21.15.116:0/1863118695 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdfbc06d810 0x7fdfbc06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:37.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.034+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1863118695 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fdfcc08d100 con 0x7fdfd00fe730 2024-03-04T16:15:37.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.034+0000 7fdfd4f78640 1 --2- 172.21.15.116:0/1863118695 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdfbc06d810 0x7fdfbc06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:37.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.034+0000 7fdfd4f78640 1 --2- 172.21.15.116:0/1863118695 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdfbc06d810 0x7fdfbc06fcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0037040 tx=0x7fdfc00058d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:37.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.038+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1863118695 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdfcc056860 con 0x7fdfd00fe730 2024-03-04T16:15:37.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.245+0000 7fdfd79dd640 1 -- 172.21.15.116:0/1863118695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7fdfd0108bf0 con 0x7fdfd00fe730 2024-03-04T16:15:37.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.245+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1863118695 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v29) v1 ==== 107+0+4119 (secure 0 0 0) 0x7fdfcc059f20 con 0x7fdfd00fe730 2024-03-04T16:15:37.246 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:37.246 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:24.428265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24257":{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":155,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:37.246 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 22 2024-03-04T16:15:37.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.247+0000 7fdfd79dd640 1 -- 172.21.15.116:0/1863118695 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdfbc06d810 msgr2=0x7fdfbc06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.247+0000 7fdfd79dd640 1 --2- 172.21.15.116:0/1863118695 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdfbc06d810 0x7fdfbc06fcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0037040 tx=0x7fdfc00058d0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.247+0000 7fdfd79dd640 1 -- 172.21.15.116:0/1863118695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd00fe730 msgr2=0x7fdfd0195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.247+0000 7fdfd79dd640 1 --2- 172.21.15.116:0/1863118695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd00fe730 0x7fdfd0195db0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00d6f0 tx=0x7fdfcc00dbb0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.247+0000 7fdfd79dd640 1 -- 172.21.15.116:0/1863118695 shutdown_connections 2024-03-04T16:15:37.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.247+0000 7fdfd79dd640 1 --2- 172.21.15.116:0/1863118695 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fdfbc06d810 0x7fdfbc06fcd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.247+0000 7fdfd79dd640 1 --2- 172.21.15.116:0/1863118695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfd01046f0 0x7fdfd01962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.247+0000 7fdfd79dd640 1 --2- 172.21.15.116:0/1863118695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd00fe730 0x7fdfd0195db0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.247+0000 7fdfd79dd640 1 -- 172.21.15.116:0/1863118695 >> 172.21.15.116:0/1863118695 conn(0x7fdfd0074250 msgr2=0x7fdfd00fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:37.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.248+0000 7fdfd79dd640 1 -- 172.21.15.116:0/1863118695 shutdown_connections 2024-03-04T16:15:37.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.248+0000 7fdfd79dd640 1 -- 172.21.15.116:0/1863118695 wait complete. 2024-03-04T16:15:37.259 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-03-04T16:15:37.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.362+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3194331985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59bc0ffc10 msgr2=0x7f59bc103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.362+0000 7f59c3ec4640 1 --2- 172.21.15.116:0/3194331985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59bc0ffc10 0x7f59bc103b70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009880 tx=0x7f59b802bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3194331985 shutdown_connections 2024-03-04T16:15:37.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 --2- 172.21.15.116:0/3194331985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59bc0ffc10 0x7f59bc103b70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 --2- 172.21.15.116:0/3194331985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59bc0ff260 0x7f59bc0ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3194331985 >> 172.21.15.116:0/3194331985 conn(0x7f59bc074a50 msgr2=0x7f59bc074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3194331985 shutdown_connections 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3194331985 wait complete. 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 Processor -- start 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 -- start start 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59bc0ff260 0x7f59bc195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59bc196310 0x7f59bc19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc196910 con 0x7f59bc196310 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.363+0000 7f59c3ec4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc196a80 con 0x7f59bc0ff260 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59c145f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59bc196310 0x7f59bc19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59c145f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59bc196310 0x7f59bc19a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32930/0 (socket says 172.21.15.116:32930) 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59c145f640 1 -- 172.21.15.116:0/3032604484 learned_addr learned my addr 172.21.15.116:0/3032604484 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59c1c60640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59bc0ff260 0x7f59bc195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59c145f640 1 -- 172.21.15.116:0/3032604484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59bc0ff260 msgr2=0x7f59bc195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59c145f640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59bc0ff260 0x7f59bc195dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59c145f640 1 -- 172.21.15.116:0/3032604484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8009560 con 0x7f59bc196310 2024-03-04T16:15:37.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59c145f640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59bc196310 0x7f59bc19a770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f59b8036040 tx=0x7f59b802c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:37.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59b2ffd640 1 -- 172.21.15.116:0/3032604484 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59b80035f0 con 0x7f59bc196310 2024-03-04T16:15:37.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59b2ffd640 1 -- 172.21.15.116:0/3032604484 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59b8003c10 con 0x7f59bc196310 2024-03-04T16:15:37.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3032604484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59bc19ad10 con 0x7f59bc196310 2024-03-04T16:15:37.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59b2ffd640 1 -- 172.21.15.116:0/3032604484 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59b8005630 con 0x7f59bc196310 2024-03-04T16:15:37.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.364+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3032604484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59bc19b270 con 0x7f59bc196310 2024-03-04T16:15:37.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.365+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3032604484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59bc04e040 con 0x7f59bc196310 2024-03-04T16:15:37.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.366+0000 7f59b2ffd640 1 -- 172.21.15.116:0/3032604484 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f59b8003750 con 0x7f59bc196310 2024-03-04T16:15:37.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.366+0000 7f59b2ffd640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f59a806d810 0x7f59a806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:37.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.366+0000 7f59b2ffd640 1 -- 172.21.15.116:0/3032604484 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f59b80af790 con 0x7f59bc196310 2024-03-04T16:15:37.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.366+0000 7f59c1c60640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f59a806d810 0x7f59a806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:37.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.367+0000 7f59c1c60640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f59a806d810 0x7f59a806fcd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f59ac009bd0 tx=0x7f59ac005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:37.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.371+0000 7f59b2ffd640 1 -- 172.21.15.116:0/3032604484 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f59b8079020 con 0x7f59bc196310 2024-03-04T16:15:37.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.588+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3032604484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f59bc197120 con 0x7f59bc196310 2024-03-04T16:15:37.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.589+0000 7f59b2ffd640 1 -- 172.21.15.116:0/3032604484 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v29) v1 ==== 107+0+4119 (secure 0 0 0) 0x7f59b807c6e0 con 0x7f59bc196310 2024-03-04T16:15:37.599 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:37.599 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:25.515194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24257":{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":0,"incarnation":20,"state":"up:active","state_seq":156,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:37.600 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 23 2024-03-04T16:15:37.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3032604484 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f59a806d810 msgr2=0x7f59a806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f59a806d810 0x7f59a806fcd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f59ac009bd0 tx=0x7f59ac005f90 comp rx=0 tx=0).stop 2024-03-04T16:15:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3032604484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59bc196310 msgr2=0x7f59bc19a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59bc196310 0x7f59bc19a770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f59b8036040 tx=0x7f59b802c750 comp rx=0 tx=0).stop 2024-03-04T16:15:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3032604484 shutdown_connections 2024-03-04T16:15:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f59a806d810 0x7f59a806fcd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59bc196310 0x7f59bc19a770 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 --2- 172.21.15.116:0/3032604484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59bc0ff260 0x7f59bc195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3032604484 >> 172.21.15.116:0/3032604484 conn(0x7f59bc074a50 msgr2=0x7f59bc0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3032604484 shutdown_connections 2024-03-04T16:15:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.591+0000 7f59c3ec4640 1 -- 172.21.15.116:0/3032604484 wait complete. 2024-03-04T16:15:37.602 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:37 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3995229244' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-03-04T16:15:37.602 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:37 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2826441595' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-04T16:15:37.602 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:37 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1863118695' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-04T16:15:37.602 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:37 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:15:37.603 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-03-04T16:15:37.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.709+0000 7f86d3aca640 1 -- 172.21.15.116:0/629606101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86cc0675c0 msgr2=0x7f86cc0679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.709+0000 7f86d3aca640 1 --2- 172.21.15.116:0/629606101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86cc0675c0 0x7f86cc0679a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009880 tx=0x7f86c802bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.709+0000 7f86d3aca640 1 -- 172.21.15.116:0/629606101 shutdown_connections 2024-03-04T16:15:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.709+0000 7f86d3aca640 1 --2- 172.21.15.116:0/629606101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86cc067f70 0x7f86cc104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.709+0000 7f86d3aca640 1 --2- 172.21.15.116:0/629606101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86cc0675c0 0x7f86cc0679a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.709+0000 7f86d3aca640 1 -- 172.21.15.116:0/629606101 >> 172.21.15.116:0/629606101 conn(0x7f86cc074a50 msgr2=0x7f86cc074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.709+0000 7f86d3aca640 1 -- 172.21.15.116:0/629606101 shutdown_connections 2024-03-04T16:15:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.709+0000 7f86d3aca640 1 -- 172.21.15.116:0/629606101 wait complete. 2024-03-04T16:15:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.709+0000 7f86d3aca640 1 Processor -- start 2024-03-04T16:15:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.709+0000 7f86d3aca640 1 -- start start 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d3aca640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86cc0675c0 0x7f86cc195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d3aca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86cc1962d0 0x7f86cc19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d3aca640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc1968d0 con 0x7f86cc1962d0 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d3aca640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc196a40 con 0x7f86cc0675c0 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d1065640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86cc1962d0 0x7f86cc19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d1065640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86cc1962d0 0x7f86cc19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32956/0 (socket says 172.21.15.116:32956) 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d1065640 1 -- 172.21.15.116:0/2675786982 learned_addr learned my addr 172.21.15.116:0/2675786982 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d1866640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86cc0675c0 0x7f86cc195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d1065640 1 -- 172.21.15.116:0/2675786982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86cc0675c0 msgr2=0x7f86cc195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d1065640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86cc0675c0 0x7f86cc195d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d1065640 1 -- 172.21.15.116:0/2675786982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8009560 con 0x7f86cc1962d0 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d1065640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86cc1962d0 0x7f86cc19a730 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00e970 tx=0x7f86bc00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2675786982 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86bc009700 con 0x7f86cc1962d0 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86d3aca640 1 -- 172.21.15.116:0/2675786982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86cc19ad30 con 0x7f86cc1962d0 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2675786982 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86bc0054e0 con 0x7f86cc1962d0 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.710+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2675786982 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86bc010430 con 0x7f86cc1962d0 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.711+0000 7f86d3aca640 1 -- 172.21.15.116:0/2675786982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86cc19b290 con 0x7f86cc1962d0 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.712+0000 7f86d3aca640 1 -- 172.21.15.116:0/2675786982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86cc19b570 con 0x7f86cc1962d0 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.712+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2675786982 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f86bc00cc90 con 0x7f86cc1962d0 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.713+0000 7f86c2ffd640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86b806d810 0x7f86b806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.713+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2675786982 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f86bc014070 con 0x7f86cc1962d0 2024-03-04T16:15:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.713+0000 7f86d1866640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86b806d810 0x7f86b806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:37.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.713+0000 7f86d1866640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86b806d810 0x7f86b806fcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f86c8006010 tx=0x7f86c8005dd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.717+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2675786982 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f86bc0571a0 con 0x7f86cc1962d0 2024-03-04T16:15:37.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:37 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3995229244' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-03-04T16:15:37.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:37 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2826441595' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-04T16:15:37.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:37 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1863118695' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-04T16:15:37.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:37 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:15:37.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.937+0000 7f86d3aca640 1 -- 172.21.15.116:0/2675786982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f86cc1970e0 con 0x7f86cc1962d0 2024-03-04T16:15:37.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.938+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2675786982 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v29) v1 ==== 107+0+4935 (secure 0 0 0) 0x7f86bc05a860 con 0x7f86cc1962d0 2024-03-04T16:15:37.938 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:37.938 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":44331,"name":"cephfs.smithi152.tlpshy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6825/3736721533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3736721533},{"type":"v1","addr":"172.21.15.152:6825","nonce":3736721533}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:25.515194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24257":{"gid":24257,"name":"cephfs.smithi152.kqoybf","rank":0,"incarnation":20,"state":"up:active","state_seq":156,"addr":"172.21.15.152:6827/1672192052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1672192052},{"type":"v1","addr":"172.21.15.152:6827","nonce":1672192052}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:37.938 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-03-04T16:15:37.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 -- 172.21.15.116:0/2675786982 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86b806d810 msgr2=0x7f86b806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86b806d810 0x7f86b806fcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f86c8006010 tx=0x7f86c8005dd0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 -- 172.21.15.116:0/2675786982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86cc1962d0 msgr2=0x7f86cc19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86cc1962d0 0x7f86cc19a730 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00e970 tx=0x7f86bc00ee30 comp rx=0 tx=0).stop 2024-03-04T16:15:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 -- 172.21.15.116:0/2675786982 shutdown_connections 2024-03-04T16:15:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86b806d810 0x7f86b806fcd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86cc1962d0 0x7f86cc19a730 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 --2- 172.21.15.116:0/2675786982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86cc0675c0 0x7f86cc195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 -- 172.21.15.116:0/2675786982 >> 172.21.15.116:0/2675786982 conn(0x7f86cc074a50 msgr2=0x7f86cc0fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 -- 172.21.15.116:0/2675786982 shutdown_connections 2024-03-04T16:15:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:37.940+0000 7f86d3aca640 1 -- 172.21.15.116:0/2675786982 wait complete. 2024-03-04T16:15:37.952 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-03-04T16:15:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.056+0000 7ffbf824b640 1 -- 172.21.15.116:0/1136605959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf01002b0 msgr2=0x7ffbf0100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.056+0000 7ffbf824b640 1 --2- 172.21.15.116:0/1136605959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf01002b0 0x7ffbf0100690 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ffbec009880 tx=0x7ffbec02bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.056+0000 7ffbf824b640 1 -- 172.21.15.116:0/1136605959 shutdown_connections 2024-03-04T16:15:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.056+0000 7ffbf824b640 1 --2- 172.21.15.116:0/1136605959 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf0100c60 0x7ffbf0104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.056+0000 7ffbf824b640 1 --2- 172.21.15.116:0/1136605959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf01002b0 0x7ffbf0100690 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.056+0000 7ffbf824b640 1 -- 172.21.15.116:0/1136605959 >> 172.21.15.116:0/1136605959 conn(0x7ffbf00fbb30 msgr2=0x7ffbf00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.056+0000 7ffbf824b640 1 -- 172.21.15.116:0/1136605959 shutdown_connections 2024-03-04T16:15:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.056+0000 7ffbf824b640 1 -- 172.21.15.116:0/1136605959 wait complete. 2024-03-04T16:15:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf824b640 1 Processor -- start 2024-03-04T16:15:38.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf824b640 1 -- start start 2024-03-04T16:15:38.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf824b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf01002b0 0x7ffbf0195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf824b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf01962e0 0x7ffbf019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf824b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf01968e0 con 0x7ffbf01002b0 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf824b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf0196a50 con 0x7ffbf01962e0 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf5fe7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf01002b0 0x7ffbf0195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf5fe7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf01002b0 0x7ffbf0195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32974/0 (socket says 172.21.15.116:32974) 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf5fe7640 1 -- 172.21.15.116:0/3371017764 learned_addr learned my addr 172.21.15.116:0/3371017764 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf5fe7640 1 -- 172.21.15.116:0/3371017764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf01962e0 msgr2=0x7ffbf019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf57e6640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf01962e0 0x7ffbf019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf5fe7640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf01962e0 0x7ffbf019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.057+0000 7ffbf5fe7640 1 -- 172.21.15.116:0/3371017764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec009560 con 0x7ffbf01002b0 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.058+0000 7ffbf5fe7640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf01002b0 0x7ffbf0195da0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ffbec009430 tx=0x7ffbec004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.058+0000 7ffbe6ffd640 1 -- 172.21.15.116:0/3371017764 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbec0035f0 con 0x7ffbf01002b0 2024-03-04T16:15:38.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.058+0000 7ffbe6ffd640 1 -- 172.21.15.116:0/3371017764 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbec003c10 con 0x7ffbf01002b0 2024-03-04T16:15:38.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.058+0000 7ffbf824b640 1 -- 172.21.15.116:0/3371017764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf019ace0 con 0x7ffbf01002b0 2024-03-04T16:15:38.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.058+0000 7ffbe6ffd640 1 -- 172.21.15.116:0/3371017764 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbec034440 con 0x7ffbf01002b0 2024-03-04T16:15:38.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.058+0000 7ffbf824b640 1 -- 172.21.15.116:0/3371017764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf019b240 con 0x7ffbf01002b0 2024-03-04T16:15:38.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.059+0000 7ffbf824b640 1 -- 172.21.15.116:0/3371017764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf004e040 con 0x7ffbf01002b0 2024-03-04T16:15:38.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.059+0000 7ffbe6ffd640 1 -- 172.21.15.116:0/3371017764 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7ffbec004fd0 con 0x7ffbf01002b0 2024-03-04T16:15:38.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.060+0000 7ffbe6ffd640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ffbdc06d810 0x7ffbdc06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:38.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.060+0000 7ffbe6ffd640 1 -- 172.21.15.116:0/3371017764 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7ffbec0afa20 con 0x7ffbf01002b0 2024-03-04T16:15:38.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.060+0000 7ffbf57e6640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ffbdc06d810 0x7ffbdc06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:38.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.060+0000 7ffbf57e6640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ffbdc06d810 0x7ffbdc06fcd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00058c0 tx=0x7ffbe0005c60 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:38.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.064+0000 7ffbe6ffd640 1 -- 172.21.15.116:0/3371017764 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffbec079180 con 0x7ffbf01002b0 2024-03-04T16:15:38.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.285+0000 7ffbf824b640 1 -- 172.21.15.116:0/3371017764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7ffbf019b4b0 con 0x7ffbf01002b0 2024-03-04T16:15:38.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.285+0000 7ffbe6ffd640 1 -- 172.21.15.116:0/3371017764 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v29) v1 ==== 107+0+4140 (secure 0 0 0) 0x7ffbec07c840 con 0x7ffbf01002b0 2024-03-04T16:15:38.286 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:38.286 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":44331,"name":"cephfs.smithi152.tlpshy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6825/3736721533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3736721533},{"type":"v1","addr":"172.21.15.152:6825","nonce":3736721533}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:38.459390+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:38.286 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 25 2024-03-04T16:15:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.287+0000 7ffbf824b640 1 -- 172.21.15.116:0/3371017764 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ffbdc06d810 msgr2=0x7ffbdc06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.287+0000 7ffbf824b640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ffbdc06d810 0x7ffbdc06fcd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00058c0 tx=0x7ffbe0005c60 comp rx=0 tx=0).stop 2024-03-04T16:15:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.287+0000 7ffbf824b640 1 -- 172.21.15.116:0/3371017764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf01002b0 msgr2=0x7ffbf0195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.287+0000 7ffbf824b640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf01002b0 0x7ffbf0195da0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ffbec009430 tx=0x7ffbec004040 comp rx=0 tx=0).stop 2024-03-04T16:15:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.287+0000 7ffbf824b640 1 -- 172.21.15.116:0/3371017764 shutdown_connections 2024-03-04T16:15:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.287+0000 7ffbf824b640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7ffbdc06d810 0x7ffbdc06fcd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.287+0000 7ffbf824b640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf01962e0 0x7ffbf019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.287+0000 7ffbf824b640 1 --2- 172.21.15.116:0/3371017764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf01002b0 0x7ffbf0195da0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.287+0000 7ffbf824b640 1 -- 172.21.15.116:0/3371017764 >> 172.21.15.116:0/3371017764 conn(0x7ffbf00fbb30 msgr2=0x7ffbf00fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:38.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.288+0000 7ffbf824b640 1 -- 172.21.15.116:0/3371017764 shutdown_connections 2024-03-04T16:15:38.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.288+0000 7ffbf824b640 1 -- 172.21.15.116:0/3371017764 wait complete. 2024-03-04T16:15:38.299 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-03-04T16:15:38.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.412+0000 7f40e1884640 1 -- 172.21.15.116:0/2112005167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40dc1002d0 msgr2=0x7f40dc1006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.412+0000 7f40e1884640 1 --2- 172.21.15.116:0/2112005167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40dc1002d0 0x7f40dc1006b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009880 tx=0x7f40cc02bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.412+0000 7f40e1884640 1 -- 172.21.15.116:0/2112005167 shutdown_connections 2024-03-04T16:15:38.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.412+0000 7f40e1884640 1 --2- 172.21.15.116:0/2112005167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc100c80 0x7f40dc104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.412+0000 7f40e1884640 1 --2- 172.21.15.116:0/2112005167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40dc1002d0 0x7f40dc1006b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.412+0000 7f40e1884640 1 -- 172.21.15.116:0/2112005167 >> 172.21.15.116:0/2112005167 conn(0x7f40dc0fbb50 msgr2=0x7f40dc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.412+0000 7f40e1884640 1 -- 172.21.15.116:0/2112005167 shutdown_connections 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.412+0000 7f40e1884640 1 -- 172.21.15.116:0/2112005167 wait complete. 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.412+0000 7f40e1884640 1 Processor -- start 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40e1884640 1 -- start start 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40e1884640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40dc1002d0 0x7f40dc195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40e1884640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc196460 0x7f40dc19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40e1884640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc196a60 con 0x7f40dc1002d0 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40e1884640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc196bd0 con 0x7f40dc196460 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40dc1002d0 0x7f40dc195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40dc1002d0 0x7f40dc195f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32998/0 (socket says 172.21.15.116:32998) 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40daffd640 1 -- 172.21.15.116:0/1922308668 learned_addr learned my addr 172.21.15.116:0/1922308668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40daffd640 1 -- 172.21.15.116:0/1922308668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc196460 msgr2=0x7f40dc19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40daffd640 1 --2- 172.21.15.116:0/1922308668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc196460 0x7f40dc19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40daffd640 1 -- 172.21.15.116:0/1922308668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc009560 con 0x7f40dc1002d0 2024-03-04T16:15:38.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40daffd640 1 --2- 172.21.15.116:0/1922308668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40dc1002d0 0x7f40dc195f20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009430 tx=0x7f40cc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:38.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.413+0000 7f40e0882640 1 -- 172.21.15.116:0/1922308668 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40cc0035f0 con 0x7f40dc1002d0 2024-03-04T16:15:38.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.414+0000 7f40e1884640 1 -- 172.21.15.116:0/1922308668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc19ae60 con 0x7f40dc1002d0 2024-03-04T16:15:38.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.414+0000 7f40e1884640 1 -- 172.21.15.116:0/1922308668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc19b3c0 con 0x7f40dc1002d0 2024-03-04T16:15:38.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.414+0000 7f40e0882640 1 -- 172.21.15.116:0/1922308668 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40cc003c10 con 0x7f40dc1002d0 2024-03-04T16:15:38.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.414+0000 7f40e0882640 1 -- 172.21.15.116:0/1922308668 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40cc034440 con 0x7f40dc1002d0 2024-03-04T16:15:38.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.414+0000 7f40e1884640 1 -- 172.21.15.116:0/1922308668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc04e040 con 0x7f40dc1002d0 2024-03-04T16:15:38.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.416+0000 7f40e0882640 1 -- 172.21.15.116:0/1922308668 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f40cc004fd0 con 0x7f40dc1002d0 2024-03-04T16:15:38.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.416+0000 7f40e0882640 1 --2- 172.21.15.116:0/1922308668 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f40c406d810 0x7f40c406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:38.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.416+0000 7f40e0882640 1 -- 172.21.15.116:0/1922308668 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f40cc0af9e0 con 0x7f40dc1002d0 2024-03-04T16:15:38.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.416+0000 7f40da7fc640 1 --2- 172.21.15.116:0/1922308668 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f40c406d810 0x7f40c406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:38.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.416+0000 7f40da7fc640 1 --2- 172.21.15.116:0/1922308668 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f40c406d810 0x7f40c406fcd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0694b0 tx=0x7f40d0006c90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:38.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.421+0000 7f40e0882640 1 -- 172.21.15.116:0/1922308668 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f40cc079140 con 0x7f40dc1002d0 2024-03-04T16:15:38.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.651+0000 7f40e1884640 1 -- 172.21.15.116:0/1922308668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f40dc19b6a0 con 0x7f40dc1002d0 2024-03-04T16:15:38.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.651+0000 7f40e0882640 1 -- 172.21.15.116:0/1922308668 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v29) v1 ==== 107+0+4185 (secure 0 0 0) 0x7f40cc07c800 con 0x7f40dc1002d0 2024-03-04T16:15:38.652 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:38.652 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":44331,"name":"cephfs.smithi152.tlpshy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6825/3736721533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3736721533},{"type":"v1","addr":"172.21.15.152:6825","nonce":3736721533}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:38.464480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34400":{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:38.652 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 26 2024-03-04T16:15:38.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.653+0000 7f40e1884640 1 -- 172.21.15.116:0/1922308668 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f40c406d810 msgr2=0x7f40c406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.653+0000 7f40e1884640 1 --2- 172.21.15.116:0/1922308668 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f40c406d810 0x7f40c406fcd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0694b0 tx=0x7f40d0006c90 comp rx=0 tx=0).stop 2024-03-04T16:15:38.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.653+0000 7f40e1884640 1 -- 172.21.15.116:0/1922308668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40dc1002d0 msgr2=0x7f40dc195f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.653+0000 7f40e1884640 1 --2- 172.21.15.116:0/1922308668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40dc1002d0 0x7f40dc195f20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009430 tx=0x7f40cc004040 comp rx=0 tx=0).stop 2024-03-04T16:15:38.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.653+0000 7f40e1884640 1 -- 172.21.15.116:0/1922308668 shutdown_connections 2024-03-04T16:15:38.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.653+0000 7f40e1884640 1 --2- 172.21.15.116:0/1922308668 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f40c406d810 0x7f40c406fcd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.653+0000 7f40e1884640 1 --2- 172.21.15.116:0/1922308668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc196460 0x7f40dc19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.653+0000 7f40e1884640 1 --2- 172.21.15.116:0/1922308668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40dc1002d0 0x7f40dc195f20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.654+0000 7f40e1884640 1 -- 172.21.15.116:0/1922308668 >> 172.21.15.116:0/1922308668 conn(0x7f40dc0fbb50 msgr2=0x7f40dc0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.654+0000 7f40e1884640 1 -- 172.21.15.116:0/1922308668 shutdown_connections 2024-03-04T16:15:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.654+0000 7f40e1884640 1 -- 172.21.15.116:0/1922308668 wait complete. 2024-03-04T16:15:38.664 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:38 smithi116 ceph-mon[94164]: pgmap v329: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:38.664 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:38 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3032604484' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-04T16:15:38.664 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:15:38.664 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:15:38.664 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:38 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2675786982' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-04T16:15:38.664 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:38.664 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:38 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3371017764' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-04T16:15:38.665 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-03-04T16:15:38.758 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:38 smithi152 ceph-mon[82845]: pgmap v329: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:38.758 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:38 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3032604484' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-04T16:15:38.758 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:15:38.759 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:15:38.759 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:38 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2675786982' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-04T16:15:38.759 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:38.759 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:38 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3371017764' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-04T16:15:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.767+0000 7fcc01f73640 1 -- 172.21.15.116:0/163596513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc0ffb50 msgr2=0x7fcbfc103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.767+0000 7fcc01f73640 1 --2- 172.21.15.116:0/163596513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc0ffb50 0x7fcbfc103b90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0009880 tx=0x7fcbf002bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.767+0000 7fcc01f73640 1 -- 172.21.15.116:0/163596513 shutdown_connections 2024-03-04T16:15:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.767+0000 7fcc01f73640 1 --2- 172.21.15.116:0/163596513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc0ffb50 0x7fcbfc103b90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.767+0000 7fcc01f73640 1 --2- 172.21.15.116:0/163596513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbfc0ff1a0 0x7fcbfc0ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.767+0000 7fcc01f73640 1 -- 172.21.15.116:0/163596513 >> 172.21.15.116:0/163596513 conn(0x7fcbfc0faa20 msgr2=0x7fcbfc0fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.767+0000 7fcc01f73640 1 -- 172.21.15.116:0/163596513 shutdown_connections 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.767+0000 7fcc01f73640 1 -- 172.21.15.116:0/163596513 wait complete. 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.768+0000 7fcc01f73640 1 Processor -- start 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.768+0000 7fcc01f73640 1 -- start start 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.768+0000 7fcc01f73640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbfc0ff1a0 0x7fcbfc194d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.768+0000 7fcc01f73640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1952c0 0x7fcbfc199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.768+0000 7fcc01f73640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc1958c0 con 0x7fcbfc1952c0 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.768+0000 7fcc01f73640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc195a30 con 0x7fcbfc0ff1a0 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.768+0000 7fcbfaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1952c0 0x7fcbfc199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.768+0000 7fcbfaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1952c0 0x7fcbfc199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33010/0 (socket says 172.21.15.116:33010) 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.768+0000 7fcbfaffd640 1 -- 172.21.15.116:0/819731044 learned_addr learned my addr 172.21.15.116:0/819731044 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:38.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.768+0000 7fcbfb7fe640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbfc0ff1a0 0x7fcbfc194d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.769+0000 7fcbfaffd640 1 -- 172.21.15.116:0/819731044 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbfc0ff1a0 msgr2=0x7fcbfc194d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.769+0000 7fcbfaffd640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbfc0ff1a0 0x7fcbfc194d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.769+0000 7fcbfaffd640 1 -- 172.21.15.116:0/819731044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf0009560 con 0x7fcbfc1952c0 2024-03-04T16:15:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.769+0000 7fcbfaffd640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1952c0 0x7fcbfc199720 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0036040 tx=0x7fcbf0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.769+0000 7fcbf8ff9640 1 -- 172.21.15.116:0/819731044 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbf00037c0 con 0x7fcbfc1952c0 2024-03-04T16:15:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.769+0000 7fcbf8ff9640 1 -- 172.21.15.116:0/819731044 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbf0003de0 con 0x7fcbfc1952c0 2024-03-04T16:15:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.769+0000 7fcbf8ff9640 1 -- 172.21.15.116:0/819731044 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbf0034440 con 0x7fcbfc1952c0 2024-03-04T16:15:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.769+0000 7fcc01f73640 1 -- 172.21.15.116:0/819731044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbfc199cc0 con 0x7fcbfc1952c0 2024-03-04T16:15:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.769+0000 7fcc01f73640 1 -- 172.21.15.116:0/819731044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbfc19a220 con 0x7fcbfc1952c0 2024-03-04T16:15:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.770+0000 7fcc01f73640 1 -- 172.21.15.116:0/819731044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbfc04e040 con 0x7fcbfc1952c0 2024-03-04T16:15:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.770+0000 7fcbf8ff9640 1 -- 172.21.15.116:0/819731044 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fcbf0004fb0 con 0x7fcbfc1952c0 2024-03-04T16:15:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.771+0000 7fcbf8ff9640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcbe406d810 0x7fcbe406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.771+0000 7fcbf8ff9640 1 -- 172.21.15.116:0/819731044 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fcbf00afa60 con 0x7fcbfc1952c0 2024-03-04T16:15:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.771+0000 7fcbfb7fe640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcbe406d810 0x7fcbe406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.771+0000 7fcbfb7fe640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcbe406d810 0x7fcbe406fcd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009890 tx=0x7fcbec008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.775+0000 7fcbf8ff9640 1 -- 172.21.15.116:0/819731044 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcbf00791c0 con 0x7fcbfc1952c0 2024-03-04T16:15:38.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.984+0000 7fcc01f73640 1 -- 172.21.15.116:0/819731044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7fcbfc1074a0 con 0x7fcbfc1952c0 2024-03-04T16:15:38.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.984+0000 7fcbf8ff9640 1 -- 172.21.15.116:0/819731044 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v29) v1 ==== 107+0+4189 (secure 0 0 0) 0x7fcbf007c880 con 0x7fcbfc1952c0 2024-03-04T16:15:38.985 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:38.985 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":44331,"name":"cephfs.smithi152.tlpshy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6825/3736721533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3736721533},{"type":"v1","addr":"172.21.15.152:6825","nonce":3736721533}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:47.319612+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34400":{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:38.985 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 27 2024-03-04T16:15:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.986+0000 7fcc01f73640 1 -- 172.21.15.116:0/819731044 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcbe406d810 msgr2=0x7fcbe406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.986+0000 7fcc01f73640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcbe406d810 0x7fcbe406fcd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009890 tx=0x7fcbec008040 comp rx=0 tx=0).stop 2024-03-04T16:15:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.986+0000 7fcc01f73640 1 -- 172.21.15.116:0/819731044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1952c0 msgr2=0x7fcbfc199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.986+0000 7fcc01f73640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1952c0 0x7fcbfc199720 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0036040 tx=0x7fcbf0004270 comp rx=0 tx=0).stop 2024-03-04T16:15:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.986+0000 7fcc01f73640 1 -- 172.21.15.116:0/819731044 shutdown_connections 2024-03-04T16:15:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.986+0000 7fcc01f73640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fcbe406d810 0x7fcbe406fcd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.986+0000 7fcc01f73640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1952c0 0x7fcbfc199720 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.986+0000 7fcc01f73640 1 --2- 172.21.15.116:0/819731044 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbfc0ff1a0 0x7fcbfc194d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.986+0000 7fcc01f73640 1 -- 172.21.15.116:0/819731044 >> 172.21.15.116:0/819731044 conn(0x7fcbfc0faa20 msgr2=0x7fcbfc0fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.987+0000 7fcc01f73640 1 -- 172.21.15.116:0/819731044 shutdown_connections 2024-03-04T16:15:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:38.987+0000 7fcc01f73640 1 -- 172.21.15.116:0/819731044 wait complete. 2024-03-04T16:15:38.998 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-03-04T16:15:39.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.101+0000 7f6e31e70640 1 -- 172.21.15.116:0/4245215794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0675c0 msgr2=0x7f6e2c0679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.101+0000 7f6e31e70640 1 --2- 172.21.15.116:0/4245215794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0675c0 0x7f6e2c0679a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009880 tx=0x7f6e1c02bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.101+0000 7f6e31e70640 1 -- 172.21.15.116:0/4245215794 shutdown_connections 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.101+0000 7f6e31e70640 1 --2- 172.21.15.116:0/4245215794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e2c067f70 0x7f6e2c104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.101+0000 7f6e31e70640 1 --2- 172.21.15.116:0/4245215794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0675c0 0x7f6e2c0679a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.101+0000 7f6e31e70640 1 -- 172.21.15.116:0/4245215794 >> 172.21.15.116:0/4245215794 conn(0x7f6e2c074a50 msgr2=0x7f6e2c074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.101+0000 7f6e31e70640 1 -- 172.21.15.116:0/4245215794 shutdown_connections 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.101+0000 7f6e31e70640 1 -- 172.21.15.116:0/4245215794 wait complete. 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e31e70640 1 Processor -- start 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e31e70640 1 -- start start 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e31e70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0675c0 0x7f6e2c195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e31e70640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e2c1963e0 0x7f6e2c19a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e31e70640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c1969e0 con 0x7f6e2c0675c0 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e31e70640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c196b50 con 0x7f6e2c1963e0 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e2b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0675c0 0x7f6e2c195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e2b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0675c0 0x7f6e2c195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33032/0 (socket says 172.21.15.116:33032) 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e2b7fe640 1 -- 172.21.15.116:0/1552906952 learned_addr learned my addr 172.21.15.116:0/1552906952 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e2affd640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e2c1963e0 0x7f6e2c19a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e2b7fe640 1 -- 172.21.15.116:0/1552906952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e2c1963e0 msgr2=0x7f6e2c19a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e2b7fe640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e2c1963e0 0x7f6e2c19a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e2b7fe640 1 -- 172.21.15.116:0/1552906952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c009560 con 0x7f6e2c0675c0 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e2affd640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e2c1963e0 0x7f6e2c19a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.102+0000 7f6e2b7fe640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0675c0 0x7f6e2c195ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009430 tx=0x7f6e1c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.103+0000 7f6e28ff9640 1 -- 172.21.15.116:0/1552906952 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e1c0035f0 con 0x7f6e2c0675c0 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.103+0000 7f6e31e70640 1 -- 172.21.15.116:0/1552906952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2c19ade0 con 0x7f6e2c0675c0 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.103+0000 7f6e28ff9640 1 -- 172.21.15.116:0/1552906952 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e1c003c10 con 0x7f6e2c0675c0 2024-03-04T16:15:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.103+0000 7f6e28ff9640 1 -- 172.21.15.116:0/1552906952 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e1c034440 con 0x7f6e2c0675c0 2024-03-04T16:15:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.103+0000 7f6e31e70640 1 -- 172.21.15.116:0/1552906952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2c19b340 con 0x7f6e2c0675c0 2024-03-04T16:15:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.104+0000 7f6e31e70640 1 -- 172.21.15.116:0/1552906952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2c04e040 con 0x7f6e2c0675c0 2024-03-04T16:15:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.104+0000 7f6e28ff9640 1 -- 172.21.15.116:0/1552906952 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f6e1c004fd0 con 0x7f6e2c0675c0 2024-03-04T16:15:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.105+0000 7f6e28ff9640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e1406d810 0x7f6e1406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.105+0000 7f6e28ff9640 1 -- 172.21.15.116:0/1552906952 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f6e1c0afa20 con 0x7f6e2c0675c0 2024-03-04T16:15:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.105+0000 7f6e2affd640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e1406d810 0x7f6e1406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.105+0000 7f6e2affd640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e1406d810 0x7f6e1406fcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6e20006fd0 tx=0x7f6e20008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.109+0000 7f6e28ff9640 1 -- 172.21.15.116:0/1552906952 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6e1c079180 con 0x7f6e2c0675c0 2024-03-04T16:15:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.329+0000 7f6e31e70640 1 -- 172.21.15.116:0/1552906952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7f6e2c108580 con 0x7f6e2c0675c0 2024-03-04T16:15:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.330+0000 7f6e28ff9640 1 -- 172.21.15.116:0/1552906952 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v29) v1 ==== 107+0+5002 (secure 0 0 0) 0x7f6e1c07c840 con 0x7f6e2c0675c0 2024-03-04T16:15:39.330 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:39.330 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi116.gkenwy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6829/786823228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":786823228},{"type":"v1","addr":"172.21.15.116:6829","nonce":786823228}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":44331,"name":"cephfs.smithi152.tlpshy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6825/3736721533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3736721533},{"type":"v1","addr":"172.21.15.152:6825","nonce":3736721533}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":44335,"name":"cephfs.smithi152.kqoybf","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6827/1659417498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1659417498},{"type":"v1","addr":"172.21.15.152:6827","nonce":1659417498}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T16:01:00.313772+0000","modified":"2024-03-04T16:11:48.156971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34400":{"gid":34400,"name":"cephfs.smithi116.pjrmzv","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.116:6827/487584748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":487584748},{"type":"v1","addr":"172.21.15.116:6827","nonce":487584748}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T16:15:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-04T16:15:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.331+0000 7f6e31e70640 1 -- 172.21.15.116:0/1552906952 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e1406d810 msgr2=0x7f6e1406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.331+0000 7f6e31e70640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e1406d810 0x7f6e1406fcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6e20006fd0 tx=0x7f6e20008040 comp rx=0 tx=0).stop 2024-03-04T16:15:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.331+0000 7f6e31e70640 1 -- 172.21.15.116:0/1552906952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0675c0 msgr2=0x7f6e2c195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.331+0000 7f6e31e70640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0675c0 0x7f6e2c195ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009430 tx=0x7f6e1c004040 comp rx=0 tx=0).stop 2024-03-04T16:15:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.331+0000 7f6e31e70640 1 -- 172.21.15.116:0/1552906952 shutdown_connections 2024-03-04T16:15:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.331+0000 7f6e31e70640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6e1406d810 0x7f6e1406fcd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.331+0000 7f6e31e70640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e2c1963e0 0x7f6e2c19a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.331+0000 7f6e31e70640 1 --2- 172.21.15.116:0/1552906952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0675c0 0x7f6e2c195ea0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.331+0000 7f6e31e70640 1 -- 172.21.15.116:0/1552906952 >> 172.21.15.116:0/1552906952 conn(0x7f6e2c074a50 msgr2=0x7f6e2c0fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.332+0000 7f6e31e70640 1 -- 172.21.15.116:0/1552906952 shutdown_connections 2024-03-04T16:15:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.332+0000 7f6e31e70640 1 -- 172.21.15.116:0/1552906952 wait complete. 2024-03-04T16:15:39.343 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-04T16:15:39.354 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-04T16:15:39.354 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T16:15:39.355 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T16:15:39.372 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T16:15:39.372 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T16:15:39.429 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.569+0000 7f9c43fa4640 1 -- 172.21.15.116:0/1355191284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c100270 msgr2=0x7f9c3c100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.569+0000 7f9c43fa4640 1 --2- 172.21.15.116:0/1355191284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c100270 0x7f9c3c100650 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009880 tx=0x7f9c3002bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.569+0000 7f9c43fa4640 1 -- 172.21.15.116:0/1355191284 shutdown_connections 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.569+0000 7f9c43fa4640 1 --2- 172.21.15.116:0/1355191284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c3c100c20 0x7f9c3c104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.569+0000 7f9c43fa4640 1 --2- 172.21.15.116:0/1355191284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c100270 0x7f9c3c100650 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.569+0000 7f9c43fa4640 1 -- 172.21.15.116:0/1355191284 >> 172.21.15.116:0/1355191284 conn(0x7f9c3c0fbad0 msgr2=0x7f9c3c0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c43fa4640 1 -- 172.21.15.116:0/1355191284 shutdown_connections 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c43fa4640 1 -- 172.21.15.116:0/1355191284 wait complete. 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c43fa4640 1 Processor -- start 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c43fa4640 1 -- start start 2024-03-04T16:15:39.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c43fa4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c3c100270 0x7f9c3c195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c43fa4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c1961f0 0x7f9c3c1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c43fa4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c1967f0 con 0x7f9c3c1961f0 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c43fa4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c196930 con 0x7f9c3c100270 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c4153f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c1961f0 0x7f9c3c1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c4153f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c1961f0 0x7f9c3c1995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33042/0 (socket says 172.21.15.116:33042) 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.570+0000 7f9c4153f640 1 -- 172.21.15.116:0/3207231443 learned_addr learned my addr 172.21.15.116:0/3207231443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.571+0000 7f9c4153f640 1 -- 172.21.15.116:0/3207231443 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c3c100270 msgr2=0x7f9c3c195cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.571+0000 7f9c4153f640 1 --2- 172.21.15.116:0/3207231443 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c3c100270 0x7f9c3c195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.571+0000 7f9c4153f640 1 -- 172.21.15.116:0/3207231443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c30009560 con 0x7f9c3c1961f0 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.571+0000 7f9c4153f640 1 --2- 172.21.15.116:0/3207231443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c1961f0 0x7f9c3c1995a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800d6f0 tx=0x7f9c3800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.571+0000 7f9c2effd640 1 -- 172.21.15.116:0/3207231443 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c38009a30 con 0x7f9c3c1961f0 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.571+0000 7f9c2effd640 1 -- 172.21.15.116:0/3207231443 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c38009b90 con 0x7f9c3c1961f0 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.571+0000 7f9c43fa4640 1 -- 172.21.15.116:0/3207231443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c3c199ba0 con 0x7f9c3c1961f0 2024-03-04T16:15:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.571+0000 7f9c2effd640 1 -- 172.21.15.116:0/3207231443 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c3800f680 con 0x7f9c3c1961f0 2024-03-04T16:15:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.571+0000 7f9c43fa4640 1 -- 172.21.15.116:0/3207231443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3c19a100 con 0x7f9c3c1961f0 2024-03-04T16:15:39.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.572+0000 7f9c43fa4640 1 -- 172.21.15.116:0/3207231443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3c19a3d0 con 0x7f9c3c1961f0 2024-03-04T16:15:39.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.572+0000 7f9c2effd640 1 -- 172.21.15.116:0/3207231443 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f9c38009d00 con 0x7f9c3c1961f0 2024-03-04T16:15:39.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.573+0000 7f9c2effd640 1 --2- 172.21.15.116:0/3207231443 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9c2806d8d0 0x7f9c2806fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:39.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.573+0000 7f9c41d40640 1 --2- 172.21.15.116:0/3207231443 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9c2806d8d0 0x7f9c2806fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:39.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.573+0000 7f9c2effd640 1 -- 172.21.15.116:0/3207231443 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f9c3808d390 con 0x7f9c3c1961f0 2024-03-04T16:15:39.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.573+0000 7f9c41d40640 1 --2- 172.21.15.116:0/3207231443 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9c2806d8d0 0x7f9c2806fd90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9c30036040 tx=0x7f9c30000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:39.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.578+0000 7f9c2effd640 1 -- 172.21.15.116:0/3207231443 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9c38056b70 con 0x7f9c3c1961f0 2024-03-04T16:15:39.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:39.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:39.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:15:39.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:39 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1922308668' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-04T16:15:39.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:39 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/819731044' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-04T16:15:39.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:39 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1552906952' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-04T16:15:39.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.752+0000 7f9c43fa4640 1 -- 172.21.15.116:0/3207231443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c3c100d10 con 0x7f9c3c1961f0 2024-03-04T16:15:39.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.753+0000 7f9c2effd640 1 -- 172.21.15.116:0/3207231443 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16947 (secure 0 0 0) 0x7f9c3805a230 con 0x7f9c3c1961f0 2024-03-04T16:15:39.754 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:39.754 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":84,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:53:48.465240+0000","modified":"2024-03-04T16:11:38.459407+0000","last_up_change":"2024-03-04T16:10:31.185124+0000","last_in_change":"2024-03-04T15:59:22.307720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:58:27.372335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-04T16:00:58.546762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-04T16:00:59.546717+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d12966f-954e-4a09-bbbd-5b0f6a04d039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6803","nonce":2502615728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6805","nonce":2502615728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6809","nonce":2502615728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6807","nonce":2502615728}]},"public_addr":"172.21.15.116:6803/2502615728","cluster_addr":"172.21.15.116:6805/2502615728","heartbeat_back_addr":"172.21.15.116:6809/2502615728","heartbeat_front_addr":"172.21.15.116:6807/2502615728","state":["exists","up"]},{"osd":1,"uuid":"f990720f-a19d-48c1-9545-81e511c2690b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6811","nonce":22677649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6813","nonce":22677649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6817","nonce":22677649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6815","nonce":22677649}]},"public_addr":"172.21.15.116:6811/22677649","cluster_addr":"172.21.15.116:6813/22677649","heartbeat_back_addr":"172.21.15.116:6817/22677649","heartbeat_front_addr":"172.21.15.116:6815/22677649","state":["exists","up"]},{"osd":2,"uuid":"fe3132c8-6b15-4a11-a884-7c6a09d42955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6819","nonce":1756826225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6821","nonce":1756826225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6825","nonce":1756826225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6823","nonce":1756826225}]},"public_addr":"172.21.15.116:6819/1756826225","cluster_addr":"172.21.15.116:6821/1756826225","heartbeat_back_addr":"172.21.15.116:6825/1756826225","heartbeat_front_addr":"172.21.15.116:6823/1756826225","state":["exists","up"]},{"osd":3,"uuid":"15930a9b-f721-4c30-9c5b-bef5ef69777c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6801","nonce":3040811013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6803","nonce":3040811013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6807","nonce":3040811013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6805","nonce":3040811013}]},"public_addr":"172.21.15.152:6801/3040811013","cluster_addr":"172.21.15.152:6803/3040811013","heartbeat_back_addr":"172.21.15.152:6807/3040811013","heartbeat_front_addr":"172.21.15.152:6805/3040811013","state":["exists","up"]},{"osd":4,"uuid":"8d201b6a-6a91-4a7e-984e-e8d7f0289793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6809","nonce":878265314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6811","nonce":878265314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6815","nonce":878265314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6813","nonce":878265314}]},"public_addr":"172.21.15.152:6809/878265314","cluster_addr":"172.21.15.152:6811/878265314","heartbeat_back_addr":"172.21.15.152:6815/878265314","heartbeat_front_addr":"172.21.15.152:6813/878265314","state":["exists","up"]},{"osd":5,"uuid":"c06ef0f5-6894-4e01-a693-bb129d248fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6817","nonce":1180242057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6819","nonce":1180242057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6823","nonce":1180242057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6821","nonce":1180242057}]},"public_addr":"172.21.15.152:6817/1180242057","cluster_addr":"172.21.15.152:6819/1180242057","heartbeat_back_addr":"172.21.15.152:6823/1180242057","heartbeat_front_addr":"172.21.15.152:6821/1180242057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:24.230215+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:52.709103+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:24.072974+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:47.684390+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:13.571654+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:39.874966+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.152:6827/1672192052":"2024-03-05T16:11:38.459377+0000","172.21.15.152:6826/1672192052":"2024-03-05T16:11:38.459377+0000","172.21.15.152:6825/287206196":"2024-03-05T16:11:14.444369+0000","172.21.15.152:6824/287206196":"2024-03-05T16:11:14.444369+0000","172.21.15.116:0/3594007286":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/4255448814":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/317318762":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/2770714257":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/3287532122":"2024-03-05T15:54:53.931186+0000","172.21.15.116:6801/2966163948":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/976694871":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6800/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6801/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6801/3356602598":"2024-03-05T16:03:33.250227+0000","172.21.15.116:6800/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3670178561":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6800/3356602598":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/2883878856":"2024-03-05T16:05:52.994136+0000","172.21.15.152:0/3493116756":"2024-03-05T16:04:41.923774+0000","172.21.15.116:6800/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.152:0/1831164264":"2024-03-05T16:04:41.923774+0000","172.21.15.116:6801/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:6827/114739798":"2024-03-05T16:01:15.387483+0000","172.21.15.116:6801/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/385581892":"2024-03-05T15:54:21.166941+0000","172.21.15.152:0/1293393009":"2024-03-05T16:04:41.923774+0000","172.21.15.152:6829/2881662614":"2024-03-05T16:04:41.923774+0000","172.21.15.152:0/1265936494":"2024-03-05T16:04:41.923774+0000","172.21.15.116:0/831649128":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3323890559":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/355050440":"2024-03-05T15:56:07.046998+0000","172.21.15.152:6828/2881662614":"2024-03-05T16:04:41.923774+0000","172.21.15.116:0/590312912":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/232576737":"2024-03-05T16:03:33.250227+0000","172.21.15.152:0/549268400":"2024-03-05T16:04:41.923774+0000","172.21.15.152:0/768114426":"2024-03-05T16:04:41.923774+0000","172.21.15.116:0/4142805454":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/1756904921":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/821201733":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6826/114739798":"2024-03-05T16:01:15.387483+0000","172.21.15.116:0/3213147898":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/2867113891":"2024-03-05T16:05:52.994136+0000","172.21.15.116:6800/2966163948":"2024-03-05T16:05:52.994136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 -- 172.21.15.116:0/3207231443 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9c2806d8d0 msgr2=0x7f9c2806fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 --2- 172.21.15.116:0/3207231443 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9c2806d8d0 0x7f9c2806fd90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9c30036040 tx=0x7f9c30000bc0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 -- 172.21.15.116:0/3207231443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c1961f0 msgr2=0x7f9c3c1995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 --2- 172.21.15.116:0/3207231443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c1961f0 0x7f9c3c1995a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800d6f0 tx=0x7f9c3800dbb0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 -- 172.21.15.116:0/3207231443 shutdown_connections 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 --2- 172.21.15.116:0/3207231443 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f9c2806d8d0 0x7f9c2806fd90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 --2- 172.21.15.116:0/3207231443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c1961f0 0x7f9c3c1995a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 --2- 172.21.15.116:0/3207231443 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c3c100270 0x7f9c3c195cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 -- 172.21.15.116:0/3207231443 >> 172.21.15.116:0/3207231443 conn(0x7f9c3c0fbad0 msgr2=0x7f9c3c0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 -- 172.21.15.116:0/3207231443 shutdown_connections 2024-03-04T16:15:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.755+0000 7f9c43fa4640 1 -- 172.21.15.116:0/3207231443 wait complete. 2024-03-04T16:15:39.767 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T15:58:27.372335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-04T16:00:58.546762+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-04T16:00:59.546717+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T16:15:39.767 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T16:15:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:15:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:39 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1922308668' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-04T16:15:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:39 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/819731044' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-04T16:15:39.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:39 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1552906952' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-04T16:15:39.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.871+0000 7fac30821640 1 -- 172.21.15.116:0/1943128671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac2c100c50 msgr2=0x7fac2c104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.871+0000 7fac30821640 1 --2- 172.21.15.116:0/1943128671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac2c100c50 0x7fac2c104c90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fac20009880 tx=0x7fac2002bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.871+0000 7fac30821640 1 -- 172.21.15.116:0/1943128671 shutdown_connections 2024-03-04T16:15:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.871+0000 7fac30821640 1 --2- 172.21.15.116:0/1943128671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac2c100c50 0x7fac2c104c90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.871+0000 7fac30821640 1 --2- 172.21.15.116:0/1943128671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac2c1002a0 0x7fac2c100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.871+0000 7fac30821640 1 -- 172.21.15.116:0/1943128671 >> 172.21.15.116:0/1943128671 conn(0x7fac2c0fbb20 msgr2=0x7fac2c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.871+0000 7fac30821640 1 -- 172.21.15.116:0/1943128671 shutdown_connections 2024-03-04T16:15:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.871+0000 7fac30821640 1 -- 172.21.15.116:0/1943128671 wait complete. 2024-03-04T16:15:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac30821640 1 Processor -- start 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac30821640 1 -- start start 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac30821640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac2c1002a0 0x7fac2c195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac30821640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac2c1962a0 0x7fac2c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac30821640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c196810 con 0x7fac2c1002a0 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac30821640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c196980 con 0x7fac2c1962a0 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac2a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac2c1002a0 0x7fac2c195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac2a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac2c1002a0 0x7fac2c195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33054/0 (socket says 172.21.15.116:33054) 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac2a59c640 1 -- 172.21.15.116:0/538042917 learned_addr learned my addr 172.21.15.116:0/538042917 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac2a59c640 1 -- 172.21.15.116:0/538042917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac2c1962a0 msgr2=0x7fac2c19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac2a59c640 1 --2- 172.21.15.116:0/538042917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac2c1962a0 0x7fac2c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac2a59c640 1 -- 172.21.15.116:0/538042917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac20009560 con 0x7fac2c1002a0 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.872+0000 7fac2a59c640 1 --2- 172.21.15.116:0/538042917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac2c1002a0 0x7fac2c195d60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00c8a0 tx=0x7fac1c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:39.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.873+0000 7fac1b7fe640 1 -- 172.21.15.116:0/538042917 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac1c007b80 con 0x7fac2c1002a0 2024-03-04T16:15:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.873+0000 7fac1b7fe640 1 -- 172.21.15.116:0/538042917 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac1c007ce0 con 0x7fac2c1002a0 2024-03-04T16:15:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.873+0000 7fac1b7fe640 1 -- 172.21.15.116:0/538042917 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac1c018810 con 0x7fac2c1002a0 2024-03-04T16:15:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.873+0000 7fac30821640 1 -- 172.21.15.116:0/538042917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2c19aca0 con 0x7fac2c1002a0 2024-03-04T16:15:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.873+0000 7fac30821640 1 -- 172.21.15.116:0/538042917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2c19b1b0 con 0x7fac2c1002a0 2024-03-04T16:15:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.874+0000 7fac30821640 1 -- 172.21.15.116:0/538042917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2c04e040 con 0x7fac2c1002a0 2024-03-04T16:15:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.874+0000 7fac1b7fe640 1 -- 172.21.15.116:0/538042917 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fac1c018a80 con 0x7fac2c1002a0 2024-03-04T16:15:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.874+0000 7fac1b7fe640 1 --2- 172.21.15.116:0/538042917 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fac1406d810 0x7fac1406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.874+0000 7fac1b7fe640 1 -- 172.21.15.116:0/538042917 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fac1c08d7c0 con 0x7fac2c1002a0 2024-03-04T16:15:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.874+0000 7fac29d9b640 1 --2- 172.21.15.116:0/538042917 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fac1406d810 0x7fac1406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.875+0000 7fac29d9b640 1 --2- 172.21.15.116:0/538042917 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fac1406d810 0x7fac1406fcd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fac20036040 tx=0x7fac20000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:39.879+0000 7fac1b7fe640 1 -- 172.21.15.116:0/538042917 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fac1c056f20 con 0x7fac2c1002a0 2024-03-04T16:15:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.050+0000 7fac30821640 1 -- 172.21.15.116:0/538042917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac2c1971a0 con 0x7fac2c1002a0 2024-03-04T16:15:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.050+0000 7fac1b7fe640 1 -- 172.21.15.116:0/538042917 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac1c05a5e0 con 0x7fac2c1002a0 2024-03-04T16:15:40.051 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T16:15:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 -- 172.21.15.116:0/538042917 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fac1406d810 msgr2=0x7fac1406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 --2- 172.21.15.116:0/538042917 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fac1406d810 0x7fac1406fcd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fac20036040 tx=0x7fac20000bc0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 -- 172.21.15.116:0/538042917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac2c1002a0 msgr2=0x7fac2c195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 --2- 172.21.15.116:0/538042917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac2c1002a0 0x7fac2c195d60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00c8a0 tx=0x7fac1c00cd60 comp rx=0 tx=0).stop 2024-03-04T16:15:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 -- 172.21.15.116:0/538042917 shutdown_connections 2024-03-04T16:15:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 --2- 172.21.15.116:0/538042917 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fac1406d810 0x7fac1406fcd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 --2- 172.21.15.116:0/538042917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac2c1962a0 0x7fac2c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 --2- 172.21.15.116:0/538042917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac2c1002a0 0x7fac2c195d60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 -- 172.21.15.116:0/538042917 >> 172.21.15.116:0/538042917 conn(0x7fac2c0fbb20 msgr2=0x7fac2c0fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 -- 172.21.15.116:0/538042917 shutdown_connections 2024-03-04T16:15:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.052+0000 7fac30821640 1 -- 172.21.15.116:0/538042917 wait complete. 2024-03-04T16:15:40.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-04T16:15:40.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.168+0000 7f6e19ff0640 1 -- 172.21.15.116:0/3495671936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e140ff390 msgr2=0x7f6e14104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.168+0000 7f6e19ff0640 1 --2- 172.21.15.116:0/3495671936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e140ff390 0x7f6e14104d10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009880 tx=0x7f6e0802bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.168+0000 7f6e19ff0640 1 -- 172.21.15.116:0/3495671936 shutdown_connections 2024-03-04T16:15:40.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.168+0000 7f6e19ff0640 1 --2- 172.21.15.116:0/3495671936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e140ff390 0x7f6e14104d10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.168+0000 7f6e19ff0640 1 --2- 172.21.15.116:0/3495671936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e140fe9e0 0x7f6e140fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.168+0000 7f6e19ff0640 1 -- 172.21.15.116:0/3495671936 >> 172.21.15.116:0/3495671936 conn(0x7f6e140741f0 msgr2=0x7f6e14074600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.168+0000 7f6e19ff0640 1 -- 172.21.15.116:0/3495671936 shutdown_connections 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.168+0000 7f6e19ff0640 1 -- 172.21.15.116:0/3495671936 wait complete. 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.168+0000 7f6e19ff0640 1 Processor -- start 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.168+0000 7f6e19ff0640 1 -- start start 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e19ff0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e140fe9e0 0x7f6e14195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e19ff0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e14196340 0x7f6e1419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e19ff0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e14196940 con 0x7f6e14196340 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e19ff0640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e14196ab0 con 0x7f6e140fe9e0 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e12ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e14196340 0x7f6e1419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e12ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e14196340 0x7f6e1419a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33074/0 (socket says 172.21.15.116:33074) 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e12ffd640 1 -- 172.21.15.116:0/2966812225 learned_addr learned my addr 172.21.15.116:0/2966812225 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e137fe640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e140fe9e0 0x7f6e14195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e12ffd640 1 -- 172.21.15.116:0/2966812225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e140fe9e0 msgr2=0x7f6e14195e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e12ffd640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e140fe9e0 0x7f6e14195e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e12ffd640 1 -- 172.21.15.116:0/2966812225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e08009560 con 0x7f6e14196340 2024-03-04T16:15:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e137fe640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e140fe9e0 0x7f6e14195e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:15:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e12ffd640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e14196340 0x7f6e1419a7a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6e08036040 tx=0x7f6e08004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e10ff9640 1 -- 172.21.15.116:0/2966812225 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e080037c0 con 0x7f6e14196340 2024-03-04T16:15:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e19ff0640 1 -- 172.21.15.116:0/2966812225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1419ad40 con 0x7f6e14196340 2024-03-04T16:15:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e10ff9640 1 -- 172.21.15.116:0/2966812225 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e08003de0 con 0x7f6e14196340 2024-03-04T16:15:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e10ff9640 1 -- 172.21.15.116:0/2966812225 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e08034440 con 0x7f6e14196340 2024-03-04T16:15:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.169+0000 7f6e19ff0640 1 -- 172.21.15.116:0/2966812225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1419b2a0 con 0x7f6e14196340 2024-03-04T16:15:40.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.171+0000 7f6e19ff0640 1 -- 172.21.15.116:0/2966812225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1404e040 con 0x7f6e14196340 2024-03-04T16:15:40.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.171+0000 7f6e10ff9640 1 -- 172.21.15.116:0/2966812225 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f6e08004fb0 con 0x7f6e14196340 2024-03-04T16:15:40.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.171+0000 7f6e10ff9640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6dfc06d810 0x7f6dfc06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:40.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.171+0000 7f6e10ff9640 1 -- 172.21.15.116:0/2966812225 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f6e080afa60 con 0x7f6e14196340 2024-03-04T16:15:40.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.171+0000 7f6e137fe640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6dfc06d810 0x7f6dfc06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:40.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.172+0000 7f6e137fe640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6dfc06d810 0x7f6dfc06fcd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6e04005fd0 tx=0x7f6e04005ea0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:40.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.176+0000 7f6e10ff9640 1 -- 172.21.15.116:0/2966812225 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6e080791c0 con 0x7f6e14196340 2024-03-04T16:15:40.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.348+0000 7f6e19ff0640 1 -- 172.21.15.116:0/2966812225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f6e141085f0 con 0x7f6e14196340 2024-03-04T16:15:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.348+0000 7f6e10ff9640 1 -- 172.21.15.116:0/2966812225 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6e0807c880 con 0x7f6e14196340 2024-03-04T16:15:40.349 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T16:15:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 -- 172.21.15.116:0/2966812225 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6dfc06d810 msgr2=0x7f6dfc06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6dfc06d810 0x7f6dfc06fcd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6e04005fd0 tx=0x7f6e04005ea0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 -- 172.21.15.116:0/2966812225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e14196340 msgr2=0x7f6e1419a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e14196340 0x7f6e1419a7a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6e08036040 tx=0x7f6e08004270 comp rx=0 tx=0).stop 2024-03-04T16:15:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 -- 172.21.15.116:0/2966812225 shutdown_connections 2024-03-04T16:15:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f6dfc06d810 0x7f6dfc06fcd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e14196340 0x7f6e1419a7a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 --2- 172.21.15.116:0/2966812225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e140fe9e0 0x7f6e14195e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 -- 172.21.15.116:0/2966812225 >> 172.21.15.116:0/2966812225 conn(0x7f6e140741f0 msgr2=0x7f6e140fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:40.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 -- 172.21.15.116:0/2966812225 shutdown_connections 2024-03-04T16:15:40.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.350+0000 7f6e19ff0640 1 -- 172.21.15.116:0/2966812225 wait complete. 2024-03-04T16:15:40.362 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-04T16:15:40.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.466+0000 7fb90616e640 1 -- 172.21.15.116:0/3480750806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9000ff180 msgr2=0x7fb9000ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.466+0000 7fb90616e640 1 --2- 172.21.15.116:0/3480750806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9000ff180 0x7fb9000ff560 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0009880 tx=0x7fb8f002bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.466+0000 7fb90616e640 1 -- 172.21.15.116:0/3480750806 shutdown_connections 2024-03-04T16:15:40.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.466+0000 7fb90616e640 1 --2- 172.21.15.116:0/3480750806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9000ffb30 0x7fb900103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.466+0000 7fb90616e640 1 --2- 172.21.15.116:0/3480750806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9000ff180 0x7fb9000ff560 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.466+0000 7fb90616e640 1 -- 172.21.15.116:0/3480750806 >> 172.21.15.116:0/3480750806 conn(0x7fb9000faa00 msgr2=0x7fb9000fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.466+0000 7fb90616e640 1 -- 172.21.15.116:0/3480750806 shutdown_connections 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.466+0000 7fb90616e640 1 -- 172.21.15.116:0/3480750806 wait complete. 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb90616e640 1 Processor -- start 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb90616e640 1 -- start start 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb90616e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9000ff180 0x7fb90019aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb90616e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb90019afb0 0x7fb900194ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb90616e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90019b4f0 con 0x7fb9000ff180 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb90616e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90019b630 con 0x7fb90019afb0 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb8ff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9000ff180 0x7fb90019aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb8ff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9000ff180 0x7fb90019aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33094/0 (socket says 172.21.15.116:33094) 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb8ff7fe640 1 -- 172.21.15.116:0/4255100613 learned_addr learned my addr 172.21.15.116:0/4255100613 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb8ff7fe640 1 -- 172.21.15.116:0/4255100613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb90019afb0 msgr2=0x7fb900194ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb8ff7fe640 1 --2- 172.21.15.116:0/4255100613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb90019afb0 0x7fb900194ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb8ff7fe640 1 -- 172.21.15.116:0/4255100613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0009560 con 0x7fb9000ff180 2024-03-04T16:15:40.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.467+0000 7fb8ff7fe640 1 --2- 172.21.15.116:0/4255100613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9000ff180 0x7fb90019aa70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0009230 tx=0x7fb8f0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:40.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.468+0000 7fb8fcff9640 1 -- 172.21.15.116:0/4255100613 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8f00035f0 con 0x7fb9000ff180 2024-03-04T16:15:40.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.468+0000 7fb90616e640 1 -- 172.21.15.116:0/4255100613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb900195080 con 0x7fb9000ff180 2024-03-04T16:15:40.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.468+0000 7fb8fcff9640 1 -- 172.21.15.116:0/4255100613 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8f0003c10 con 0x7fb9000ff180 2024-03-04T16:15:40.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.468+0000 7fb8fcff9640 1 -- 172.21.15.116:0/4255100613 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8f0034440 con 0x7fb9000ff180 2024-03-04T16:15:40.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.468+0000 7fb90616e640 1 -- 172.21.15.116:0/4255100613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9001955e0 con 0x7fb9000ff180 2024-03-04T16:15:40.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.469+0000 7fb90616e640 1 -- 172.21.15.116:0/4255100613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90004e040 con 0x7fb9000ff180 2024-03-04T16:15:40.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.469+0000 7fb8fcff9640 1 -- 172.21.15.116:0/4255100613 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fb8f0004fd0 con 0x7fb9000ff180 2024-03-04T16:15:40.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.470+0000 7fb8fcff9640 1 --2- 172.21.15.116:0/4255100613 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8e806d810 0x7fb8e806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:40.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.470+0000 7fb8fcff9640 1 -- 172.21.15.116:0/4255100613 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fb8f00af9f0 con 0x7fb9000ff180 2024-03-04T16:15:40.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.470+0000 7fb8feffd640 1 --2- 172.21.15.116:0/4255100613 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8e806d810 0x7fb8e806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:40.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.470+0000 7fb8feffd640 1 --2- 172.21.15.116:0/4255100613 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8e806d810 0x7fb8e806fcd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb900069810 tx=0x7fb8f4005c70 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:40.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.474+0000 7fb8fcff9640 1 -- 172.21.15.116:0/4255100613 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb8f0079150 con 0x7fb9000ff180 2024-03-04T16:15:40.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.645+0000 7fb90616e640 1 -- 172.21.15.116:0/4255100613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fb9001958f0 con 0x7fb9000ff180 2024-03-04T16:15:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.646+0000 7fb8fcff9640 1 -- 172.21.15.116:0/4255100613 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb8f007c810 con 0x7fb9000ff180 2024-03-04T16:15:40.646 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T16:15:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 -- 172.21.15.116:0/4255100613 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8e806d810 msgr2=0x7fb8e806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 --2- 172.21.15.116:0/4255100613 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8e806d810 0x7fb8e806fcd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb900069810 tx=0x7fb8f4005c70 comp rx=0 tx=0).stop 2024-03-04T16:15:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 -- 172.21.15.116:0/4255100613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9000ff180 msgr2=0x7fb90019aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 --2- 172.21.15.116:0/4255100613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9000ff180 0x7fb90019aa70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0009230 tx=0x7fb8f0004040 comp rx=0 tx=0).stop 2024-03-04T16:15:40.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 -- 172.21.15.116:0/4255100613 shutdown_connections 2024-03-04T16:15:40.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 --2- 172.21.15.116:0/4255100613 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fb8e806d810 0x7fb8e806fcd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 --2- 172.21.15.116:0/4255100613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb90019afb0 0x7fb900194ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 --2- 172.21.15.116:0/4255100613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9000ff180 0x7fb90019aa70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 -- 172.21.15.116:0/4255100613 >> 172.21.15.116:0/4255100613 conn(0x7fb9000faa00 msgr2=0x7fb9000fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:40.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 -- 172.21.15.116:0/4255100613 shutdown_connections 2024-03-04T16:15:40.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.648+0000 7fb90616e640 1 -- 172.21.15.116:0/4255100613 wait complete. 2024-03-04T16:15:40.658 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:40 smithi116 ceph-mon[94164]: pgmap v330: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:40.658 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:40 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3207231443' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:15:40.658 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:40 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/538042917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:15:40.658 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:40 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2966812225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-04T16:15:40.659 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T16:15:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.761+0000 7fd3c21cb640 1 -- 172.21.15.116:0/1057037020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc100280 msgr2=0x7fd3bc100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.761+0000 7fd3c21cb640 1 --2- 172.21.15.116:0/1057037020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc100280 0x7fd3bc100660 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009880 tx=0x7fd3ac02bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.761+0000 7fd3c21cb640 1 -- 172.21.15.116:0/1057037020 shutdown_connections 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.761+0000 7fd3c21cb640 1 --2- 172.21.15.116:0/1057037020 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc100c30 0x7fd3bc104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.761+0000 7fd3c21cb640 1 --2- 172.21.15.116:0/1057037020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc100280 0x7fd3bc100660 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.761+0000 7fd3c21cb640 1 -- 172.21.15.116:0/1057037020 >> 172.21.15.116:0/1057037020 conn(0x7fd3bc0fbb20 msgr2=0x7fd3bc0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.761+0000 7fd3c21cb640 1 -- 172.21.15.116:0/1057037020 shutdown_connections 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.761+0000 7fd3c21cb640 1 -- 172.21.15.116:0/1057037020 wait complete. 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.761+0000 7fd3c21cb640 1 Processor -- start 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3c21cb640 1 -- start start 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3c21cb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc100280 0x7fd3bc0725a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3c21cb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc072ae0 0x7fd3bc072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3c21cb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc1a1e40 con 0x7fd3bc100280 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3c21cb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc1a1f80 con 0x7fd3bc072ae0 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc100280 0x7fd3bc0725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc100280 0x7fd3bc0725a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33118/0 (socket says 172.21.15.116:33118) 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3bb7fe640 1 -- 172.21.15.116:0/3870542534 learned_addr learned my addr 172.21.15.116:0/3870542534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3baffd640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc072ae0 0x7fd3bc072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3bb7fe640 1 -- 172.21.15.116:0/3870542534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc072ae0 msgr2=0x7fd3bc072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3bb7fe640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc072ae0 0x7fd3bc072f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3bb7fe640 1 -- 172.21.15.116:0/3870542534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac009560 con 0x7fd3bc100280 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3bb7fe640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc100280 0x7fd3bc0725a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac036040 tx=0x7fd3ac02c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3b8ff9640 1 -- 172.21.15.116:0/3870542534 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3ac004520 con 0x7fd3bc100280 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3b8ff9640 1 -- 172.21.15.116:0/3870542534 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3ac003ab0 con 0x7fd3bc100280 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3c21cb640 1 -- 172.21.15.116:0/3870542534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3bc1a20c0 con 0x7fd3bc100280 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3b8ff9640 1 -- 172.21.15.116:0/3870542534 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3ac034440 con 0x7fd3bc100280 2024-03-04T16:15:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.762+0000 7fd3c21cb640 1 -- 172.21.15.116:0/3870542534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3bc1a2570 con 0x7fd3bc100280 2024-03-04T16:15:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.764+0000 7fd3c21cb640 1 -- 172.21.15.116:0/3870542534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3bc04e040 con 0x7fd3bc100280 2024-03-04T16:15:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.764+0000 7fd3b8ff9640 1 -- 172.21.15.116:0/3870542534 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fd3ac003c20 con 0x7fd3bc100280 2024-03-04T16:15:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.764+0000 7fd3b8ff9640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3a406d8d0 0x7fd3a406fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.764+0000 7fd3b8ff9640 1 -- 172.21.15.116:0/3870542534 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fd3ac0af810 con 0x7fd3bc100280 2024-03-04T16:15:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.764+0000 7fd3baffd640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3a406d8d0 0x7fd3a406fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.765+0000 7fd3baffd640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3a406d8d0 0x7fd3a406fd90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0005fd0 tx=0x7fd3b0005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:40.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.769+0000 7fd3b8ff9640 1 -- 172.21.15.116:0/3870542534 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd3ac078ff0 con 0x7fd3bc100280 2024-03-04T16:15:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:40 smithi152 ceph-mon[82845]: pgmap v330: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:40 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3207231443' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:15:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:40 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/538042917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:15:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:40 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2966812225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-04T16:15:40.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.953+0000 7fd3c21cb640 1 -- 172.21.15.116:0/3870542534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd3bc108580 con 0x7fd3bc100280 2024-03-04T16:15:40.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.953+0000 7fd3b8ff9640 1 -- 172.21.15.116:0/3870542534 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 43 entries v84) v1 ==== 81+0+2575 (secure 0 0 0) 0x7fd3ac07c6b0 con 0x7fd3bc100280 2024-03-04T16:15:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6827/1672192052 2024-03-05T16:11:38.459377+0000 2024-03-04T16:15:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6826/1672192052 2024-03-05T16:11:38.459377+0000 2024-03-04T16:15:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6825/287206196 2024-03-05T16:11:14.444369+0000 2024-03-04T16:15:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6824/287206196 2024-03-05T16:11:14.444369+0000 2024-03-04T16:15:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3594007286 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/4255448814 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/317318762 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2770714257 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3287532122 2024-03-05T15:54:53.931186+0000 2024-03-04T16:15:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/2966163948 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/976694871 2024-03-05T15:54:21.166941+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/961078374 2024-03-05T15:54:21.166941+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/2020358480 2024-03-05T15:56:07.046998+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3356602598 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/1340152843 2024-03-05T15:54:53.931186+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3670178561 2024-03-05T15:56:07.046998+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3356602598 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2883878856 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/3493116756 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/2020358480 2024-03-05T15:56:07.046998+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/1831164264 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/1340152843 2024-03-05T15:54:53.931186+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6827/114739798 2024-03-05T16:01:15.387483+0000 2024-03-04T16:15:40.955 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/961078374 2024-03-05T15:54:21.166941+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/385581892 2024-03-05T15:54:21.166941+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/1293393009 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6829/2881662614 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/1265936494 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/831649128 2024-03-05T15:54:53.931186+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3323890559 2024-03-05T15:54:21.166941+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/355050440 2024-03-05T15:56:07.046998+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6828/2881662614 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/590312912 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/232576737 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/549268400 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/768114426 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:40.956 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/4142805454 2024-03-05T15:54:53.931186+0000 2024-03-04T16:15:40.957 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1756904921 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:40.957 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/821201733 2024-03-05T15:56:07.046998+0000 2024-03-04T16:15:40.957 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6826/114739798 2024-03-05T16:01:15.387483+0000 2024-03-04T16:15:40.957 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3213147898 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:40.957 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2867113891 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:40.957 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/2966163948 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:listed 43 entries 2024-03-04T16:15:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.955+0000 7fd3c21cb640 1 -- 172.21.15.116:0/3870542534 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3a406d8d0 msgr2=0x7fd3a406fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.955+0000 7fd3c21cb640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3a406d8d0 0x7fd3a406fd90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0005fd0 tx=0x7fd3b0005850 comp rx=0 tx=0).stop 2024-03-04T16:15:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.955+0000 7fd3c21cb640 1 -- 172.21.15.116:0/3870542534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc100280 msgr2=0x7fd3bc0725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.955+0000 7fd3c21cb640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc100280 0x7fd3bc0725a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac036040 tx=0x7fd3ac02c850 comp rx=0 tx=0).stop 2024-03-04T16:15:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.955+0000 7fd3c21cb640 1 -- 172.21.15.116:0/3870542534 shutdown_connections 2024-03-04T16:15:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.955+0000 7fd3c21cb640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fd3a406d8d0 0x7fd3a406fd90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.955+0000 7fd3c21cb640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc072ae0 0x7fd3bc072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.955+0000 7fd3c21cb640 1 --2- 172.21.15.116:0/3870542534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc100280 0x7fd3bc0725a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:40.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.955+0000 7fd3c21cb640 1 -- 172.21.15.116:0/3870542534 >> 172.21.15.116:0/3870542534 conn(0x7fd3bc0fbb20 msgr2=0x7fd3bc0fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:40.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.956+0000 7fd3c21cb640 1 -- 172.21.15.116:0/3870542534 shutdown_connections 2024-03-04T16:15:40.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:40.956+0000 7fd3c21cb640 1 -- 172.21.15.116:0/3870542534 wait complete. 2024-03-04T16:15:40.967 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T16:15:40.967 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T16:15:40.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T16:15:41.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.123+0000 7fe72addd640 1 -- 172.21.15.116:0/55383030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe724101150 msgr2=0x7fe724101530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.123+0000 7fe72addd640 1 --2- 172.21.15.116:0/55383030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe724101150 0x7fe724101530 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe7140098b0 tx=0x7fe71402bcd0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.123+0000 7fe72addd640 1 -- 172.21.15.116:0/55383030 shutdown_connections 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.123+0000 7fe72addd640 1 --2- 172.21.15.116:0/55383030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe724101b00 0x7fe724109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.123+0000 7fe72addd640 1 --2- 172.21.15.116:0/55383030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe724101150 0x7fe724101530 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.123+0000 7fe72addd640 1 -- 172.21.15.116:0/55383030 >> 172.21.15.116:0/55383030 conn(0x7fe7240faa00 msgr2=0x7fe7240fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.123+0000 7fe72addd640 1 -- 172.21.15.116:0/55383030 shutdown_connections 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.123+0000 7fe72addd640 1 -- 172.21.15.116:0/55383030 wait complete. 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe72addd640 1 Processor -- start 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe72addd640 1 -- start start 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe72addd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe724101150 0x7fe72419ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe72addd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe724101b00 0x7fe72419bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe72addd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe724195ac0 con 0x7fe724101b00 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe72addd640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe724195c30 con 0x7fe724101150 2024-03-04T16:15:41.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe724101b00 0x7fe72419bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe724101b00 0x7fe72419bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33148/0 (socket says 172.21.15.116:33148) 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe723fff640 1 -- 172.21.15.116:0/1776142533 learned_addr learned my addr 172.21.15.116:0/1776142533 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe723fff640 1 -- 172.21.15.116:0/1776142533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe724101150 msgr2=0x7fe72419ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe723fff640 1 --2- 172.21.15.116:0/1776142533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe724101150 0x7fe72419ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe723fff640 1 -- 172.21.15.116:0/1776142533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe714009560 con 0x7fe724101b00 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.124+0000 7fe723fff640 1 --2- 172.21.15.116:0/1776142533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe724101b00 0x7fe72419bf90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe71800d8d0 tx=0x7fe71800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.125+0000 7fe721ffb640 1 -- 172.21.15.116:0/1776142533 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe718009c00 con 0x7fe724101b00 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.125+0000 7fe721ffb640 1 -- 172.21.15.116:0/1776142533 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe718009d60 con 0x7fe724101b00 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.125+0000 7fe72addd640 1 -- 172.21.15.116:0/1776142533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe724195f10 con 0x7fe724101b00 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.125+0000 7fe721ffb640 1 -- 172.21.15.116:0/1776142533 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7180049d0 con 0x7fe724101b00 2024-03-04T16:15:41.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.125+0000 7fe72addd640 1 -- 172.21.15.116:0/1776142533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe724196470 con 0x7fe724101b00 2024-03-04T16:15:41.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.126+0000 7fe72addd640 1 -- 172.21.15.116:0/1776142533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72404e040 con 0x7fe724101b00 2024-03-04T16:15:41.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.126+0000 7fe721ffb640 1 -- 172.21.15.116:0/1776142533 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fe718004c90 con 0x7fe724101b00 2024-03-04T16:15:41.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.127+0000 7fe721ffb640 1 --2- 172.21.15.116:0/1776142533 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe70c06d8d0 0x7fe70c06fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:41.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.127+0000 7fe721ffb640 1 -- 172.21.15.116:0/1776142533 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fe71808d280 con 0x7fe724101b00 2024-03-04T16:15:41.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.127+0000 7fe728b79640 1 --2- 172.21.15.116:0/1776142533 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe70c06d8d0 0x7fe70c06fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:41.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.127+0000 7fe728b79640 1 --2- 172.21.15.116:0/1776142533 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe70c06d8d0 0x7fe70c06fd90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe714036040 tx=0x7fe714000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.131+0000 7fe721ffb640 1 -- 172.21.15.116:0/1776142533 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe7180569e0 con 0x7fe724101b00 2024-03-04T16:15:41.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.309+0000 7fe72addd640 1 -- 172.21.15.116:0/1776142533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7241966f0 con 0x7fe724101b00 2024-03-04T16:15:41.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.311+0000 7fe721ffb640 1 -- 172.21.15.116:0/1776142533 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16947 (secure 0 0 0) 0x7fe71805a0a0 con 0x7fe724101b00 2024-03-04T16:15:41.311 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T16:15:41.311 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":84,"fsid":"4960653c-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:53:48.465240+0000","modified":"2024-03-04T16:11:38.459407+0000","last_up_change":"2024-03-04T16:10:31.185124+0000","last_in_change":"2024-03-04T15:59:22.307720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:58:27.372335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-04T16:00:58.546762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-04T16:00:59.546717+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d12966f-954e-4a09-bbbd-5b0f6a04d039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6803","nonce":2502615728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6805","nonce":2502615728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6809","nonce":2502615728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2502615728},{"type":"v1","addr":"172.21.15.116:6807","nonce":2502615728}]},"public_addr":"172.21.15.116:6803/2502615728","cluster_addr":"172.21.15.116:6805/2502615728","heartbeat_back_addr":"172.21.15.116:6809/2502615728","heartbeat_front_addr":"172.21.15.116:6807/2502615728","state":["exists","up"]},{"osd":1,"uuid":"f990720f-a19d-48c1-9545-81e511c2690b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6811","nonce":22677649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6813","nonce":22677649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6817","nonce":22677649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":22677649},{"type":"v1","addr":"172.21.15.116:6815","nonce":22677649}]},"public_addr":"172.21.15.116:6811/22677649","cluster_addr":"172.21.15.116:6813/22677649","heartbeat_back_addr":"172.21.15.116:6817/22677649","heartbeat_front_addr":"172.21.15.116:6815/22677649","state":["exists","up"]},{"osd":2,"uuid":"fe3132c8-6b15-4a11-a884-7c6a09d42955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6819","nonce":1756826225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6821","nonce":1756826225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6825","nonce":1756826225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1756826225},{"type":"v1","addr":"172.21.15.116:6823","nonce":1756826225}]},"public_addr":"172.21.15.116:6819/1756826225","cluster_addr":"172.21.15.116:6821/1756826225","heartbeat_back_addr":"172.21.15.116:6825/1756826225","heartbeat_front_addr":"172.21.15.116:6823/1756826225","state":["exists","up"]},{"osd":3,"uuid":"15930a9b-f721-4c30-9c5b-bef5ef69777c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6801","nonce":3040811013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6803","nonce":3040811013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6807","nonce":3040811013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3040811013},{"type":"v1","addr":"172.21.15.152:6805","nonce":3040811013}]},"public_addr":"172.21.15.152:6801/3040811013","cluster_addr":"172.21.15.152:6803/3040811013","heartbeat_back_addr":"172.21.15.152:6807/3040811013","heartbeat_front_addr":"172.21.15.152:6805/3040811013","state":["exists","up"]},{"osd":4,"uuid":"8d201b6a-6a91-4a7e-984e-e8d7f0289793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6809","nonce":878265314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6811","nonce":878265314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6815","nonce":878265314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":878265314},{"type":"v1","addr":"172.21.15.152:6813","nonce":878265314}]},"public_addr":"172.21.15.152:6809/878265314","cluster_addr":"172.21.15.152:6811/878265314","heartbeat_back_addr":"172.21.15.152:6815/878265314","heartbeat_front_addr":"172.21.15.152:6813/878265314","state":["exists","up"]},{"osd":5,"uuid":"c06ef0f5-6894-4e01-a693-bb129d248fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6817","nonce":1180242057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6819","nonce":1180242057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6823","nonce":1180242057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1180242057},{"type":"v1","addr":"172.21.15.152:6821","nonce":1180242057}]},"public_addr":"172.21.15.152:6817/1180242057","cluster_addr":"172.21.15.152:6819/1180242057","heartbeat_back_addr":"172.21.15.152:6823/1180242057","heartbeat_front_addr":"172.21.15.152:6821/1180242057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:24.230215+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:57:52.709103+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:24.072974+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:47.684390+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:13.571654+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:39.874966+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.152:6827/1672192052":"2024-03-05T16:11:38.459377+0000","172.21.15.152:6826/1672192052":"2024-03-05T16:11:38.459377+0000","172.21.15.152:6825/287206196":"2024-03-05T16:11:14.444369+0000","172.21.15.152:6824/287206196":"2024-03-05T16:11:14.444369+0000","172.21.15.116:0/3594007286":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/4255448814":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/317318762":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/2770714257":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/3287532122":"2024-03-05T15:54:53.931186+0000","172.21.15.116:6801/2966163948":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/976694871":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6800/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:6801/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6801/3356602598":"2024-03-05T16:03:33.250227+0000","172.21.15.116:6800/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3670178561":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6800/3356602598":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/2883878856":"2024-03-05T16:05:52.994136+0000","172.21.15.152:0/3493116756":"2024-03-05T16:04:41.923774+0000","172.21.15.116:6800/2020358480":"2024-03-05T15:56:07.046998+0000","172.21.15.152:0/1831164264":"2024-03-05T16:04:41.923774+0000","172.21.15.116:6801/1340152843":"2024-03-05T15:54:53.931186+0000","172.21.15.116:6827/114739798":"2024-03-05T16:01:15.387483+0000","172.21.15.116:6801/961078374":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/385581892":"2024-03-05T15:54:21.166941+0000","172.21.15.152:0/1293393009":"2024-03-05T16:04:41.923774+0000","172.21.15.152:6829/2881662614":"2024-03-05T16:04:41.923774+0000","172.21.15.152:0/1265936494":"2024-03-05T16:04:41.923774+0000","172.21.15.116:0/831649128":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/3323890559":"2024-03-05T15:54:21.166941+0000","172.21.15.116:0/355050440":"2024-03-05T15:56:07.046998+0000","172.21.15.152:6828/2881662614":"2024-03-05T16:04:41.923774+0000","172.21.15.116:0/590312912":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/232576737":"2024-03-05T16:03:33.250227+0000","172.21.15.152:0/549268400":"2024-03-05T16:04:41.923774+0000","172.21.15.152:0/768114426":"2024-03-05T16:04:41.923774+0000","172.21.15.116:0/4142805454":"2024-03-05T15:54:53.931186+0000","172.21.15.116:0/1756904921":"2024-03-05T16:05:52.994136+0000","172.21.15.116:0/821201733":"2024-03-05T15:56:07.046998+0000","172.21.15.116:6826/114739798":"2024-03-05T16:01:15.387483+0000","172.21.15.116:0/3213147898":"2024-03-05T16:03:33.250227+0000","172.21.15.116:0/2867113891":"2024-03-05T16:05:52.994136+0000","172.21.15.116:6800/2966163948":"2024-03-05T16:05:52.994136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:15:41.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 -- 172.21.15.116:0/1776142533 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe70c06d8d0 msgr2=0x7fe70c06fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 --2- 172.21.15.116:0/1776142533 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe70c06d8d0 0x7fe70c06fd90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe714036040 tx=0x7fe714000bc0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 -- 172.21.15.116:0/1776142533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe724101b00 msgr2=0x7fe72419bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 --2- 172.21.15.116:0/1776142533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe724101b00 0x7fe72419bf90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe71800d8d0 tx=0x7fe71800dd90 comp rx=0 tx=0).stop 2024-03-04T16:15:41.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 -- 172.21.15.116:0/1776142533 shutdown_connections 2024-03-04T16:15:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 --2- 172.21.15.116:0/1776142533 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe70c06d8d0 0x7fe70c06fd90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 --2- 172.21.15.116:0/1776142533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe724101b00 0x7fe72419bf90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 --2- 172.21.15.116:0/1776142533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe724101150 0x7fe72419ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 -- 172.21.15.116:0/1776142533 >> 172.21.15.116:0/1776142533 conn(0x7fe7240faa00 msgr2=0x7fe724104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 -- 172.21.15.116:0/1776142533 shutdown_connections 2024-03-04T16:15:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.313+0000 7fe72addd640 1 -- 172.21.15.116:0/1776142533 wait complete. 2024-03-04T16:15:41.325 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T15:58:27.372335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-04T16:00:58.546762+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-04T16:00:59.546717+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T16:15:41.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T16:15:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.430+0000 7f8889dcb640 1 -- 172.21.15.116:0/3847719794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8884067f70 msgr2=0x7f8884104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.430+0000 7f8889dcb640 1 --2- 172.21.15.116:0/3847719794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8884067f70 0x7f8884104c70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8874009880 tx=0x7f887402bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.430+0000 7f8889dcb640 1 -- 172.21.15.116:0/3847719794 shutdown_connections 2024-03-04T16:15:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.430+0000 7f8889dcb640 1 --2- 172.21.15.116:0/3847719794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8884067f70 0x7f8884104c70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.430+0000 7f8889dcb640 1 --2- 172.21.15.116:0/3847719794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88840675c0 0x7f88840679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.430+0000 7f8889dcb640 1 -- 172.21.15.116:0/3847719794 >> 172.21.15.116:0/3847719794 conn(0x7f8884074a50 msgr2=0x7f8884074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:41.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.430+0000 7f8889dcb640 1 -- 172.21.15.116:0/3847719794 shutdown_connections 2024-03-04T16:15:41.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.430+0000 7f8889dcb640 1 -- 172.21.15.116:0/3847719794 wait complete. 2024-03-04T16:15:41.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.431+0000 7f8889dcb640 1 Processor -- start 2024-03-04T16:15:41.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.431+0000 7f8889dcb640 1 -- start start 2024-03-04T16:15:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.431+0000 7f8889dcb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88840675c0 0x7f8884195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.431+0000 7f8889dcb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88841962c0 0x7f888419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.431+0000 7f8889dcb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88841968c0 con 0x7f88841962c0 2024-03-04T16:15:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.431+0000 7f8889dcb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8884196a30 con 0x7f88840675c0 2024-03-04T16:15:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.431+0000 7f8882ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88841962c0 0x7f888419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.431+0000 7f8882ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88841962c0 0x7f888419a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33160/0 (socket says 172.21.15.116:33160) 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.431+0000 7f8882ffd640 1 -- 172.21.15.116:0/178460986 learned_addr learned my addr 172.21.15.116:0/178460986 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f88837fe640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88840675c0 0x7f8884195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f8882ffd640 1 -- 172.21.15.116:0/178460986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88840675c0 msgr2=0x7f8884195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f8882ffd640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88840675c0 0x7f8884195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f8882ffd640 1 -- 172.21.15.116:0/178460986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8874009560 con 0x7f88841962c0 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f88837fe640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88840675c0 0x7f8884195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f8882ffd640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88841962c0 0x7f888419a720 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8874036040 tx=0x7f887402c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f8880ff9640 1 -- 172.21.15.116:0/178460986 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88740035f0 con 0x7f88841962c0 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f8889dcb640 1 -- 172.21.15.116:0/178460986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888419acc0 con 0x7f88841962c0 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f8880ff9640 1 -- 172.21.15.116:0/178460986 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8874003c10 con 0x7f88841962c0 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f8880ff9640 1 -- 172.21.15.116:0/178460986 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8874005630 con 0x7f88841962c0 2024-03-04T16:15:41.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.432+0000 7f8889dcb640 1 -- 172.21.15.116:0/178460986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888419b220 con 0x7f88841962c0 2024-03-04T16:15:41.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.433+0000 7f8889dcb640 1 -- 172.21.15.116:0/178460986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888404e040 con 0x7f88841962c0 2024-03-04T16:15:41.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.434+0000 7f8880ff9640 1 -- 172.21.15.116:0/178460986 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f8874003750 con 0x7f88841962c0 2024-03-04T16:15:41.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.434+0000 7f8880ff9640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f886c06d810 0x7f886c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:41.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.434+0000 7f8880ff9640 1 -- 172.21.15.116:0/178460986 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f88740af950 con 0x7f88841962c0 2024-03-04T16:15:41.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.434+0000 7f88837fe640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f886c06d810 0x7f886c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:41.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.435+0000 7f88837fe640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f886c06d810 0x7f886c06fcd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8878009b50 tx=0x7f8878005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:41.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.439+0000 7f8880ff9640 1 -- 172.21.15.116:0/178460986 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f88740790b0 con 0x7f88841962c0 2024-03-04T16:15:41.493 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:41 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/4255100613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-04T16:15:41.493 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:41 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/3870542534' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:15:41.493 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:41 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1776142533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:15:41.493 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:41 smithi116 ceph-mon[94164]: pgmap v331: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:41.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.623+0000 7f8889dcb640 1 -- 172.21.15.116:0/178460986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88841970d0 con 0x7f88841962c0 2024-03-04T16:15:41.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.624+0000 7f8880ff9640 1 -- 172.21.15.116:0/178460986 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f887407c770 con 0x7f88841962c0 2024-03-04T16:15:41.624 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 -- 172.21.15.116:0/178460986 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f886c06d810 msgr2=0x7f886c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f886c06d810 0x7f886c06fcd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8878009b50 tx=0x7f8878005f90 comp rx=0 tx=0).stop 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 -- 172.21.15.116:0/178460986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88841962c0 msgr2=0x7f888419a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88841962c0 0x7f888419a720 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8874036040 tx=0x7f887402c750 comp rx=0 tx=0).stop 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 -- 172.21.15.116:0/178460986 shutdown_connections 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f886c06d810 0x7f886c06fcd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88841962c0 0x7f888419a720 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 --2- 172.21.15.116:0/178460986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88840675c0 0x7f8884195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 -- 172.21.15.116:0/178460986 >> 172.21.15.116:0/178460986 conn(0x7f8884074a50 msgr2=0x7f88840fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 -- 172.21.15.116:0/178460986 shutdown_connections 2024-03-04T16:15:41.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.626+0000 7f8889dcb640 1 -- 172.21.15.116:0/178460986 wait complete. 2024-03-04T16:15:41.638 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-04T16:15:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.746+0000 7f870f116640 1 -- 172.21.15.116:0/890193872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708100c90 msgr2=0x7f8708104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.746+0000 7f870f116640 1 --2- 172.21.15.116:0/890193872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708100c90 0x7f8708104cd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009880 tx=0x7f86f402bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 -- 172.21.15.116:0/890193872 shutdown_connections 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 --2- 172.21.15.116:0/890193872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708100c90 0x7f8708104cd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 --2- 172.21.15.116:0/890193872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87081002e0 0x7f87081006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 -- 172.21.15.116:0/890193872 >> 172.21.15.116:0/890193872 conn(0x7f87080fbb80 msgr2=0x7f87080fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 -- 172.21.15.116:0/890193872 shutdown_connections 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 -- 172.21.15.116:0/890193872 wait complete. 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 Processor -- start 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 -- start start 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87081002e0 0x7f8708195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708196300 0x7f870819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8708196900 con 0x7f8708196300 2024-03-04T16:15:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f870f116640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8708196a70 con 0x7f87081002e0 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f8707fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708196300 0x7f870819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.747+0000 7f8707fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708196300 0x7f870819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33168/0 (socket says 172.21.15.116:33168) 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.748+0000 7f8707fff640 1 -- 172.21.15.116:0/1329313434 learned_addr learned my addr 172.21.15.116:0/1329313434 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.748+0000 7f8707fff640 1 -- 172.21.15.116:0/1329313434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87081002e0 msgr2=0x7f8708195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.748+0000 7f8707fff640 1 --2- 172.21.15.116:0/1329313434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87081002e0 0x7f8708195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.748+0000 7f8707fff640 1 -- 172.21.15.116:0/1329313434 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f4009560 con 0x7f8708196300 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.748+0000 7f8707fff640 1 --2- 172.21.15.116:0/1329313434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708196300 0x7f870819a760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f86f4036040 tx=0x7f86f402c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.748+0000 7f8705ffb640 1 -- 172.21.15.116:0/1329313434 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86f40035f0 con 0x7f8708196300 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.748+0000 7f8705ffb640 1 -- 172.21.15.116:0/1329313434 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86f4003c10 con 0x7f8708196300 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.748+0000 7f870f116640 1 -- 172.21.15.116:0/1329313434 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870819ad00 con 0x7f8708196300 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.748+0000 7f8705ffb640 1 -- 172.21.15.116:0/1329313434 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86f4034440 con 0x7f8708196300 2024-03-04T16:15:41.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.748+0000 7f870f116640 1 -- 172.21.15.116:0/1329313434 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870819b260 con 0x7f8708196300 2024-03-04T16:15:41.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.749+0000 7f870f116640 1 -- 172.21.15.116:0/1329313434 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870804e040 con 0x7f8708196300 2024-03-04T16:15:41.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.750+0000 7f8705ffb640 1 -- 172.21.15.116:0/1329313434 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f86f4003750 con 0x7f8708196300 2024-03-04T16:15:41.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.750+0000 7f8705ffb640 1 --2- 172.21.15.116:0/1329313434 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86f806d810 0x7f86f806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:41.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.750+0000 7f8705ffb640 1 -- 172.21.15.116:0/1329313434 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f86f40af930 con 0x7f8708196300 2024-03-04T16:15:41.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.750+0000 7f870ceb2640 1 --2- 172.21.15.116:0/1329313434 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86f806d810 0x7f86f806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:41.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.751+0000 7f870ceb2640 1 --2- 172.21.15.116:0/1329313434 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86f806d810 0x7f86f806fcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f86fc005fd0 tx=0x7f86fc005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:41.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.755+0000 7f8705ffb640 1 -- 172.21.15.116:0/1329313434 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f86f4079090 con 0x7f8708196300 2024-03-04T16:15:41.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:41 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/4255100613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-04T16:15:41.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:41 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/3870542534' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:15:41.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:41 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1776142533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:15:41.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:41 smithi152 ceph-mon[82845]: pgmap v331: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.945+0000 7f870f116640 1 -- 172.21.15.116:0/1329313434 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f87081085e0 con 0x7f8708196300 2024-03-04T16:15:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.946+0000 7f8705ffb640 1 -- 172.21.15.116:0/1329313434 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f86f4081e60 con 0x7f8708196300 2024-03-04T16:15:41.946 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T16:15:41.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 -- 172.21.15.116:0/1329313434 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86f806d810 msgr2=0x7f86f806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 --2- 172.21.15.116:0/1329313434 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86f806d810 0x7f86f806fcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f86fc005fd0 tx=0x7f86fc005850 comp rx=0 tx=0).stop 2024-03-04T16:15:41.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 -- 172.21.15.116:0/1329313434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708196300 msgr2=0x7f870819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:41.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 --2- 172.21.15.116:0/1329313434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708196300 0x7f870819a760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f86f4036040 tx=0x7f86f402c850 comp rx=0 tx=0).stop 2024-03-04T16:15:41.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 -- 172.21.15.116:0/1329313434 shutdown_connections 2024-03-04T16:15:41.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 --2- 172.21.15.116:0/1329313434 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f86f806d810 0x7f86f806fcd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 --2- 172.21.15.116:0/1329313434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708196300 0x7f870819a760 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 --2- 172.21.15.116:0/1329313434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87081002e0 0x7f8708195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:41.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 -- 172.21.15.116:0/1329313434 >> 172.21.15.116:0/1329313434 conn(0x7f87080fbb80 msgr2=0x7f87080fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:41.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 -- 172.21.15.116:0/1329313434 shutdown_connections 2024-03-04T16:15:41.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:41.948+0000 7f870f116640 1 -- 172.21.15.116:0/1329313434 wait complete. 2024-03-04T16:15:41.959 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-04T16:15:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 -- 172.21.15.116:0/2625571575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43801002d0 msgr2=0x7f43801006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 --2- 172.21.15.116:0/2625571575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43801002d0 0x7f43801006b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4370009880 tx=0x7f437002bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 -- 172.21.15.116:0/2625571575 shutdown_connections 2024-03-04T16:15:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 --2- 172.21.15.116:0/2625571575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4380100c80 0x7f4380104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 --2- 172.21.15.116:0/2625571575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43801002d0 0x7f43801006b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 -- 172.21.15.116:0/2625571575 >> 172.21.15.116:0/2625571575 conn(0x7f43800fbb50 msgr2=0x7f43800fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 -- 172.21.15.116:0/2625571575 shutdown_connections 2024-03-04T16:15:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 -- 172.21.15.116:0/2625571575 wait complete. 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 Processor -- start 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 -- start start 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43801002d0 0x7f4380195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43801962e0 0x7f438019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43801968e0 con 0x7f43801962e0 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.065+0000 7f438845f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4380196a50 con 0x7f43801002d0 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43859fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43801962e0 0x7f438019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43859fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43801962e0 0x7f438019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33188/0 (socket says 172.21.15.116:33188) 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43859fa640 1 -- 172.21.15.116:0/634627243 learned_addr learned my addr 172.21.15.116:0/634627243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43861fb640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43801002d0 0x7f4380195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43859fa640 1 -- 172.21.15.116:0/634627243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43801002d0 msgr2=0x7f4380195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43859fa640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43801002d0 0x7f4380195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43859fa640 1 -- 172.21.15.116:0/634627243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370009560 con 0x7f43801962e0 2024-03-04T16:15:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43859fa640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43801962e0 0x7f438019a740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f437c00d8d0 tx=0x7f437c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43777fe640 1 -- 172.21.15.116:0/634627243 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f437c00bea0 con 0x7f43801962e0 2024-03-04T16:15:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43777fe640 1 -- 172.21.15.116:0/634627243 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f437c009c00 con 0x7f43801962e0 2024-03-04T16:15:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f438845f640 1 -- 172.21.15.116:0/634627243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438019ad40 con 0x7f43801962e0 2024-03-04T16:15:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f43777fe640 1 -- 172.21.15.116:0/634627243 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f437c010460 con 0x7f43801962e0 2024-03-04T16:15:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.066+0000 7f438845f640 1 -- 172.21.15.116:0/634627243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438019b2a0 con 0x7f43801962e0 2024-03-04T16:15:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.067+0000 7f438845f640 1 -- 172.21.15.116:0/634627243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438004e040 con 0x7f43801962e0 2024-03-04T16:15:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.068+0000 7f43777fe640 1 -- 172.21.15.116:0/634627243 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7f437c003f10 con 0x7f43801962e0 2024-03-04T16:15:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.068+0000 7f43777fe640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f436c06d810 0x7f436c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.069+0000 7f43777fe640 1 -- 172.21.15.116:0/634627243 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f437c08d360 con 0x7f43801962e0 2024-03-04T16:15:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.069+0000 7f43861fb640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f436c06d810 0x7f436c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.069+0000 7f43861fb640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f436c06d810 0x7f436c06fcd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4370036040 tx=0x7f4370005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:42.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.073+0000 7f43777fe640 1 -- 172.21.15.116:0/634627243 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f437c056ac0 con 0x7f43801962e0 2024-03-04T16:15:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.262+0000 7f438845f640 1 -- 172.21.15.116:0/634627243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f43801085d0 con 0x7f43801962e0 2024-03-04T16:15:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.262+0000 7f43777fe640 1 -- 172.21.15.116:0/634627243 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f437c05a180 con 0x7f43801962e0 2024-03-04T16:15:42.262 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T16:15:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 -- 172.21.15.116:0/634627243 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f436c06d810 msgr2=0x7f436c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f436c06d810 0x7f436c06fcd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4370036040 tx=0x7f4370005fd0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 -- 172.21.15.116:0/634627243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43801962e0 msgr2=0x7f438019a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43801962e0 0x7f438019a740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f437c00d8d0 tx=0x7f437c00dd90 comp rx=0 tx=0).stop 2024-03-04T16:15:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 -- 172.21.15.116:0/634627243 shutdown_connections 2024-03-04T16:15:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7f436c06d810 0x7f436c06fcd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43801962e0 0x7f438019a740 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 --2- 172.21.15.116:0/634627243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43801002d0 0x7f4380195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 -- 172.21.15.116:0/634627243 >> 172.21.15.116:0/634627243 conn(0x7f43800fbb50 msgr2=0x7f43800fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 -- 172.21.15.116:0/634627243 shutdown_connections 2024-03-04T16:15:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.264+0000 7f438845f640 1 -- 172.21.15.116:0/634627243 wait complete. 2024-03-04T16:15:42.275 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T16:15:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.379+0000 7fe6c130a640 1 -- 172.21.15.116:0/1104655930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc0fe720 msgr2=0x7fe6bc0feb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.379+0000 7fe6c130a640 1 --2- 172.21.15.116:0/1104655930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc0fe720 0x7fe6bc0feb80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009880 tx=0x7fe6ac02bce0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.379+0000 7fe6c130a640 1 -- 172.21.15.116:0/1104655930 shutdown_connections 2024-03-04T16:15:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.379+0000 7fe6c130a640 1 --2- 172.21.15.116:0/1104655930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc0fe720 0x7fe6bc0feb80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.379+0000 7fe6c130a640 1 --2- 172.21.15.116:0/1104655930 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6bc106200 0x7fe6bc1065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.379+0000 7fe6c130a640 1 -- 172.21.15.116:0/1104655930 >> 172.21.15.116:0/1104655930 conn(0x7fe6bc0752f0 msgr2=0x7fe6bc075700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.379+0000 7fe6c130a640 1 -- 172.21.15.116:0/1104655930 shutdown_connections 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.379+0000 7fe6c130a640 1 -- 172.21.15.116:0/1104655930 wait complete. 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.379+0000 7fe6c130a640 1 Processor -- start 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6c130a640 1 -- start start 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6c130a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6bc0fe720 0x7fe6bc195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6c130a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc106200 0x7fe6bc196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6c130a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc196920 con 0x7fe6bc106200 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6c130a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc19a6f0 con 0x7fe6bc0fe720 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6ba7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc106200 0x7fe6bc196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6ba7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc106200 0x7fe6bc196290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33196/0 (socket says 172.21.15.116:33196) 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6ba7fc640 1 -- 172.21.15.116:0/2980062893 learned_addr learned my addr 172.21.15.116:0/2980062893 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T16:15:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6baffd640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6bc0fe720 0x7fe6bc195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6ba7fc640 1 -- 172.21.15.116:0/2980062893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6bc0fe720 msgr2=0x7fe6bc195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6ba7fc640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6bc0fe720 0x7fe6bc195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6ba7fc640 1 -- 172.21.15.116:0/2980062893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac009560 con 0x7fe6bc106200 2024-03-04T16:15:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6baffd640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6bc0fe720 0x7fe6bc195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T16:15:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.380+0000 7fe6ba7fc640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc106200 0x7fe6bc196290 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009230 tx=0x7fe6ac004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.381+0000 7fe6a3fff640 1 -- 172.21.15.116:0/2980062893 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6ac0035f0 con 0x7fe6bc106200 2024-03-04T16:15:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.381+0000 7fe6c130a640 1 -- 172.21.15.116:0/2980062893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6bc19a970 con 0x7fe6bc106200 2024-03-04T16:15:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.381+0000 7fe6a3fff640 1 -- 172.21.15.116:0/2980062893 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6ac003c10 con 0x7fe6bc106200 2024-03-04T16:15:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.381+0000 7fe6a3fff640 1 -- 172.21.15.116:0/2980062893 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6ac005670 con 0x7fe6bc106200 2024-03-04T16:15:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.381+0000 7fe6c130a640 1 -- 172.21.15.116:0/2980062893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6bc19ae70 con 0x7fe6bc106200 2024-03-04T16:15:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.382+0000 7fe6c130a640 1 -- 172.21.15.116:0/2980062893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6bc04e040 con 0x7fe6bc106200 2024-03-04T16:15:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.383+0000 7fe6a3fff640 1 -- 172.21.15.116:0/2980062893 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 37) v1 ==== 91485+0+0 (secure 0 0 0) 0x7fe6ac004dd0 con 0x7fe6bc106200 2024-03-04T16:15:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.383+0000 7fe6a3fff640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe6a406d870 0x7fe6a406fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T16:15:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.383+0000 7fe6a3fff640 1 -- 172.21.15.116:0/2980062893 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fe6ac0af950 con 0x7fe6bc106200 2024-03-04T16:15:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.383+0000 7fe6baffd640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe6a406d870 0x7fe6a406fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T16:15:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.383+0000 7fe6baffd640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe6a406d870 0x7fe6a406fd30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00058c0 tx=0x7fe6b0005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T16:15:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.387+0000 7fe6a3fff640 1 -- 172.21.15.116:0/2980062893 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe6ac0790b0 con 0x7fe6bc106200 2024-03-04T16:15:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.576+0000 7fe6c130a640 1 -- 172.21.15.116:0/2980062893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe6bc1970a0 con 0x7fe6bc106200 2024-03-04T16:15:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.577+0000 7fe6a3fff640 1 -- 172.21.15.116:0/2980062893 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 43 entries v84) v1 ==== 81+0+2575 (secure 0 0 0) 0x7fe6ac07c770 con 0x7fe6bc106200 2024-03-04T16:15:42.577 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6827/1672192052 2024-03-05T16:11:38.459377+0000 2024-03-04T16:15:42.577 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6826/1672192052 2024-03-05T16:11:38.459377+0000 2024-03-04T16:15:42.577 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6825/287206196 2024-03-05T16:11:14.444369+0000 2024-03-04T16:15:42.577 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6824/287206196 2024-03-05T16:11:14.444369+0000 2024-03-04T16:15:42.577 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3594007286 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:42.577 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/4255448814 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/317318762 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2770714257 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3287532122 2024-03-05T15:54:53.931186+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/2966163948 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/976694871 2024-03-05T15:54:21.166941+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/961078374 2024-03-05T15:54:21.166941+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/2020358480 2024-03-05T15:56:07.046998+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3356602598 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/1340152843 2024-03-05T15:54:53.931186+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3670178561 2024-03-05T15:56:07.046998+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3356602598 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2883878856 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:42.578 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/3493116756 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/2020358480 2024-03-05T15:56:07.046998+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/1831164264 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/1340152843 2024-03-05T15:54:53.931186+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6827/114739798 2024-03-05T16:01:15.387483+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/961078374 2024-03-05T15:54:21.166941+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/385581892 2024-03-05T15:54:21.166941+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/1293393009 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6829/2881662614 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/1265936494 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/831649128 2024-03-05T15:54:53.931186+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3323890559 2024-03-05T15:54:21.166941+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/355050440 2024-03-05T15:56:07.046998+0000 2024-03-04T16:15:42.579 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:6828/2881662614 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/590312912 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/232576737 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/549268400 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.152:0/768114426 2024-03-05T16:04:41.923774+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/4142805454 2024-03-05T15:54:53.931186+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1756904921 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/821201733 2024-03-05T15:56:07.046998+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6826/114739798 2024-03-05T16:01:15.387483+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3213147898 2024-03-05T16:03:33.250227+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2867113891 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:42.580 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/2966163948 2024-03-05T16:05:52.994136+0000 2024-03-04T16:15:42.581 INFO:teuthology.orchestra.run.smithi116.stderr:listed 43 entries 2024-03-04T16:15:42.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.578+0000 7fe6c130a640 1 -- 172.21.15.116:0/2980062893 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe6a406d870 msgr2=0x7fe6a406fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:42.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.578+0000 7fe6c130a640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe6a406d870 0x7fe6a406fd30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00058c0 tx=0x7fe6b0005850 comp rx=0 tx=0).stop 2024-03-04T16:15:42.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.578+0000 7fe6c130a640 1 -- 172.21.15.116:0/2980062893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc106200 msgr2=0x7fe6bc196290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T16:15:42.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.578+0000 7fe6c130a640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc106200 0x7fe6bc196290 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009230 tx=0x7fe6ac004040 comp rx=0 tx=0).stop 2024-03-04T16:15:42.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.579+0000 7fe6c130a640 1 -- 172.21.15.116:0/2980062893 shutdown_connections 2024-03-04T16:15:42.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.579+0000 7fe6c130a640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.116:6800/2954839814,v1:172.21.15.116:6801/2954839814] conn(0x7fe6a406d870 0x7fe6a406fd30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.579+0000 7fe6c130a640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc106200 0x7fe6bc196290 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.579+0000 7fe6c130a640 1 --2- 172.21.15.116:0/2980062893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6bc0fe720 0x7fe6bc195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T16:15:42.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.579+0000 7fe6c130a640 1 -- 172.21.15.116:0/2980062893 >> 172.21.15.116:0/2980062893 conn(0x7fe6bc0752f0 msgr2=0x7fe6bc0fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T16:15:42.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.579+0000 7fe6c130a640 1 -- 172.21.15.116:0/2980062893 shutdown_connections 2024-03-04T16:15:42.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T16:15:42.579+0000 7fe6c130a640 1 -- 172.21.15.116:0/2980062893 wait complete. 2024-03-04T16:15:42.588 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:42 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/178460986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:15:42.589 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:42 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/1329313434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-04T16:15:42.589 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:42 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/634627243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-04T16:15:42.590 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2024-03-04T16:15:42.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T16:15:42.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-04T16:15:42.617 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T16:15:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:42 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/178460986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:15:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:42 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/1329313434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-04T16:15:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:42 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/634627243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-04T16:15:43.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:43 smithi116 ceph-mon[94164]: from='client.? 172.21.15.116:0/2980062893' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:15:43.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:43 smithi116 ceph-mon[94164]: pgmap v332: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:43.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:43 smithi152 ceph-mon[82845]: from='client.? 172.21.15.116:0/2980062893' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:15:43.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:43 smithi152 ceph-mon[82845]: pgmap v332: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:46 smithi116 ceph-mon[94164]: pgmap v333: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:46.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:46 smithi152 ceph-mon[82845]: pgmap v333: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:48 smithi116 ceph-mon[94164]: pgmap v334: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:48.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:48 smithi152 ceph-mon[82845]: pgmap v334: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:50.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:49 smithi152 ceph-mon[82845]: pgmap v335: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:49 smithi116 ceph-mon[94164]: pgmap v335: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:52 smithi116 ceph-mon[94164]: pgmap v336: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:52 smithi152 ceph-mon[82845]: pgmap v336: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:53.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:53.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:54.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:54 smithi152 ceph-mon[82845]: pgmap v337: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:54 smithi116 ceph-mon[94164]: pgmap v337: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:56.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:56 smithi116 ceph-mon[94164]: pgmap v338: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:56.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:56 smithi152 ceph-mon[82845]: pgmap v338: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:15:58 smithi116 ceph-mon[94164]: pgmap v339: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:15:58.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:15:58 smithi152 ceph-mon[82845]: pgmap v339: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:00 smithi116 ceph-mon[94164]: pgmap v340: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:00.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:00 smithi152 ceph-mon[82845]: pgmap v340: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:02.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:02 smithi116 ceph-mon[94164]: pgmap v341: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:02.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:02 smithi152 ceph-mon[82845]: pgmap v341: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:04.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:04 smithi116 ceph-mon[94164]: pgmap v342: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:04.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:04 smithi152 ceph-mon[82845]: pgmap v342: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:06.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:06 smithi116 ceph-mon[94164]: pgmap v343: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:06.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:06 smithi152 ceph-mon[82845]: pgmap v343: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:08.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:08 smithi116 ceph-mon[94164]: pgmap v344: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:08.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:08 smithi152 ceph-mon[82845]: pgmap v344: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:09.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:09 smithi116 ceph-mon[94164]: pgmap v345: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:09.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:09 smithi152 ceph-mon[82845]: pgmap v345: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:12.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:12 smithi116 ceph-mon[94164]: pgmap v346: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:12.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:12 smithi152 ceph-mon[82845]: pgmap v346: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:14.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:14 smithi116 ceph-mon[94164]: pgmap v347: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:14.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:14 smithi152 ceph-mon[82845]: pgmap v347: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:16 smithi116 ceph-mon[94164]: pgmap v348: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:16.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:16 smithi152 ceph-mon[82845]: pgmap v348: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:18.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:18 smithi116 ceph-mon[94164]: pgmap v349: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:18.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:18 smithi152 ceph-mon[82845]: pgmap v349: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:20 smithi116 ceph-mon[94164]: pgmap v350: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:20.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:20 smithi152 ceph-mon[82845]: pgmap v350: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:22 smithi116 ceph-mon[94164]: pgmap v351: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:22 smithi152 ceph-mon[82845]: pgmap v351: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:23.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:24.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:24 smithi116 ceph-mon[94164]: pgmap v352: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:24.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:24 smithi152 ceph-mon[82845]: pgmap v352: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:26.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:26 smithi116 ceph-mon[94164]: pgmap v353: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:26.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:26 smithi152 ceph-mon[82845]: pgmap v353: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:28 smithi116 ceph-mon[94164]: pgmap v354: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:28.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:28 smithi152 ceph-mon[82845]: pgmap v354: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:29.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:29 smithi116 ceph-mon[94164]: pgmap v355: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:29.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:29 smithi152 ceph-mon[82845]: pgmap v355: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:32.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:32 smithi116 ceph-mon[94164]: pgmap v356: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:32.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:32 smithi152 ceph-mon[82845]: pgmap v356: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:34.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:34 smithi116 ceph-mon[94164]: pgmap v357: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:34.741 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:34 smithi152 ceph-mon[82845]: pgmap v357: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:36.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:36 smithi116 ceph-mon[94164]: pgmap v358: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:36.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:36 smithi152 ceph-mon[82845]: pgmap v358: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:38 smithi116 ceph-mon[94164]: pgmap v359: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:38 smithi152 ceph-mon[82845]: pgmap v359: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:39.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:16:39.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:16:39.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:39 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:16:39.791 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:16:39.791 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:16:39.791 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:39 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:16:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:40 smithi116 ceph-mon[94164]: pgmap v360: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:40 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:16:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:40 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:16:40.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:40 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:16:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:40 smithi152 ceph-mon[82845]: pgmap v360: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:40 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:16:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:40 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:16:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:40 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:16:42.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:42 smithi116 ceph-mon[94164]: pgmap v361: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:42 smithi152 ceph-mon[82845]: pgmap v361: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:44 smithi116 ceph-mon[94164]: pgmap v362: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:44.837 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:44 smithi152 ceph-mon[82845]: pgmap v362: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:45.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:45 smithi152 ceph-mon[82845]: pgmap v363: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:45 smithi116 ceph-mon[94164]: pgmap v363: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:48 smithi116 ceph-mon[94164]: pgmap v364: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:48.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:48 smithi152 ceph-mon[82845]: pgmap v364: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:50 smithi116 ceph-mon[94164]: pgmap v365: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:50.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:50 smithi152 ceph-mon[82845]: pgmap v365: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:52 smithi116 ceph-mon[94164]: pgmap v366: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:52 smithi152 ceph-mon[82845]: pgmap v366: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:53.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:53.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:54 smithi116 ceph-mon[94164]: pgmap v367: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:54.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:54 smithi152 ceph-mon[82845]: pgmap v367: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:56.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:56 smithi116 ceph-mon[94164]: pgmap v368: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:56.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:56 smithi152 ceph-mon[82845]: pgmap v368: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:16:58 smithi116 ceph-mon[94164]: pgmap v369: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:16:58.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:16:58 smithi152 ceph-mon[82845]: pgmap v369: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:00 smithi116 ceph-mon[94164]: pgmap v370: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:00.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:00 smithi152 ceph-mon[82845]: pgmap v370: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:01.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:01 smithi116 ceph-mon[94164]: pgmap v371: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:01.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:01 smithi152 ceph-mon[82845]: pgmap v371: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:04.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:04 smithi116 ceph-mon[94164]: pgmap v372: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:04.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:04 smithi152 ceph-mon[82845]: pgmap v372: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:06.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:06 smithi116 ceph-mon[94164]: pgmap v373: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:06.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:06 smithi152 ceph-mon[82845]: pgmap v373: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:08.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:08 smithi116 ceph-mon[94164]: pgmap v374: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:08.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:08 smithi152 ceph-mon[82845]: pgmap v374: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:10.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:10 smithi116 ceph-mon[94164]: pgmap v375: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:10.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:10 smithi152 ceph-mon[82845]: pgmap v375: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:12.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:12 smithi116 ceph-mon[94164]: pgmap v376: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:12.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:12 smithi152 ceph-mon[82845]: pgmap v376: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:14.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:14 smithi116 ceph-mon[94164]: pgmap v377: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:14.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:14 smithi152 ceph-mon[82845]: pgmap v377: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:16 smithi116 ceph-mon[94164]: pgmap v378: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:16.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:16 smithi152 ceph-mon[82845]: pgmap v378: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:18.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:18 smithi116 ceph-mon[94164]: pgmap v379: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:18.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:18 smithi152 ceph-mon[82845]: pgmap v379: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:19.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:19 smithi152 ceph-mon[82845]: pgmap v380: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:19.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:19 smithi116 ceph-mon[94164]: pgmap v380: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:22 smithi116 ceph-mon[94164]: pgmap v381: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:22 smithi152 ceph-mon[82845]: pgmap v381: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:23.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:24.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:24 smithi116 ceph-mon[94164]: pgmap v382: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:24.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:24 smithi152 ceph-mon[82845]: pgmap v382: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:26.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:26 smithi116 ceph-mon[94164]: pgmap v383: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:26.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:26 smithi152 ceph-mon[82845]: pgmap v383: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:28 smithi116 ceph-mon[94164]: pgmap v384: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:28.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:28 smithi152 ceph-mon[82845]: pgmap v384: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:30.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:30 smithi152 ceph-mon[82845]: pgmap v385: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:30.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:30 smithi116 ceph-mon[94164]: pgmap v385: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:32.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:32 smithi116 ceph-mon[94164]: pgmap v386: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:32.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:32 smithi152 ceph-mon[82845]: pgmap v386: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:33.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:33 smithi152 ceph-mon[82845]: pgmap v387: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:33.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:33 smithi116 ceph-mon[94164]: pgmap v387: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:36.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:36 smithi116 ceph-mon[94164]: pgmap v388: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:36.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:36 smithi152 ceph-mon[82845]: pgmap v388: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:38 smithi116 ceph-mon[94164]: pgmap v389: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:38 smithi152 ceph-mon[82845]: pgmap v389: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:40 smithi116 ceph-mon[94164]: pgmap v390: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:40 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:17:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:40 smithi152 ceph-mon[82845]: pgmap v390: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:40.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:40 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:17:41.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:17:41.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:17:41.626 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:17:41.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:17:41.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:17:41.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:17:42.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:42 smithi116 ceph-mon[94164]: pgmap v391: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:42 smithi152 ceph-mon[82845]: pgmap v391: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:44 smithi116 ceph-mon[94164]: pgmap v392: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:44.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:44 smithi152 ceph-mon[82845]: pgmap v392: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:46 smithi116 ceph-mon[94164]: pgmap v393: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:46.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:46 smithi152 ceph-mon[82845]: pgmap v393: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:48.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:48 smithi116 ceph-mon[94164]: pgmap v394: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:48.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:48 smithi152 ceph-mon[82845]: pgmap v394: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:49.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:49 smithi152 ceph-mon[82845]: pgmap v395: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:49.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:49 smithi116 ceph-mon[94164]: pgmap v395: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:52.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:52 smithi116 ceph-mon[94164]: pgmap v396: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:52 smithi152 ceph-mon[82845]: pgmap v396: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:53.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:53.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:54 smithi116 ceph-mon[94164]: pgmap v397: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:54.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:54 smithi152 ceph-mon[82845]: pgmap v397: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:56.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:56 smithi116 ceph-mon[94164]: pgmap v398: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:56.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:56 smithi152 ceph-mon[82845]: pgmap v398: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:57.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:17:57 smithi152 ceph-mon[82845]: pgmap v399: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:17:57.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:17:57 smithi116 ceph-mon[94164]: pgmap v399: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:00 smithi116 ceph-mon[94164]: pgmap v400: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:00.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:00 smithi152 ceph-mon[82845]: pgmap v400: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:02.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:02 smithi116 ceph-mon[94164]: pgmap v401: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:02.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:02 smithi152 ceph-mon[82845]: pgmap v401: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:04.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:04 smithi116 ceph-mon[94164]: pgmap v402: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:04.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:04 smithi152 ceph-mon[82845]: pgmap v402: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:06.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:06 smithi116 ceph-mon[94164]: pgmap v403: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:06.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:06 smithi152 ceph-mon[82845]: pgmap v403: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:08.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:08 smithi116 ceph-mon[94164]: pgmap v404: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:08.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:08 smithi152 ceph-mon[82845]: pgmap v404: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:10.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:10 smithi116 ceph-mon[94164]: pgmap v405: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:10.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:10 smithi152 ceph-mon[82845]: pgmap v405: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:11.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:11 smithi116 ceph-mon[94164]: pgmap v406: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:11.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:11 smithi152 ceph-mon[82845]: pgmap v406: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:14.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:14 smithi116 ceph-mon[94164]: pgmap v407: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:14.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:14 smithi152 ceph-mon[82845]: pgmap v407: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:16 smithi116 ceph-mon[94164]: pgmap v408: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:16.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:16 smithi152 ceph-mon[82845]: pgmap v408: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:18.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:18 smithi116 ceph-mon[94164]: pgmap v409: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:18.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:18 smithi152 ceph-mon[82845]: pgmap v409: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:20 smithi116 ceph-mon[94164]: pgmap v410: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:20.771 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:20 smithi152 ceph-mon[82845]: pgmap v410: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:22 smithi152 ceph-mon[82845]: pgmap v411: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:22.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:22 smithi116 ceph-mon[94164]: pgmap v411: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:23.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:23.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:24.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:24 smithi152 ceph-mon[82845]: pgmap v412: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:24.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:24 smithi116 ceph-mon[94164]: pgmap v412: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:25.820 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:25 smithi152 ceph-mon[82845]: pgmap v413: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:25.849 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:25 smithi116 ceph-mon[94164]: pgmap v413: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:28 smithi116 ceph-mon[94164]: pgmap v414: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:28.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:28 smithi152 ceph-mon[82845]: pgmap v414: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:30.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:30 smithi116 ceph-mon[94164]: pgmap v415: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:30.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:30 smithi152 ceph-mon[82845]: pgmap v415: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:32.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:32 smithi116 ceph-mon[94164]: pgmap v416: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:32.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:32 smithi152 ceph-mon[82845]: pgmap v416: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:34.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:34 smithi152 ceph-mon[82845]: pgmap v417: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:34.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:34 smithi116 ceph-mon[94164]: pgmap v417: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:35.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:35 smithi152 ceph-mon[82845]: pgmap v418: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:35.956 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:35 smithi116 ceph-mon[94164]: pgmap v418: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:38 smithi116 ceph-mon[94164]: pgmap v419: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:38.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:38 smithi152 ceph-mon[82845]: pgmap v419: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:40 smithi116 ceph-mon[94164]: pgmap v420: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:40.856 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:40 smithi152 ceph-mon[82845]: pgmap v420: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:41.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:18:41.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:18:41.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:18:41.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:18:41.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:18:41.973 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:18:41.973 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:18:41.973 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:18:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:42 smithi152 ceph-mon[82845]: pgmap v421: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:42.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:42 smithi116 ceph-mon[94164]: pgmap v421: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:44.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:44 smithi152 ceph-mon[82845]: pgmap v422: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:44.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:44 smithi116 ceph-mon[94164]: pgmap v422: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:45.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:45 smithi152 ceph-mon[82845]: pgmap v423: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:45 smithi116 ceph-mon[94164]: pgmap v423: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:48 smithi116 ceph-mon[94164]: pgmap v424: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:48.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:48 smithi152 ceph-mon[82845]: pgmap v424: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:50.357 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:50 smithi152 ceph-mon[82845]: pgmap v425: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:50 smithi116 ceph-mon[94164]: pgmap v425: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:52 smithi116 ceph-mon[94164]: pgmap v426: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:52 smithi152 ceph-mon[82845]: pgmap v426: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:53.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:53.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:54.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:54 smithi152 ceph-mon[82845]: pgmap v427: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:54.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:54 smithi116 ceph-mon[94164]: pgmap v427: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:55.858 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:55 smithi152 ceph-mon[82845]: pgmap v428: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:55.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:55 smithi116 ceph-mon[94164]: pgmap v428: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:58.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:18:58 smithi116 ceph-mon[94164]: pgmap v429: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:18:58.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:18:58 smithi152 ceph-mon[82845]: pgmap v429: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:00.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:00 smithi152 ceph-mon[82845]: pgmap v430: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:00.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:00 smithi116 ceph-mon[94164]: pgmap v430: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:02.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:02 smithi152 ceph-mon[82845]: pgmap v431: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:02.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:02 smithi116 ceph-mon[94164]: pgmap v431: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:03.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:03 smithi152 ceph-mon[82845]: pgmap v432: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:03 smithi116 ceph-mon[94164]: pgmap v432: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:06.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:06 smithi116 ceph-mon[94164]: pgmap v433: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:06.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:06 smithi152 ceph-mon[82845]: pgmap v433: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:08 smithi152 ceph-mon[82845]: pgmap v434: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:08.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:08 smithi116 ceph-mon[94164]: pgmap v434: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:08.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:10.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:10 smithi152 ceph-mon[82845]: pgmap v435: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:10.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:10 smithi116 ceph-mon[94164]: pgmap v435: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:11.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:11 smithi152 ceph-mon[82845]: pgmap v436: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:11.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:11 smithi116 ceph-mon[94164]: pgmap v436: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:14.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:14 smithi152 ceph-mon[82845]: pgmap v437: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:14.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:14 smithi116 ceph-mon[94164]: pgmap v437: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:16.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:16 smithi152 ceph-mon[82845]: pgmap v438: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:16 smithi116 ceph-mon[94164]: pgmap v438: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:17.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:17 smithi152 ceph-mon[82845]: pgmap v439: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:17.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:17 smithi116 ceph-mon[94164]: pgmap v439: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:20.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:20 smithi152 ceph-mon[82845]: pgmap v440: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:20.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:20 smithi116 ceph-mon[94164]: pgmap v440: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:22.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:22 smithi152 ceph-mon[82845]: pgmap v441: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:22.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:22 smithi116 ceph-mon[94164]: pgmap v441: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:23.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:23.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:24.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:24 smithi152 ceph-mon[82845]: pgmap v442: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:24.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:24 smithi116 ceph-mon[94164]: pgmap v442: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:25.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:25 smithi152 ceph-mon[82845]: pgmap v443: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:25.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:25 smithi116 ceph-mon[94164]: pgmap v443: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:28.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:28 smithi152 ceph-mon[82845]: pgmap v444: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:28.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:28 smithi116 ceph-mon[94164]: pgmap v444: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:30.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:30 smithi152 ceph-mon[82845]: pgmap v445: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:30.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:30 smithi116 ceph-mon[94164]: pgmap v445: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:31.542 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:31 smithi116 ceph-mon[94164]: pgmap v446: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:31.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:31 smithi152 ceph-mon[82845]: pgmap v446: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:34.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:34 smithi152 ceph-mon[82845]: pgmap v447: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:34.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:34 smithi116 ceph-mon[94164]: pgmap v447: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:35.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:35 smithi152 ceph-mon[82845]: pgmap v448: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:35.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:35 smithi116 ceph-mon[94164]: pgmap v448: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:38 smithi152 ceph-mon[82845]: pgmap v449: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:38.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:38 smithi116 ceph-mon[94164]: pgmap v449: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:38.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:39 smithi152 ceph-mon[82845]: pgmap v450: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:39.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:39 smithi116 ceph-mon[94164]: pgmap v450: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:41.649 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:41 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:19:41.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:41 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:19:42.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:42 smithi116 ceph-mon[94164]: pgmap v451: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:42.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:19:42.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:19:42.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:42 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:19:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:42 smithi152 ceph-mon[82845]: pgmap v451: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:19:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:19:42.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:42 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' 2024-03-04T16:19:44.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:44 smithi152 ceph-mon[82845]: pgmap v452: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:44.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:44 smithi116 ceph-mon[94164]: pgmap v452: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:45.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:45 smithi152 ceph-mon[82845]: pgmap v453: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:45 smithi116 ceph-mon[94164]: pgmap v453: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:48.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:48 smithi152 ceph-mon[82845]: pgmap v454: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:48.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:48 smithi116 ceph-mon[94164]: pgmap v454: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:49.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:49 smithi152 ceph-mon[82845]: pgmap v455: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:49.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:49 smithi116 ceph-mon[94164]: pgmap v455: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:52.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:52 smithi152 ceph-mon[82845]: pgmap v456: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:52.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:52 smithi116 ceph-mon[94164]: pgmap v456: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:53.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:53 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:53.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:53 smithi152 ceph-mon[82845]: pgmap v457: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:53.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:53 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:53.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:53 smithi116 ceph-mon[94164]: pgmap v457: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:56.806 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:56 smithi116 ceph-mon[94164]: pgmap v458: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:56.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:56 smithi152 ceph-mon[82845]: pgmap v458: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:57.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:57 smithi152 ceph-mon[82845]: pgmap v459: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:57.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:57 smithi116 ceph-mon[94164]: pgmap v459: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:59.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:19:59 smithi152 ceph-mon[82845]: pgmap v460: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:19:59.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:19:59 smithi116 ceph-mon[94164]: pgmap v460: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:00.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:00 smithi152 ceph-mon[82845]: overall HEALTH_OK 2024-03-04T16:20:00.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:00 smithi116 ceph-mon[94164]: overall HEALTH_OK 2024-03-04T16:20:01.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:01 smithi152 ceph-mon[82845]: pgmap v461: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:01.859 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:01 smithi116 ceph-mon[94164]: pgmap v461: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:03.860 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:03 smithi152 ceph-mon[82845]: pgmap v462: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:03 smithi116 ceph-mon[94164]: pgmap v462: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:05.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:05 smithi152 ceph-mon[82845]: pgmap v463: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:05.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:05 smithi116 ceph-mon[94164]: pgmap v463: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:07.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:07 smithi152 ceph-mon[82845]: pgmap v464: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:07.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:07 smithi116 ceph-mon[94164]: pgmap v464: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:08.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:08 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:08.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:08 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:09.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:09 smithi152 ceph-mon[82845]: pgmap v465: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:09.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:09 smithi116 ceph-mon[94164]: pgmap v465: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:11.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:11 smithi152 ceph-mon[82845]: pgmap v466: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:11.967 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:11 smithi116 ceph-mon[94164]: pgmap v466: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:13.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:13 smithi152 ceph-mon[82845]: pgmap v467: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:13.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:13 smithi116 ceph-mon[94164]: pgmap v467: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:15.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:15 smithi152 ceph-mon[82845]: pgmap v468: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:15.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:15 smithi116 ceph-mon[94164]: pgmap v468: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:17.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:17 smithi152 ceph-mon[82845]: pgmap v469: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:17.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:17 smithi116 ceph-mon[94164]: pgmap v469: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:19.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:19 smithi152 ceph-mon[82845]: pgmap v470: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:19.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:19 smithi116 ceph-mon[94164]: pgmap v470: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:21.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:21 smithi152 ceph-mon[82845]: pgmap v471: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:21.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:21 smithi116 ceph-mon[94164]: pgmap v471: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:23 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:23.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:23 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:24 smithi116 ceph-mon[94164]: pgmap v472: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:24.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:24 smithi152 ceph-mon[82845]: pgmap v472: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:25.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:25 smithi152 ceph-mon[82845]: pgmap v473: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:25.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:25 smithi116 ceph-mon[94164]: pgmap v473: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:27.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:27 smithi152 ceph-mon[82845]: pgmap v474: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:27.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:27 smithi116 ceph-mon[94164]: pgmap v474: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:29.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:29 smithi152 ceph-mon[82845]: pgmap v475: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:29.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:29 smithi116 ceph-mon[94164]: pgmap v475: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:31.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:31 smithi152 ceph-mon[82845]: pgmap v476: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:31.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:31 smithi116 ceph-mon[94164]: pgmap v476: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:33.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:33 smithi152 ceph-mon[82845]: pgmap v477: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:33.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:33 smithi116 ceph-mon[94164]: pgmap v477: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:35.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:35 smithi152 ceph-mon[82845]: pgmap v478: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:35.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:35 smithi116 ceph-mon[94164]: pgmap v478: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:37.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:37 smithi152 ceph-mon[82845]: pgmap v479: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:37.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:37 smithi116 ceph-mon[94164]: pgmap v479: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:38.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:38 smithi152 ceph-mon[82845]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:38.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:38 smithi116 ceph-mon[94164]: from='mgr.34102 172.21.15.116:0/3152864169' entity='mgr.smithi116.jwdhop' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:39.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:39 smithi152 ceph-mon[82845]: pgmap v480: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:39.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:39 smithi116 ceph-mon[94164]: pgmap v480: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:41.686 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-03-04T16:20:41.687 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-04T16:20:41.688 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-04T16:20:41.700 INFO:tasks.cephadm:Teardown begin 2024-03-04T16:20:41.700 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks/cephadm.py", line 1867, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-04T16:20:41.702 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:20:41.735 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:20:41.771 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-04T16:20:41.771 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T16:20:41.789 DEBUG:teuthology.orchestra.run.smithi152:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T16:20:41.819 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:41 smithi116 ceph-mon[94164]: pgmap v481: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:41.828 INFO:tasks.cephadm:Stopping all daemons... 2024-03-04T16:20:41.828 INFO:tasks.cephadm.mon.smithi116:Stopping mon.smithi116... 2024-03-04T16:20:41.828 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi116 2024-03-04T16:20:41.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:41 smithi152 ceph-mon[82845]: pgmap v481: 65 pgs: 65 active+clean; 212 MiB data, 900 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:20:42.159 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:41 smithi116 systemd[1]: Stopping Ceph mon.smithi116 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:20:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:42 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116[94159]: 2024-03-04T16:20:42.158+0000 7fa4b8962700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi116 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:20:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:42 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116[94159]: 2024-03-04T16:20:42.158+0000 7fa4b8962700 -1 mon.smithi116@0(leader) e3 *** Got Signal Terminated *** 2024-03-04T16:20:42.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:42 smithi116 podman[154492]: 2024-03-04 16:20:42.291031474 +0000 UTC m=+0.261705887 container died 4ecbaf85d8a2575b809ea886bbfd181d6aac4fe12b6de9e19635f8eae67005b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:20:42.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:42 smithi116 podman[154492]: 2024-03-04 16:20:42.782196006 +0000 UTC m=+0.752870418 container cleanup 4ecbaf85d8a2575b809ea886bbfd181d6aac4fe12b6de9e19635f8eae67005b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-04T16:20:42.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 04 16:20:42 smithi116 bash[154492]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi116 2024-03-04T16:20:43.157 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi116.service' 2024-03-04T16:20:43.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:20:43.206 INFO:tasks.cephadm.mon.smithi116:Stopped mon.smithi116 2024-03-04T16:20:43.206 INFO:tasks.cephadm.mon.smithi152:Stopping mon.smithi152... 2024-03-04T16:20:43.206 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi152 2024-03-04T16:20:43.500 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:43 smithi152 systemd[1]: Stopping Ceph mon.smithi152 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:20:43.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:43 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152[82841]: 2024-03-04T16:20:43.499+0000 7f13c9052700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi152 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:20:43.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:43 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152[82841]: 2024-03-04T16:20:43.499+0000 7f13c9052700 -1 mon.smithi152@1(peon) e3 *** Got Signal Terminated *** 2024-03-04T16:20:43.857 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:43 smithi152 podman[124680]: 2024-03-04 16:20:43.622352337 +0000 UTC m=+0.231345020 container died 9129fd088663105002dc11995866534db45077524cf445bac5cc1f76c1f3ae8f (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-03-04T16:20:44.352 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:44 smithi152 podman[124680]: 2024-03-04 16:20:44.10910095 +0000 UTC m=+0.718093684 container cleanup 9129fd088663105002dc11995866534db45077524cf445bac5cc1f76c1f3ae8f (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD) 2024-03-04T16:20:44.352 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Mar 04 16:20:44 smithi152 bash[124680]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-mon-smithi152 2024-03-04T16:20:44.474 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@mon.smithi152.service' 2024-03-04T16:20:44.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:20:44.526 INFO:tasks.cephadm.mon.smithi152:Stopped mon.smithi152 2024-03-04T16:20:44.526 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-04T16:20:44.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.0 2024-03-04T16:20:44.930 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:44 smithi116 systemd[1]: Stopping Ceph osd.0 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:20:45.224 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:44 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[102822]: 2024-03-04T16:20:44.929+0000 7f6d5a534700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:20:45.224 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:44 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[102822]: 2024-03-04T16:20:44.929+0000 7f6d5a534700 -1 osd.0 84 *** Got signal Terminated *** 2024-03-04T16:20:45.224 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:44 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0[102822]: 2024-03-04T16:20:44.929+0000 7f6d5a534700 -1 osd.0 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:20:50.622 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:50 smithi116 podman[154726]: 2024-03-04 16:20:50.357256046 +0000 UTC m=+5.637012185 container died 98c52b3190c9ebc68c49bafa084891627f7a06db09fd320d99fc907163652d0a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True) 2024-03-04T16:20:50.923 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:50 smithi116 podman[154726]: 2024-03-04 16:20:50.706415335 +0000 UTC m=+5.986171515 container cleanup 98c52b3190c9ebc68c49bafa084891627f7a06db09fd320d99fc907163652d0a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:20:50.923 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:50 smithi116 bash[154726]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0 2024-03-04T16:20:51.224 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:50 smithi116 podman[154743]: 2024-03-04 16:20:50.923566987 +0000 UTC m=+0.562743706 container remove 98c52b3190c9ebc68c49bafa084891627f7a06db09fd320d99fc907163652d0a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-04T16:20:51.619 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:51 smithi116 podman[154838]: 2024-03-04 16:20:51.270866372 +0000 UTC m=+0.018925897 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:20:51.619 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:51 smithi116 podman[154838]: 2024-03-04 16:20:51.391282989 +0000 UTC m=+0.139342540 container create 1e83408b4f3c74342fcf7a90cc5af6dc0813b391c3764650f52095726e51ef68 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-04T16:20:51.619 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:51 smithi116 podman[154838]: 2024-03-04 16:20:51.61958246 +0000 UTC m=+0.367641973 container init 1e83408b4f3c74342fcf7a90cc5af6dc0813b391c3764650f52095726e51ef68 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-04T16:20:51.921 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:51 smithi116 podman[154838]: 2024-03-04 16:20:51.623104658 +0000 UTC m=+0.371164169 container start 1e83408b4f3c74342fcf7a90cc5af6dc0813b391c3764650f52095726e51ef68 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-04T16:20:51.921 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:51 smithi116 podman[154838]: 2024-03-04 16:20:51.711109367 +0000 UTC m=+0.459168905 container attach 1e83408b4f3c74342fcf7a90cc5af6dc0813b391c3764650f52095726e51ef68 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:20:52.222 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 16:20:51 smithi116 podman[154838]: 2024-03-04 16:20:51.922023983 +0000 UTC m=+0.670083499 container died 1e83408b4f3c74342fcf7a90cc5af6dc0813b391c3764650f52095726e51ef68 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-04T16:20:52.556 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.0.service' 2024-03-04T16:20:52.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:20:52.598 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-04T16:20:52.598 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-04T16:20:52.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.1 2024-03-04T16:20:52.926 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:52 smithi116 systemd[1]: Stopping Ceph osd.1 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:20:53.224 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:52 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[108248]: 2024-03-04T16:20:52.926+0000 7f8f8f424700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:20:53.224 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:52 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[108248]: 2024-03-04T16:20:52.926+0000 7f8f8f424700 -1 osd.1 84 *** Got signal Terminated *** 2024-03-04T16:20:53.224 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:52 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1[108248]: 2024-03-04T16:20:52.926+0000 7f8f8f424700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:20:58.724 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:58 smithi116 podman[154947]: 2024-03-04 16:20:58.453046177 +0000 UTC m=+5.626453169 container died 3e403d77dd721861150a9374548e56db48d1c4d0a572cef21d8caa8db8227e47 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-04T16:20:59.094 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:58 smithi116 podman[154947]: 2024-03-04 16:20:58.826971288 +0000 UTC m=+6.000378280 container cleanup 3e403d77dd721861150a9374548e56db48d1c4d0a572cef21d8caa8db8227e47 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:20:59.094 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:58 smithi116 bash[154947]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1 2024-03-04T16:20:59.454 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:59 smithi116 podman[154965]: 2024-03-04 16:20:59.094750513 +0000 UTC m=+0.644920125 container remove 3e403d77dd721861150a9374548e56db48d1c4d0a572cef21d8caa8db8227e47 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-04T16:20:59.705 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:59 smithi116 podman[155054]: 2024-03-04 16:20:59.357313955 +0000 UTC m=+0.017181952 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:20:59.705 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:59 smithi116 podman[155054]: 2024-03-04 16:20:59.479070463 +0000 UTC m=+0.138938492 container create 55d30ae25ba8f5581144b1d6ad745bd1d225176e200800d1c031decdda2adfc2 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:20:59.706 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:59 smithi116 podman[155054]: 2024-03-04 16:20:59.706371933 +0000 UTC m=+0.366239924 container init 55d30ae25ba8f5581144b1d6ad745bd1d225176e200800d1c031decdda2adfc2 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-04T16:20:59.974 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:59 smithi116 podman[155054]: 2024-03-04 16:20:59.709814861 +0000 UTC m=+0.369682858 container start 55d30ae25ba8f5581144b1d6ad745bd1d225176e200800d1c031decdda2adfc2 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:20:59.974 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:20:59 smithi116 podman[155054]: 2024-03-04 16:20:59.781562135 +0000 UTC m=+0.441430141 container attach 55d30ae25ba8f5581144b1d6ad745bd1d225176e200800d1c031decdda2adfc2 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:21:00.292 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 04 16:21:00 smithi116 podman[155054]: 2024-03-04 16:21:00.004608372 +0000 UTC m=+0.664476369 container died 55d30ae25ba8f5581144b1d6ad745bd1d225176e200800d1c031decdda2adfc2 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) 2024-03-04T16:21:00.629 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.1.service' 2024-03-04T16:21:00.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:21:00.671 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-04T16:21:00.671 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-04T16:21:00.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.2 2024-03-04T16:21:00.974 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:00 smithi116 systemd[1]: Stopping Ceph osd.2 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:21:01.291 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:01 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[114198]: 2024-03-04T16:21:01.022+0000 7ff234680700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:21:01.291 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:01 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[114198]: 2024-03-04T16:21:01.022+0000 7ff234680700 -1 osd.2 84 *** Got signal Terminated *** 2024-03-04T16:21:01.291 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:01 smithi116 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2[114198]: 2024-03-04T16:21:01.022+0000 7ff234680700 -1 osd.2 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:21:06.622 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:06 smithi116 podman[155161]: 2024-03-04 16:21:06.343512788 +0000 UTC m=+5.449568085 container died 43927e90a69fbe4f1217710261b7422787a96d4155963a6a99bd66f72f7c5579 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD) 2024-03-04T16:21:06.974 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:06 smithi116 podman[155161]: 2024-03-04 16:21:06.713327703 +0000 UTC m=+5.819383008 container cleanup 43927e90a69fbe4f1217710261b7422787a96d4155963a6a99bd66f72f7c5579 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:21:06.974 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:06 smithi116 bash[155161]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2 2024-03-04T16:21:07.282 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:06 smithi116 podman[155181]: 2024-03-04 16:21:06.981258564 +0000 UTC m=+0.631287068 container remove 43927e90a69fbe4f1217710261b7422787a96d4155963a6a99bd66f72f7c5579 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-04T16:21:07.551 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:07 smithi116 podman[155273]: 2024-03-04 16:21:07.185841017 +0000 UTC m=+0.018169978 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:21:07.551 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:07 smithi116 podman[155273]: 2024-03-04 16:21:07.316088449 +0000 UTC m=+0.148417442 container create 0ac5b71052bf7b7047ba47a332ca6d981dda2cb49ffb29566482fbf89dde3043 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-04T16:21:07.551 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:07 smithi116 podman[155273]: 2024-03-04 16:21:07.552095502 +0000 UTC m=+0.384424457 container init 0ac5b71052bf7b7047ba47a332ca6d981dda2cb49ffb29566482fbf89dde3043 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-04T16:21:07.848 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:07 smithi116 podman[155273]: 2024-03-04 16:21:07.55579437 +0000 UTC m=+0.388123329 container start 0ac5b71052bf7b7047ba47a332ca6d981dda2cb49ffb29566482fbf89dde3043 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-04T16:21:07.848 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:07 smithi116 podman[155273]: 2024-03-04 16:21:07.651972375 +0000 UTC m=+0.484301348 container attach 0ac5b71052bf7b7047ba47a332ca6d981dda2cb49ffb29566482fbf89dde3043 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-04T16:21:08.138 INFO:journalctl@ceph.osd.2.smithi116.stdout:Mar 04 16:21:07 smithi116 podman[155273]: 2024-03-04 16:21:07.849270008 +0000 UTC m=+0.681598950 container died 0ac5b71052bf7b7047ba47a332ca6d981dda2cb49ffb29566482fbf89dde3043 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-04T16:21:08.425 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.2.service' 2024-03-04T16:21:08.466 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:21:08.466 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-04T16:21:08.466 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-04T16:21:08.466 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.3 2024-03-04T16:21:08.758 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:08 smithi152 systemd[1]: Stopping Ceph osd.3 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:21:09.107 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:08 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[92960]: 2024-03-04T16:21:08.804+0000 7f68f10da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:21:09.107 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:08 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[92960]: 2024-03-04T16:21:08.804+0000 7f68f10da700 -1 osd.3 84 *** Got signal Terminated *** 2024-03-04T16:21:09.107 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:08 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[92960]: 2024-03-04T16:21:08.804+0000 7f68f10da700 -1 osd.3 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:21:11.607 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:11 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[92960]: 2024-03-04T16:21:11.326+0000 7f68e9ec7700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.181393+0000 front 2024-03-04T16:20:47.181376+0000 (oldest deadline 2024-03-04T16:21:10.681258+0000) 2024-03-04T16:21:11.607 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:11 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:11.275+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:12.607 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[92960]: 2024-03-04T16:21:12.339+0000 7f68e9ec7700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.181393+0000 front 2024-03-04T16:20:47.181376+0000 (oldest deadline 2024-03-04T16:21:10.681258+0000) 2024-03-04T16:21:12.607 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:12.295+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:13.292 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:12 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:12.861+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:13.607 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:13 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3[92960]: 2024-03-04T16:21:13.299+0000 7f68e9ec7700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.181393+0000 front 2024-03-04T16:20:47.181376+0000 (oldest deadline 2024-03-04T16:21:10.681258+0000) 2024-03-04T16:21:13.607 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:13 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:13.292+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:14.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:13 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:13.813+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:14.440 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:14 smithi152 podman[124807]: 2024-03-04 16:21:14.163320623 +0000 UTC m=+5.500480949 container died 6b01e97c8f9b3597992f79c8e52df9cd368618e231cf196a74e53d5c11e464bf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS) 2024-03-04T16:21:14.440 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:14 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:14.316+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:14.788 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:14 smithi152 podman[124807]: 2024-03-04 16:21:14.551603184 +0000 UTC m=+5.888763503 container cleanup 6b01e97c8f9b3597992f79c8e52df9cd368618e231cf196a74e53d5c11e464bf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True) 2024-03-04T16:21:14.788 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:14 smithi152 bash[124807]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3 2024-03-04T16:21:15.107 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:14 smithi152 podman[124828]: 2024-03-04 16:21:14.810795226 +0000 UTC m=+0.638568755 container remove 6b01e97c8f9b3597992f79c8e52df9cd368618e231cf196a74e53d5c11e464bf (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) 2024-03-04T16:21:15.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:14 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:14.787+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:15.452 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:15 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:15.333+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:15.453 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:15 smithi152 podman[124923]: 2024-03-04 16:21:15.026585792 +0000 UTC m=+0.017925515 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:21:15.453 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:15 smithi152 podman[124923]: 2024-03-04 16:21:15.141782537 +0000 UTC m=+0.133122293 container create 0d579fc69ffe1300d826c15b685672f6d950703a8e3ede8978ca8fea219ccb97 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-04T16:21:15.453 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:15 smithi152 podman[124923]: 2024-03-04 16:21:15.369518613 +0000 UTC m=+0.360858327 container init 0d579fc69ffe1300d826c15b685672f6d950703a8e3ede8978ca8fea219ccb97 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-03-04T16:21:15.453 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:15 smithi152 podman[124923]: 2024-03-04 16:21:15.372627136 +0000 UTC m=+0.363966852 container start 0d579fc69ffe1300d826c15b685672f6d950703a8e3ede8978ca8fea219ccb97 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS) 2024-03-04T16:21:15.453 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:15 smithi152 podman[124923]: 2024-03-04 16:21:15.45304164 +0000 UTC m=+0.444381355 container attach 0d579fc69ffe1300d826c15b685672f6d950703a8e3ede8978ca8fea219ccb97 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-04T16:21:15.788 INFO:journalctl@ceph.osd.3.smithi152.stdout:Mar 04 16:21:15 smithi152 podman[124923]: 2024-03-04 16:21:15.666076301 +0000 UTC m=+0.657416021 container died 0d579fc69ffe1300d826c15b685672f6d950703a8e3ede8978ca8fea219ccb97 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-04T16:21:16.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:15 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:15.787+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:16.248 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.3.service' 2024-03-04T16:21:16.290 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:21:16.291 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-04T16:21:16.291 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-04T16:21:16.291 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.4 2024-03-04T16:21:16.533 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:16 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:16.353+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:16.533 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:16 smithi152 systemd[1]: Stopping Ceph osd.4 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:21:16.798 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:16 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:16.669+0000 7efcaeb58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:21:16.798 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:16 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:16.669+0000 7efcaeb58700 -1 osd.4 84 *** Got signal Terminated *** 2024-03-04T16:21:16.798 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:16 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:16.669+0000 7efcaeb58700 -1 osd.4 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:21:17.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:16 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:16.797+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:17.747 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:17 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:17.402+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:18.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:17 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:17.747+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:18.790 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:18 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:18.374+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:19.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:18 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:18.790+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:19.804 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:19 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:19.415+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:20.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:19 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:19.804+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:20.757 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:20 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:20.408+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:20.757 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:20 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:20.408+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2024-03-04T16:20:57.864843+0000 front 2024-03-04T16:20:57.864891+0000 (oldest deadline 2024-03-04T16:21:20.164769+0000) 2024-03-04T16:21:21.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:20 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:20.758+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:21.774 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:21 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:21.386+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:45.563646+0000 front 2024-03-04T16:20:45.563753+0000 (oldest deadline 2024-03-04T16:21:10.863527+0000) 2024-03-04T16:21:21.774 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:21 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4[97857]: 2024-03-04T16:21:21.386+0000 7efca7945700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2024-03-04T16:20:57.864843+0000 front 2024-03-04T16:20:57.864891+0000 (oldest deadline 2024-03-04T16:21:20.164769+0000) 2024-03-04T16:21:22.093 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:21 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:21.774+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:22.357 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:22 smithi152 podman[125029]: 2024-03-04 16:21:22.093236808 +0000 UTC m=+5.553366786 container died 5ecc30e5a86ccd93283723000d5b0979b90653f857da0dade47bc6a81149e57d (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-03-04T16:21:22.720 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:22 smithi152 podman[125029]: 2024-03-04 16:21:22.461050757 +0000 UTC m=+5.921180736 container cleanup 5ecc30e5a86ccd93283723000d5b0979b90653f857da0dade47bc6a81149e57d (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4, GIT_CLEAN=True, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:21:22.720 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:22 smithi152 bash[125029]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4 2024-03-04T16:21:23.036 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:22 smithi152 podman[125056]: 2024-03-04 16:21:22.720920199 +0000 UTC m=+0.621075407 container remove 5ecc30e5a86ccd93283723000d5b0979b90653f857da0dade47bc6a81149e57d (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:21:23.036 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:22 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:22.750+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:23.332 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:23 smithi152 podman[125149]: 2024-03-04 16:21:22.939600775 +0000 UTC m=+0.018377479 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:21:23.332 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:23 smithi152 podman[125149]: 2024-03-04 16:21:23.046890593 +0000 UTC m=+0.125667288 container create ec70ef3d3445414d14909633ed26b3ddaacfc81c9261213e775804df87e91582 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:21:23.332 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:23 smithi152 podman[125149]: 2024-03-04 16:21:23.266862522 +0000 UTC m=+0.345639232 container init ec70ef3d3445414d14909633ed26b3ddaacfc81c9261213e775804df87e91582 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-04T16:21:23.332 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:23 smithi152 podman[125149]: 2024-03-04 16:21:23.27027593 +0000 UTC m=+0.349052635 container start ec70ef3d3445414d14909633ed26b3ddaacfc81c9261213e775804df87e91582 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-04T16:21:23.333 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:23 smithi152 podman[125149]: 2024-03-04 16:21:23.332926629 +0000 UTC m=+0.411703343 container attach ec70ef3d3445414d14909633ed26b3ddaacfc81c9261213e775804df87e91582 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:21:23.607 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 04 16:21:23 smithi152 podman[125149]: 2024-03-04 16:21:23.56390816 +0000 UTC m=+0.642684860 container died ec70ef3d3445414d14909633ed26b3ddaacfc81c9261213e775804df87e91582 (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:21:24.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:23 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:23.758+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:24.153 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.4.service' 2024-03-04T16:21:24.193 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:21:24.193 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-04T16:21:24.194 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-04T16:21:24.194 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.5 2024-03-04T16:21:24.541 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:24 smithi152 systemd[1]: Stopping Ceph osd.5 for 4960653c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:21:24.857 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:24 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:24.540+0000 7ff9bfb35700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:21:24.857 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:24 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:24.540+0000 7ff9bfb35700 -1 osd.5 84 *** Got signal Terminated *** 2024-03-04T16:21:24.857 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:24 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:24.540+0000 7ff9bfb35700 -1 osd.5 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:21:24.857 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:24 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:24.732+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:24.857 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:24 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:24.732+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2024-03-04T16:20:56.334954+0000 front 2024-03-04T16:20:56.334928+0000 (oldest deadline 2024-03-04T16:21:23.935333+0000) 2024-03-04T16:21:26.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:25 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:25.755+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:26.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:25 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:25.755+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2024-03-04T16:20:56.334954+0000 front 2024-03-04T16:20:56.334928+0000 (oldest deadline 2024-03-04T16:21:23.935333+0000) 2024-03-04T16:21:27.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:26 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:26.740+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:27.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:26 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:26.740+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2024-03-04T16:20:56.334954+0000 front 2024-03-04T16:20:56.334928+0000 (oldest deadline 2024-03-04T16:21:23.935333+0000) 2024-03-04T16:21:27.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:26 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:26.740+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2024-03-04T16:21:04.436297+0000 front 2024-03-04T16:21:04.436351+0000 (oldest deadline 2024-03-04T16:21:26.736172+0000) 2024-03-04T16:21:28.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:27 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:27.771+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:28.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:27 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:27.771+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2024-03-04T16:20:56.334954+0000 front 2024-03-04T16:20:56.334928+0000 (oldest deadline 2024-03-04T16:21:23.935333+0000) 2024-03-04T16:21:28.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:27 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:27.771+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2024-03-04T16:21:04.436297+0000 front 2024-03-04T16:21:04.436351+0000 (oldest deadline 2024-03-04T16:21:26.736172+0000) 2024-03-04T16:21:29.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:28 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:28.775+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T16:20:47.533950+0000 front 2024-03-04T16:20:47.534110+0000 (oldest deadline 2024-03-04T16:21:12.233794+0000) 2024-03-04T16:21:29.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:28 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:28.775+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2024-03-04T16:20:56.334954+0000 front 2024-03-04T16:20:56.334928+0000 (oldest deadline 2024-03-04T16:21:23.935333+0000) 2024-03-04T16:21:29.107 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:28 smithi152 ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5[102760]: 2024-03-04T16:21:28.775+0000 7ff9b8922700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2024-03-04T16:21:04.436297+0000 front 2024-03-04T16:21:04.436351+0000 (oldest deadline 2024-03-04T16:21:26.736172+0000) 2024-03-04T16:21:30.200 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:29 smithi152 podman[125255]: 2024-03-04 16:21:29.940896876 +0000 UTC m=+5.525829967 container died 1083700b476d11f516829fbff21ed94dc01632106053c9336e91960db89bc22a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:21:30.466 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:30 smithi152 podman[125255]: 2024-03-04 16:21:30.282076147 +0000 UTC m=+5.867009246 container cleanup 1083700b476d11f516829fbff21ed94dc01632106053c9336e91960db89bc22a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-03-04T16:21:30.466 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:30 smithi152 bash[125255]: ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5 2024-03-04T16:21:30.857 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:30 smithi152 podman[125277]: 2024-03-04 16:21:30.466602647 +0000 UTC m=+0.522491911 container remove 1083700b476d11f516829fbff21ed94dc01632106053c9336e91960db89bc22a (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-04T16:21:31.162 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:30 smithi152 podman[125365]: 2024-03-04 16:21:30.776478023 +0000 UTC m=+0.018515740 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:21:31.162 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:30 smithi152 podman[125365]: 2024-03-04 16:21:30.90973559 +0000 UTC m=+0.151773345 container create 7255aea7ee581fb3544d177ce5d2b4639f29189c6989fce5ca5f64744c22146f (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:21:31.162 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:31 smithi152 podman[125365]: 2024-03-04 16:21:31.162394525 +0000 UTC m=+0.404432246 container init 7255aea7ee581fb3544d177ce5d2b4639f29189c6989fce5ca5f64744c22146f (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD) 2024-03-04T16:21:31.461 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:31 smithi152 podman[125365]: 2024-03-04 16:21:31.165628906 +0000 UTC m=+0.407666616 container start 7255aea7ee581fb3544d177ce5d2b4639f29189c6989fce5ca5f64744c22146f (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:21:31.461 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:31 smithi152 podman[125365]: 2024-03-04 16:21:31.245807628 +0000 UTC m=+0.487845354 container attach 7255aea7ee581fb3544d177ce5d2b4639f29189c6989fce5ca5f64744c22146f (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=) 2024-03-04T16:21:31.756 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 16:21:31 smithi152 podman[125365]: 2024-03-04 16:21:31.462522156 +0000 UTC m=+0.704559869 container died 7255aea7ee581fb3544d177ce5d2b4639f29189c6989fce5ca5f64744c22146f (image=quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-4960653c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-04T16:21:32.038 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4960653c-da3f-11ee-95c6-87774f69a715@osd.5.service' 2024-03-04T16:21:32.081 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:21:32.081 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-04T16:21:32.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4960653c-da3f-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T16:21:32.241 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:21:36.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[78279]: fuse finished with error 0 and tester_r 0 2024-03-04T16:21:54.467 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4960653c-da3f-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T16:21:54.628 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:22:09.707 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:22:09.742 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:22:09.772 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-04T16:22:09.774 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235/remote/smithi116/crash 2024-03-04T16:22:09.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/crash -- . 2024-03-04T16:22:09.811 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T16:22:09.812 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now 2024-03-04T16:22:09.813 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235/remote/smithi152/crash 2024-03-04T16:22:09.814 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/crash -- . 2024-03-04T16:22:09.840 INFO:teuthology.orchestra.run.smithi152.stderr:tar: /var/lib/ceph/4960653c-da3f-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T16:22:09.840 INFO:teuthology.orchestra.run.smithi152.stderr:tar: Error is not recoverable: exiting now 2024-03-04T16:22:09.841 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-04T16:22:09.841 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-04T16:22:09.886 INFO:teuthology.orchestra.run.smithi116.stdout:2024-03-04T16:10:00.000214+0000 mon.smithi116 (mon.0) 477 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 22/231 objects degraded (9.524%), 7 pgs degraded 2024-03-04T16:22:09.886 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-04T16:22:09.886 INFO:tasks.cephadm:Compressing logs... 2024-03-04T16:22:09.887 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T16:22:09.929 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T16:22:09.954 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T16:22:09.955 INFO:teuthology.orchestra.run.smithi116.stderr:: No such file or directory 2024-03-04T16:22:09.955 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-04T16:22:09.955 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T16:22:09.955 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi116.pjrmzv.log 2024-03-04T16:22:09.955 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.1.log 2024-03-04T16:22:09.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mon.smithi116.log 2024-03-04T16:22:09.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi116.pjrmzv.log: /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi116.gkenwy.log 2024-03-04T16:22:09.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mon.smithi116.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.0.log 2024-03-04T16:22:09.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi116.gkenwy.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T16:22:09.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.0.log: 91.5% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T16:22:09.964 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.log 2024-03-04T16:22:09.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log: 86.1% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T16:22:09.964 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.2.log 2024-03-04T16:22:09.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.log: 94.4% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi116.gkenwy.log.gz 2024-03-04T16:22:09.965 INFO:teuthology.orchestra.run.smithi152.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T16:22:09.965 INFO:teuthology.orchestra.run.smithi152.stderr:: No such file or directory 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi152.tlpshy.log 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.1.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mgr.smithi152.aawieo.log 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi152.tlpshy.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mon.smithi152.log 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mgr.smithi152.aawieo.log: /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.4.log 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mon.smithi152.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi152.kqoybf.log 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.5.log 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5%/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi152.kqoybf.log: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.3.log 2024-03-04T16:22:09.966 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.5.log: 91.6% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T16:22:09.967 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T16:22:09.968 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T16:22:09.968 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T16:22:09.968 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mgr.smithi116.jwdhop.log 2024-03-04T16:22:09.970 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-volume.log: 88.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T16:22:09.970 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-client.ceph-exporter.smithi116.log 2024-03-04T16:22:09.976 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.log 2024-03-04T16:22:09.977 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log: 86.0% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T16:22:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mgr.smithi116.jwdhop.log: /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-client.ceph-exporter.smithi116.log: 91.1% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T16:22:09.986 INFO:teuthology.orchestra.run.smithi116.stderr: 94.2% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-client.ceph-exporter.smithi116.log.gz 2024-03-04T16:22:09.992 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T16:22:09.995 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.log: 88.4% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T16:22:10.001 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-client.ceph-exporter.smithi152.log 2024-03-04T16:22:10.005 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-volume.log: 89.3% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mgr.smithi152.aawieo.log.gz 2024-03-04T16:22:10.023 INFO:teuthology.orchestra.run.smithi152.stderr: 91.0%/var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-client.ceph-exporter.smithi152.log: -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T16:22:10.025 INFO:teuthology.orchestra.run.smithi152.stderr: 94.3% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-client.ceph-exporter.smithi152.log.gz 2024-03-04T16:22:10.255 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mgr.smithi116.jwdhop.log.gz 2024-03-04T16:22:10.407 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mon.smithi152.log.gz 2024-03-04T16:22:11.198 INFO:teuthology.orchestra.run.smithi116.stderr: 90.5% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mon.smithi116.log.gz 2024-03-04T16:22:15.437 INFO:teuthology.orchestra.run.smithi116.stderr: 93.5% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.1.log.gz 2024-03-04T16:22:15.473 INFO:teuthology.orchestra.run.smithi152.stderr: 93.7% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.5.log.gz 2024-03-04T16:22:15.907 INFO:teuthology.orchestra.run.smithi116.stderr: 94.7% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi116.pjrmzv.log.gz 2024-03-04T16:22:15.944 INFO:teuthology.orchestra.run.smithi152.stderr: 93.4% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.4.log.gz 2024-03-04T16:22:15.995 INFO:teuthology.orchestra.run.smithi116.stderr: 93.5% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.2.log.gz 2024-03-04T16:22:16.105 INFO:teuthology.orchestra.run.smithi152.stderr: 93.6% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.3.log.gz 2024-03-04T16:22:16.499 INFO:teuthology.orchestra.run.smithi116.stderr: 93.5% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-osd.0.log.gz 2024-03-04T16:22:17.845 INFO:teuthology.orchestra.run.smithi152.stderr: 94.8% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi152.kqoybf.log.gz 2024-03-04T16:22:24.155 INFO:teuthology.orchestra.run.smithi116.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-04T16:22:24.157 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-04T16:22:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:real 0m14.216s 2024-03-04T16:22:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:user 0m38.457s 2024-03-04T16:22:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m1.277s 2024-03-04T16:22:26.157 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-03-04T16:22:26.157 INFO:teuthology.orchestra.run.smithi152.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-03-04T16:23:36.013 INFO:teuthology.orchestra.run.smithi152.stderr: 92.7% -- replaced with /var/log/ceph/4960653c-da3f-11ee-95c6-87774f69a715/ceph-mds.cephfs.smithi152.tlpshy.log.gz 2024-03-04T16:23:36.015 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-04T16:23:36.015 INFO:teuthology.orchestra.run.smithi152.stderr:real 1m26.071s 2024-03-04T16:23:36.015 INFO:teuthology.orchestra.run.smithi152.stderr:user 2m4.189s 2024-03-04T16:23:36.015 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m3.801s 2024-03-04T16:23:36.016 INFO:tasks.cephadm:Archiving logs... 2024-03-04T16:23:36.016 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235/remote/smithi116/log 2024-03-04T16:23:36.017 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T16:23:41.382 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235/remote/smithi152/log 2024-03-04T16:23:41.383 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T16:24:00.502 INFO:tasks.cephadm:Removing cluster... 2024-03-04T16:24:00.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4960653c-da3f-11ee-95c6-87774f69a715 --force 2024-03-04T16:24:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:24:00.955 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4960653c-da3f-11ee-95c6-87774f69a715 --force 2024-03-04T16:24:01.113 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: 4960653c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:24:01.504 INFO:tasks.cephadm:Removing cephadm ... 2024-03-04T16:24:01.505 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T16:24:01.522 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T16:24:01.538 INFO:tasks.cephadm:Teardown complete 2024-03-04T16:24:01.538 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-04T16:24:01.550 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-04T16:24:01.551 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-04T16:24:01.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T16:24:01.565 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T16:24:01.654 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T16:24:01.665 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T16:24:01.665 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T16:24:01.668 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T16:24:01.680 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-04T16:24:01.682 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-04T16:24:01.683 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T16:24:01.684 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-04T16:24:01.684 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 -1797us[-1797us] +/- 139ms 2024-03-04T16:24:01.684 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 7 377 45 -480us[ -347us] +/- 76ms 2024-03-04T16:24:01.684 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +1654us[+1654us] +/- 72ms 2024-03-04T16:24:01.684 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:24:01.685 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T16:24:01.685 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-04T16:24:01.685 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 -1902us[-1902us] +/- 139ms 2024-03-04T16:24:01.685 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -618us[ -612us] +/- 76ms 2024-03-04T16:24:01.686 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +1535us[+1535us] +/- 72ms 2024-03-04T16:24:01.686 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:24:01.687 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T16:24:01.697 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T16:24:01.698 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T16:24:01.723 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T16:24:01.810 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T16:24:01.890 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T16:24:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:01.921 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T16:24:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:01.952 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T16:24:01.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T16:24:01.978 INFO:teuthology.task.internal:Duration was 2245.953967 seconds 2024-03-04T16:24:01.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T16:24:01.989 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T16:24:01.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T16:24:01.993 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T16:24:02.033 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T16:24:02.036 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T16:24:02.267 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-04T16:24:02.268 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T16:24:02.268 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T16:24:02.328 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-03-04T16:24:02.328 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T16:24:02.363 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-04T16:24:02.363 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T16:24:02.371 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T16:24:02.432 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-04T16:24:02.432 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T16:24:02.439 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T16:24:03.092 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T16:24:03.104 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T16:24:03.104 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T16:24:03.131 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T16:24:03.159 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T16:24:03.172 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:24:03.174 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:24:03.197 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-04T16:24:03.225 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-03-04T16:24:03.270 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:24:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:03.324 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:24:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:03.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T16:24:03.350 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T16:24:03.351 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235/remote/smithi116 2024-03-04T16:24:03.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T16:24:03.416 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235/remote/smithi152 2024-03-04T16:24:03.416 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T16:24:03.457 INFO:teuthology.task.internal:Removing archive directory... 2024-03-04T16:24:03.457 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T16:24:03.460 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T16:24:03.506 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T16:24:03.518 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T16:24:03.518 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T16:24:03.530 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T16:24:03.530 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T16:24:03.532 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T16:24:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 4 16:24 /home/ubuntu/cephtest 2024-03-04T16:24:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: 528469 4 d--------- 2 ubuntu ubuntu 4096 Mar 4 16:01 /home/ubuntu/cephtest/mnt.0 2024-03-04T16:24:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-03-04T16:24:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-04T16:24:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:03.548 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-04T16:24:03.549 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T16:24:03.561 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T16:24:03.574 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-04T16:24:03.575 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T16:24:03.605 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-04T16:24:03.605 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T16:24:03.627 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:37:10.749215', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPQOb5Abv1OZIbkRjvIYIsiGaWcSXRg72uD1itWszrWmuDYmlbWwlZZ1NkclDfTumZcyG//l2WDWYSNbadXBoo='} 2024-03-04T16:24:03.631 DEBUG:teuthology.nuke:shortname: smithi152 2024-03-04T16:24:03.631 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T16:24:03.651 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:37:10.747864', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEzoCGrb5maYMP2IHy53/n2pgqD7FOQ/cAlr36Z5wmwvFEyl0DSC1ka+4yUgvjuq0E89DjowqdWUO/gWhKO0WsA='} 2024-03-04T16:24:03.682 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-04T16:24:03.683 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T16:24:03.698 INFO:teuthology.orchestra.console.smithi152:Power off 2024-03-04T16:24:03.699 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T16:24:03.710 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-04T16:24:03.711 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:24:03.721 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2024-03-04T16:24:03.722 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:24:03.734 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-04T16:24:03.744 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-04T16:24:07.736 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:24:07.745 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:24:07.848 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-04T16:24:07.855 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-04T16:24:11.852 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:24:11.860 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:24:11.963 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-04T16:24:11.970 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-04T16:24:15.968 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:24:15.976 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:24:16.078 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-04T16:24:16.085 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2024-03-04T16:24:16.180 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-04T16:24:16.186 INFO:teuthology.orchestra.console.smithi152:Power off completed 2024-03-04T16:24:16.292 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 2245.953966856003 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_tonay@teuthology status: fail success: false 2024-03-04T16:24:16.292 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T16:24:16.383 INFO:teuthology.run:FAIL